Image processing device, image processing method, and program

Abstract

To provide an image processing device, an image processing method, and a program which can achieve automation of a dredge work.SOLUTION: An image processing device includes: a feature detection part for detecting three-dimensional positional information of an object in a construction area of a work machine, using distance information obtained by measuring the construction area by a distance measurement device for measuring a distance between a photographic image obtained by photographing the construction area by the photographic device and a subject photographed by the photographic device; a construction state calculation part for calculating the position and the attitude of the object using the three-dimensional positional information of the object detected by the feature detection part; and a notification part for notifying the position and the attitude of the object calculated by the construction state calculation part.SELECTED DRAWING: Figure 2B

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] A technology has been disclosed in which a work machine such as a grab bucket is equipped with an acceleration sensor and an angular velocity sensor, and the position signal of the grab bucket on the bottom of the water from the acceleration sensor and the tilt signal and orientation signal of the grab bucket on the bottom of the water from the angular velocity sensor are input together with reference data into a data processing calculation device (an example of an image processing device) for calculation processing, to obtain calculation data including the actual position signal, tilt signal, and orientation signal of the grab bucket on the bottom of the water, and to operate the grab bucket based on this calculation data (see Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0003] However, with the above technology, dredging work, in which a grab bucket is used to excavate the bottom of a specified body of water, is difficult to automate because the excavation work location and the location where the excavated soil is released change continuously due to the influence of the surrounding environment, such as waves and wind, and operation by a skilled operator is required, leaving room for improvement.

[0004] The present invention has been made in view of the above, and has an object to provide an image processing device, an image processing method, and a program that can realize automation of dredging work. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems and achieve the object, the present invention comprises a feature detection unit that detects three-dimensional position information of an object within the construction area using a captured image obtained by photographing a construction area of ​​a work machine with a photographing device and distance information obtained by measuring the construction area with a distance measuring device that measures the distance to the subject photographed by the photographing device, a construction status calculation unit that calculates the position and attitude of the object using the three-dimensional position information of the object detected by the feature detection unit, and a notification unit that notifies the position and attitude of the object calculated by the construction status calculation unit. [Effects of the Invention]

[0006] According to the present invention, it is possible to realize automation of dredging work. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of a crane system according to this embodiment. [Figure 2A] FIG. 2A is a block diagram showing an example of a hardware configuration of an image processing device according to this embodiment. [Figure 2B] FIG. 2B is a diagram illustrating an example of functional blocks in the image processing device included in the crane system according to the present embodiment. [Figure 3] FIG. 3 is a diagram for explaining an example of an image captured by a measuring instrument of the crane system according to this embodiment. [Figure 4] FIG. 4 is a diagram for explaining an example of a result of feature detection by the image processing device according to the present embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of a detection result of the position and orientation of a detection object in the image processing device according to the present embodiment. [Figure 6] FIG. 6 is a diagram for explaining an example of the detection result of the earth and sand loading state of a barge by the image processing device according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of an image processing device, an image processing method, and a crane system to which a program is applied will be described in detail with reference to the accompanying drawings.

[0009] FIG. 1 is a diagram illustrating an example of the hardware configuration of a crane system according to this embodiment. As shown in FIG. 1, the crane system according to this embodiment includes a measuring instrument 1, an image processing device 2, a crane control device 3, and a display device 4. In this embodiment, the measuring instrument 1, the image processing device 2, the crane control device 3, and the display device 4 are communicatively connected to each other via the image processing device 2, but they may also be communicatively connected to each other via a communication network. Here, the communication network is constructed using the Internet, a mobile communication network, a LAN (Local Area Network), or the like. The communication network may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution).

[0010] The measuring device 1 and the crane control device 3 are installed at the crane work site. The measuring device 1 and the crane control device 3 may each be configured integrally with the crane. An operator at the site can check the work being done at the site using the crane control device 3. A manager at a remote location can also check the work being done at the site using a communication terminal.

[0011] The measuring instrument 1 has a camera 1a, a three-dimensional measuring device 1b, etc., and is a device for measuring three-dimensional space. The measuring instrument 1 may be one that measures three-dimensional space using two or more cameras 1a, or may be one that combines the camera 1a with a three-dimensional sensor such as a LiDAR (Light Detection and Ranging) or RADAR. Here, the camera 1a is an example of an imaging device that captures an image of a construction area of ​​a work machine such as a crane. Also, here, the three-dimensional measuring device 1b is an example of a distance measuring device that measures the construction area of ​​a work machine and obtains distance information.

[0012] The camera 1a photographs the construction area and acquires the photographed image (two-dimensional data) including image data, color, brightness information, etc. The three-dimensional measuring device 1b measures the distance to the subject photographed by the photographing device and acquires distance information (three-dimensional data) such as point cloud data.

[0013] The camera 1a may capture time-lapse images or videos, in which case the three-dimensional measuring device 1b acquires distance information in synchronization with the frame rate of the camera 1a.

[0014] The three-dimensional measuring device 1b is equipped with a stereo camera capable of three-dimensional measurement using two or more cameras 1a, and acquires captured images and distance information using the stereo camera. Alternatively, the three-dimensional measuring device 1b may be a distance sensor such as a ToF (Time of Flight) sensor. In this case, the three-dimensional measuring device 1b measures the distance from the light source to the object by irradiating the object with laser light from the light source and measuring the scattered and reflected light.

[0015] Alternatively, the three-dimensional measuring device 1b may be a LiDAR sensor. LiDAR is a method of measuring the time of flight of light using pulses, but as another method of ToF sensors, distance may be measured using a phase difference detection method. In the phase difference detection method, a laser beam amplitude-modulated at a fundamental frequency is irradiated onto a measurement range, and the phase difference between the irradiated light and the reflected light is measured. The distance is calculated by multiplying this time by the speed of light.

[0016] The image processing device 2 is an example of an image processing device that processes captured images of the working state of a work machine. Specifically, the image processing device 2 is a device that acquires the position, posture, and other conditions of a detection object (an example of an object) based on the three-dimensional space measured by the measuring device 1. The image processing device 2 also transmits the acquired position, posture, and other conditions of the detection object to the crane control device 3. Furthermore, the image processing device 2 displays the acquired position, posture, and other conditions of the detection object (detection results) on the display device 4.

[0017] The crane control device 3 is a device that controls a crane in accordance with information transmitted from the image processing device 2 (for example, the position and posture of the detection target).

[0018] The display device 4 is a monitor, a tablet, a PC (Personal Computer), or the like, and is a device that displays the contents of drawing from the image processing device 2 (for example, the position and posture of the detection target object).

[0019] 2A is a block diagram showing an example of a hardware configuration of an image processing device according to this embodiment. As shown in FIG. 2A, the image processing device 2 includes an image capture device I / F (Interface) 901, a sensor device I / F 902, a bus line 910, a CPU (Central Processing Unit) 911, a ROM (Read Only Memory) 912, a RAM (Random Access Memory) 913, a HD (Hard Disk) 915, an HDD (Hard Disk Drive) controller 914, a network I / F 916, an external device connection I / F 923, and a timer 924.

[0020] Of these, the imaging device I / F 901 is an interface for transmitting and receiving various data or information to and from the camera 1a of the measuring instrument 1. The sensor device I / F 902 is an interface for transmitting and receiving various data or information to and from the three-dimensional measuring device 1b of the measuring instrument 1. The bus line 910 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 911 shown in FIG. 2.

[0021] Furthermore, the CPU 911 controls the overall operation of the monitoring device 9. The ROM 912 stores programs used to drive the CPU 911, such as the IPL. The RAM 913 is used as a work area for the CPU 911. The HD 915 stores various data such as programs. The HDD controller 914 controls the reading and writing of various data from and to the HD 915 under the control of the CPU 911. The network I / F 916 is an interface for data communication using the communication network 100.

[0022] The program executed by the image processing device 2 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The program executed by the image processing device 2 of this embodiment may be provided or distributed via a network such as the Internet. The program executed by the image processing device 2 of this embodiment may be provided by being pre-installed in a ROM or the like.

[0023] FIG. 2B is a diagram illustrating an example of functional blocks in an image processing device included in the crane system according to this embodiment. In this embodiment, as shown in FIG. 2B, the image processing device 2 includes a data receiving unit 201, a feature detecting unit 202, a construction status calculation unit 203, and a display control unit 204. In this embodiment, the image processing device 2 includes these units. Each of these units is a function or a means for performing a function that is realized when any of the components shown in FIG. 2A operates in response to an instruction from the CPU 911 in accordance with a program loaded from the HD 915 onto the RAM 913. Each block fulfills the role described below to notify the crane control device 3 of the construction status and to draw on the display device 4.

[0024] Data receiving unit 201 is an example of a data receiving unit that receives captured images and distance information used to detect three-dimensional position information of a detection target from measuring instrument 1. In this embodiment, data receiving unit 201 receives the results of measurement by measuring instrument 1 (measurement results of three-dimensional space).

[0025] The feature detection unit 202 is an example of a feature detection unit that uses the captured image and distance information to detect three-dimensional position information of a target object (an example of an object) in the construction area. Here, the detection object is a grab bucket, a mining work site (pollution prevention frame), a site where excavated soil and sand are released (barge), etc.

[0026] In this embodiment, the feature detection unit 202 detects features of the detection object using the captured image and distance information, and detects three-dimensional position information based on the detected features. When the construction status calculation unit 203 detects the position and orientation of the detection object, the feature detection unit 202 detects at least three features of the detection object. Furthermore, the feature detection unit 202 may use a learning algorithm that detects three-dimensional position information of the detection object based on a model created in advance.

[0027] For example, feature detection unit 202 detects features (markers) of the detection target object based on color and brightness information of the captured image captured by camera 1a of measuring instrument 1. The feature detection method may use a detection algorithm that is deductively designed using the feature amount of the feature in the captured image, or may use a learning algorithm that detects features based on similarity with a model created in advance.

[0028] Generally, the position and orientation of a detection object can be uniquely determined by the three-dimensional coordinates (three-dimensional position information) of three points on the detection object. Therefore, to detect the entire position and orientation of the detection object, at least three features (markers) are required. If the markers are on a single line, the orientation of the detection object becomes indefinite. Therefore, in order to accurately detect the orientation of the detection object, it is desirable for the feature detection unit 202 to detect three or more features that are not on a single line (for example, features located on two or more sides). In this way, it is possible to estimate the rotation of the detection object about three axes with high accuracy. In other words, it is desirable for the feature detection unit 202 to detect features that exist on two or more sides of the detection object.

[0029] In this embodiment, the feature detection unit 202 receives the measurement results of the three-dimensional space received by the data receiving unit 201 and detects features (e.g., markers) from the measurement results of the three-dimensional space. The feature detection method may be a method of detecting features using a detection algorithm designed deductively using the feature amount on the image that the feature (e.g., marker) has, or a method of using a learning algorithm that detects features based on similarity with a pre-created model. In addition, the feature detection unit 202 obtains position information of the detection target from the median or average value of the three-dimensional information of the detected feature (e.g., within a frame).

[0030] The construction status calculation unit 203 is an example of a construction status calculation unit that calculates the position and posture of a detected object using three-dimensional position information of the detected object detected by the feature detection unit 202. In this embodiment, the construction status calculation unit 203 calculates the position, posture, etc. of a detected object such as a grab bucket, a mining work site (pollution prevention frame), a site where excavated soil and sand is discharged (barge), etc., based on three-dimensional information of features (e.g., markers) detected by the feature detection unit 202. Furthermore, the construction status calculation unit 203 may calculate the distribution of soil and sand in the construction area using the three-dimensional position information of the detected object.

[0031] It is preferable that the distance between the detection objects be approximately equal to or greater than the distance from the camera 1a and three-dimensional measuring device 1b of the measuring device 1 to the construction area. This allows for highly accurate estimation of the six axes. For example, if the barge is 35 m away and the markers are 35 m apart, even with an installation error of 10 mm and a three-dimensional position detection error of 10%, it is possible to calculate the position and orientation of the detection object within an angle of 1.5 degrees and within 2 m of the center of the barge.

[0032] Furthermore, when the reliability of some of the three-dimensional position information of the features of the detected object is low, the construction status calculation unit 203 may calculate the position and orientation of the detected object based on the three-dimensional position information of the features with high reliability, the positional relationships between the features that have been determined in advance, such as dimensional values, and the distortion-corrected captured image. Furthermore, by feeding back the position and orientation of the detected object obtained by the construction status calculation unit 203 to crane control, automation of dredging work can be realized.

[0033] The display control unit 204 is an example of a notification unit that notifies the position and posture of the detected object obtained by the construction status calculation unit 203. In this embodiment, the display control unit 204 displays the position, posture, etc. of the detected object calculated by the construction status calculation unit 203 on the display device 4.

[0034] Fig. 3 is a diagram illustrating an example of a captured image taken by a measuring instrument of a crane system according to this embodiment. In this embodiment, measuring instrument 1 acquires a captured image in which a grab bucket, a pollution prevention frame, a barge, etc., which are examples of detection objects, are reflected. The captured image shown in Fig. 3 is an example in which all detection objects (e.g., pollution prevention frame, grab bucket, barge) are captured by one camera 1a (an example of measuring instrument 1), but depending on the angle of view and installation position of camera 1a and three-dimensional measuring instrument 1b, multiple cameras 1a and three-dimensional measuring instruments 1b may be provided, and detection objects such as pollution prevention frames, grab buckets, barges, etc. may be captured by separate cameras 1a and three-dimensional measuring instruments 1b.

[0035] In this embodiment, markers are placed around the detection target, such as a pollution prevention frame, a barge hold (where earth and sand is loaded), etc. The positional relationship between the features (markers M1 and M2) and the detection target is assumed to be known.

[0036] Fig. 4 is a diagram for explaining an example of the result of feature detection by the image processing device according to this embodiment. Specifically, the image shown in Fig. 4 is an example of an image obtained by detecting, as features, the vicinity P1 of the hanging point of the grab bucket and the markers M1 and M2 installed on the pollution prevention frame and the barge from the photographed image shown in Fig. 3 by the feature detection unit 202. In the image shown in Fig. 4, the detected features are indicated by frames.

[0037] The feature detection unit 202 acquires three-dimensional position information of the features from the three-dimensional data corresponding to the features detected from the image. When the reliability of the data from the measuring device 1 (e.g., three-dimensional measuring device 1b) for some features decreases due to the feature position and disturbances such as weather, or when the angle of view of three-dimensional measuring device 1b is narrower than that of camera 1a, and all features are captured by camera 1a but only some of the features are captured by the angle of view of three-dimensional measuring device 1b, it is possible to use camera 1a with distortion correction. This makes it possible to estimate the three-dimensional position information of unmeasurable features from the three-dimensional information of features that can be measured with high reliability (measurable features), the positional relationships between features, and the image positions of features that cannot be measured with high reliability (unmeasurable features).

[0038] In this embodiment, the feature detection unit 202 detects the pollution prevention frame and the barge by attaching dedicated markers M1 and M2 to them, but as in the example of a grab bucket, it is also possible to detect a location P1 that has a distinctive shape and is easy to detect as a feature without attaching markers. Also, in this embodiment, the feature detection unit 202 detects one to multiple features for one detection object.

[0039] In this embodiment, the feature detection unit 202 detects at least one feature when it is desired to detect only the position of the detection object. In this embodiment, the feature detection unit 202 detects at least three features when it is desired to detect both the position and orientation of the detection object. In other words, the feature detection unit 202 determines the placement of markers M1 and M2, the features to be detected, etc., depending on the content to be detected.

[0040] In this embodiment, the coordinate system of the three-dimensional measuring device 1b is expressed as a coordinate system 0m-XmYmZm. For the Xm coordinate, the center of the image is x=0m, and the left side of the drawing is expressed as a minus sign and the right side as a plus sign. For the Ym coordinate, the center of the image is y=0m, and the top side is expressed as a minus sign and the bottom side is expressed as a plus sign. For the Zm coordinate, all distances from the measuring device 1 (for example, camera 1a) are expressed as plus signs.

[0041] Here, an example of calculating the position and posture of a barge, which is an example of a detection target, will be described. The barge coordinate system is represented as the barge coordinate system Ob-XbYbZbOb. The Xb direction is parallel to the ship's width. The Yb direction is parallel to the ship's depth. The Zb direction is parallel to the ship's length. The origin Ob is the position of the marker on the front left. The position of marker M1 on the barge coordinate system and the hold size are known.

[0042] In this embodiment, using the left front marker position P1_m, the left rear marker position P2_m, and the right rear marker position P3_m as viewed from the coordinate system Om-XmYmZm, the basis vectors ex'_m, ey'_m, and ez'_m of the barge coordinate system Ob-XbYbZbOb as viewed from the coordinate system Om-XmYmZm are expressed by the following equations (1) to (3). ex'_m=(P2_m→P3_m) / |(P2_m→P3_m)|···(1) ez'_m=(P1_m→P2_m) / |(P1_m→P2_m)|···(2) ey'_m=ez'_m×ex'_m···(3)

[0043] Therefore, the transformation equations from the position of point P_b on the barge coordinate system Ob-XbYbZb to point P_m as viewed from the coordinate system Om-XmYmZm are expressed by the following equations (4) and (5). P_m={ex'_m,ey'_m,ez'_m}P_b+Om→Ob...(4) =[R]P_b+P1_m (5) Here, [R] is a rotation matrix from the barge coordinate system Ob-XbYbZb to the coordinate system Om-XmYmZm, and represents the attitude. If the attitude is expressed using ZYX Euler angles (θx, θy, θz), the relationship between the rotation matrix [R] and θx, θy, θz is expressed by the following equation (6).

number

[0044] From equation (6), the ZYX Euler angles θx, θy, and θz can be obtained from each component of the rotation matrix [R].

[0045] Generally, the position and orientation of a detection target object can be uniquely determined with the 3D position information of three features, so a minimum of three features are required to detect the position and orientation of all detection targets. As shown in the example of the pollution prevention frame, more than three features can be set, and three highly reliable features can be selected depending on the scene and used to calculate the position and orientation. Furthermore, to mitigate errors in the 3D position information, the position and orientation can be calculated using an optimization method or the like from the 3D position information of more than three features.

[0046] In this embodiment, the vectors between the feature positions are set to coincide with the base vectors of the barge coordinate system Ob-XbYbZb, but similar detection can be achieved if the arrangement of the features on the coordinate system of the object to be detected is known.

[0047] Furthermore, when detecting partial information about the position and orientation of the detection target, the number of features may be two or less. For example, if a grab bucket is suspended approximately parallel to the Y axis of a measuring device 1 such as a camera 1a, and only the position and the angle of rotation about the Y axis are considered to change, the position and the angle of rotation about the Y axis can be obtained from the three-dimensional data of two features, as shown in the example in the figure.

[0048] Fig. 5 is a diagram for explaining an example of a result of detecting the position and orientation of a detection object in the image processing device according to this embodiment. Specifically, Fig. 5 is an example of an image resulting from detecting a detection object from an image captured by a camera 1a during dredging work.

[0049] In the example shown in Fig. 5, the construction status calculation unit 203 calculates the position and posture of each detected object and notifies the calculation results to the crane control device 3. Based on the calculated position and posture, the crane control device 3 controls the movement of the grab between the pollution prevention frame area and the barge hold. The image processing device 2 displays the position and posture of the detected object on the display device 4, making it easier for workers to understand the situation in the construction area.

[0050] Fig. 6 is a diagram for explaining an example of the detection result of the sediment loading state of a barge by the image processing device according to this embodiment. Specifically, Fig. 6 shows an example of the detection result of the sediment loading state of a barge. For example, the sediment loading state of a barge may be detected for the purpose of automating dredging work.

[0051] The construction status calculation unit 203 acquires three-dimensional position information P_m in the area inside the hold of the barge detected by the process shown in Fig. 5. For example, the construction status calculation unit 203 converts it into three-dimensional position information P_b in the barge coordinate system Ob-XbYbZb using the following equations (7) and (8). P_m=[R]P_b+P1_m (7) P_b=inv([R])·(P_m-P1_m)···(8) Here, the Yb coordinate component of P_b is the height of the soil as seen from the barge.

[0052] As shown in Figure 6, the construction status calculation unit 203 divides the interior of the hold into multiple areas and determines whether the height of the representative soil and sand within the area is higher than the set height threshold (×), close to the height threshold (△), or lower than the height threshold (◯).

[0053] In this way, according to the crane system of this embodiment, the object to be detected is detected three-dimensionally using a measuring device capable of three-dimensional measurement (e.g., a stereo camera), and the three-dimensional position information of the detected object is fed back to the crane control, thereby realizing automation of dredging work.

[0054] Although the embodiments of the present invention have been described above, they are presented as examples and are not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents described in the claims.

[0055] For example, aspects of the present invention are as follows. <1> a feature detection unit that detects three-dimensional position information of an object within the construction area using a photographed image obtained by photographing the construction area of ​​the work machine with a photographing device and distance information obtained by measuring the construction area with a distance measuring device that measures the distance to the object photographed by the photographing device; and a construction status calculation unit that calculates the position and posture of the object using the three-dimensional position information of the object detected by the feature detection unit; a notification unit that notifies the position and posture of the object calculated by the construction status calculation unit; An image processing device comprising: <2> a data receiving unit that receives the captured image and the distance information; <1> The image processing device according to claim 1. <3> the feature detection unit detects at least three features of the object, and detects the three-dimensional position information based on the detection results of the three features. <1> or <2> The image processing device according to claim 1. <4> the feature detection unit detects features present on two or more sides of the object; <1> from <3> 10. The image processing device according to claim 9, wherein <5> The distance between the objects is approximately equal to or greater than the distance from the imaging device and the distance measuring device to the construction area. <1> from <4> 10. The image processing device according to claim 9, wherein <6> The construction status calculation unit calculates the distribution of soil and sand in the construction area using the three-dimensional position information of the object. <1> from <5> 10. The image processing device according to claim 9, wherein <7> the feature detection unit uses a learning algorithm to detect the three-dimensional position information of the object based on a model created in advance; <1> from <6> 10. The image processing device according to claim 9, wherein <8> When the reliability of a part of the three-dimensional position information of the object is low, the construction status calculation unit calculates the position and orientation of the object based on the three-dimensional position information of the object having high reliability, the positional relationship between the objects, and the distortion-corrected captured image. <1> from <7> 10. The image processing device according to claim 9, wherein <9> An image processing method executed by an image processing device, comprising: a step of detecting three-dimensional position information of an object within the construction area using a photographed image obtained by photographing the construction area of ​​the work machine with a photographing device and distance information obtained by measuring the construction area with a distance measuring device that measures the distance to the object photographed by the photographing device; calculating a position and orientation of the detected object using the three-dimensional position information of the object; a step of reporting the calculated position and orientation of the object; An image processing method comprising: <10> Computer, a feature detection unit that detects three-dimensional position information of an object within the construction area using a photographed image obtained by photographing the construction area of ​​the work machine with a photographing device and distance information obtained by measuring the construction area with a distance measuring device that measures the distance to the object photographed by the photographing device; and a construction status calculation unit that calculates the position and posture of the object using the three-dimensional position information of the object detected by the feature detection unit; a notification unit that notifies the position and posture of the object calculated by the construction status calculation unit; A program to make it function as such. [Explanation of symbols]

[0056] 1 Measuring instrument 1a Camera 1b Coordinate measuring device 2. Image processing device 3 Crane control device 4 Display Devices 201 Data receiving unit 202 Feature detection unit 203 Construction status calculation unit 204 Display control unit 911 CPU 912 ROM 913 RAM M1, M2 markers [Prior art documents] [Patent documents]

[0057] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-248719

Claims

1. a feature detection unit that detects three-dimensional position information of an object within the construction area using a photographed image obtained by photographing the construction area of ​​the work machine with a photographing device and distance information obtained by measuring the construction area with a distance measuring device that measures the distance to the object photographed by the photographing device; and a construction status calculation unit that calculates the position and posture of the object using the three-dimensional position information of the object detected by the feature detection unit; a notification unit that notifies the position and posture of the object calculated by the construction status calculation unit; An image processing device comprising:

2. The image processing device according to claim 1 , further comprising a data receiving unit that receives the captured image and the distance information.

3. The image processing device according to claim 1 , wherein the feature detection unit detects at least three features of the object, and detects the three-dimensional position information based on the detection results of the three features.

4. The image processing device according to claim 1 , wherein the feature detection unit detects features present on two or more sides of the object.

5. The image processing device according to claim 1 , wherein the distance between the objects is approximately equal to or greater than the distance from the photographing device and the distance measuring device to the construction area.

6. The image processing device according to claim 1 , wherein the construction status calculation unit calculates a distribution of soil and sand in the construction area using the three-dimensional position information of the object.

7. The image processing device according to claim 1 , wherein the feature detection unit uses a learning algorithm that detects the three-dimensional position information of the object based on a model created in advance.

8. 2. The image processing device according to claim 1, wherein, when the reliability of part of the three-dimensional position information of the object is low, the construction status calculation unit calculates the position and posture of the object based on the three-dimensional position information of the object with high reliability, the positional relationship between the objects, and the distortion-corrected captured image.

9. An image processing method executed by an image processing device, comprising: a step of detecting three-dimensional position information of an object within the construction area using a photographed image obtained by photographing the construction area of ​​the work machine with a photographing device and distance information obtained by measuring the construction area with a distance measuring device that measures the distance to the object photographed by the photographing device; calculating a position and orientation of the detected object using the three-dimensional position information of the object; a step of reporting the calculated position and orientation of the object; An image processing method comprising:

10. Computer, a feature detection unit that detects three-dimensional position information of an object within the construction area using a photographed image obtained by photographing the construction area of ​​the work machine with a photographing device and distance information obtained by measuring the construction area with a distance measuring device that measures the distance to the object photographed by the photographing device; and a construction status calculation unit that calculates the position and posture of the object using the three-dimensional position information of the object detected by the feature detection unit; a notification unit that notifies the position and posture of the object calculated by the construction status calculation unit; A program to make it function as such.

Images