Image processing device, image processing method and program

Abstract

To specify reliable information from an output of a learned model to be used for three-dimensional image measurement.SOLUTION: An image processing device incudes: an image acquisition part for acquiring a first viewpoint image and a second viewpoint image; an inference part for inputting a first image and a second image and using a first learned model for associating and outputting, for each point in the first image, a first estimated parallax with first position coordinates in the first image to input the first viewpoint image as the first image and the second viewpoint image as the second image in the first learned model to acquire a first estimated parallax associated with the first position coordinates; and high reliability area specification means for calculating, for each point in the first viewpoint image, an estimated position of a correspondence point in the second viewpoint image from the first position coordinates and the first estimated parallax, generating association data obtained by associating the first position coordinates, the first estimated parallax, and the estimated position of the correspondence point in the second viewpoint image with one another, and specifying a set of the first position coordinates and the first estimated parallax having high reliability of the first estimated parallax from a group of the association data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] There is a known technology for performing 3D image measurement to acquire 3D information such as depth in an image based on the parallax between multiple images with different viewpoints, such as stereo images. This type of technology estimates corresponding points in one image, i.e., points in another image, to determine the parallax between the images. Furthermore, as shown in Non-Patent Document 1, for example, it has been proposed to apply machine learning technology to 3D image measurement processing. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] L. Lipson, Z. Teed and J. Deng, "RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching," 2021 International Conference on 3D Vision (3DV), London, United Kingdom, 2021, pp. 218-227, doi: 10.1109 / 3DV53792.2021.00032. [online]<URL:https: / / arxiv.org / pdf / 2109.07547.pd> ,[Searched on February 28, 2020] Summary of the Invention [Problem to be solved by the invention]

[0004] In 3D image measurement, it is theoretically impossible to calculate correct parallax for points that appear in only one image, because it is not possible to identify corresponding points in other images. For example, it is not possible to determine corresponding points for a point that appears in one image but is blocked by another object in the other image, making it a blind spot.

[0005] However, when a trained model developed through machine learning is used for 3D image measurement, the trained model can calculate disparity even in areas where corresponding points are uncertain by using learned information to complement the data.As a result, the output of the trained model may contain a mixture of highly reliable information based on areas where corresponding points are certain and less reliable information calculated from areas where corresponding points are uncertain, and it has been difficult to distinguish between them.

[0006] The present invention has been made in consideration of the above, and aims to provide an image processing device, an image processing method, and a program that are capable of identifying reliable information from the output of a trained model used for 3D image measurement. [Means for solving the problem]

[0007] The image processing device according to the present invention is an image processing device that performs three-dimensional image measurement processing based on the disparity between a plurality of images, and includes an image acquisition unit that acquires a first viewpoint image and a second viewpoint image captured from a first viewpoint and a second viewpoint, respectively, and a first trained model that inputs the first image and the second image, and outputs a first estimated disparity, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate that represents the position of the point in the first image, and The system includes an inference unit that inputs the second viewpoint image as a second image into a first trained model and acquires a first estimated disparity associated with a first position coordinate; and a high-reliability area identification means that, for each point in the first viewpoint image, determines an estimated position of a corresponding point in the second viewpoint image from the first position coordinate and the first estimated disparity, generates association data in which the first position coordinate, the first estimated disparity, and the estimated position of the corresponding point in the second viewpoint image are associated, and identifies, from the collection of association data, a pair of first position coordinate and first estimated disparity for which the reliability of the first estimated disparity is high.

[0008] The image processing device according to the present invention is an image processing device that performs three-dimensional image measurement processing based on the disparity between a plurality of images, and includes an image acquisition unit that acquires a first viewpoint image and a second viewpoint image captured from a first viewpoint and a second viewpoint, respectively, and a first trained model that inputs the first image and the second image, and outputs a first estimated disparity, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate that represents the position of the point in the first image, and the first viewpoint image is a first image, and the second viewpoint image is a second viewpoint image. and an inference unit that inputs the first image into a first trained model as a second image and acquires a first estimated disparity associated with a first position coordinate. Also, for each point in the first viewpoint image, a high-reliability area identification means is provided that calculates an estimated position of a corresponding point in the second viewpoint image from the first position coordinate and the first estimated disparity, calculates a first depth corresponding to the first estimated disparity, generates association data in which the first position coordinate, the first depth, and the estimated position of the corresponding point in the second viewpoint image are associated, and identifies a pair of first position coordinate and first depth with a high reliability for the first depth from the set of association data.

[0009] An image processing method according to the present invention is an image processing method for performing three-dimensional image measurement processing based on disparity between multiple images, and includes the steps of acquiring a first viewpoint image and a second viewpoint image captured from a first viewpoint and a second viewpoint, respectively; inputting the first image and the second image, and using a trained model that outputs a first estimated disparity, which is a machine-learned inference value of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate representing the position of the point in the first image; inputting the first viewpoint image as the first image and the second viewpoint image as the second image into the trained model to acquire the first estimated disparity associated with the first position coordinate; and determining, for each point in the first viewpoint image, an estimated position of the corresponding point in the second viewpoint image from the first position coordinate and the first estimated disparity, generating association data in which the first position coordinate, the first estimated disparity, and the estimated position of the corresponding point in the second viewpoint image are associated with each other; and identifying, from the collection of association data, a pair of first position coordinates and first estimated disparity that has a high reliability for the first estimated disparity.

[0010] The program of the present invention is a program for performing 3D image measurement processing based on disparity between multiple images, and causes a computer to execute the following processes: a process for acquiring a first viewpoint image and a second viewpoint image captured from a first viewpoint and a second viewpoint, respectively; a trained model is available that inputs the first and second images and outputs a first estimated disparity, which is a machine-learned inference value of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate representing the position of the point in the first image; a process for inputting the first viewpoint image as the first image and the second viewpoint image as the second image into the trained model to acquire the first estimated disparity associated with the first position coordinate; and a process for determining, for each point in the first viewpoint image, an estimated position of a corresponding point in the second viewpoint image from the first position coordinate and the first estimated disparity, generating association data in which the first position coordinate, the first estimated disparity, and the estimated position of the corresponding point in the second viewpoint image are associated with each other, and identifying, from the collection of association data, a pair of first position coordinates and first estimated disparity with a high reliability of the first estimated disparity. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an image processing device, an image processing method, and a program that are capable of identifying reliable information from the output of a trained model used for three-dimensional image measurement. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram schematically illustrating an overview of an image processing apparatus according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating calculation of depth information. [Figure 3] FIG. 3 is a block diagram illustrating a computer system according to an embodiment. [Figure 4] FIG. 4 is a block diagram showing a specific example of the configuration of the image processing device according to the first embodiment. [Figure 5] FIG. 5 is a schematic diagram showing a specific example for explaining the difference between the areas captured in the first viewpoint image and the second viewpoint image. [Figure 6] FIG. 6 is a schematic diagram showing the first viewpoint image and the second viewpoint image acquired in the specific example shown in FIG. [Figure 7] FIG. 7 is a schematic diagram showing a specific example of the first estimated disparity output by the first trained model when the first viewpoint image and the second viewpoint image are input. [Figure 8] FIG. 8 is a schematic diagram illustrating corresponding points. [Figure 9] FIG. 9 is a schematic diagram showing the first estimated parallax and the second estimated parallax. [Figure 10] FIG. 10 is a schematic diagram showing a first example of obtaining the second estimated parallax. [Figure 11] FIG. 11 is a schematic diagram showing a second example of obtaining the second estimated parallax. [Figure 12] FIG. 12 is a schematic diagram showing a third example of obtaining the second estimated parallax. [Figure 13] FIG. 13 is a schematic diagram showing the relationship between the first estimated disparity, corresponding points, and the second estimated disparity. [Figure 14] FIG. 14 is a schematic diagram showing an example of reliability information. [Figure 15] FIG. 15 is a diagram illustrating a method for acquiring reliability information by the high-reliability region specifying means according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a method for calculating depth information in the depth calculation unit. [Figure 17] FIG. 17 is a diagram schematically illustrating another example of a method for calculating depth information in the depth calculation unit. [Figure 18] FIG. 18 is a flowchart showing the image processing method according to the first embodiment. [Figure 19] FIG. 19 is a flowchart showing the reliability determination process according to the first embodiment. [Figure 20] FIG. 20 is a schematic diagram for explaining a reliability determination method according to the second embodiment. [Figure 21] FIG. 21 is a flowchart showing the reliability determination process according to the second embodiment. [Figure 22] FIG. 22 is a diagram illustrating calculation of depth information by a depth calculation unit according to the third embodiment. [Figure 23] FIG. 23 is a diagram illustrating the determination of reliability by the high-reliability region specifying means according to the third embodiment. [Figure 24] FIG. 24 is a flowchart showing an image processing method according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of an image processing device, an image processing method, and a program according to the present invention will be described with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the components in the following embodiments include those that are easily replaceable by those skilled in the art, or those that are substantially identical.

[0014] [First embodiment] (Image Processing Device Overview) Fig. 1 is a diagram schematically illustrating an overview of an image processing device 100 according to a first embodiment. The image processing device 100 illustrated in Fig. 1 is an image processing device that performs three-dimensional image measurement processing based on the parallax between multiple images.

[0015] The image processing device 100 acquires disparity information between multiple images using a machine-learned trained model, and acquires depth information 70 representing the depth of each point in the image based on the acquired disparity information. The depth information 70 for each point in the image makes it possible to grasp the three-dimensional position of an object appearing in the image.

[0016] The image processing device 100 shown in FIG. 1 includes an image acquisition unit 1, an inference unit 2, a highly reliable region identification means 3, a depth calculation unit 4, an output unit 5, and a storage unit 6.

[0017] The image acquisition unit 1 acquires multiple images to be subjected to 3D image measurement processing. The multiple images are images of the same object (subject) captured from different viewpoints. The number of images, i.e., the number of viewpoints, is not particularly limited as long as there is a plurality of images. The image acquisition unit 1 acquires at least a first viewpoint image 41 and a second viewpoint image 42 captured from a first viewpoint and a second viewpoint, respectively. The first viewpoint image 41 and the second viewpoint image 42 are, for example, stereo images. The stereo images are images captured from two viewpoints (the first viewpoint and the second viewpoint) that are offset in parallel, and constitute a pair of images. The direction in which the viewpoints of the stereo images shift is not particularly limited, but in one example, it coincides with the horizontal (left-right) direction of the images. The direction in which the viewpoints shift is the direction in which parallax occurs between the images, and is referred to as the parallax direction in this specification. In addition, the imaging conditions (camera focal length, number of images, angle of view, etc.) of the two stereo images are the same.

[0018] The stereo image is captured by an imaging device 20 including, for example, a left camera 21 and a right camera 22. The left camera 21 and the right camera 22 include, but are not limited to, imaging elements such as a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor). In this embodiment, the left camera 21 and the right camera 22 have the same and known optical characteristics (focal length, angle of view, etc.). The left camera 21 and the right camera 22 have parallel optical axes and are arranged at a predetermined interval in the left-right direction of the captured image. The baseline length, which is the distance between the optical axes of the left camera 21 and the right camera 22, is known. The left camera 21 and the right camera 22 can capture still images at the same time. The imaging fields of the left camera 21 and the right camera 22 overlap each other. In the following description, the origin of the horizontal position in the image is defined as the left edge of the image. The left camera 21 and the right camera 22 are also defined as being at the same vertical position in the image.

[0019] The image processing device 100 and the imaging device 20 can communicate with each other via a wired or wireless connection. The image processing device 100 may also be able to communicate with the imaging device 20 via a network such as the Internet. The image acquisition unit 1 acquires a first viewpoint image 41 and a second viewpoint image 42 captured by the imaging device 20. The first viewpoint image 41 is, for example, a left image captured by the left camera 21. In this case, the first viewpoint is the position of the optical axis of the left camera 21. The second viewpoint image 42 is, for example, a right image captured by the right camera 22. In this case, the second viewpoint is the position of the optical axis of the right camera 22. Note that the first viewpoint image 41 may be a right image and the second viewpoint image 42 may be a left image.

[0020] The inference unit 2 acquires information on disparity between images from multiple images acquired by the image acquisition unit 1. The inference unit 2 acquires disparity using a trained model generated by machine learning. The inference unit 2 is able to use a first trained model 12. The first trained model 12 receives a first image and a second image, and outputs a first estimated disparity 51, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point in the second image, in association with first position coordinates representing the position of the point in the first image. The inference unit 2 inputs the first viewpoint image 41 as the first image and the second viewpoint image 42 as the second image into the first trained model 12, and acquires the first estimated disparity 51 associated with the first position coordinates.

[0021] Disparity is the displacement of coordinates in the parallax direction between a pair of corresponding points in one viewpoint image and another viewpoint image. A spatial position captured at a point of interest P (see FIG. 2), which is a pixel in one viewpoint image, is captured at a corresponding point Q (see FIG. 2), which is a pixel shifted in the parallax direction, in the other viewpoint image. When the point of interest P is used as a reference, disparity is the amount of displacement of the position coordinates of the corresponding point Q relative to the position coordinates of the point of interest P. The trained model may output only an estimated disparity based on one viewpoint image as a reference, or may output both an estimated disparity based on one viewpoint image as a reference and an estimated disparity based on the other viewpoint image as a reference. In this embodiment, the inference unit 2 outputs at least a first estimated disparity 51 (see FIG. 7) of each point in the first viewpoint image 41. The first estimated disparity 51 is information indicating the amount of displacement (amount of pixel shift) of the corresponding point Q in the second viewpoint image 42 relative to each point (point of interest P) in the first viewpoint image 41.

[0022] The inference unit 2 may output the estimated disparity in any representation format. The inference unit 2 may output a disparity image in which the estimated disparity at each point is represented in an image format. The pixel value of each pixel in the disparity image represents the disparity. The inference unit 2 may output the estimated disparity at each point in any data format, such as CSV (comma-separated values).

[0023] The high-reliability region identifying means 3 identifies a pair of first position coordinates and the first estimated disparity 51 for which the reliability of the first estimated disparity 51 is high. Here, a set of first position coordinates for which the reliability of the first estimated disparity 51 is high, which is identified by the high-reliability region identifying means 3, is referred to as a high-reliability region. The high-reliability region identifying means 3 outputs reliability information 60 for identifying the high-reliability region. The reliability information 60 is information identifying a region in the first viewpoint image 41 that has high reliability. The reliability information 60 may identify the high-reliability region by including only the first position coordinates of points for which the reliability of the first estimated disparity 51 is high. The reliability information 60 may identify the high-reliability region by including information indicating whether the reliability of all points in the first viewpoint image 41 is high. The reliability information 60 may identify the high-reliability region by including information indicating whether the reliability of each point in the first viewpoint image 41 is high. The reliability information 60 is binary information consisting of, for example, either "1" indicating high reliability or "0" indicating low reliability. In this case, points in the first viewpoint image 41 that are assigned data of "1" are high reliability areas.

[0024] The depth calculation unit 4 calculates depth information 70 representing the depth of each point in the first viewpoint image 41 based on the first estimated parallax 51 and the reliability information 60. The depth calculation unit 4 calculates the depth information 70 based on the first viewpoint image 41, the first estimated parallax 51 of each pixel in the first viewpoint image 41, and camera parameters 13. The camera parameters 13 are optical characteristics of the image capture device 20 that captured the first viewpoint image 41 and the second viewpoint image 42, and are information used to calculate the depth information 70. The camera parameters 13 include information on the baseline length B of the image capture device 20, information on the focal length f, etc.

[0025] 2 is a diagram illustrating the calculation of the depth information 70. The point of interest P of the first viewpoint image 41 (left image) is the origin O of the left camera 21. L The coordinates in the left camera coordinate system (u L ,v L The corresponding point Q in the second viewpoint image 42 (right image) is expressed as the origin O of the right camera 22. R The coordinates in the right camera coordinate system (u R ,vR The point of interest P in the first viewpoint image 41 and the corresponding point Q in the second viewpoint image 42 are the same point M in space. L =(X,Y,Z) to point M L represents the coordinate value of point M in the left camera coordinate system. In parallel stereo images, the corresponding point pair between the first viewpoint image 41 (left image) and the second viewpoint image 42 (right image) is at the same position in the vertical direction in the image, so v L and v R In the following description, the vertical position will not be mentioned.

[0026] The relationship between the target point P, the corresponding point Q, and the point M in space is expressed by equations (1) and (2).

number

number

[0027] This allows the depth calculation unit 4 to calculate depth information 70 representing the depth Z for each point in the first viewpoint image 41. In the present embodiment, the depth calculation unit 4 calculates depth information 70 representing the depth Z for each point identified as a pair of the first position coordinate and the first estimated disparity 51 for which the reliability of the first estimated disparity 51 is high. The depth calculation unit 4 does not need to output depth information 70 for points in the first viewpoint image 41 that are not highly reliable. For points that are not highly reliable, the depth calculation unit 4 may output the depth information 70 together with the calculated depth Z, adding supplemental information indicating that the reliability is not high.

[0028] The depth calculation unit 4 may output the depth information 70 in any format. The depth calculation unit 4 may output a depth image that represents the depth Z of each point in an image format as the depth information 70. The depth calculation unit 4 may output the depth information 70 of each point in any data format, such as CSV.

[0029] The output unit 5 outputs information generated in the image processing device 100 to the data utilization device DU. The output unit 5 outputs depth information 70 calculated by the depth calculation unit 4 to the data utilization device DU. The output unit 5 may output reliability information 60 and disparity information in addition to the depth information 70. The image processing device 100 and the data utilization device DU can communicate with each other via a wired or wireless connection. The image processing device 100 may also be able to communicate with the data utilization device DU via a network such as the Internet.

[0030] The data utilization device DU is not particularly limited. For example, the data utilization device DU may be a control device for controlling a robot based on the depth information 70, or a mobile object control device for setting a route for a mobile object based on the depth information 70. The data utilization device DU may also be a SLAM (Simultaneous Localization and Mapping) calculation device for creating an environmental map and estimating a self-position based on the depth information 70. The data utilization device DU may be provided in the image processing device 100. In other words, the image processing device 100 may include, as the data utilization device DU, a device for performing calculation processing using the depth information 70.

[0031] The storage unit 6 stores various types of information used in the image processing device 100. The storage unit 6 stores a program 11, a first trained model 12, and camera parameters 13. The program 11 is an application program that causes a computer to function as an image acquisition unit 1, an inference unit 2, a high-reliability region identification means 3, a depth calculation unit 4, and an output unit 5. The storage unit 6 includes a rewritable auxiliary storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive).

[0032] FIG. 3 is a block diagram showing a computer system 1000 according to an embodiment. The image processing device 100 described above includes the computer system 1000. The computer system 1000 includes a processor 1001 such as a central processing unit (CPU), a main memory 1002 including a nonvolatile memory such as a read-only memory (ROM) and a volatile memory such as a random access memory (RAM), a storage 1003, and an interface 1004 including an input / output circuit. The functions of the image processing device 100 are stored in the storage 1003 as a program 11. The storage 1003 implements the memory unit 6 of the image processing device 100. The processor 1001 reads the program 11 from the storage 1003, loads it into the main memory 1002, and executes various processes in accordance with the program 11. The program 11 causes the processor 1001 to function as an image acquisition unit 1, an inference unit 2, a high-reliability region identification unit 3, a depth calculation unit 4, and an output unit 5. The program 11 may be distributed to the computer system 1000 via a network.

[0033] (Specific configuration example of image processing device) 4 is a block diagram showing a specific configuration example of the image processing device 100 according to the first embodiment. In FIG. 4, an example in which the image processing device 100 according to the first embodiment is applied to a robot system 110 is shown.

[0034] The robot system 110 includes the image processing device 100 according to the first embodiment, an imaging device 20, a camera-moving robot 30, a robot control device 31 that controls the camera-moving robot 30, a work robot 32, and a robot control device 33 that controls the work robot 32. That is, the robot system 110 includes a robot vision system that performs three-dimensional image measurement processing on stereo images captured by the imaging device 20 using the image processing device 100 and performs object recognition based on the resulting depth information 70. The robot system 110 then executes a predetermined task using the work robot 32 based on the results of the object recognition.

[0035] The working robot 32 has a drive source such as a motor and a movable part that is operated by the drive source, and performs a predetermined task by moving the movable part. There are no particular limitations on the type of working robot 32, and the content of the task performed by the working robot 32 is also not limited. Examples of types of working robot 32 include vertical articulated robots, SCARA robots, parallel robots, Cartesian robots, dual-arm or multi-arm robots, and humanoid robots. Examples of the content of the task include picking, palletizing, transporting, parts assembly, processing, joining work such as welding, and collaborative work with workers.

[0036] The camera moving robot 30 has a drive source such as a motor and a movable part operated by the drive source, and moves while holding the imaging device 20. The camera moving robot 30 moves the imaging device 20 so as to capture images of the entire working area of ​​the work robot 32 and the area around the work robot 32. There are no particular limitations on the type of camera moving robot 30. The camera moving robot 30 may be an electric platform that can change the angle, such as panning and tilting, or may be an articulated robot that can change the angle and position.

[0037] The image processing device 100 includes an object recognition unit 7 and an imaging control unit 8 in addition to the above-mentioned image acquisition unit 1, inference unit 2, high-reliability region identification means 3, depth calculation unit 4, output unit 5, and storage unit 6. In this example, the object recognition unit 7 corresponds to the data utilization device DU.

[0038] The object recognition unit 7 recognizes an object from the image based on the depth information 70 calculated by the depth calculation unit 4. The object recognition unit 7 acquires CAD data 14 of the object to be recognized from the memory unit 6. The CAD data 14 defines the three-dimensional shape of the object. The object recognition unit 7 performs a three-dimensional matching process of the object defined by the CAD data 14 against the depth information 70 to acquire the position and orientation of the recognition target. The object to be recognized is, for example, an object to be worked on by the work robot 32.

[0039] Based on the reliability information 60, the depth calculation unit 4 outputs depth information 70 for each point identified as a pair of first position coordinates and first estimated disparity 51 with high reliability for the first estimated disparity 51. The object recognition unit 7 performs 3D matching based on the depth information 70 for the highly reliable points. If depth information 70 for points with low reliability exists, the object recognition unit 7 does not use the depth information 70 for the low reliability points in the 3D matching process. When CAD data 14 is used for object recognition, the outer shape of the object to be recognized is known. Therefore, the depth information 70 is used to identify the position and orientation of the object to be recognized, rather than to measure the outer shape of the object to be recognized. In this case, it is sufficient to obtain enough position information to identify the position and orientation of the object. Therefore, even if the number of available depth information 70 points is reduced, the recognition accuracy of the object's position and orientation is improved by using only the depth information 70 for points with high reliability. The object recognition unit 7 may use depth information 70 of not only highly reliable points but also less reliable points. For example, a greater weight may be assigned to the depth information 70 of highly reliable points and a smaller weight may be assigned to the depth information 70 of less reliable points, and object recognition processing may be performed based on the weighting. The output unit 5 outputs the position and orientation information of the recognition target generated by the object recognition unit 7 to the robot control device 33.

[0040] The robot controller 33 controls the working robot 32. The robot controller 33 operates the working robot 32 based on the information on the position and orientation of the recognition target output from the output unit 5.

[0041] The imaging control unit 8 generates control information for the robot control device 31 based on the depth information 70 calculated by the depth calculation unit 4 or the position and orientation information of the recognition target generated by the object recognition unit 7. The output unit 5 outputs the control information generated by the imaging control unit 8 to the robot control device 31. The robot control device 31 controls the camera-moving robot 30. The robot control device 31 operates the camera-moving robot 30 based on the control information output from the output unit 5.

[0042] The field of view of the imaging device 20 may be fixed, in which case the camera moving robot 30, robot control device 31 and imaging control unit 8 may not be provided.

[0043] (Detailed explanation of reliability information) Fig. 5 is a schematic diagram showing a specific example for explaining the difference in the areas captured in a first viewpoint image 41 and a second viewpoint image 42. Fig. 6 is a schematic diagram showing the first viewpoint image 41 and the second viewpoint image 42 acquired in the specific example shown in Fig. 5. For convenience of explanation, Fig. 5 shows the imaging system of the imaging device 20 as a pinhole camera model. For convenience of explanation, Fig. 6 simplifies the image by reducing the number of pixels of the captured image.

[0044] The left camera 21 acquires an image projected onto a virtual image plane 25 within the imaging field of view 23 as a first viewpoint image 41. The right camera 22 acquires an image projected onto a virtual image plane 26 within the imaging field of view 24 as a second viewpoint image 42. FIG. 5 shows an example in which an object 90 is imaged by the imaging device 20. Of the front of the object 90, a region 91 is included in both the imaging field of view 23 and the imaging field of view 24. A region 92 of the object 90 is outside the imaging field of view 23 of the left camera 21 and is included only in the imaging field of view 24. A region 93, which is the side of the object 90, is in a blind spot blocked by the front from the imaging field of view 24 of the right camera 22 and is therefore included only in the imaging field of view 23 of the left camera 21.

[0045] 6, a region 91 of the object 90 is commonly captured in the first viewpoint image 41 and the second viewpoint image 42. The first viewpoint image 41 does not capture a region 92, but captures a region 93 of the object 90. The second viewpoint image 42 does not capture a region 93 of the object 90, but captures a region 92.

[0046] (Obtaining the first estimated disparity) FIG. 7 is a schematic diagram showing a specific example of a first estimated disparity 51 output by the first trained model 12 when a first viewpoint image 41 and a second viewpoint image 42 are input. The first trained model 12 has a left channel Lch that inputs a left image (first image) of the stereo images, a right channel Rch that inputs a right image (second image) of the stereo images, and an output channel Och that outputs an estimated disparity that is an inference result. The inference unit 2 inputs the first viewpoint image 41 to the left channel Lch of the first trained model 12 and inputs the second viewpoint image 42 to the right channel Rch of the first trained model 12. The first trained model 12 outputs, from the output channel Och, the first estimated disparity 51 for each point (each pixel) of the image input to the left channel Lch, i.e., the first viewpoint image 41.

[0047] There is no particular limitation on the type of the first trained model 12. In summary, the first trained model 12 is designed to perform a feature extraction process for extracting feature vectors of each of the input first viewpoint image 41 and second viewpoint image 42, a corresponding point matching process for finding the most similar corresponding point pair between the feature vectors of the first viewpoint image 41 and the second viewpoint image 42, and an output process for determining a disparity d representing a displacement in the disparity direction from the obtained corresponding point pair and outputting the result in a set data format, and is machine-trained to obtain an output with a desired accuracy.

[0048] The disparity is expressed as d(u,v). The disparity with the left image (first image) as the reference is d l (u l ,v l ) As mentioned above, if the vertical direction of each image is considered to be the same, then the same row (same Y coordinate v L、 v R ), the first estimated disparity 51 is expressed in the following data format D1. [Data format D1] (u l , d l * ) where u l is the position (first position coordinate) in the left image (first viewpoint image 41), d l *is the estimated disparity output by the first trained model 12. l , estimated disparity d l * 7 shows an example in which the first estimated parallax 51 is output in the form of a parallax image, for convenience's sake, and each pixel of the first estimated parallax 51 represents the parallax (amount of pixel shift to the corresponding point Q in the second viewpoint image 42) with respect to the first viewpoint image 41.

[0049] The first trained model 12 is designed and trained to output the disparity of each point in the first image (here, the first viewpoint image 41) input to the left channel Lch. In order to output the disparity of each point, the disparity of points for which the disparity cannot be calculated is interpolated with a value estimated to be most likely by machine learning. Therefore, in the first estimated disparity 51, it is highly likely that an accurate disparity has been calculated for an area 91 that appears in both the first viewpoint image 41 and the second viewpoint image 42. However, for the disparity of an area that appears only in the first viewpoint image 41 (here, an area 93 of the object 90), the value is interpolated by the first trained model 12, and therefore the output value may not be accurate.

[0050] Therefore, the high-reliability region identifying means 3 according to this embodiment generates association data in which, for each point in each first viewpoint image 41, first position coordinates, a first estimated parallax 51, and an estimated position of a corresponding point Q in the second viewpoint image 42 are associated with each other. From the collection of association data, the high-reliability region identifying means 3 identifies a pair of first position coordinates and first estimated parallax 51 for which the reliability of the first estimated parallax 51 is high. In the example of FIG. 7 , the high-reliability region identifying means 3 identifies a point (pixel) containing the region 91 and its estimated parallax as a pair of first position coordinates and first estimated parallax 51 for which the reliability of the first estimated parallax 51 is high. The high-reliability region identifying means 3 outputs reliability information 60 identifying the region 91 as a high-reliability region. The high-reliability region identifying means 3 makes it possible to identify a region with high reliability among the points in the first viewpoint image 41 using the reliability information 60. The high-reliability region identifying process performed by the high-reliability region identifying means 3 will now be described.

[0051] (Getting corresponding points) The high-reliability region specifying means 3 uses the first position coordinates (u l ) and the first estimated disparity 51(d l * ) to obtain an estimated position of the corresponding point Q in the second viewpoint image 42. From equation (3), the position of the corresponding point Q is expressed by equation (5).

number

[0052] FIG. 8 is a schematic diagram illustrating the corresponding point Q. In FIG. 8, the numerical value of each point in the first viewpoint image 41, which is the left image, is the first position coordinate (u l The first estimated parallax 51 is the parallax (d l * ) is shown. For example, the point of interest P(u l = 5), the corresponding point Q in the right image (second viewpoint image 42) is the disparity (d l * =2) is subtracted from the point (u r * = 3). By calculating the corresponding points Q for all points in the first viewpoint image 41, the estimated positions (u r * That is, the highly reliable region specifying means 3 adds the estimated position (u r * The associated data is obtained as data format D2 with the addition of [Data format D2] (u l , d l * , u r * ) In FIG. 8, the pixel value of each point in the first viewpoint image 41 is calculated based on the estimated position (u r *) is shown. The corresponding point map 45 is an image of the first position coordinate (u l ) and its first position coordinate (u l ) and the estimated position of the corresponding point Q (u r * ) is used to identify the high-reliability region. r * ) is sufficient, so there is no need to generate an image (correspondence point map 45) that shows the estimated positions of the correspondence points Q.

[0053] (Obtaining the second estimated disparity) In the first embodiment, the high-reliability area identification means 3 acquires a second estimated parallax 52 relating to the three-dimensional position of each point in the second viewpoint image 42 as a comparison object for identifying highly reliable data in the first estimated parallax 51.

[0054] 9 is a schematic diagram showing the first estimated parallax 51 and the second estimated parallax 52. The second estimated parallax 52 represents the parallax (amount of pixel shift to a corresponding point in the first viewpoint image 41) with the second viewpoint image 42 as the reference. In other words, the first estimated parallax 51 is information indicating the amount of displacement from a pixel in the first viewpoint image 41 as the point of interest P to a corresponding point Q in the second viewpoint image 42, whereas the second estimated parallax 52 is information indicating the amount of displacement from a pixel in the second viewpoint image 42 as the point of interest P to a corresponding point Q in the first viewpoint image 41. The parallax with the right image as the reference is d r * (u r ,v r )

[0055] The second estimated disparity 52 is represented in the following data format D3. [Data format D3] (u r , d r * ) However, u r is the second position coordinate representing the position in the right image (second viewpoint image 42). r * is the estimated parallax (value of the second estimated parallax 52) based on the right image (second viewpoint image 42). r, estimated disparity d r * 7 shows an example in which the second estimated parallax 52 is output in the form of a parallax image, and the pixel value of each pixel of the second estimated parallax 52 is expressed as the second position coordinate (u r 9 , the points included in the same row of the first estimated parallax 51 and the second estimated parallax 52 represent the estimated parallax of the corresponding point pair in the first viewpoint image 41 and the second viewpoint image 42, respectively, based on the left image and the right image.

[0056] There are no particular limitations on the method for obtaining the second estimated parallax 52. In the first embodiment, three examples of the method for obtaining the second estimated parallax 52 are shown.

[0057] (First example of obtaining second estimated disparity 52) FIG. 10 is a schematic diagram showing a first acquisition example of the second estimated disparity 52. ​​In the first acquisition example, the first estimated disparity 51 and the second estimated disparity 52 are acquired using separate trained models (12A, 12B). Specifically, the inference unit 2 outputs the first estimated disparity 51 using the first trained model 12A. The first trained model 12A is the same as the above-mentioned first trained model 12. The inference unit 2 outputs the second estimated disparity 52 using the second trained model 12B. That is, the inference unit 2 inputs the first image and the second image in addition to the first trained model 12A, and acquires the second estimated disparity 52 (d r * ) into the second position coordinates (u r The inference unit 2 inputs the first viewpoint image 41 as the first image and the second viewpoint image 42 as the second image into the second trained model 12B, and outputs the second position coordinates (u r ) and the second estimated disparity 52 (d r * ) to get the

[0058] The inference unit 2 inputs a first viewpoint image 41, which is a left image (first image), to the left channel Lch of the first trained model 12A and the second trained model 12B, respectively, and inputs a second viewpoint image 42, which is a right image (second image), to the right channel Rch of the first trained model 12A and the second trained model 12B, respectively. The first trained model 12A outputs a first estimated disparity 51 from the output channel Och, and the second trained model 12B outputs a second estimated disparity 52 from the output channel Och. As a result, the high-reliability region identification means 3 acquires the first estimated disparity 51 generated by the first trained model 12A and the second estimated disparity 52 generated by the second trained model 12B.

[0059] (Second example of obtaining second estimated disparity 52) FIG. 11 is a schematic diagram showing a second acquisition example of the second estimated disparity 52. ​​In the second acquisition example, a first estimated disparity 51 and a second estimated disparity 52 are acquired using one first trained model 12C. Specifically, the first trained model 12C not only outputs the first estimated disparity 51 in association with the position of a point in the first image, but also outputs a second estimated disparity 52 (d r * ) into the second position coordinates (u r The inference unit 2 inputs the first viewpoint image 41 as a left image (first image) and the second viewpoint image 42 as a right image (second image) to the first trained model 12, and outputs the first position coordinates (u l ) and the first estimated disparity 51 (d l * ), and the second position coordinate (u r ) and the second estimated disparity 52 (d r * ) to get the

[0060] The first trained model 12C has a left channel Lch that inputs the left image (first image) of the stereo image, a right channel Rch that inputs the right image (second image) of the stereo image, a first output channel O1ch that outputs a first estimated disparity 51 that is the inference result, and a second output channel O2ch that outputs a second estimated disparity 52 that is the inference result. The inference unit 2 inputs a first viewpoint image 41 that is the left image (first image) to the left channel Lch of the first trained model 12C, and inputs a second viewpoint image 42 that is the right image (second image) to the right channel Rch of the first trained model 12C. The first trained model 12C outputs the first estimated disparity 51 from the first output channel O1ch and outputs the second estimated disparity 52 from the second output channel O2ch. As a result, the high-reliability region identification means 3 acquires the first estimated disparity 51 and the second estimated disparity 52 generated by the first trained model 12C.

[0061] (Third example of obtaining second estimated disparity 52) FIG. 12 is a schematic diagram showing a third example of acquiring the second estimated disparity 52. ​​Note that the schematic diagram of the image and estimated disparity shown in FIG. 12 is shown for convenience in distinguishing between left and right of the image, and does not reflect the image or disparity of the object 90. The third example of acquiring the second estimated disparity 52 shows an example of acquiring the second estimated disparity 52 using the first trained model 12 that has learned the process of outputting the first estimated disparity 51. The first trained model 12 has learned the process of inputting the first viewpoint image 41 and the second viewpoint image 42 and outputting the first estimated disparity 51. Therefore, when acquiring the first estimated disparity 51, the inference unit 2 inputs the first viewpoint image 41 to the left channel Lch of the first trained model 12 and inputs the second viewpoint image 42 to the right channel Rch of the first trained model 12, as shown in FIG. 7, thereby outputting the first estimated disparity 51 from the output channel Och.

[0062] On the other hand, when acquiring the second estimated disparity 52, as shown in FIG. 12, the inference unit 2 performs input image swapping processing and disparity direction inversion processing on the first trained model 12. The inference unit 2 performs inversion processing to invert the first viewpoint image 41, which is the left image (first image), and the second viewpoint image 42, which is the right image (second image), in the parallax direction (left-right direction), respectively, to obtain a first inverted image 43 and a second inverted image 44. The first inverted image 43 is an image obtained by horizontally inverting the first viewpoint image 41, and the second inverted image 44 is an image obtained by horizontally inverting the second viewpoint image 42. The inference unit 2 then inputs the second inverted image 44, which is obtained by horizontally inverting the second viewpoint image 42, as the first image, and the first inverted image 43, which is obtained by horizontally inverting the first viewpoint image 41, as the second image, to the first trained model 12. That is, the inference unit 2 inputs the second inverted image 44 to the left channel Lch of the first trained model 12, and inputs the first inverted image 43 to the right channel Rch of the first trained model 12. As a result, the first trained model 12 outputs an inverted estimated disparity 53. The inverted estimated disparity 53 represents the pixel shift amount to the corresponding point Q of the second inverted image 44, based on each point of the first inverted image 43, which is the image input to the left channel Lch of the first trained model 12. The inference unit 2 performs a disparity direction inversion process on the obtained inverted estimated disparity 53, thereby obtaining a second estimated disparity 52.

[0063] In this way, in the third example of obtaining the second estimated disparity 52, the inference unit 2 inputs the second inverted image 44 obtained by inverting the second viewpoint image 42 in the disparity direction as the first image, and the first inverted image 43 obtained by inverting the first viewpoint image 41 in the disparity direction as the second image into the first trained model 12 to obtain a first estimated disparity associated with the first position coordinates, inverts the first position coordinates in the disparity direction to obtain the second position coordinates, and inverts the sign of the first estimated disparity to obtain the second estimated disparity 52.

[0064] In the first embodiment, the second estimated disparity 52 may be acquired by any of the methods of the first acquisition example, the second acquisition example, and the third acquisition example. In the first acquisition example, two trained models must be created, and in the second acquisition example, only one trained model is required, but the scale of the trained model increases, resulting in increased computational costs for inference and learning. On the other hand, in the third acquisition example, only image inversion processing and input image swapping processing are performed, so only one trained model is required, and the scale of the trained model is small, thereby reducing computational costs for inference and learning.

[0065] As a result of the above, the highly reliable area specifying means 3 generates the correspondence data (u l , d l * , u r * ) and data in a data format D3 including second position coordinates and second estimated disparity 52 (u r , d r * ) to obtain the following data format D4. [Data format D4] (u l , d l * , u r * , u r , d r * ) In the data format D4, data (u l , d l * , u r * ) are data related to each other. r , d r * ) are data that are associated with each other.

[0066] (Comparison of the first estimated disparity at the point of interest and the second estimated disparity at the corresponding point) 13 is a schematic diagram showing the relationship between a first estimated disparity 51, a corresponding point Q, and a second estimated disparity 52. ​​The first estimated disparity 51 represents the disparity based on the first image at each point in a first viewpoint image 41, which is a left image (first image), and the second estimated disparity 52 represents the disparity based on the second image at each point in a second viewpoint image 42, which is a right image (second image). Here, the relationship between the point of interest P and the corresponding point Q is based on the first estimated disparity 51. The high-reliability region identification means 3 determines the reliability of the first estimated disparity 51 based on the first estimated disparity 51 at the point of interest P and the second estimated disparity 52 at the corresponding point Q.

[0067] If the first estimated disparity 51 and the second estimated disparity 52 are correct, the true disparity d l and the true disparity d r is expressed by the relationship in equation (6).

number

number

[0068] That is, if the inference in the inference unit 2 is correct, the first estimated disparity 51 of the point of interest P, that is, the estimated disparity d l * and the estimated disparity d r * and have the same value but opposite signs.

[0069] Therefore, the first position coordinate of the point of interest P (u l ) estimated disparity d l * (u l ) and the estimated position u of the corresponding point Q r * The estimated disparity d r * (u r * ) satisfies the formula (8), the inference in the inference unit 2 can be said to be correct.

number

[0070] Therefore, in the first embodiment, the high-reliability region specifying means 3 uses the association data (u l , d l * , u r * ) at the first position coordinate (u l ) is the estimated position (u r * ) and the same second position coordinate (u r ) is identified, and the second estimated disparity 52 associated with the l , d l * , u r * ) at the first position coordinate (u l ) the magnitude d of the first estimated disparity 51 associated with l * (u l ) and the magnitude of the second estimated disparity 52, d r * (u r * ) is smaller than a predetermined threshold, the corresponding data (u l , d l * , u r * ) at the first position coordinate (u l ) has a high reliability. That is, when the left side of equation (9) for the point of interest P is smaller than a threshold, the high-reliability region identification means 3 determines that the reliability of that point is high. When the left side of equation (9) for the point of interest P is equal to or greater than a threshold, the high-reliability region identification means 3 determines that the reliability of that point is not high.

number

[0071] For example, in the example shown in FIG. 13, as shown by the solid line, the target point P1(u l = 9) estimated disparity d l* (u l ) is "2", and the estimated position u r * is "7". From the second estimated disparity 52, the estimated disparity d r * (u r * ) is "-2". In this case, the error between the first estimated disparity 51 of the point of interest P1 and the second estimated disparity 52 of the corresponding point Q1 shown on the left side of equation (9) is 0, and it can be determined that the reliability of the first estimated disparity 51 for the point of interest P1 is high.

[0072] On the other hand, as shown by the dotted line, the target point P2 (u l = 4) estimated disparity d l * (u l ) is "1", and the estimated position u r * is "3". From the second estimated disparity 52, the estimated disparity d r * (u r * ) is "-2". In this case, the error between the first estimated disparity 51 of the point of interest P2 and the second estimated disparity 52 of the corresponding point Q2 is 1. For example, when the threshold value on the right side of equation (9) is "1", it is determined that the reliability of the first estimated disparity 51 for the point of interest P2 is not high.

[0073] The highly reliable region specifying means 3 determines the first position coordinates (u l ), the reliability of the first estimated disparity 51 is determined based on Equation (9), and the pair (u l, d l *) is identified. The high-reliability region identification means 3 outputs reliability information 60 indicating a high-reliability region, which is a set of first position coordinates for which the reliability of the first estimated disparity 51 is high. As described above, the data format of the reliability information 60 is not particularly limited. However, as an example, the reliability information 60 is assumed to be binary information indicating the level of reliability. That is, the reliability information 60 includes a variable R for each point in the first viewpoint image 41, where R=1 when formula (9) is satisfied and R=0 when formula (9) is not satisfied. R=1 indicates that the error between the first estimated disparity 51 at the point of interest P and the second estimated disparity 52 at the corresponding point Q is less than a threshold, and the reliability of the first estimated disparity 51 is high. R=0 indicates that the error between the first estimated disparity 51 at the point of interest P and the second estimated disparity 52 at the corresponding point Q is equal to or greater than a threshold, and the reliability of the first estimated disparity 51 is not high. The high-reliability region specifying means 3 adds the variable R of each point, which is the reliability information 60, to the data format D4 to obtain data represented in a data format D5. [Data format D5] (u l , d l * , u r * , u r , d r * , R) In the data format D5, data (u l , d l * , u r * , R) are data related to each other. r , d r * ) are data that are associated with each other.

[0074] 14 is a schematic diagram showing an example of the reliability information 60. The representation format of the reliability information 60 is not particularly limited, but in FIG. 14, the reliability information 60 is represented as an image (u l, R) is shown. In the case of FIG. 13, if the threshold value of equation (9) is set to "1", then as shown in FIG. 14, R=1 for pixels that include the background and region 91, and R=0 for pixels that include region 93. The reliability information 60 shows that, among the pixel regions that include object 90, points that include region 91 are high-reliability regions, and points that include region 93 are low-reliability regions. In this way, the reliability information 60 can distinguish high-reliability regions, among the points in the first viewpoint image 41, from other regions.

[0075] (Calculation of depth information based on reliability information) Next, calculation of the depth information 70 based on the reliability information 60 will be described. Fig. 15 is a diagram schematically showing a method for acquiring the reliability information 60 by the high-reliability area identification means 3 according to the first embodiment. Fig. 16 is a diagram schematically showing an example of a method for calculating the depth information 70 in the depth calculation unit 4.

[0076] 15, the high-reliability region identifying means 3 performs a reliability calculation process based on Equation (9) using the first estimated disparity 51 and the second estimated disparity 52, and outputs reliability information 60. As shown in FIG. 14, the high-reliability region identifying means 3 can output the reliability information 60 as mask data representing the high-reliability region of each point in the first viewpoint image 41. The mask data is a binary image having a reliability value (R=1 or R=0) for the first estimated disparity 51 of each first position coordinate in the first viewpoint image 41.

[0077] As shown in FIG. 16, the depth calculation unit 4 calculates depth information 70 based on the first estimated disparity 51 and the reliability information 60. For example, the depth calculation unit 4 performs an AND (logical product) operation on the first estimated disparity 51 and the reliability information 60, which is mask data. For points with high reliability (R=1) in the reliability information 60, the first estimated disparity 51 is calculated as a disparity (d l * ) value is stored, and for points where the reliability of the reliability information 60 is not high (R=0), the disparity (d l *) becomes 0. The depth calculation unit 4 obtains the depth Z of each point using equation (4) based on the first estimated disparity 51 after the AND operation and the camera parameters 13. The depth calculation unit 4 outputs depth information 70, for example, in the form of a depth image in which the pixel value of each pixel represents the depth Z. The output depth information 70 does not include depth information (depth Z) for points with low reliability, but includes only depth information (depth Z) for high-reliability areas with high reliability. This prevents depth information calculated from the disparity of points with low reliability from being output.

[0078] 17 is a diagram schematically illustrating another example of a calculation method of depth information 70 in the depth calculation unit 4. In FIG. 17, when calculating depth information 70 for each point in the first viewpoint image 41, the depth calculation unit 4 refers to the reliability information 60 to obtain first position coordinates of each point belonging to a high-reliability region. Of the points in the first viewpoint image 41, the depth calculation unit 4 uses points that belong to a high-reliability region (holding R=1) as use points for depth calculation, and excludes points that do not belong to a high-reliability region (holding R=0) as non-use points from the depth calculation. For non-use points, the depth calculation unit 4 may assign, for example, "0" to the depth Z, or may assign information indicating that the point is non-use (for example, "NaN").

[0079] (Image processing method) Next, an image processing method according to the first embodiment will be described. Fig. 18 is a flowchart showing the image processing method according to the first embodiment. The image processing method according to the embodiment is an image processing method that performs three-dimensional image measurement processing based on parallax between multiple images. The image processing method according to the embodiment is implemented by an image processing device 100. In other words, a program 11 according to the first embodiment causes a computer system 1000 to execute processing of each step according to Fig. 18.

[0080] As shown in FIG. 18, the image processing method according to the embodiment includes step S1 of acquiring a first viewpoint image 41 and a second viewpoint image 42, step S2 of acquiring an estimated parallax, and step S3 of generating association data (u l , d l* , u r * ), a set of first position coordinates and first estimated disparity 51 with high reliability of the first estimated disparity 51 (u l, d l * 18, the image processing method further includes step S4 of calculating depth information 70 representing the depth Z of each point in first viewpoint image 41, step S5 of performing object recognition based on the depth information 70, and step S6 of outputting control information based on the object recognition result.

[0081] Step S1 of acquiring the first viewpoint image 41 and the second viewpoint image 42 acquires the first viewpoint image 41 and the second viewpoint image 42 captured at the first viewpoint and the second viewpoint, respectively. That is, the image acquisition unit 1 acquires the first viewpoint image 41 captured by the left camera 21 and the second viewpoint image 42 captured by the right camera 22 from the imaging device 20. The image acquisition unit 1 outputs the acquired first viewpoint image 41 and second viewpoint image 42 to the inference unit 2.

[0082] In step S2 of outputting the estimated disparity, the first trained model 12 receives the first viewpoint image 41 and the second viewpoint image 42 as input and outputs an estimated disparity, thereby outputting at least a first estimated disparity 51 for each point in the first viewpoint image 41. In the first embodiment, in addition to the first estimated disparity 51 for each point in the first viewpoint image 41, a second estimated disparity 52 for each point in the second viewpoint image 42 is acquired. That is, the inference unit 2 inputs the first viewpoint image 41 and the second viewpoint image 42 to the first trained model 12, thereby acquiring the first estimated disparity 51 as an output of the first trained model 12. The inference unit 2 acquires the second estimated disparity 52 using one of the first, second, and third acquisition methods for the second estimated disparity 52, or another method. The inference unit 2 outputs the acquired first estimated disparity 51 to the high-reliability region identification means 3 and the depth calculation unit 4. The inference unit 2 outputs the acquired second estimated disparity 52 to the high-reliability region identification means 3.

[0083] In step S3 of performing reliability determination, for each point in the first viewpoint image 41, an estimated position of a corresponding point Q in the second viewpoint image 42 is obtained from the first position coordinates and the first estimated parallax 51, and correspondence data (u l , d l * , u r * ) and generate the mapping data (u l , d l * , u r * ), a set of first position coordinates and first estimated disparity 51 with high reliability of the first estimated disparity 51 (u l, d l * ) is identified. The reliability determination process (step S3) according to the first embodiment will be described with reference to FIG.

[0084] FIG. 19 is a flowchart showing the reliability determination process according to the first embodiment. The high-reliability region specifying means 3 acquires the first estimated parallax 51 output from the inference unit 2 (step S11). For each point in the first viewpoint image 41, the high-reliability region specifying means 3 calculates the estimated position (u r * ) is calculated, and the first position coordinate, the first estimated parallax 51, and the estimated position of the corresponding point Q in the second viewpoint image 42 are correlated to each other to obtain the correspondence data (u l , d l * , u r * The high-reliability region identifying means 3 acquires the second estimated disparity 52 output from the inference unit 2 (step S12). The high-reliability region identifying means 3 determines the reliability of the first estimated disparity 51 associated with the first position coordinates (step S13).

[0085] The high-reliability region specifying means 3 uses the association data (u l , d l * , u r *) at the first position coordinate (u l ) is the estimated position (u r * ) and the same second position coordinate (u r ) is identified, and the magnitude (d l * ) and the magnitude of the second estimated disparity 52 (d r * ) is smaller than a predetermined threshold, the reliability of the first estimated disparity 51 associated with the first position coordinates is determined to be high. That is, according to equation (9), if the error between the first estimated disparity 51 and the second estimated disparity 52 is less than a threshold, the high-reliability region identification means 3 determines that the reliability of the point of interest P is high (R=1). If the error between the first estimated disparity 51 and the second estimated disparity 52 is equal to or greater than a threshold, the high-reliability region identification means 3 determines that the reliability of the point of interest P is not high (R=0). The high-reliability region identification means 3 calculates a reliability variable R for all points (pixels) in the first viewpoint image 41 to obtain reliability information 60. The high-reliability region identification means 3 outputs the reliability information 60 of each point to the depth calculation unit 4 (step S14). 18, the high-reliability area specifying means 3 acquires the reliability information 60 for each point in the first viewpoint image 41 and outputs it to the depth calculation unit 4. In step S3 of FIG.

[0086] As shown in FIG. 18, in step S4 of calculating depth information 70, the depth information 70 representing the depth of each point in the first viewpoint image 41 is calculated based on the first estimated parallax 51 and the reliability information 60. That is, the depth calculation unit 4 calculates the depth information 70 representing the depth of each point in the first viewpoint image 41 based on the first estimated parallax 51 and the reliability information 60. l, d l * ) is calculated by, for example, the method illustrated in Fig. 16 or 17. The depth calculation unit 4 then outputs to the object recognition unit 7 output depth information 71 that does not include depth information 70 for points with low reliability but includes depth information 70 for points with high reliability.

[0087] In step S5 of performing object recognition, the position and orientation of the object 90 to be recognized are recognized based on the output depth information 71 (depth information 70 of points with high reliability among the points in the first viewpoint image 41) and the CAD data 14. That is, the object recognition unit 7 acquires three-dimensional position information of the object 90 and orientation information, which is the rotation angle around each three-dimensional axis, by performing a matching process in which each point on the surface of the object 90 represented by the CAD data 14 is matched with each point in space determined from the depth Z of each point in the output depth information 71. The object recognition unit 7 outputs the acquired information on the position and orientation of the object 90 to the output unit 5. When the field of view of the imaging device 20 is changed according to the position and orientation of the object 90, the object recognition unit 7 may output the acquired information on the position and orientation of the object 90 to the imaging control unit 8.

[0088] In step S6 of outputting control information based on the object recognition result, information on the position and orientation of the object 90, which is the object recognition result, is output to the robot control device 33. That is, the output unit 5 outputs the information on the position and orientation of the object 90 output from the object recognition unit 7 to the robot control device 33 via communication. The robot control device 33 controls the operation of the work robot 32 based on the acquired information on the position and orientation of the object 90.

[0089] When the field of view of the imaging device 20 is changed according to the position and posture of the object 90, the imaging control unit 8 outputs the information on the position and posture of the object 90 output from the object recognition unit 7 to the robot control device 31 via communication from the output unit 5. Based on the acquired information on the position and posture of the object 90, the robot control device 31 controls the operation of the camera-moving robot 30 so as to perform, for example, tracking of the moving object.

[0090] As described above, the image processing device 100 according to the first embodiment is an image processing device that performs three-dimensional image measurement processing based on the parallax between a plurality of images, and includes an image acquisition unit 1 that acquires a first viewpoint image 41 and a second viewpoint image 42 captured from a first viewpoint and a second viewpoint, respectively, and a first estimated parallax 51 that is an inferred value by machine learning of the parallax between each point in the first image and a corresponding point in the second image, and outputs the first estimated parallax 51 in association with first position coordinates that indicate the position of the point in the first image. The inference unit 2 is capable of using a first trained model 12 that uses the first viewpoint image 41 as a first image and the second viewpoint image 42 as a second image to input the first trained model 12 and acquire a first estimated parallax 51 associated with the first position coordinates; and the inference unit 2 is capable of calculating, for each point in the first viewpoint image 41, an estimated position of a corresponding point Q in the second viewpoint image 42 from the first position coordinates and the first estimated parallax 51, and acquiring association data (u l , d l * , u r * ) and generate the mapping data (u l , d l * , u r * ), a set of first position coordinates and first estimated disparities 51 with high reliability is selected. l, d l * and a high-reliability region specifying means 3 for specifying the region.

[0091] According to this configuration, the first position coordinates, the first estimated parallax 51, and the correspondence data (u l , d l * , u r * ), a set of first position coordinates and first estimated disparities 51 with high reliability is selected. l, d l * ) can be identified. This makes it possible to identify reliable information from the output of the first trained model 12 used for three-dimensional image measurement.

[0092] In the image processing device 100 according to the first embodiment, in a first example of acquiring the second estimated disparity 52, the inference unit 2 is further enabled to use a second trained model 12B that inputs the first image and the second image and outputs the second estimated disparity 52, which is an inferred value by machine learning of the disparity between each point in the second image and a corresponding point Q in the first image, in association with second position coordinates representing the position of the point in the second image, and inputs the first viewpoint image 41 as the first image and the second viewpoint image 42 as the second image to the second trained model 12B to acquire the second estimated disparity 52 associated with the second position coordinates. The high-reliability area identification means 3 uses the association data (u l , d l * , u r * ), the second estimated disparity 52 associated with the second position coordinates that are the same as the estimated position of the corresponding point Q in the second viewpoint image 42 associated with the first position coordinates is identified, and if the difference between the magnitude of the first estimated disparity 51 associated with the first position coordinates and the magnitude of the identified second estimated disparity 52 is smaller than a predetermined threshold, the reliability of the first estimated disparity 51 associated with the first position coordinates is determined to be high. As a result, the first estimated disparity 51 and the second estimated disparity 52 can be obtained by inputting the first viewpoint image 41 and the second viewpoint image 42 into the first trained model 12A and the second trained model 12B, respectively. Then, by comparing the magnitude of the first estimated disparity 51 associated with the first position coordinates with the magnitude of the identified second estimated disparity 52 as in Equation (9), appropriate reliability can be easily determined.

[0093] In the image processing device 100 according to the first embodiment, in a second example of acquiring the second estimated disparity 52, the first trained model 12C not only outputs the first estimated disparity 51 in association with the position of a point in the first image, but also outputs the second estimated disparity 52, which is an inferred value by machine learning of the disparity between each point in the second image and a corresponding point Q in the first image, in association with second position coordinates representing the position of the point in the second image, and the inference unit 2 inputs the first viewpoint image 41 as the first image and the second viewpoint image 42 as the second image into the first trained model 12 to acquire the first estimated disparity 51 associated with the first position coordinates and the second estimated disparity 52 associated with the second position coordinates. The high-reliability area identification means 3 uses the association data (u l , d l * , u r * ), the second estimated disparity 52 associated with the second position coordinates that are the same as the estimated position of the corresponding point Q in the second viewpoint image 42 associated with the first position coordinates is identified, and if the difference between the magnitude of the first estimated disparity 51 associated with the first position coordinates and the magnitude of the identified second estimated disparity 52 is smaller than a predetermined threshold, the reliability of the first estimated disparity 51 associated with the first position coordinates is determined to be high. This makes it possible to acquire both the first estimated disparity 51 and the second estimated disparity 52 using a single first trained model 12C. Then, by comparing the magnitude of the first estimated disparity 51 associated with the first position coordinates with the magnitude of the identified second estimated disparity 52 as in Equation (9), appropriate reliability determination can be easily performed.

[0094] In the image processing device 100 according to the first embodiment, in a third example of acquiring the second estimated disparity 52, the inference unit 2 inputs a second inverted image 44 obtained by inverting the second viewpoint image 42 in the disparity direction as the first image, and a first inverted image 43 obtained by inverting the first viewpoint image 41 in the disparity direction as the second image to the first trained model 12, and acquires an inverted estimated disparity 53 associated with the position of a point in the second inverted image 44, and then inverts the acquired inverted estimated disparity 53 and the position of the point associated with the inverted estimated disparity 53 in the disparity direction, thereby acquiring a second estimated disparity 52 associated with the second position coordinates representing the position of the point in the second image. l , d l * , u r * ), the second estimated disparity 52 associated with the second position coordinates that are the same as the estimated position of the corresponding point Q in the second viewpoint image 42 associated with the first position coordinates is identified, and if the difference between the magnitude of the first estimated disparity 51 associated with the first position coordinates and the magnitude of the identified second estimated disparity 52 is smaller than a predetermined threshold, the reliability of the first estimated disparity 51 associated with the first position coordinates is determined to be high. This makes it possible to acquire both the first estimated disparity 51 and the second estimated disparity 52 using a single first trained model 12C. Then, by comparing the magnitude of the first estimated disparity 51 associated with the first position coordinates with the magnitude of the identified second estimated disparity 52 as in Equation (9), appropriate reliability determination can be easily performed.

[0095] An image processing device 100 according to the first embodiment is an image processing method for performing three-dimensional image measurement processing based on disparity between a plurality of images, the image processing method including: step S1 of acquiring a first viewpoint image 41 and a second viewpoint image 42 captured from a first viewpoint and a second viewpoint, respectively; a trained model is available for inputting the first image and the second image, and outputting a first estimated disparity 51, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point Q in the second image, in association with first position coordinates representing the position of the point in the first image; step S2 of inputting the first viewpoint image 41 as the first image and the second viewpoint image 42 as the second image into the trained model to acquire the first estimated disparity 51 associated with the first position coordinates; and step S3 of calculating, for each point in the first viewpoint image 41, an estimated position of the corresponding point Q in the second viewpoint image 42 from the first position coordinates and the first estimated disparity 51, and outputting association data (u l , d l * , u r * ) and generate the mapping data (u l , d l * , u r * ), a set of first position coordinates and first estimated disparities 51 with high reliability is selected. l, d l * and step S3 of identifying the

[0096] According to this configuration, the first position coordinates, the first estimated parallax 51, and the correspondence data (u l , d l * , u r * ), a set of first position coordinates and first estimated disparities 51 with high reliability is selected. l, d l * ) can be identified. This makes it possible to identify reliable information from the output of the first trained model 12 used for three-dimensional image measurement.

[0097] The program 11 according to the first embodiment is a program for performing three-dimensional image measurement processing based on the parallax between a plurality of images, and includes the following steps: a process (step S1) for acquiring a first viewpoint image 41 and a second viewpoint image 42 captured from a first viewpoint and a second viewpoint, respectively; a process (step S2) for inputting the first and second images, using a trained model that outputs a first estimated parallax 51, which is an inferred value by machine learning of the parallax between each point in the first image and a corresponding point Q in the second image, in association with a first position coordinate representing the position of the point in the first image; and a process (step S3) for inputting the first viewpoint image 41 as the first image and the second viewpoint image 42 as the second image into the trained model to acquire the first estimated parallax 51 associated with the first position coordinate. l , d l * , u r * ) and generate the mapping data (u l , d l * , u r * ), a set of first position coordinates and first estimated disparities 51 with high reliability is selected. l, d l * ) (Step S3).

[0098] According to this configuration, the first position coordinates, the first estimated parallax 51, and the correspondence data (u l , d l * , u r * ), a set of first position coordinates and first estimated disparities 51 with high reliability is selected. l, d l *) can be identified. This makes it possible to identify reliable information from the output of the first trained model 12 used for three-dimensional image measurement.

[0099] In the first embodiment, an example was shown in which the first trained model 12 outputs a low-reliability estimated disparity (an estimated disparity interpolated without being based on an image) due to the presence of a blind spot in the image from one of the first and second viewpoints in FIGS. 5 and 6 . However, in the present embodiment, the cause of the low-reliability estimated disparity output is not particularly limited. For example, even for a portion (such as region 91 of object 90 in FIG. 5 ) that appears in both the first viewpoint image 41 and the second viewpoint image 42, there are cases where corresponding point matching is locally difficult due to light reflection or refraction. Specifically, even if a portion occurs in either the first viewpoint image 41 or the second viewpoint image 42 due to local pixel saturation (so-called blown-out highlights) caused by light reflection, making corresponding point matching difficult, the first trained model 12 interpolates the estimated disparity at the pixel saturation location. Even in such cases, according to this embodiment, it is possible to distinguish and identify locations where estimated disparity has been calculated by appropriate corresponding point matching (locations with high reliability) from locations where disparity has been supplemented without being based on an image (locations with low reliability).

[0100] [Second embodiment] A second embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be given the same reference numerals, and descriptions of those components will be simplified or omitted.

[0101] In the first embodiment described above, an example has been shown in which the high-reliability region identification means 3 acquires the second estimated disparity 52, and acquires the reliability information 60 based on the first estimated disparity 51 at the point of interest P and the second estimated disparity 52 at the corresponding point Q. In contrast, in the second embodiment, an example will be described in which the high-reliability region identification means 3 acquires the reliability information 60 based on the correspondence between the point of interest P and the corresponding point Q without acquiring the second estimated disparity 52.

[0102] As in the first embodiment, the high-reliability region identifying means 3 acquires a corresponding point Q in the second viewpoint image 42 that corresponds to the point of interest P in the first viewpoint image 41 based on the first estimated parallax 51. Specifically, the high-reliability region identifying means 3 receives from the inference unit 2 the [data format D1](u l , d l * Next, the highly reliable region specifying means 3 obtains the estimated position (u r * ) is acquired. As a result, the high-reliability region specifying means 3 acquires association data in a data format D2 in which the estimated position of the corresponding point Q is added to the data in the data format D1. [Data format D2] (u l , d l * , u r * )

[0103] FIG. 20 is a schematic diagram for explaining a method for determining reliability according to the second embodiment. FIG. 20 shows the correspondence data (u l , d l * , u r * As an example of the representation format of (u), a correspondence point map 45 showing the distribution of correspondence points Q is shown. Each point on the correspondence point map 45 is l、 u r * )

[0104] In the second embodiment, the first viewpoint image 41 and the second viewpoint image 42 are parallel stereo images, and corresponding point pairs are assumed to be at the same position (same row) in the vertical direction in the images. The position of each point on the corresponding point map 45 is expressed by the first position coordinates (u l The value of each point in the correspondence point map 45 is expressed as l ) in the second viewpoint image 42. r *) from equation (6). If the first estimated disparity 51 output by the first trained model 12 of the inference unit 2 is correct, the point of interest P(u l ) and the corresponding point Q(u r * ) has a one-to-one correspondence, and the same corresponding point Q(u r * ) indicates the target point P(u l ) does not exist. On the other hand, when the first trained model 12 complements the first estimated disparity 51 for a point for which a corresponding point pair cannot be determined, the target point P(u l ) and the corresponding point Q(u r * ) does not have a one-to-one correspondence, and the same corresponding point Q(u r * ) indicates the target point P(u l In other words, when the first trained model 12 generates the first estimated disparity 51 by interpolation, there may be multiple estimated positions (u r * ) may exist at multiple points with the same value.

[0105] Therefore, the high-reliability region specifying means 3 uses the correspondence data (u l , d l * , u r * ) is the estimated position (u r * ) and the estimated position (u r * ) and other mapping data (u l , d l * , u r * ) exist in a predetermined number or more, the estimated position (u r * ) including a predetermined number of correspondence data (u l , d l * , u r *) is assumed to have low reliability.

[0106] The high-reliability region specifying means 3 r * The number of data m containing the value of u r * For example, in Figure 20, when we look at a certain row (the second row), u r * = 2 (u l =2, u l =3) exists, u r * = 3 (u l =4, u l = 5) exist (see dotted line). Each of these points has a data count m = 2. The high-reliability area identification means 3 acquires data in data format D6 by adding the data count m acquired for each point of the corresponding point map (which is the same as each point of the first viewpoint image 41) to the data format D2. [Data format D6] (u l , d l * , u r * , m) In the data format D6, data (u l , d l * , u r * , m) are data associated with each other.

[0107] Then, the high-reliability area specifying means 3 determines the reliability based on, for example, a comparison between the number m of points that point to the same corresponding point Q at each point in the first viewpoint image 41 and a threshold value. The reliability determination result can be obtained, for example, by equations (10) and (11).

number

[0108] The high-reliability region identification means 3 acquires reliability information 60 for each point in the first viewpoint image 41 based on equations (10) and (11). As described above, the format of the reliability information 60 is not particularly limited. As an example, in the second embodiment, the reliability information 60 includes a variable S for each point in the first viewpoint image 41. The variable S is S=1 when equation (11) is satisfied, and S=0 when equation (11) is not satisfied. S=1 indicates that the left side of equation (11), which is based on the number m of points that point to the same corresponding point Q, is equal to or greater than a threshold, and the reliability of the first estimated disparity 51 is high. S=0 indicates that the left side of equation (11) is less than a threshold, and the reliability of the first estimated disparity 51 is not high.

[0109] In equation (11), if the threshold value on the right side is set to "1", S=1 only when the number m of points pointing to the same corresponding point Q is 1. The threshold value on the right side may be less than 1. For example, if the threshold value is set to 0.5, S=1 even when the number m of points pointing to the same corresponding point Q is 2. If the number m is allowed to be 2 or more, the estimated disparity d l * The value of d may be used as it is, or the estimated disparity d l * For the point with the largest disparity, S=1, and for the other points, S=0. l *The point with the largest value is the point closest to the image capture device 20 and is the point with the lowest possibility of being in a blind spot in the field of view, and therefore can be evaluated as having a relatively high reliability. Therefore, when the threshold value on the right side is less than 1, the high-reliability region specifying means 3 uses the correspondence data (u l , d l * , u r * ) is the estimated position (u r * ) and the estimated position (u r * ) and other mapping data (u l , d l * , u r * If there are multiple u l , d l * , u r * ) the first estimated disparity 51 (d l * ) is considered to be highly reliable.

[0110] The high-reliability region specifying means 3 adds the variable S of each point, which is the reliability information 60, to the data format D2 to obtain data represented in the data format D7. [Data format D7] (u l , d l * , u r * , S) In the data format D7, data (u l , d l * , u r * , S) are data associated with each other. The high-reliability region specifying means 3 outputs reliability information 60 including the variable S from the data format D7 in an arbitrary representation format.

[0111] Fig. 21 is a flowchart showing reliability determination processing according to the second embodiment. In the second embodiment, processing other than the reliability determination processing is the same as each processing of the image processing method shown in Fig. 18. The second embodiment differs only in the processing of step S3 of the image processing method shown in Fig. 18.

[0112] As shown in FIG. 21 , the high-reliability region identifying means 3 acquires the first estimated disparity 51 output from the inference unit 2 (step S21). Based on the first estimated disparity 51, the high-reliability region identifying means 3 acquires a corresponding point Q in the second viewpoint image 42 corresponding to the point of interest P in the first viewpoint image 41, and counts the number of points that point to the same corresponding point Q for each line in the disparity direction (step S22). The high-reliability region identifying means 3 determines the reliability based on the number of points that point to the same corresponding point Q among the points aligned in the disparity direction with the point of interest P (step S23). That is, the high-reliability region identifying means 3 compares the number m of points that point to the same corresponding point Q with a threshold using equation (11). The high-reliability region identifying means 3 determines that the reliability of points whose left-hand side of equation (11) is equal to or greater than the threshold is high (S=1). The high-reliability region identifying means 3 determines that the reliability of points whose left-hand side of equation (11) is less than the threshold is low (S=0). The high-reliability region identifying means 3 obtains the reliability variable S for all points (pixels) of the first estimated disparity 51 to obtain reliability information 60. The high-reliability region identifying means 3 outputs the reliability information 60 for each point to the depth calculation unit 4 (step S24). As a result, in step S3 of FIG. 18 , the high-reliability region identifying means 3 obtains the reliability information 60 for each point of the first viewpoint image 41 and outputs it to the depth calculation unit 4.

[0113] As described above, in the image processing device 100 according to the second embodiment, the high-reliability region specifying means 3 uses the correspondence data (u l , d l * , u r * The estimated position (u r * ) and other mapping data (u l , d l* , u r * ) exist in a predetermined number or more, the estimated position (u r * ) including a predetermined number of correspondence data (u l , d l * , u r * ) has a low reliability. As a result, since the point of interest P and the corresponding point Q theoretically have a one-to-one correspondence relationship, the reliability of the first estimated disparity 51, which is the output of the first trained model 12, can be appropriately evaluated based on the number of points that point to the same corresponding point Q.

[0114] In the image processing device 100 according to the second embodiment, the high-reliability region specifying means 3 uses the correspondence data (u l , d l * , u r * ) and the estimated position of the corresponding point Q in the second viewpoint image 42 included in the other correspondence data (u l , d l * , u r * If there are multiple u l , d l * , u r * ), only the first estimated disparity 51 with the largest absolute value among the first estimated disparities 51 included in the set (u l, d l * ) can be identified.

[0115] Other effects of the second embodiment are the same as those of the first embodiment.

[0116] [Third embodiment] A third embodiment will now be described. In the following description, the same or equivalent components as those in the first and second embodiments will be denoted by the same reference numerals, and the description of those components will be simplified or omitted.

[0117] In the above first and second embodiments, an example was shown in which the inference unit 2 acquires a corresponding point Q from the estimated disparity output by the first trained model 12, and acquires reliability information 60 based on the point of interest P and the corresponding point Q. In contrast, in the third embodiment, an example will be described in which the high-reliability area identification means 3 determines reliability based on depth information.

[0118] Specifically, in the third embodiment, the high-reliability region specifying means 3 calculates the first position coordinates (u l ) and the first estimated disparity (d l * ) to estimate the position (u r * ) and calculate the first estimated disparity (d l * ) and calculate the first depth (Z1) corresponding to the first position coordinate (u l ), the first depth (Z1), the estimated position (u r * ) are associated with each other, and from the set of association data, the first position coordinate (u l ) and the first depth (Z1).

[0119] In the third embodiment, before determining the reliability, the depth calculation unit 4 calculates depth information in advance based on the estimated disparity output from the inference unit 2. The calculated depth information may include a depth Z calculated from a highly reliable estimated disparity and a depth Z calculated from a less reliable estimated disparity. In the third embodiment, the high-reliability area identification means 3 determines the reliability of the depth Z for each point of the depth information calculated by the depth calculation unit 4.

[0120] Fig. 22 is a diagram illustrating calculation of depth information by the depth calculation unit 4 according to the third embodiment. Fig. 23 is a diagram illustrating determination of reliability by the high-reliability area identification means 3 according to the third embodiment.

[0121] As shown in FIG. 22, in the third embodiment, the inference unit 2 outputs a first estimated parallax 51 and a second estimated parallax 52. As described above, the method for acquiring the second estimated parallax 52 is not particularly limited, and the second estimated parallax 52 may be acquired by any of the first acquisition example, the second acquisition example, and the third acquisition example. The first estimated parallax 51 is calculated based on the first position coordinates (u l ) estimated disparity d l * The second estimated parallax 52 is calculated by dividing the second position coordinates (u r ) estimated disparity d r * As a result, the depth calculation unit 4 obtains information in the data format D8. [Data format D8] (u l , d l * , u r , d r * ) (u l , d l * ) are data related to each other, and (u r , d r * ) are data that are associated with each other.

[0122] The high-reliability region specifying means 3 uses the depth calculation unit 4 to calculate the first estimated parallax (d l * The highly reliable region specifying means 3 calculates the first depth (Z1) corresponding to the second estimated disparity (d r *) and calculates a second depth (Z2) corresponding to the first estimated disparity 51 and the second estimated disparity 52. ​​The depth calculation unit 4 calculates first depth information 70A representing the depth of each point in the first viewpoint image 41 and second depth information 70B representing the depth of each point in the second viewpoint image 42 based on the first estimated disparity 51 and the second estimated disparity 52. ​​The first depth information 70A represents the depth (first depth) Z1 of each point in the first viewpoint image 41 calculated using equation (4) based on the first estimated disparity 51 and the camera parameters 13. The second depth information 70B represents the depth (second depth) Z2 of each point in the second viewpoint image 42 calculated using equation (4) based on the second estimated disparity 52 and the camera parameters 13. The depth calculation unit 4 outputs the first depth information 70A and the second depth information 70B to the high-reliability area identification means 3. As a result, the high-reliability area identification means 3 obtains information in data format D9. [Data format D9] (u l , d l * , Z1, u r , d r * , Z2) (u l , d l * , Z1) are data related to each other, and (u r , d r * , Z2) are data that are associated with each other.

[0123] As shown in FIG. 23, the high-reliability area specifying means 3 determines a set of first position coordinates and first depths (u) of points with high reliability at the first depth (Z1) based on the first depth information 70A and the second depth information 70B. l, The high-reliability area identification means 3 performs a viewpoint conversion process to convert either the first depth information 70A of the first viewpoint or the second depth information 70B of the second viewpoint into the depth information of the other viewpoint, using the camera parameters 13. In the example of FIG. 23, the high-reliability area identification means 3 converts the second depth information 70B of the second viewpoint into the depth information of the first viewpoint, based on the camera parameters 13 and the relationship shown in equations (1) to (4). The second depth information 70B after the viewpoint conversion process is calculated based on the second estimated parallax 52 for each point in the first viewpoint image 41. *The data has the following structure:

[0124] The highly reliable area specifying means 3 calculates the depth (first depth) Z1 of each point in the first depth information 70A of the first viewpoint and the depth Z2 of the same point in the second depth information 70B after the viewpoint conversion process. * For example, the reliability information 60 may be obtained by dividing the depth (first depth) Z1 and the depth Z2 of the same point. * If the error between Z1 and Z2 is less than the threshold, T=1, and the depth (first depth) Z1 and Z2 of the same point are * The parameter T is set to 0 when the error between the first depth (Z1) and the first position coordinates (u) is greater than or equal to the threshold value. T=1 indicates that the depth error is less than the threshold value and the reliability of the estimated disparity is high. T=0 indicates that the depth error is greater than or equal to the threshold value and the reliability of the estimated disparity is not high. As a result, the first position coordinates and the first depth pair (u) of the point (where T=1) with high reliability of the first depth (Z1) are set to 0. l, Z1) is identified.

[0125] The high-reliability region specifying means 3 adds the variable T of each point, which is the reliability information 60, to the data format D9 to obtain data represented in the data format D10. [Data format D10] (u l , d l * , Z1, u r , d r * , Z 2、 T)

[0126] The high-reliability area identifying means 3 outputs reliability information 60 including a variable T from the data format D10 in an arbitrary representation format. In the example of FIG. 23, since the first viewpoint is used as a reference, the reliability information 60 includes information indicating the level of reliability for each point of the first depth information 70A. For example, the high-reliability area identifying means 3 can output the reliability information 60 as mask data indicating the reliability of each point of the first viewpoint image 41. The mask data is a binary image having the reliability value (T=1 or T=0) of each point that is the same as each point of the first viewpoint image 41. The points included in each row of the mask data are (u l、 It is expressed as T).

[0127] The depth calculation unit 4 acquires the reliability information 60 from the high-reliability area identification means 3. Based on the reliability information 60, the depth calculation unit 4 outputs output depth information 71 for points with high reliability among the points in the first depth information 70A, excluding points with low reliability. Specifically, the depth calculation unit 4 performs an AND (logical product) operation on the first depth information 70A and the reliability information 60, which is mask data, to acquire the output depth information 71. The output depth information 71 retains a depth (first depth) Z1 after the operation for points with high reliability (holding T=1) among the reliability information 60, and the depth (first depth) Z1 becomes 0 after the operation for points with low reliability (holding T=0) among the reliability information 60. As a result, the output depth information 71 does not include depth information (first depth Z1) for points with low reliability, but only includes depth information (first depth Z1) for points with high reliability. Therefore, depth information calculated from the parallax of a point with low reliability is prevented from being output.

[0128] FIG. 24 is a flowchart showing an image processing method according to the third embodiment.

[0129] The image acquisition unit 1 acquires a first viewpoint image 41 and a second viewpoint image 42 captured from the first viewpoint and the second viewpoint, respectively (step S31). The inference unit 2 inputs the first viewpoint image 41 as a first image and the second viewpoint image 42 as a second image into the trained model, and calculates a first estimated parallax 51 (u l , d l * ), and a second estimated parallax 52 (u r , d r *) and outputs it (step S32). The depth calculation unit 4 calculates first depth information 70A representing the depth (first depth) of each point in the first viewpoint image 41 and second depth information 70B representing the depth (second depth) of each point in the second viewpoint image 42 based on the first estimated disparity 51 and the second estimated disparity 52 (step S33). The high-reliability area identification means 3 determines the reliability of each point in the first depth information 70A based on the first depth information 70A and the second depth information 70B (step S34). The high-reliability area identification means 3 outputs reliability information 60 as the determination result. The depth calculation unit 4 outputs output depth information 71 of points in the first depth information 70A with high reliability, excluding points with low reliability, based on the reliability information 60 (step S35).

[0130] The object recognition unit 7 acquires position information and orientation information of the object 90 to be recognized based on the CAD data 14 and the output depth information 71 (step S5). The output unit 5 outputs the position and orientation information of the object 90 output from the object recognition unit 7 to the robot control device 33 via communication (step S6). The robot control device 33 controls the operation of the work robot 32 based on the acquired position and orientation information of the object 90.

[0131] As described above, the image processing device 100 according to the third embodiment is an image processing device that performs three-dimensional image measurement processing based on the parallax between a plurality of images, and includes an image acquisition unit 1 that acquires a first viewpoint image 41 and a second viewpoint image 42 that are captured from a first viewpoint and a second viewpoint, respectively, and an image acquisition unit 2 that inputs the first image and the second image and calculates a first estimated parallax 51 (d l * ) into the first position coordinates (u l ) and outputs the first trained model 12 in association with the first position coordinates (u l ) and the inference unit 2 obtains a first estimated parallax 51 associated with the first position coordinates (ul ) and the first estimated disparity 51(d l * ) to estimate the position (u r * ) and calculate the first estimated disparity 51(d l * ) and calculate the first depth (Z1) corresponding to the first position coordinate (u l ), the first depth (Z1), the estimated position (u r * ) is associated with the associated data (u l , d l * , Z1), and from the set of association data, a pair of first position coordinates and first depth (u l、 Z1).

[0132] According to this configuration, the reliability (T) of each point in the first viewpoint image 41 can be determined based on the relationship that the first depth information 70A of the first viewpoint and the second depth information 70B of the second viewpoint theoretically match in a corresponding point pair. As a result, the first position coordinates and the first depth (Z1) of the point with the highest reliability (u l、 Z1), which makes it possible to identify reliable information from the output of a trained model used for 3D image measurement.

[0133] In the image processing device 100 according to the third embodiment, the depth calculation unit 4 calculates a set (u) of the first position coordinate and the first depth (Z1) having a high reliability of the first depth (Z1) from among the points in the first depth information 70A. l、 The depth information 71 is output, which is depth information 70 for each point identified as the first depth Z1. This makes it possible to output depth information (first depth Z1) for points with high reliability, and not output depth information (first depth Z1) for points with low reliability. As a result, when performing object recognition using depth information, only depth information that is likely to be highly reliable and accurate can be used, thereby improving the accuracy of information processing using depth information.

[0134] Other effects of the third embodiment are the same as those of the first embodiment.

[0135] The technical scope of the present invention is not limited to the above-described embodiment, and appropriate modifications can be made without departing from the spirit of the present invention. For example, in the above-described embodiment, an example was shown in which each unit of the image processing device 100 is realized by the processor 1001 of the computer system 1000. However, the image processing device 100 may be realized by one or more server devices constituting a computer system based on a client-server model. For example, the server device may receive the first viewpoint image 41 and the second viewpoint image 42 from a client terminal, acquire reliability information 60, and output depth information 70 acquired based on the reliability information 60 to the client terminal.

[0136] In the above embodiment, an example has been shown in which a stereo image consisting of two viewpoint images, a first viewpoint image 41 and a second viewpoint image 42, is handled, but the image processing device 100 may acquire three or more viewpoint images. The viewpoint images do not have to be parallel in the left-right direction of the image. In other words, the parallax direction may be other than the left-right direction of the image. Furthermore, in the above embodiment, an example has been shown in which the optical characteristics of the left camera 21 and the right camera 22 are identical to each other, but as long as the optical characteristics are known, the optical characteristics of the cameras acquiring each viewpoint image may be different from each other. In this case, a process may be performed to correct the difference in optical characteristics for each viewpoint image.

[0137] In the first embodiment, an example was shown in which a high-reliability region was identified from the estimated disparity and depth information for each point belonging to the identified high-reliability region was calculated. However, depth information may be calculated from the estimated disparity before identifying the high-reliability region, and the high-reliability region may be identified by comparing the depths as in the third embodiment. Alternatively, a high-reliability region may be identified from the estimated disparity without calculating depth information, and the estimated disparity for each point belonging to the identified high-reliability region may be output. In the first embodiment, the data output by the image processing device 100 to the data utilization device DU may be estimated disparity or depth. The same applies to the second embodiment, in which the image processing device 100 may output estimated disparity for each point belonging to the identified high-reliability region, or may calculate depth information from the estimated disparity for each point belonging to the high-reliability region and output the calculated depth information.

[0138] In the above embodiment, an example was shown in which the image processing device 100 includes the depth calculation unit 4, but the image processing device 100 does not necessarily need to include the depth calculation unit 4. [Explanation of symbols]

[0139] 1...image acquisition unit, 2...inference unit, 3...high-reliability region identification means, 4...depth calculation unit, 5...output unit, 6...memory unit, 7...object recognition unit, 8...imaging control unit, 11...program, 12...first trained model, 12A...first trained model, 12B...second trained model, 12C...first trained model, 13...camera parameters, 14...CAD data, 20...imaging device, 21...left camera, 22...right camera, 23...imaging field of view, 24...imaging field of view, 25...image surface, 26...image surface, 30...camera moving robot, 31...robot control device, 32...working robot, 33...robot control device, 41...first viewpoint Image, 42...second viewpoint image, 43...first inverted image, 44...second inverted image, 45...corresponding point map, 51...first estimated disparity, 52...second estimated disparity, 53...inverted estimated disparity, 60...reliability information, 70...depth information, 70A...first depth information, 70B...second depth information, 71...output depth information, 90...object, 91...region, 92...region, 93...region, 100...image processing device, 110...robot system, 1000...computer system, 1001...processor, 1002...main memory, 1003...storage, 1004...interface, P, P1, P2...point of interest, Q, Q1, Q2...corresponding points.

Claims

1. An image processing device that performs three-dimensional image measurement processing based on parallax between a plurality of images, an image acquisition unit that acquires a first viewpoint image and a second viewpoint image captured at the first viewpoint and the second viewpoint, respectively; an inference unit that is capable of using a first trained model that receives a first image and a second image, and that outputs a first estimated disparity, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate that represents the position of the point in the first image, and that inputs the first viewpoint image as the first image and the second viewpoint image as the second image into the first trained model to acquire the first estimated disparity associated with the first position coordinate; and a high-reliability area specifying means for determining, for each point in the first viewpoint image, an estimated position of a corresponding point in the second viewpoint image from the first position coordinates and the first estimated parallax, generating correspondence data in which the first position coordinates, the first estimated parallax, and the estimated positions of the corresponding points in the second viewpoint image are associated with each other, and specifying, from the set of correspondence data, a pair of the first position coordinates and the first estimated parallax in which the reliability of the first estimated parallax is high. Image processing device.

2. The inference unit is further configured to use a second trained model that receives the first image and the second image, and outputs a second estimated disparity, which is an inferred value by machine learning of the disparity between each point in the second image and a corresponding point in the first image, in association with second position coordinates that represent the position of the point in the second image; the first viewpoint image is input as the first image, and the second viewpoint image is input as the second image to the second trained model to obtain a second estimated disparity that is associated with the second position coordinates; the high-reliability region specifying means specifies the second estimated disparity associated with the second position coordinates that are identical to the estimated position of the corresponding point in the second viewpoint image that is associated with the first position coordinates in the association data, and determines that the reliability of the first estimated disparity associated with the first position coordinates is high when a difference between a magnitude of the first estimated disparity associated with the first position coordinates and a magnitude of the specified second estimated disparity is smaller than a predetermined threshold value. The image processing device according to claim 1 .

3. the first trained model outputs the first estimated disparity in association with the position of the point in the first image, and in addition, outputs a second estimated disparity, which is an inferred value by machine learning of the disparity between each point in the second image and a corresponding point in the first image, in association with second position coordinates representing the position of the point in the second image; the inference unit inputs the first viewpoint image as the first image and the second viewpoint image as the second image into the first trained model to acquire the first estimated disparity associated with the first position coordinates and the second estimated disparity associated with the second position coordinates; the high-reliability region specifying means specifies the second estimated disparity associated with the second position coordinates that are identical to the estimated position of the corresponding point in the second viewpoint image that is associated with the first position coordinates in the association data, and determines that the reliability of the first estimated disparity associated with the first position coordinates is high when a difference between a magnitude of the first estimated disparity associated with the first position coordinates and a magnitude of the specified second estimated disparity is smaller than a predetermined threshold value. The image processing device according to claim 1 .

4. the inference unit inputs a second inverted image obtained by inverting the second viewpoint image in a parallax direction as the first image, a first inverted image obtained by inverting the first viewpoint image in the parallax direction as the second image into the first trained model to acquire the first estimated parallax associated with the first position coordinates, inverts the first position coordinates in the parallax direction to acquire second position coordinates, and inverts a sign of the first estimated parallax to acquire a second estimated parallax, the high-reliability area specifying means specifies the second estimated disparity associated with the second position coordinates that are identical to the estimated position of the corresponding point in the second viewpoint image that is associated with the first position coordinates in the association data, and determines that the reliability of the first estimated disparity associated with the first position coordinates in the association data is high when a difference between a magnitude of the first estimated disparity associated with the first position coordinates in the association data and a magnitude of the specified second estimated disparity is smaller than a predetermined threshold value. The image processing device according to claim 1 .

5. When there are a predetermined number or more of other correspondence data including the same estimated positions of the corresponding points in the second viewpoint image as the estimated positions of the corresponding points in the second viewpoint image included in the correspondence data of interest, the high-reliability region specifying means determines that the reliability of at least a part of the first estimated parallaxes included in the predetermined number or more of correspondence data including the same estimated positions of the corresponding points in the second viewpoint image is not high. The image processing device according to claim 1 .

6. When there are a plurality of other correspondence data including estimated positions of corresponding points in the second viewpoint image that are identical to the estimated positions of corresponding points in the second viewpoint image included in the correspondence data of interest, the high-reliability region specifying means determines that only the first estimated disparity having the largest absolute value among the first estimated disparities included in the correspondence data is highly reliable. The image processing device according to claim 5 .

7. a depth calculation unit that calculates depth information representing a depth of each point identified as a pair of the first position coordinates and the first estimated disparity, where the reliability of the first estimated disparity is high; The image processing device according to claim 1 .

8. An image processing device that performs three-dimensional image measurement processing based on parallax between a plurality of images, an image acquisition unit that acquires a first viewpoint image and a second viewpoint image captured at the first viewpoint and the second viewpoint, respectively; an inference unit that is capable of using a first trained model that receives a first image and a second image, and that outputs a first estimated disparity, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate that represents the position of the point in the first image, and that inputs the first viewpoint image as the first image and the second viewpoint image as the second image into the first trained model to acquire the first estimated disparity associated with the first position coordinate; and a high-reliability area specifying means for determining, for each point in the first viewpoint image, an estimated position of a corresponding point in the second viewpoint image from the first position coordinates and the first estimated parallax, determining a first depth corresponding to the first estimated parallax, generating correspondence data in which the first position coordinates, the first depth, and the estimated position of the corresponding point in the second viewpoint image are associated with each other, and specifying, from the set of correspondence data, a pair of the first position coordinates and the first depth for which the reliability of the first depth is high. Image processing device.

9. An image processing method for performing three-dimensional image measurement processing based on parallax between a plurality of images, acquiring a first viewpoint image and a second viewpoint image captured at a first viewpoint and a second viewpoint, respectively; a trained model is available that inputs a first image and a second image, and outputs a first estimated disparity, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate representing the position of the point in the first image; and inputs the first viewpoint image as the first image and the second viewpoint image as the second image into the trained model to obtain the first estimated disparity associated with the first position coordinate; for each point in the first viewpoint image, determining an estimated position of a corresponding point in the second viewpoint image from the first position coordinates and the first estimated parallax, generating correspondence data in which the first position coordinates, the first estimated parallax, and the estimated position of the corresponding point in the second viewpoint image are associated with each other, and identifying, from the set of correspondence data, a pair of the first position coordinates and the first estimated parallax in which the reliability of the first estimated parallax is high. Image processing methods.

10. A program for performing three-dimensional image measurement processing based on parallax between multiple images, A process of acquiring a first viewpoint image and a second viewpoint image captured at the first viewpoint and the second viewpoint, respectively; a trained model is available for inputting a first image and a second image, and outputting a first estimated disparity, which is an inferred value by machine learning of the disparity between each point in the first image and a corresponding point in the second image, in association with a first position coordinate representing the position of the point in the first image; and a process of inputting the first viewpoint image as the first image and the second viewpoint image as the second image into the trained model to acquire the first estimated disparity associated with the first position coordinate; a process of determining, for each point in the first viewpoint image, an estimated position of a corresponding point in the second viewpoint image from the first position coordinates and the first estimated parallax, generating correspondence data in which the first position coordinates, the first estimated parallax, and the estimated position of the corresponding point in the second viewpoint image are associated with each other, and identifying, from the collection of correspondence data, a pair of the first position coordinates and the first estimated parallax for which the reliability of the first estimated parallax is high; A program that causes a computer to execute the following.

Images