Image processing device, three-dimensional measurement system, image processing method

Abstract

An image processing apparatus includes a map generation unit that generates a map using an image obtained by projecting pattern light on an object and taking an image, the map being data in which information on a depth distance is associated with each pixel; an edge detection unit that detects an edge of the object using an image obtained by taking an image without projecting pattern light on the object; and a correction unit that corrects the map based on the detected edge so that a position where the depth distance becomes discontinuous coincides with a position of the edge of the object.

Description

Technical Field

[0001] This invention relates to an image processing apparatus, and more particularly to a technique for obtaining high-precision three-dimensional information from an image. Background Technology

[0002] One method for acquiring three-dimensional information of an object using images is called active measurement (see Non-Patent Document 1). Active measurement has the advantage of projecting patterned light onto the object's surface, thus enabling high-precision three-dimensional measurement even of objects with few image features (such as objects without surface texture), making it far more practical than passive measurement.

[0003] Existing technical documents

[0004] Non-patent literature

[0005] Non-patent literature 1: P. Vuylsteke and A. Oosterlinck, Range Image Acquisition with a Single Binary-Encoded Light Pattern, IEEE PAMI 12(2), pp.148-164, 1990. Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] However, active measurement has an inherent technical problem: measurement accuracy tends to decrease near the edges of objects. Because the height of an object becomes discontinuous at the edges, the projected pattern also becomes discontinuous along with the edges. This makes it easy for pattern misidentification to occur near the edges, either misidentifying points on the object's surface as points in the background, or vice versa, resulting in the output of incorrect height information. Figure 18 This schematically illustrates the deviation between the three-dimensional information (point group data) obtained through active measurement and the actual edge (shape) of the object. It shows where point groups extend beyond the edge and where point groups are missing.

[0008] Depending on the purpose and intended use of active measurement, instances where minute deviations (hereinafter referred to as "measurement errors") between the three-dimensional information and the actual shape cannot be ignored may arise. For example, in situations where the shape of an object is identified through active measurement and the identification results are used for the gripping control of a picking robot, measurement errors may lead to picking or releasing malfunctions. Specifically, malfunctions such as: recognizing the object's shape as too large, resulting in an incorrect determination that there is no gap between the object and its surroundings for inserting the hand, thus interrupting the picking or releasing action; or conversely, recognizing the object's shape as too small, resulting in hand-object interference when gripping the object. Furthermore, in situations where the position and orientation of an object are identified through active measurement and parts are fitted and assembled, the aforementioned measurement errors become fatal due to the need for micrometer-level positioning. Additionally, in components where positioning marks (protrusions, grooves, etc.) are placed on the surface, it may be impossible to distinguish whether the surface irregularities appearing in the three-dimensional information obtained through active measurement are positioning marks or irregularities caused by measurement errors, thus failing to identify the positioning marks.

[0009] The present invention is made in view of the above-mentioned actual situation, and its object is to provide a technique for eliminating or even reducing measurement errors caused by misidentification of patterns at the edge portions of an object.

[0010] Solutions for solving technical problems

[0011] One aspect of the present invention provides an image processing apparatus, characterized in that it comprises: a map generation unit that generates a map using an image obtained by projecting patterned light onto an object and taking a picture, the map being data obtained by associating depth distance-related information with each pixel; an edge detection unit that detects the edges of the object using an image obtained by taking a picture without projecting patterned light onto the object; and a correction unit that corrects the map based on the detected edges so that the positions where the depth distance becomes discontinuous match the positions of the edges of the object.

[0012] The map generated by the map generation unit reconstructs the depth distance from an image obtained by projecting patterned light onto the object and then photographing it. Therefore, in principle, it may include measurement errors caused by misidentification of the pattern. On the other hand, the edge information generated by the edge detection unit is obtained from an image obtained by photographing it without projecting patterned light, and thus does not include errors caused by misidentification of the pattern. Therefore, by correcting the map based on the positions of the detected edges, measurement errors included in the map can be eliminated or even reduced, resulting in a more accurate map compared to previous methods.

[0013] The method by which the map generation unit generates the map is not limited. For example, the map generation unit could also be a unit that generates the map by stereo matching multiple images obtained from different perspectives, projecting patterned light onto the object. Stereo matching is a method of calculating the depth distance based on the parallax between two images obtained from different perspectives using the principle of triangulation. According to stereo matching, high-precision three-dimensional measurements with high spatial resolution can be performed. In the case of stereo matching, the map can be a parallax map, a depth map, etc.

[0014] Alternatively, the map generation unit can also be a unit that generates the map using a spatially encoded pattern method by projecting patterned light onto the object and capturing the image. The spatially encoded pattern method calculates the depth distance based on the relationship between camera pixels and corresponding points of the encoded pattern using the principle of triangulation, enabling the acquisition of three-dimensional information from a single image. The spatially encoded pattern method allows for high-speed and high-precision three-dimensional measurement. In the case of the spatially encoded pattern method, the map can also be a depth map, etc.

[0015] Alternatively, the map generation unit may also include: an acquisition unit that acquires an image pair consisting of a first image and a second image obtained by projecting patterned light onto the object and taking pictures from different perspectives; a disparity prediction unit that predicts the disparity between the first image and the second image in a manner different from stereo matching; a setting unit that sets the search range for corresponding points in stereo matching based on the predicted disparity; and a stereo matching unit that performs stereo matching within the set search range using the first image and the second image. In general stereo matching, since corresponding points are searched from all comparison images, the processing time inevitably increases if high-resolution images are used. In contrast, in the above configuration, the search range for corresponding points is limited based on the predicted disparity. Therefore, since the search range can be greatly narrowed, the time required to search for corresponding points can be significantly shortened.

[0016] The disparity prediction unit can also predict disparity based on distance information obtained through spatial coding patterning. This is because, using an image sensor with the same resolution as stereo matching, spatial coding patterning allows for significantly shorter processing time compared to stereo matching when acquiring distance information. It should be noted that while the spatial resolution of ranging using spatial coding patterning is lower than that of stereo matching, it is sufficient for the purpose of predicting disparity.

[0017] The map generation unit may also include a disparity map synthesis unit, which generates a synthetic disparity map by synthesizing multiple disparity maps generated by the stereo matching unit from each of the multiple image pairs. In this case, the correction unit may also correct the synthetic disparity map or the depth map converted from the synthetic disparity map. By synthesizing the disparity map obtained from multiple image pairs, measurement bias can be reduced, and a stable, high-precision, and highly reliable map can be obtained.

[0018] The disparity prediction unit can generate multiple predicted disparities, and the setting unit can use the synthesized predicted disparity obtained by synthesizing the multiple predicted disparities to set the search range of the multiple image pairs. By generating (measuring) the predicted disparities multiple times in this way, accuracy and robustness can be improved.

[0019] An image obtained by taking a picture without projecting patterned light onto the object can also be obtained by projecting uniform light onto the object and then taking a picture. For example, a uniform white light source can also be used.

[0020] A second aspect of the present invention provides a three-dimensional measurement system, characterized by comprising: a sensor unit having at least a first projection section having projected pattern light and one or more cameras; and an image processing apparatus according to the first aspect, which processes images acquired from the sensor unit. According to this configuration, it is possible to obtain three-dimensional information that eliminates or even reduces measurement errors caused by misidentification of patterns at the edge portions of an object.

[0021] The present invention can also be understood as an image processing apparatus or a three-dimensional measurement system having at least a portion of the above-described units. Furthermore, the present invention can be understood as an image processing method, a three-dimensional measurement method, a ranging method, a control method for an image processing apparatus, a control method for a three-dimensional measurement system, etc., including at least a portion of the above-described processes; or the present invention can be understood as a program for implementing the method, or a recording medium on which the program is not temporarily recorded. It should be noted that the above-described units and processes can be combined with each other to constitute the present invention to the extent possible.

[0022] Invention Effects

[0023] According to the present invention, measurement errors caused by misidentification of patterns at the edge portions of an object can be eliminated or even reduced. Attached Figure Description

[0024] Figure 1 The diagram illustrates an example of the configuration of a three-dimensional measurement system, which is one of the application examples of the present invention.

[0025] Figure 2This is a flowchart illustrating the measurement processing flow of a three-dimensional measurement system.

[0026] Figure 3 The diagram illustrates the images and maps used in the measurement process.

[0027] Figure 4 This is a flowchart of map correction processing performed by an image processing device.

[0028] Figure 5 The diagram illustrates an example of map correction processing.

[0029] Figure 6 This is a functional block diagram of the three-dimensional measurement system according to the first embodiment of the present invention.

[0030] Figure 7 This diagram illustrates an example of the configuration of the pattern projection section.

[0031] Figure 8 This is a functional block diagram of a three-dimensional measurement system according to the second embodiment of the present invention.

[0032] Figure 9 A flowchart illustrating the measurement process of the second embodiment.

[0033] Figure 10 A flowchart illustrating the post-processing flow of the second embodiment.

[0034] Figure 11 This is a functional block diagram of a three-dimensional measurement system according to the third embodiment of the present invention.

[0035] Figure 12 A flowchart illustrating the measurement process of the third embodiment.

[0036] Figure 13 This is a functional block diagram of the three-dimensional measurement system according to the fourth embodiment of the present invention.

[0037] Figure 14 A flowchart illustrating the measurement process of the fourth embodiment.

[0038] Figure 15 This is a functional block diagram of the three-dimensional measurement system according to the fifth embodiment of the present invention.

[0039] Figure 16 A flowchart illustrating the measurement process of the fifth embodiment.

[0040] Figure 17 This is a timing diagram for the measurement process in the fifth embodiment.

[0041] Figure 18A diagram illustrating the deviation between the three-dimensional information (point group data) obtained through active measurement and the actual edge (shape) of the object. Detailed Implementation

[0042] <Application Examples>

[0043] Figure 1 The diagram illustrates a configuration example of a three-dimensional measurement system, one of the application examples of the present invention. The three-dimensional measurement system 1 is a system for measuring the three-dimensional shape of an object 12 by image sensing. Generally speaking, it consists of a sensor unit 10 and an image processing device 11. The sensor unit 10 includes at least a first projection unit that projects patterned light, a second projection unit that projects unpatterned light, and one or more cameras (also called image sensors or imaging devices), and may also include other sensors and illumination as needed. The output of the sensor unit 10 is acquired by the image processing device 11. The image processing device 11 is a device that performs various processing on the data acquired from the sensor unit 10. Processing by the image processing device 11 may include, for example, distance measurement (range finding), three-dimensional shape recognition, object recognition, scene recognition, etc. The processing results of the image processing device 11 may be output to an output device such as a display, or transmitted externally and used for inspection, control of other devices, etc. Such a three-dimensional measurement system 1, for example, is represented by computer vision, robot vision, and machine vision, and is applied in a wide range of fields. In particular, since the three-dimensional measurement system 1 disclosed herein can obtain high-precision three-dimensional information of the object 12, it can be preferably applied to the control of various industrial robots for picking up, positioning, assembling, inspecting, etc. of objects.

[0044] Figure 1 The configuration is just one example; its hardware configuration can be designed appropriately according to the application of the 3D measurement system 1. For example, the sensor unit 10 and the image processing device 11 can be connected wirelessly, or they can be configured as a single unit. Alternatively, the sensor unit 10 and the image processing device 11 can be connected via a wide area network such as a LAN or the Internet. Furthermore, multiple sensor units 10 can be provided to one image processing device 11, or conversely, the output of one sensor unit 10 can be provided to multiple image processing devices 11. Moreover, the viewpoint of the sensor unit 10 can be moved by installing the sensor unit 10 on a robot, a moving body, etc.

[0045] <Measurement Processing Flow>

[0046] Reference Figure 2 and Figure 3 The general process of measurement processing of the three-dimensional measurement system 1 is described below. Figure 2A flowchart illustrating the measurement processing flow of the three-dimensional measurement system 1 is provided. Figure 3 The diagram illustrates the images and maps used in the measurement process.

[0047] In step S200, patterned light is projected from sensor unit 10 (first projection unit) onto object 12. The pattern of the patterned light can be a fixed pattern, a random pattern, or a striped pattern used in phase transfer methods, etc. A suitable pattern can be selected by combining the restoration algorithm with the three-dimensional information.

[0048] In step S201, the sensor unit 10 (camera) takes a picture of the object 12 on which the patterned light is projected. The resulting image 30 (hereinafter referred to as a "patterned image" in the sense of an image with a projected pattern) is acquired by the image processing device 11.

[0049] In step S202, the image processing device 11 (map generation unit) restores the three-dimensional information from the pattern image 30 obtained in step S201 and generates a map 31. The map 31 is data obtained by associating depth distance-related information with each pixel. The "depth distance-related information" can be the depth distance itself or information that can be converted into depth distance (e.g., parallax information).

[0050] Next, in step S203, non-patterned light is projected from the sensor unit 10 (second projection unit) onto the object 12. For example, the object 12 can be illuminated with uniform white light. It should be noted that in sufficiently bright environments, illumination of the object 12 may not be necessary.

[0051] In step S204, the sensor unit 10 (camera) takes a picture of the object 12. The resulting image 32 (hereinafter also referred to as a "non-patterned image" in the sense of an image without a projected pattern) is acquired by the image processing device 11.

[0052] In step S205, the image processing device 11 (edge ​​detection unit) uses the non-patterned image 32 acquired in step S204 to detect the edges of the object 12. In edge detection, any method can be applied, such as a differential filter or a Laplace filter.

[0053] In step S206, the image processing device 11 (correction unit) corrects the map 31 obtained in step S202 based on the information of the edge 33 obtained in step S205. If the 3D information is correctly restored in step S202, the location where the depth distance becomes discontinuous in the map 31 should correspond to the location of the edge 33 detected in step S205. However, if the pattern is incorrectly identified at the edge of the object 12, and errors occur in the restoration of the 3D information, the location where the depth distance becomes discontinuous in the map 31 will deviate from the location of the edge. Therefore, in step S206, map 31 is corrected to match the location where the depth distance becomes discontinuous with the location of the edge.

[0054] In step S207, the image processing device 11 (3D information generation unit) generates output 3D information (e.g., point cluster data) based on the corrected map 34. This processing enables the generation of high-precision and highly reliable 3D information because it eliminates or even reduces measurement errors caused by misidentification of patterns.

[0055] <Map Correction>

[0056] Reference Figure 4 and Figure 5 Here is a specific example of the map correction process in step S206. Figure 4 This is a flowchart of the map correction process performed by the image processing device 11. Figure 5 The diagram illustrates an example of map correction processing.

[0057] In step S400, the image processing device 11 segments the map 50 based on depth distance and extracts the region 51 corresponding to the object 12. The segmentation method is not limited. For example, the map 50 can be binarized or N-valued based on depth distance before marking, or regions with substantially the same or continuous depth distances and spatial continuity can be extracted.

[0058] In step S401, the image processing device 11 will perform edge detection ( Figure 2 In step S205, the edge 52 obtained coincides with the region 51 extracted from the map 50. Then, the image processing device 11 segments the region 51 according to the edge 52 (step S402) and calculates the area (number of pixels) S of each segmented region 53a to 53c (step S403).

[0059] In step S404, the image processing device 11 compares the area S of each segmented region 53a to 53c with a predetermined threshold X, and determines the segmented regions 53a and 53c whose area S is smaller than the threshold X as measurement errors. In step S405, the image processing device 11 applies a correction to the map 50 to remove the segmented regions 53a and 53c that are determined to be measurement errors, and outputs the corrected map 54. The operation of removing segmented regions can be, for example, replacing the depth distance value of the pixels in the segmented region with the depth distance value of the background region, and can also be an operation of deleting the point groups in the segmented region if the map 50 is point group data (that is, a set of points with depth distance information). Through the above processing, the measurement error of the map caused by the misidentification of the pattern can be eliminated or even reduced.

[0060] <First Implementation>

[0061] Reference Figure 6 Here, a configuration example of the three-dimensional measurement system 1 according to the first embodiment of the present invention will be described. In the first embodiment, the three-dimensional information of the object 12 is generated by means of a spatial coding pattern.

[0062] (Sensor Unit)

[0063] The sensor unit 10 includes a first camera 101, a pattern projection unit (first projection unit) 103, an illumination unit (second projection unit) 104, an image transmission unit 105, and a drive control unit 106.

[0064] The first camera 101 is a camera device that takes pictures of the object 12 to obtain image data. The first camera 101 can be either a black and white camera or a color camera.

[0065] The pattern projection unit 103 is a projection device, also known as a projector, used to project pattern light used in spatially encoded pattern ranging onto the object 12. In spatially encoded pattern ranging, for example, a regular pattern (encoded pattern) is used to specify a distance. Figure 7The diagram schematically illustrates an example of the configuration of the pattern projection unit 103. The pattern projection unit 103 is, for example, composed of a light source unit 180, a light guide lens 181, a pattern generation unit 182, and a projection lens 183. An LED, laser, or VCSEL (Vertical Cavity Surface-emitting Laser) can be used as the light source unit 180. The light guide lens 181 is an optical element used to guide light from the light source unit 180 to the pattern generation unit 182, and can be a lens or a glass rod. The pattern generation unit 182 is a component or even a device for generating composite patterns, and can use a photomask, a diffractive optical element (e.g., DOE (Diffractive Optical Element)), or a light modulation element (e.g., DLP (Digital Light Processing), LCD (Liquid Crystal Display), LCoS (Liquid Crystal on Silicon), MEMS (Micro Electromechanical Systems)). The projection lens 183 is an optical element that magnifies and projects the generated pattern.

[0066] The illumination unit 104 provides uniform illumination for capturing general visible light images. For example, it may use white LED illumination. Alternatively, it may use illumination with the same wavelength as the pattern projection unit 103.

[0067] The image transmission unit 105 transmits image data captured by the first camera 101 to the image processing device 11. The drive control unit 106 is a unit that controls the first camera 101, the pattern projection unit 103, and the illumination unit 104. It should be noted that the image transmission unit 105 and the drive control unit 106 may also be provided on the image processing device 11 side, instead of on the sensor unit 10 side.

[0068] (Image processing device)

[0069] The image processing apparatus 11 includes an image acquisition unit 110, a pattern decoding unit (map generation unit) 111, an edge detection unit (edge ​​detection unit) 112, and a map correction unit (correction unit) 113.

[0070] The image acquisition unit 110 has the function of acquiring the required image data from the sensor unit 10. The image acquisition unit 110 sends the image (pattern image) obtained by projecting patterned light and capturing it to the pattern decoding unit 111, and sends the image (non-pattern image) obtained by projecting non-patterned light and capturing it to the edge detection unit 112. The pattern decoding unit 111 acquires depth information (depth information) from the pattern image using a spatially encoded pattern method. A depth map is output from the pattern decoding unit 111. The edge detection unit 112 detects the edges of the object 12 from the non-pattern image. The map correction unit 113 corrects the depth map based on the detected edges and outputs the corrected depth map. This depth map is used, for example, in object shape recognition, object recognition, etc.

[0071] The image processing apparatus 11 is, for example, a computer equipped with a CPU (processor), RAM (memory), non-volatile storage (hard disk, SSD, etc.), input devices, and output devices. In this case, the CPU expands the program stored in the non-volatile storage in the RAM and executes the program to achieve the various functions described above. However, the configuration of the image processing apparatus 11 is not limited to this; all or part of the above functions can be implemented using dedicated circuits such as FPGAs and ASICs, or through cloud computing and distributed computing. The same applies to subsequent embodiments.

[0072] According to the configuration of this embodiment described above, because a spatially encoded pattern method is used, three-dimensional measurement can be performed at high speed and with high accuracy. Furthermore, because the depth map is corrected based on the position of edges detected from non-patterned images, measurement errors caused by misidentification of patterns can be eliminated or even reduced, resulting in a higher-precision depth map compared to conventional methods.

[0073] <Second Implementation>

[0074] Reference Figure 8 Here, a configuration example of the three-dimensional measurement system 1 according to the second embodiment of the present invention will be described. In the second embodiment, three-dimensional information of the object 12 is generated by stereo matching.

[0075] (Sensor Unit)

[0076] The sensor unit 10 includes a first camera 101, a second camera 102, a pattern projection unit (first projection unit) 103, an illumination unit (second projection unit) 104, an image transmission unit 105, and a drive control unit 106.

[0077] The first camera 101 and the second camera 102 are a camera pair constituting a so-called stereo camera, configured to be separated by a predetermined distance. By simultaneously taking pictures with the two cameras 101 and 102, it is possible to obtain a pair of images taken from different perspectives (the image from the first camera 101 is referred to as the first image, and the image from the second camera 102 is referred to as the second image). The two cameras 101 and 102 can also be configured such that their optical axes intersect each other, and their horizontal (or vertical) lines lie on the same plane. By adopting such a configuration, since the epipolar lines become parallel to the horizontal (or vertical) lines of the image, corresponding points in stereo matching can be searched within the horizontal (or vertical) lines at the same position, thus simplifying the search process. It should be noted that both black-and-white and color cameras can be used as cameras 101 and 102.

[0078] The pattern projection unit 103 is a projection device, also called a projector, for projecting patterned light used in so-called active stereo ranging onto the object 12. In active stereo ranging, for example, a regular pattern (encoded pattern) for specifying a distance can be used, as in the first embodiment, or an irregular pattern such as random dots can be used. The illumination unit 104 is uniform illumination for capturing general visible images. The specific configuration of the pattern projection unit 103 and the illumination unit 104 can be the same as in the first embodiment.

[0079] The image transmission unit 105 transmits data of the first image captured by the first camera 101 and data of the second image captured by the second camera 102 to the image processing device 11. The image transmission unit 105 can transmit the first and second images as separate image data, or it can combine the first and second images to generate a side-by-side image for single image data transmission. The drive control unit 106 controls the first camera 101, the second camera 102, the pattern projection unit 103, and the illumination unit 104. It should be noted that the image transmission unit 105 and the drive control unit 106 can also be located on the image processing device 11 side, instead of the sensor unit 10 side.

[0080] (Image processing device)

[0081] The image processing apparatus 11 includes an image acquisition unit 110, a preprocessing unit 800, a corresponding point search unit (map generation unit) 801, an edge detection unit (edge ​​detection unit) 112, a map correction unit (correction unit) 113, a parallax map postprocessing unit 802, and a depth map generation unit 803.

[0082] The image acquisition unit 110 sends a stereoscopic image consisting of a first image and a second image obtained by projecting patterned light and capturing the image to the preprocessing unit 800, and sends an image obtained by projecting non-patterned light and capturing the image to the edge detection unit 112. The preprocessing unit 800 has the function of performing necessary preprocessing on the first image and the second image. The corresponding point search unit 801 has the function of searching for corresponding points between the first image and the second image and generating a disparity map based on the search results. The edge detection unit 112 has the function of detecting the edges of the object 12 from the non-patterned image, and the map correction unit 113 has the function of correcting the disparity map based on the detected edges and outputting the corrected disparity map. The disparity map post-processing unit 802 has the function of performing necessary post-processing on the disparity map. The depth map generation unit 803 has the function of converting the disparity information of the disparity map into distance information and generating a depth map.

[0083] (Map generation)

[0084] Reference Figure 9 The process of map generation in the second embodiment will be explained. Figure 9 To show Figure 2 The flowchart details steps S202, S205, and S206.

[0085] In step S900, the image acquisition unit 110 acquires a first image and a second image from the sensor unit 10. Both the first and second images are captured by the first camera 101 and the second camera 102 while patterned light is projected onto the object 12 from the patterned light projection unit 103. It should be noted that when data in the form of side-by-side images is acquired from the sensor unit 10, the image acquisition unit 110 divides the side-by-side images into a first image and a second image. The image acquisition unit 110 then sends the first and second images to the preprocessing unit 800.

[0086] In step S901, the preprocessing unit 800 performs parallelization processing (correction) on the first image and the second image. Parallelization processing is a geometric transformation of one or both images to ensure that corresponding points between the two images lie on the same horizontal (or vertical) line in the images. Since the epipolar lines become parallel to the horizontal (or vertical) lines of the images through parallelization processing, the subsequent corresponding point search process becomes simpler. It should be noted that if the parallelism of the images acquired from the sensor unit 10 is sufficiently high, the parallelization processing in step S901 can be omitted.

[0087] In step S902, the preprocessing unit 800 calculates hash feature values ​​for each pixel of the parallelized first and second images, and replaces the value of each pixel with a hash feature value. A hash feature value is a feature value representing the brightness characteristics of a local region centered on the pixel of interest; here, a hash feature value consisting of a bit string of 8 elements is used. In this way, by converting the brightness values ​​of each image into hash feature values ​​beforehand, the subsequent similarity calculation of local brightness features in the corresponding point search becomes extremely efficient.

[0088] In step S903, the corresponding point search unit 801 searches for corresponding points between the first image and the second image, and calculates the disparity of each pixel. The corresponding point search unit 801 generates disparity data, in which disparity information is associated with the coordinates of the points (pixels) where corresponding points were successfully detected. This information is a disparity map.

[0089] Next, in step S910, the image acquisition unit 110 acquires a non-patterned image from the sensor unit 10. The non-patterned image is an image captured by the first camera 101 or the second camera 102 while non-patterned light (uniform illumination, etc.) is projected from the illumination unit 104 onto the object 12.

[0090] In step S911, the edge detection unit 112 performs parallelization processing (correction) on the non-patterned image. It should be noted that if the parallelism of the image is sufficiently high, the parallelization processing can be omitted. In step S912, the edge detection unit 112 detects the edges of the object 12 from the non-patterned image.

[0091] Then, in step S913, the map correction unit 113 corrects the disparity map generated in step S903 based on the edges detected in step S912. Details of the map correction process are as follows... Figure 4 As shown.

[0092] (Post-processing)

[0093] Reference Figure 10 The post-processing flow in the second embodiment will be explained. Figure 10 To show Figure 2 A flowchart detailing step S207.

[0094] In step S1000, the disparity map post-processing unit 802 performs post-processing of the disparity map. Since the disparity map inferred through corresponding point search includes mismeasured points, measurement omissions, etc., as post-processing, it corrects mismeasured points and supplements measurement omissions based on the disparity information of surrounding pixels.

[0095] In step S1001, the depth map generation unit 803 converts the disparity information of each pixel in the disparity map into depth distance information and generates a depth map. This depth map (point group data) is used, for example, for object shape recognition, object recognition, etc.

[0096] Based on the configuration of this embodiment described above, high-precision three-dimensional measurements with high spatial resolution can be performed by utilizing stereo matching. Furthermore, because the disparity map is corrected based on the position of edges detected from non-patterned images, measurement errors caused by misidentification of patterns can be eliminated or even reduced, resulting in a higher-precision disparity map compared to conventional methods. It should be noted that although map correction processing is applied to the disparity map in this embodiment, it is also possible to first perform a conversion process from the disparity map to a depth map, and then apply map correction processing to the depth map.

[0097] <Third Implementation Method>

[0098] Reference Figure 11 Hereinafter, an example of the configuration of the three-dimensional measurement system 1 according to the third embodiment of the present invention will be described. In the third embodiment, stereo matching is performed on multiple image pairs, and the measurement accuracy and robustness are improved by synthesizing disparity maps generated from each image pair. In the following description, the configuration that differs from the second embodiment will be mainly described, and the configuration that is the same as the second embodiment will be omitted.

[0099] (Sensor Unit)

[0100] The sensor unit 10 includes a first camera 101, a second camera 102, a third camera 1100, a fourth camera 1101, a pattern projection unit (first projection unit) 103, an illumination unit (second projection unit) 104, an image transmission unit 105, and a drive control unit 106. The sensor unit 10 of this embodiment has four cameras, enabling it to simultaneously capture images from four different perspectives in a single projection.

[0101] The third camera 1100 and the fourth camera 1101 are a camera pair constituting a so-called stereo camera, configured to be separated by a predetermined distance (the image from the third camera 1100 is referred to as the third image, and the image from the fourth camera 1101 is referred to as the fourth image). The two cameras 1100 and 1101 can be configured such that their optical axes intersect and their horizontal (or vertical) lines lie on the same plane. By adopting such a configuration, since the epipolar lines become parallel to the horizontal (or vertical) lines of the image, corresponding points in stereo matching can be searched within the horizontal (or vertical) lines at the same position, thus simplifying the search process. It should be noted that both black-and-white and color cameras can be used as cameras 1100 and 1101.

[0102] The image transmission unit 105 transmits the data of the first image to the fourth image obtained by the first camera 101 to the fourth camera 1101 to the image processing device 11. The image transmission unit 105 can transmit the first image to the fourth image as individual image data, or it can combine the first image to the fourth image to generate a side-by-side image for single image data transmission.

[0103] (Image processing device)

[0104] The image processing apparatus 11 includes an image acquisition unit 110, a preprocessing unit 800, a corresponding point search unit (map generation unit) 801, a disparity map synthesis unit (disparity map synthesis unit) 1103, an edge detection unit (edge ​​detection unit) 112, a map correction unit (correction unit) 113, a disparity map postprocessing unit 802, and a depth map generation unit 803. The disparity map synthesis unit 1103 has the function of generating a synthesized disparity map by synthesizing multiple disparity maps generated from each of multiple image pairs.

[0105] (Map generation)

[0106] Reference Figure 12 The process of map generation in the third embodiment is described. Figure 12 To show Figure 2 The flowchart details steps S202, S205, and S206.

[0107] In step S1200, the image acquisition unit 110 acquires the first image to the fourth image from the sensor unit 10. The first image to the fourth image are images captured by the first camera 101, the second camera 102, the third camera 1100, and the fourth camera 1101, respectively, while patterned light is projected from the pattern projection unit 103 onto the object 12. It should be noted that when data in the form of side-by-side images is acquired from the sensor unit 10, the image acquisition unit 110 divides the side-by-side images into the first image to the fourth image. In this embodiment, the first image and the second image are used as a first image pair, and the third image and the fourth image are used as a second image pair, and these are utilized in stereo matching.

[0108] In step S1201, the preprocessing unit 800 performs parallelization processing (correction) on the first image to the fourth image. In step S1202, the preprocessing unit 800 calculates the hash feature value for each pixel of the parallelized first image to the fourth image, and replaces the value of each pixel with the hash feature value.

[0109] In step S1203, the corresponding point search unit 801 searches for corresponding points between the first image and the second image, which are the first image pair, and generates a parallax distance. Figure 1 Next, in step S1204, the corresponding point search unit 801 searches for corresponding points between the third and fourth images, which are the second image pair, and generates a parallax distance. Figure 2 .

[0110] In step S1205, the disparity map synthesis unit 1103 synthesizes the disparity map obtained from the first image pair. Figure 1 and the parallax obtained from the second image pair Figure 2 This process generates a composite parallax map. The composite method is not particularly limited; for example, it could be done within the parallax map... Figure 1 and parallax Figure 2 In both cases where parallax is obtained, their average value is set as the composite parallax; however, when parallax is obtained only in relation to parallax... Figure 1 and parallax Figure 2 If the disparity is obtained from either of the values, then the value is directly set as the composite disparity.

[0111] Next, in step S1210, the image acquisition unit 110 acquires a non-patterned image from the sensor unit 10. The non-patterned image is an image captured by the first camera 101 or the second camera 102 while non-patterned light (uniform illumination, etc.) is projected from the illumination unit 104 onto the object 12.

[0112] In step S1211, the edge detection unit 112 performs parallelization processing (correction) on the non-patterned image. It should be noted that if the parallelism of the image is sufficiently high, the parallelization processing can be omitted. In step S1212, the edge detection unit 112 detects the edges of the object 12 from the non-patterned image.

[0113] Then, in step S1213, the map correction unit 113 corrects the synthetic parallax map generated in step S1205 based on the edges detected in step S1212. Details of the map correction process are as follows... Figure 4 As shown.

[0114] According to the configuration of this embodiment described above, by synthesizing a disparity map obtained from multiple image pairs, measurement bias can be reduced. Therefore, compared to the configuration of the second embodiment, a more stable and reliable map can be obtained.

[0115] It should be noted that, although map correction processing is applied to the synthetic parallax map in this embodiment, it is also possible to first perform the conversion processing from the synthetic parallax map to the depth map, and then apply map correction processing to the depth map. Furthermore, although two image pairs are used in this embodiment, three or more image pairs can also be used. Additionally, the same image can be used in two or more image pairs. For example, the first image to the third image can be used to form three image pairs: a first image and a second image pair, a first image and a third image pair, and a second image and a third image pair. Furthermore, the shooting conditions can be different for each image pair. For example, in the configuration of this embodiment, by differentiating the exposure times of the first and second cameras from those of the third and fourth cameras, the effect of shooting based on so-called high dynamic range (HDR) can be obtained, which can improve the robustness of the measurement. Alternatively, multiple image pairs can be obtained by taking multiple shots with one set of stereo cameras, instead of using two sets of stereo cameras as in this embodiment. In this case, the robustness of the measurement can also be improved by varying the shooting conditions (exposure time, illumination time of the pattern light, etc.) for each shot.

[0116] Furthermore, when projecting non-patterned light, multiple non-patterned images are acquired by simultaneously capturing images with multiple cameras, and high-precision edge information is preferably generated by synthesizing edge information detected from each non-patterned image. Improving the accuracy of the edge information can further enhance the accuracy and reliability of the corrected map.

[0117] <Fourth Implementation>

[0118] Reference Figure 13 Hereinafter, an example of the configuration of the three-dimensional measurement system 1 according to the fourth embodiment of the present invention will be described. In the fourth embodiment, a combination of spatial coding pattern method and stereo matching is used to obtain the three-dimensional information of the object 12 at high speed and with high accuracy.

[0119] The configuration of the sensor unit 10 is the same as in the second embodiment. The image processing apparatus 11 includes an image acquisition unit 110, a pattern decoding unit 1300, a disparity prediction unit 1301, a preprocessing unit 800, a search range setting unit 1302, a corresponding point search unit 801, an edge detection unit (edge ​​detection unit) 112, a map correction unit (correction unit) 113, a disparity map postprocessing unit 802, and a depth map generation unit 803. The pattern decoding unit 1300 has the function of acquiring distance information from the first image using a spatially encoded pattern. The disparity prediction unit 1301 has the function of predicting the disparity between the first image and the second image based on the distance information obtained by the pattern decoding unit 1300 and outputting a reference disparity map. The search range setting unit 1302 has the function of setting the search range of corresponding points based on the predicted disparity. Other than this, the configuration is basically the same as in the second embodiment, so its description is omitted.

[0120] (Map generation)

[0121] Reference Figure 14 The process of map generation in the fourth embodiment will be explained. Figure 14 To show Figure 2 The flowchart details steps S202, S205, and S206. It should be noted that, for the map generation process of the second embodiment (…),… Figure 9 The same processing is marked with the same step number, and the detailed description is discarded.

[0122] In step S900, the image acquisition unit 110 acquires a first image and a second image from the sensor unit 10. In step S901, the preprocessing unit 800 performs parallelization processing (correction) on the first image and the second image. In step S902, the preprocessing unit 800 calculates a hash feature value for each pixel of the parallelized first image and the second image, and replaces the value of each pixel with the hash feature value.

[0123] In step S1400, the pattern decoding unit 1300 obtains distance information in the depth direction at multiple points on the first image by parsing the first image and decoding the pattern.

[0124] In step S1401, the disparity prediction unit 1301 calculates the two-dimensional coordinates of each point when the first image, parallelized, is projected onto the image coordinate system, and the two-dimensional coordinates of the second image, parallelized with the same points, are projected onto the image coordinate system, based on the distance information of each point obtained in step S1400. It then calculates the difference between the coordinates of the two images. This difference is the predicted disparity. The disparity prediction unit 1301 calculates the predicted disparity for all points for which distance information was obtained in step S1400, and outputs this data as a reference disparity map.

[0125] In step S1402, the search range setting unit 1302 sets the search range for corresponding points in the first image and the second image based on the predicted disparity. The size of the search range is determined considering the prediction error. For example, if the prediction error is ±10 pixels, it is considered sufficient to set the search range to approximately ±20 pixels centered on the predicted disparity, even including the margin. Assuming the horizontal line is 640 pixels, by focusing the search range on ±20 pixels (i.e., 40 pixels), the search processing can be easily reduced to 1 / 16 compared to searching the entire horizontal line.

[0126] In step S1403, the corresponding point search unit 801 searches for corresponding points between the first image and the second image within a predefined search range, and calculates the disparity of each pixel. The corresponding point search unit 801 generates disparity data, in which disparity information is associated with the coordinates of points (pixels) where corresponding points were successfully detected. This information is a disparity map.

[0127] Next, in step S1410, the image acquisition unit 110 acquires a non-patterned image from the sensor unit 10. The non-patterned image is an image captured by the first camera 101 or the second camera 102 while non-patterned light (uniform illumination, etc.) is projected from the illumination unit 104 onto the object 12.

[0128] In step S1411, the edge detection unit 112 performs parallelization processing (correction) on the non-patterned image. It should be noted that if the parallelism of the image is sufficiently high, the parallelization processing can be omitted. In step S1412, the edge detection unit 112 detects the edges of the object 12 from the non-patterned image.

[0129] Then, in step S1413, the map correction unit 113 corrects the disparity map generated in step S1403 based on the edges detected in step S1412. Details of the map correction process are as follows... Figure 4 As shown.

[0130] In the embodiment described above, the search range for corresponding points is limited based on the predicted disparity. Therefore, since the search range can be greatly narrowed, the time required to search for corresponding points can be significantly reduced. Furthermore, improvements in the success rate and accuracy of stereo matching, i.e., corresponding point search, are also expected. Thus, according to this embodiment, high-speed and high-precision three-dimensional measurements with high spatial resolution can be performed.

[0131] It should be noted that although spatially encoded pattern ranging was used in this embodiment to predict parallax, any method that can measure distance or predict parallax faster than stereo matching can also be used for ranging or parallax prediction using methods other than spatially encoded pattern ranging. Examples include: time-encoded pattern projection, Moa topography (contour line method), illuminance difference stereo method (photometric stereo vision), illuminance difference method, laser confocal method, white confocal method, optical interference method, visual volume cross method (Shape from silhouette method), factorization method, depth from motion (Structure from Motion) method, depth from shading method, depth from focusing method, depth from defocus method, depth from zoom method, time of light (TOF) measurement method, and time of light (TOF) phase difference measurement method. In addition, although map correction processing is applied to the parallax map in this embodiment, it is also possible to first perform the conversion processing from the parallax map to the depth map, and then apply map correction processing to the depth map.

[0132] Alternatively, the configuration described in the third embodiment (such as improving the accuracy of the parallax map by using multiple image pairs, or improving the accuracy of edge information by using multiple non-patterned images) can be combined into the fourth embodiment.

[0133] <Fifth Implementation>

[0134] Reference Figure 15 Hereinafter, a configuration example of the three-dimensional measurement system 1 according to the fifth embodiment of the present invention will be described. The fifth embodiment is a variation of the method of the fourth embodiment (a combination of spatial coding pattern method and stereo matching), which further improves the accuracy and robustness of the measurement by performing the generation of predicted disparity and stereo matching (generation of disparity map) multiple times together. In the following description, the configurations that are different from those of the fourth embodiment will be described in detail, and the configurations that are the same as those of the fourth embodiment will be omitted.

[0135] The configuration of the sensor unit 10 can be the same as in the fourth embodiment. The image processing apparatus 11 includes an image acquisition unit 110, a pattern decoding unit 1300, a disparity prediction unit 1301, a preprocessing unit 800, a search range setting unit 1302, a corresponding point search unit 801, a disparity map synthesis unit 1103, an edge detection unit (edge ​​detection unit) 112, a map correction unit (correction unit) 113, a disparity map postprocessing unit 802, and a depth map generation unit 803. The disparity map synthesis unit 1103 has the function of generating a synthesized disparity map by synthesizing multiple disparity maps generated from each of multiple image pairs.

[0136] (Map generation)

[0137] Reference Figure 16 and Figure 17 The process of map generation in the fifth embodiment will be explained. Figure 16 To show Figure 2 The flowchart details steps S202, S205, and S206. Figure 17 This is a timing diagram.

[0138] The first measurement is triggered by a start signal from the drive control unit 106. First, the pattern projection unit 103 is illuminated to project a predetermined pattern onto the object 12. Then, images are simultaneously captured by the first camera 101 and the second camera 102, and the first image and the second image are transmitted from the image transmission unit 105. In steps S1600 and S1601, the image acquisition unit 110 acquires a first image pair consisting of the first image and the second image. The image acquisition unit 110 sends the first image to the pattern decoding unit 1300 and sends the first image and the second image to the preprocessing unit 800.

[0139] Next, triggered by a start signal from the drive control unit 106, a second measurement is performed. First, the pattern projection unit 103 is illuminated, and a predetermined pattern illumination is projected onto the object 12. Then, images are simultaneously captured by the first camera 101 and the second camera 102, and the first image and the second image are transmitted from the image transmission unit 105. In steps S1610 and S1611, the image acquisition unit 110 acquires a second image pair consisting of the first image and the second image. The image acquisition unit 110 sends the second image to the pattern decoding unit 1300 and sends the first image and the second image to the preprocessing unit 800.

[0140] Image processing of the first image pair obtained through the first measurement begins concurrently with the second measurement. In step S1602, the preprocessing unit 800 performs parallelization processing (correction) on the first and second images. In step S1603, the preprocessing unit 800 calculates hash feature values ​​for each pixel of the parallelized first and second images and replaces the value of each pixel with a hash feature value.

[0141] In step S1604, the pattern decoding unit 1300 obtains distance information in the depth direction at multiple points on the first image by parsing the first image and decoding the pattern.

[0142] In step S1605, the disparity prediction unit 1301 calculates the predicted disparity based on the distance information of each point obtained in step S1604, and outputs the data as a reference disparity map. It should be noted that processing of the first image can also begin from the point in time when the transmission of the first image is completed.

[0143] If the transmission of the second image pair obtained through the second measurement is completed, image processing of the second image pair begins. In step S1612, the preprocessing unit 800 performs parallelization processing (correction) on the first and second images. In step S1613, the preprocessing unit 800 calculates hash feature values ​​for each pixel of the parallelized first and second images and replaces the values ​​of each pixel with hash feature values. In step S1614, the pattern decoding unit 1300 obtains distance information based on the second image by parsing the second image and decoding the pattern. Then, in step S1615, the disparity prediction unit 1301 calculates predicted disparity based on the distance information.

[0144] Theoretically, the predicted disparity obtained in the first step (step S1605) should be the same as the predicted disparity obtained in the second step (step S1615), but in reality, they are not exactly the same. Since images taken with different cameras (viewpoints) are used in the first and second steps, there are differences in the appearance of the images (i.e., image information). Therefore, there may be a difference in the predicted disparity values ​​between the first and second steps, or one disparity prediction may succeed while the other fails. Therefore, in step S1620, the disparity prediction unit 1301 synthesizes the predicted disparity from the first step and the predicted disparity from the second step, and calculates the synthesized predicted disparity. The synthesis method is not particularly limited; for example, if predicted disparities are obtained in both the first and second steps, their average value is set as the synthesized predicted disparity; if predicted disparities are obtained only in either the first or second step, that value can be directly set as the synthesized predicted disparity.

[0145] In step S1621, the search range setting unit 1302 sets the search range of corresponding points based on each of the first image pair and the second image pair in the synthetic prediction disparity pair.

[0146] In step S1622, the corresponding point search unit 801 searches for corresponding points among the first image pairs, calculates the disparity of each pixel, and generates a disparity map. Figure 1 Similarly, in step S1623, the corresponding point search unit 801 searches for corresponding points between the second image pairs to generate a parallax distance. Figure 2 .

[0147] In step S1624, the disparity map synthesis unit 1103 synthesizes the disparity map obtained from the first image pair. Figure 1 and the parallax obtained from the second image pair Figure 2 This generates a composite parallax map. The composite method is not particularly limited; for example, in parallax mapping... Figure 1 and parallax Figure 2 When parallax is obtained in both cases, their average value is set as the composite parallax; however, when parallax is obtained only in the case of parallax... Figure 1 and parallax Figure 2 If the disparity is obtained from either of the values, then the value is directly set as the composite disparity.

[0148] Next, the start signal from the drive control unit 106 is used as a trigger to capture a non-patterned image. First, the illumination unit 104 is turned on to project uniform illumination onto the object 12. Then, images are captured by the first camera 101 or the second camera 102, and the non-patterned image is transmitted from the image transmission unit 105. Figure 17 (The sequence of capturing a non-patterned image using the first camera is shown). In step S1630, the image acquisition unit 110 acquires the non-patterned image. The image acquisition unit 110 sends the non-patterned image to the edge detection unit 112.

[0149] In step S1631, the edge detection unit 112 performs parallelization processing (correction) on the non-patterned image. It should be noted that if the parallelism of the image is sufficiently high, the parallelization processing can be omitted. In step S1632, the edge detection unit 112 detects the edges of the object 12 from the non-patterned image.

[0150] Then, in step S1633, the map correction unit 113 corrects the synthetic parallax map generated in step S1624 based on the edges detected in step S1632. Subsequent processing is the same as in the aforementioned embodiment.

[0151] Based on the configuration of this embodiment described above, in addition to the same effects as the fourth embodiment, it also has the following advantages: First, because a disparity map obtained from multiple image pairs is synthesized, measurement bias can be reduced. Second, by predicting disparity from both the first and second images, high-precision predicted disparity can be obtained. Even if information in one image is missing, the probability of predicting disparity using the other image is high. Therefore, the search range for corresponding points can be set more appropriately, further improving accuracy and robustness.

[0152] <Variation Example>

[0153] The above embodiments are merely illustrative examples illustrating the configuration of the present invention. The present invention is not limited to the specific embodiments described above, and various modifications can be made within the scope of its technical concept. For example, in the above embodiments, although the object 12 is illuminated with non-patterned light when capturing the image for edge detection, in sufficiently bright environments, the image for edge detection can also be captured without such dedicated illumination (i.e., without illuminating the non-patterned light). That is, the image for edge detection only needs to be captured in a state where no pattern for distance measurement is projected; whether or not non-patterned light is illuminating the image for edge detection is not important.

[0154] Furthermore, in the above embodiments, three methods were exemplified as examples of active measurement: spatially encoded pattern method (first embodiment), stereo matching (second and third embodiments), and a hybrid of spatially encoded pattern method and stereo matching (fourth and fifth embodiments). However, the present invention can also be applied to active measurements other than these. For example, as in the first embodiment, in a measurement system consisting of a light-projecting device and a camera, random dot patterns, grayscale patterns, time-encoded patterns, etc., can also be used as pattern light.

[0155] In the above embodiments, hash features are used in stereo matching, but other methods can also be used to evaluate the similarity of corresponding points. For example, as similarity evaluation metrics, there are pixel similarity calculation methods for left and right images based on SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), NC (Normalized Correlation), etc. Furthermore, although images from a common camera are used in the above embodiments for disparity prediction and stereo matching, images from different cameras used for 3D measurement can also be used.

[0156] <Postscript>

[0157] An image processing apparatus (11) is characterized by having:

[0158] The map generation unit (111, 801) generates a map by projecting patterned light onto the object (12) and taking a picture of the image. The map is data obtained by associating depth distance related information with each pixel.

[0159] The edge detection unit (112) detects the edges of the object using an image captured in a manner that does not project patterned light onto the object (12); and

[0160] The correction unit (113) corrects the map based on the detected edge so that the position where the depth distance becomes discontinuous matches the position of the edge of the object (12).

[0161] Explanation of reference numerals in the attached figures

[0162] 1: Three-dimensional measurement system

[0163] 10: Sensor Unit

[0164] 11: Image processing device

[0165] 12: Object.

Claims

1. An image processing apparatus, characterized in that, have: The map generation unit reconstructs the three-dimensional information of the image obtained by projecting patterned light onto the object and taking a picture, and generates a map. The map is data of three-dimensional information obtained by associating depth distance information with each pixel. The edge detection unit detects the edges of the object using an image obtained by projecting uniform light onto the object and capturing it. as well as The correction unit corrects the map based on the detected edges so that the positions where the depth distance becomes discontinuous match the positions of the object's edges. The map generation unit has: The acquisition unit acquires an image pair consisting of a first image and a second image obtained by projecting patterned light onto the object and taking pictures from different perspectives; The disparity prediction unit predicts the disparity between the first image and the second image in a different way than stereo matching. The setting unit sets the search range of the corresponding point in stereo matching based on the predicted disparity; as well as The stereo matching unit uses the first image and the second image to perform stereo matching within the set search range. The disparity prediction unit predicts disparity based on distance information obtained through spatial coding patterns.

2. The image processing apparatus according to claim 1, characterized in that, The correction unit extracts the region of the object from the map, divides the region into multiple segmented regions based on the detected edges, and corrects the map by removing segmented regions smaller than a predetermined threshold from the multiple segmented regions.

3. The image processing apparatus according to claim 1, characterized in that, The map generation unit is a unit that generates the map by using multiple images obtained by projecting patterned light onto the object and taking pictures from different perspectives, and then performing stereo matching.

4. The image processing apparatus according to claim 1, characterized in that, The map generation unit is a unit that generates the map by using an image obtained by projecting patterned light onto the object and taking a picture, and then generating the map using a spatially encoded pattern.

5. The image processing apparatus according to claim 1, characterized in that, The map generation unit also includes a disparity map synthesis unit, which generates a synthesized disparity map by synthesizing multiple disparity maps generated by the stereo matching unit from each of multiple image pairs.

6. The image processing apparatus according to claim 5, characterized in that, The correction unit corrects the synthetic parallax map or the depth map converted from the synthetic parallax map.

7. The image processing apparatus according to claim 5, characterized in that, The disparity prediction unit generates multiple predicted disparities. The setting unit uses the synthesized predicted disparity obtained by synthesizing the multiple predicted disparities to set the search range of the multiple image pairs.

8. A three-dimensional measurement system, characterized in that, have: The sensor unit has at least a first projection part for projecting patterned light and one or more cameras; as well as The image processing apparatus according to any one of claims 1 to 7 processes an image acquired from the sensor unit.

9. A program product for enabling a computer to function as a unit of the image processing apparatus according to any one of claims 1 to 7.

10. An image processing method, characterized in that, It includes the following steps: The three-dimensional information is restored by projecting patterned light onto the object and taking pictures, and a map is generated. The map is three-dimensional information data obtained by associating depth distance information with each pixel. The edges of the object are detected using an image obtained by projecting uniform light onto the object and taking a picture; as well as The map is corrected based on the detected edges so that the discontinuous depth distances match the positions of the object's edges. The image processing method has the following characteristics: The acquisition step involves acquiring an image pair consisting of a first image and a second image obtained by projecting patterned light onto the object and taking pictures from different perspectives; The disparity prediction step predicts the disparity between the first image and the second image in a different way than stereo matching. The steps are set up to determine the search range of the corresponding point in stereo matching based on the predicted disparity; as well as The stereo matching step involves using the first image and the second image to perform stereo matching within the predefined search range. In the disparity prediction step, disparity is predicted based on distance information obtained through spatial coding patterns.

Images