Processing apparatus, processing method, and program
By combining the generation and correction components with a convolutional neural network, the problem of sparse depth maps caused by different viewpoints of 3D distance measurement devices and cameras was solved, achieving high-precision and high-density 3D distance information acquisition.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CANON KK
- Filing Date
- 2024-08-19
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies result in different occlusion relationships when the 3D distance measurement equipment and camera viewpoint are at different positions, leading to sparse depth maps that cannot accurately reproduce the 3D distance information of the subject's boundary.
Distance information from a first viewpoint is generated by generating components, and depth images are corrected based on distance values of neighboring regions using correction components. A high-density depth map is then generated by combining convolutional neural networks.
It achieves high-precision and high-density acquisition of three-dimensional distance information, accurately reproducing the depth map of the subject boundary.
Smart Images
Figure CN122162074A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to processing equipment, processing methods, and procedures. Background Technology
[0002] Recently, there has been a demand for high-precision and high-density 3D distance information for various applications. 3D distance measurement devices, such as LiDAR sensors, are known as components for acquiring high-precision 3D distance information.
[0003] On the other hand, high-density 3D distance information requires long acquisition times, making it difficult to apply to moving subjects, and it also involves a large amount of information, thus increasing the processing load. Non-Patent Document 1 discloses a method for converting 3D distance information into a depth map format. A depth map is image data in which each pixel stores a value corresponding to its distance to the subject (such as a value proportional to the distance). Typically, since 3D distance measurement devices acquire data at a constant sampling interval, pixels without pixel values may appear when the data is projected onto a depth map. Inputting such a sparse depth map along with an image captured by a camera into a convolutional neural network (CNN) can provide a dense depth map storing pixel values for all pixels. Existing technical documents Non-patent literature
[0004] Non-patent literature 1: Xinjing Cheng, Peng Wang and Ruigang Yang, “DepthEstimation via Affinity Learned with Convolutional Spatial PropagationNetwork” Summary of the Invention The problem the invention aims to solve
[0005] When the viewpoints of the 3D distance measuring device and the camera are different, the occlusion relationships between subjects may differ when viewed from either the 3D distance measuring device or the camera. That is, a distant subject that is not occluded when viewed from the 3D distance measuring device may be occluded by a neighboring subject when viewed from the camera. In this case, when generating a depth map viewed from the camera, because the depth map is sparse, values corresponding to the distances of distant subjects can be stored in the depth map. Therefore, when generating a dense depth map using the method disclosed in Non-Patent Document 1, the distance values near the subject boundaries are inaccurate, and accurate 3D distance information cannot be reproduced.
[0006] The purpose of this invention is to provide a processing device capable of acquiring high-precision and high-density three-dimensional distance information. Solution to this problem
[0007] A processing apparatus according to one aspect of the invention includes: a generation unit configured to generate a distance image viewed from a second viewpoint by using distance information to a subject viewed from a first viewpoint; and a correction unit configured to determine whether to correct the distance value of the pixels included in the neighboring region based on the distance value of a pixel in the distance image that is in a neighboring region. Invention Effects
[0008] This invention provides a processing device capable of acquiring high-precision and high-density three-dimensional distance information. Attached Figure Description
[0009] Figure 1 This is a block diagram of a three-dimensional distance information processing device according to the first embodiment. Figure 2 This is a flowchart illustrating the processing of three-dimensional distance information. Figure 3 This is a conceptual diagram illustrating the acquisition of three-dimensional distance information and images. Figure 4 This is a flowchart illustrating the depth correction process according to the first embodiment. Figure 5 This is an explanatory diagram illustrating the effect of the configuration according to the first embodiment. Figure 6 This is a conceptual diagram illustrating the operation of the depth corrector according to the second embodiment. Figure 7 This is a flowchart illustrating the depth correction process according to the second embodiment. Detailed Implementation
[0010] Embodiments according to the present invention will now be described in detail with reference to the accompanying drawings. Corresponding elements in the various drawings will be designated by the same reference numerals, and repeated descriptions thereof will be omitted. First Embodiment
[0011] Figure 1 This is a block diagram of a three-dimensional distance information processing device (processing device) 100 according to this embodiment. The three-dimensional distance information processing device 100 includes a three-dimensional distance measurement unit 101, a camera unit 102, a system memory 109, a non-volatile memory 110, and a control unit 120.
[0012] The three-dimensional distance measurement unit 101 includes, for example, a LiDAR sensor. The LiDAR sensor includes: a light output unit that outputs laser light and illuminates the surface of an object with light (irradiation light); and a receiver that receives reflected light from the surface of the object. Distance data to the structural surface in the irradiation direction is acquired using the time until the reflected light returns or the phase difference between the irradiation light and the reflected light, and three-dimensional distance information (distance information to the subject viewed from a first viewpoint) is acquired by combining this distance data with information in the irradiation direction. The three-dimensional distance measurement unit 101 is not limited to a LiDAR sensor and can be a device that uses electromagnetic waves or sound waves other than laser light for distance measurement. It is assumed that the three-dimensional distance information includes at least one of point clouds, voxels, polygons, meshes, implicit surface representations, depth maps, disparity maps, and depth images.
[0013] The imaging unit 102 includes, for example, a camera. The camera includes a lens unit, an image sensor that converts an optical image into an electrical signal, and an A / D converter that converts an analog signal into a digital signal. An optical image is input to the image sensor via the lens unit, and the electrical signal converted by the image sensor is converted into a digital signal, thereby acquiring an image (taking a picture).
[0014] System memory 109 is a rewritable volatile memory such as DRAM, and contains constants, variables, and data read from non-volatile memory 110 during the operation of control unit 120.
[0015] The non-volatile memory 110 includes a 3D distance information memory 103, a system memory 104, and an image memory 105. The non-volatile memory 110 is an electrically erasable and writable memory such as an EEPROM. The 3D distance information memory 103 stores 3D distance information acquired from the 3D distance measurement unit 101 in a predefined format (e.g., point cloud format). The system memory 104 stores operating programs and constants for operation, including various blocks within the control unit 120. The programs mentioned herein are programs for executing the various flowcharts described later in this embodiment. The image memory 105 stores images acquired from the imaging unit 102.
[0016] The control unit 120 includes at least one processor (one or more processors) and controls the entire three-dimensional distance information processing device 100. The control unit 120 includes a projection unit (generation component) 106, a depth corrector (correction component, setting unit) 107, and a depth interpolation unit (interpolation unit) 108. The control unit 120 implements the various processes of this embodiment, described later, by executing programs stored in the system memory 104. The various types of control performed by the control unit 120 can be performed by a single piece of hardware, or can be distributed and performed by multiple pieces of hardware, such as multiple processors or circuits.
[0017] The projection unit 106 generates a depth map (distance image viewed from a second viewpoint) based on the three-dimensional distance information recorded in the three-dimensional distance information memory 103 and the relative position information between the three-dimensional distance measurement unit 101 and the imaging unit 102 recorded in the system memory 104. Here, the depth map is image data that stores the value corresponding to the distance to the subject (such as a value proportional to the distance) as pixel values for each pixel. In this embodiment, the pixel value is the value obtained by projecting the position vector of the subject seen from the imaging unit 102 onto the optical axis direction, but the length of the position vector can be used instead. Although this embodiment discusses the case where the distance image is a depth map, a parallax map or a depth image can also be used.
[0018] In this embodiment, pixel value 0 is assigned to the pixel corresponding to a distance of 0 [mm] to the subject, and pixel value 255 is assigned to the pixel corresponding to a distance to the subject equal to or greater than a threshold (such as 100000 [mm] = 100 [m]). For pixels without a corresponding distance to the subject, pixel value 0 is stored. In the following description, in order to distinguish them from pixel values in the image acquired by camera unit 102, pixel values on the depth map will be referred to as distance values. These values may not correspond to physical distances.
[0019] Because LiDAR sensors scan the laser based on a constant angular resolution, the resulting depth map typically includes a number of pixels missing distance values. While increasing the number of scans can result in finer sampling, acquiring 3D distance information takes longer, and this method is unsuitable for situations involving moving subjects. The amount of 3D distance information becomes enormous. In the following description, a depth map including pixels with missing distance values will be referred to as a sparse depth map. The number of pixels in the depth map may differ from the number of pixels in the image acquired by camera unit 102. The sparse depth map is stored in system memory 104 and system memory 109.
[0020] The depth corrector 107 determines whether to correct the distance value based on the distance values of the pixels included in the neighborhood region surrounding the reference pixel of the sparse depth map output from the projection unit 106. For example, the depth corrector 107 can determine whether to correct the distance value based on the difference between the distance values of each pixel included in the neighborhood region and the distance value of the reference pixel (reference distance value) or based on the magnitude of the distance values of the pixels included in the neighborhood region.
[0021] The depth interpolation unit 108 generates a depth map (hereinafter referred to as a dense depth map) based on the sparse depth map corrected by the depth corrector 107 and the image acquired by the camera unit 102, storing distance values for all pixels. For example, a convolutional neural network (CNN) can be used to generate the dense depth map from a combination of the image and the corresponding sparse depth map. Interpolation can also be performed using at least one of filters, machine learning, super-resolution, and conditional random fields.
[0022] Figure 2 This is a flowchart illustrating the processing of three-dimensional distance information. In step S201, the image memory 105 acquires and records an image from the camera unit 102. In step S202, the three-dimensional distance information memory 103 acquires and stores three-dimensional distance information from the three-dimensional distance measurement unit 101. In step S203, the projection unit 106 generates a depth map viewed from the camera unit 102 based on the three-dimensional distance information recorded in the three-dimensional distance information memory 103 and the relative position information between the three-dimensional distance measurement unit 101 and the camera unit 102 recorded in the system memory 104. In this embodiment, it is assumed that the relative position information between the three-dimensional distance measurement unit 101 and the camera unit 102 is known, but this disclosure is not limited to this embodiment. For example, the relative position information can be calculated using known calibration methods by matching feature points in the three-dimensional point cloud and the image. In step S204, the depth corrector 107 performs depth correction processing to correct the distance values of the depth map. In step S205, the depth interpolation unit 108 generates a dense depth map based on the corrected depth map and the image.
[0023] Next, we will describe Figure 2 The processing in step S204 (depth correction processing). Figure 3 This is a conceptual diagram illustrating the use of a three-dimensional distance measurement unit 101 and a camera unit 102 to acquire three-dimensional distance information and images. Figure 3 In (a), region 303 is the area on wall 301 that is obscured by subject 302 when viewed from camera unit 102. Figure 3In (b), the region 320 of the depth map 310 generated by the projection unit 106 and viewed from the camera unit 102 corresponds to the subject 302. For ease of explanation, distance values outside region 320 are exemplified as 0, but other values are actually stored. Within region 320, pixel 311 stores a distance value corresponding to the distance to wall 301, and pixel 312 stores a distance value corresponding to the distance to subject 302.
[0024] Because the 3D distance measurement unit 101 and the camera unit 102 are located at different positions in space, the parallax differs between the wall 301 and the subject 302. That is, when viewed from the camera unit 102, region 303 would not normally appear on the depth map because it is occluded by the subject 302. However, since the data acquired by the 3D distance measurement unit 101 is sparse, distance values originating from the wall 301 and distance values originating from the subject 302 are mixed in region 320. Therefore, when interpolating, the distance values in region 320 are averaged, and distance values that differ from the actual distance values are stored. Figure 3 In (a) and (b), the parallax between the three-dimensional distance measurement unit 101 and the camera unit 102 is emphasized. However, the actual parallax is not as large as illustrated. Therefore, especially for subjects closer to the camera unit 102, the distance values become inaccurate near the boundary.
[0025] To solve the above problems, this embodiment is based on Figure 4 The process is used to correct the distance values originating from wall 301 within region 320. Figure 4 This is a flowchart illustrating the depth correction process according to this embodiment.
[0026] In step S401, the depth corrector 107 obtains the distance value (reference distance value) D of the attention pixel (focus pixel, reference pixel) in the depth map.
[0027] In step S402, the depth corrector 107 determines whether the distance value D is greater than 0. If the distance value D is greater than 0, the process proceeds to step S404. If the distance value D is equal to 0, the process proceeds to step S403.
[0028] In step S403, the depth corrector 107 updates the attention pixels.
[0029] In step S404, the depth corrector 107 determines whether the distance value d of the neighboring pixels included in the neighborhood region of the attention pixel meets a predetermined condition. In this embodiment, the depth corrector 107 determines whether the difference dD between the distance value d of the neighboring pixel and the distance value D is greater than a threshold (first predetermined value) Th1. Here, the neighborhood region is set by the depth corrector 107 and is a square region of (2b+1)×(2b+1) pixels centered on the attention pixel, and may include the attention pixel. For example, b is 1 [pixel]. The shape of the neighborhood region is not limited to a square, and may be other shapes such as rectangles or circles. The size of the neighborhood region can be changed according to the distance value D. For example, when the three-dimensional distance measurement unit 101 and the camera unit 102 are approximately on the same plane, the parallax between them is proportional to the value 1 / D, so the size of the neighborhood region can be set to be proportional to the value 1 / D. This takes advantage of the characteristic that almost no parallax occurs between distant subjects. If it is determined that the difference dD is greater than the threshold Th1, the process proceeds to step S405. If the difference dD is determined to be less than the threshold Th1, the process proceeds to step S406. If the difference dD is equal to the threshold Th1, the step to be executed can be arbitrarily set. In this embodiment, if the distance value d is greater than the distance value D by a threshold, the distance value d of neighboring pixels is corrected. Alternatively, the distance value can be corrected if the distance value d is greater than another threshold (a second predetermined value). This corresponds to correcting the distance value corresponding to the distance to a subject existing at a distance greater than a specific distance.
[0030] In step S405, the depth corrector 107 corrects the distance value d of neighboring pixels. In this embodiment, the distance value d is corrected to 0. This corresponds to deleting the distance value corresponding to the distance to a distant subject present in the neighboring area. The reason why the distance value d can be set to 0 is that appropriate distance values are filled from neighboring pixels through the processing (interpolation processing) in step S205. The distance value d can instead be corrected to a value close to the distance value D.
[0031] In step S406, the depth corrector 107 determines whether the above processing has been performed on all pixels that serve as attention pixels. If it is determined that the processing has been performed on all pixels, the process ends. Otherwise, the process returns to step S403.
[0032] As described above, this embodiment corrects the sparse depth map viewed from the camera unit 102, thereby obtaining a dense depth map that reproduces the boundary of the subject with high accuracy through interpolation processing.
[0033] When the distance value D of the attention pixel is equal to or greater than a predetermined value (the third predetermined value), parallax is suppressed, so depth correction processing can be configured not to be performed.
[0034] Although this embodiment has described an example of correcting distance values in neighboring regions, the invention is not limited to this embodiment. For example, if the difference D-d_min between the distance value D of the reference pixel and the minimum distance value d_min (or median or average distance value) in the neighboring regions is greater than a first predetermined threshold, the distance value D of the reference pixel can be corrected to the minimum value d_min.
[0035] Figure 5 This is an explanatory diagram illustrating the effect of the configuration in this embodiment. Image 501 is a captured image obtained by capturing the subject 510 using the imaging unit 102. By performing depth correction processing, compared to the depth map 502 obtained without performing depth correction processing, defects in the subject are reduced, and a high-precision and high-density depth map 503 can be obtained.
[0036] As described above, the configuration of this embodiment can acquire high-precision and high-density three-dimensional distance information. Second Embodiment
[0037] In this embodiment, the depth corrector 107 also uses information related to the image acquired by the camera unit 102 to determine whether to correct the depth map. Hereinafter, only configurations different from those in the first embodiment will be described, and descriptions of configurations identical to those in the first embodiment will be omitted.
[0038] Figure 6 This is a conceptual diagram illustrating the operation of the depth corrector 107 according to this embodiment, and shows the state where region 320 is superimposed on image 501 and depth map 503. The depth corrector 107 corrects the distance values within region 320 to the distance values of the subject 510. As a result of interpolation processing, such as... Figure 6 As illustrated, the region with the distance value of subject 510 can extend outward beyond the original subject boundary. This problem becomes more pronounced, especially when the subject boundary is complex. Therefore, in this embodiment, in image 501, correction is toggled by comparing the pixel values of the pixel of interest with those of neighboring pixels included in the neighboring region. In the following description, it is assumed that image 501 and depth map 503 have the same number of pixels, but this number can be different. In this case, the pixel values on image 501 corresponding to the pixels on depth map 503 are obtained.
[0039] Figure 7 This is a flowchart illustrating the depth correction process according to this embodiment.
[0040] The processing of steps S701 to S703 and steps S705 to S707 are respectively related to Figure 4The processes of steps S401 to S406 are the same, so their descriptions will be omitted.
[0041] In step S704, the depth corrector 107 determines whether the absolute value |Ii| of the difference between the pixel value I of the attention pixel in the image 501 acquired by the camera unit 102 and the pixel value i of the neighboring pixels included in the neighboring region is greater than a threshold (fourth predetermined value) Th2. If the absolute value |Ii| is determined to be greater than the threshold Th2, the depth corrector 107 performs the processing of step S705. If the absolute value |Ii| is determined to be less than the threshold Th2, the processing of step S703 is performed. The step performed can be arbitrarily set to be either the absolute difference |Ii| or the threshold Th2. Furthermore, when comparing pixel values, values other than the absolute value of the difference can be used. The purpose of this step is to identify the subject boundary in the image 501, therefore the magnitude of the difference is important, and the sign can be positive or negative. On the other hand, in the processing of step S705, the sign of the difference of distance values is important in order to identify subjects that are relatively far away.
[0042] This embodiment uses brightness as the pixel value, but RGB information can also be used. In this case, for example, the average value of the results from each RGB channel can be used. This disclosure is not limited to the above method; any method that allows comparison of differences between pixel values is acceptable.
[0043] This embodiment has described an example of determining whether to correct the depth map 503 based on the pixel values of image 501, but the shape of the neighboring regions can be changed. In this case, in step S704, pixels whose absolute difference is greater than the threshold Th2 can be excluded from the neighboring regions.
[0044] As described above, in addition to the effects of the first embodiment, the configuration of this embodiment can also reproduce the subject boundary with high precision. [Other Embodiments] The present invention can also be implemented by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and causing one or more processors of the computer of the system or device to read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.
[0045] While preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and variations can be made within the spirit and scope of the present invention.
Claims
1. A processing apparatus, comprising: The generating component is configured to generate a distance image when viewed from a second viewpoint by using distance information to the subject when viewed from a first viewpoint; as well as A correction component is configured to determine whether to correct the distance value of the pixels included in the neighboring region based on the distance value of the pixels included in the neighboring region of the reference pixel in the distance image.
2. The processing apparatus according to claim 1, characterized in that, If the difference between the distance value and the reference distance value of the reference pixel is greater than a first predetermined value, the correction component corrects the distance value; if the difference is less than the first predetermined value, the correction component does not correct the distance value.
3. The processing apparatus according to claim 1, characterized in that, If the distance value is greater than the second predetermined value, the correction component corrects the distance value; if the distance value is less than the second predetermined value, the correction component does not correct the distance value.
4. The processing apparatus according to any one of claims 1 to 3, characterized in that, The distance information is obtained using at least one of laser, electromagnetic waves, and sound waves.
5. The processing apparatus according to any one of claims 1 to 4, characterized in that, The distance information includes at least one of point cloud, voxel, polygon, mesh, implicit function representation, depth map, disparity map, and distance map. The distance image includes any one of a depth map, a disparity map, and a distance map.
6. The processing apparatus according to any one of claims 1 to 5, characterized in that, The neighboring region includes a plurality of pixels containing the reference pixel. The correction component corrects the distance value of at least one of the plurality of pixels.
7. The processing apparatus according to any one of claims 1 to 6, characterized in that, If the reference distance value of the reference pixel is equal to or greater than a third predetermined value, the correction component does not perform the processing for correcting the distance value.
8. The processing apparatus according to any one of claims 1 to 7, further comprising a setting unit configured to set the adjacent area.
9. The processing apparatus according to claim 8, characterized in that, The setting unit sets the size of the neighboring region based on the reference distance value of the reference pixel.
10. The processing apparatus according to any one of claims 1 to 9, characterized in that, The second viewpoint is the viewpoint of the camera component configured to acquire captured images.
11. The processing apparatus according to claim 10, characterized in that, The correction component determines whether to perform processing to correct the distance value based on the pixel values of the captured image.
12. The processing apparatus according to claim 11, characterized in that, The correction component determines whether to perform the processing based on the difference between the pixel value of the pixel corresponding to the reference pixel in the captured image and the pixel value of the pixel corresponding to the pixel included in the neighboring region.
13. The processing apparatus according to claim 12, characterized in that, If the absolute value of the difference is greater than a fourth predetermined value, the correction component does not perform the processing; and if the absolute value of the difference is less than the fourth predetermined value, the correction component performs the processing.
14. The processing apparatus according to any one of claims 10 to 13, further comprising a setting unit configured to set the adjacent area, Its features are, The setting unit sets the neighboring area based on the pixel values of the captured image.
15. The processing apparatus according to any one of claims 10 to 14, further comprising an interpolation unit configured to interpolate the distance values of pixels with missing distance values by using a distance image corrected by the correction member and the captured image.
16. The processing apparatus according to claim 15, characterized in that, The interpolation unit uses at least one of filters, machine learning, super-resolution, and conditional random fields for interpolation.
17. A processing method, comprising: Distance information to the subject viewed from a first viewpoint is converted into a distance image viewed from a second viewpoint; as well as The distance value is corrected based on the distance values of pixels included in the neighboring region of the reference pixel in the distance image.
18. A program for causing a computer to perform the processing method according to claim 17.