Distance information processing device, imaging device, distance information processing method, and program

Abstract

Provided is a distance information processing device characterized by comprising: a detection means for detecting a defect region in which the density of a distance value is less than a threshold value, such detection performed in a distance map of a second viewpoint generated on the basis of distance information acquired by a distance measurement means for measuring the distance of one or more subjects at a first viewpoint; and an interpolation means for interpolating the distance value of the defect region by different methods depending on whether or not the cause of the defect region is the parallax between the first viewpoint and the second viewpoint.

Description

Distance information processing device, imaging device, distance information processing method, and program

[0001] The present invention relates to a distance information processing device, an imaging device, a distance information processing method, and a program.

[0002] In recent years, high-precision three-dimensional information (distance information) has become increasingly important for map generation in autonomous driving, object recognition for robots, video production, game production, and the like. To acquire high-precision three-dimensional information, three-dimensional ranging devices such as LiDAR sensors are commonly used. LiDAR sensors acquire distance information to a subject by emitting laser light and measuring the time it takes for the reflected light to return from the subject. Therefore, distance information may not be acquired for subjects with low reflectivity or subjects that are very far away.

[0003] In addition, in order to increase the density or render depth information, there are increasing opportunities to capture images with a separate camera and process the images and depth information together. In such cases, if the viewpoints of the camera and the LiDAR sensor are different, the occlusion relationship of the subject will differ, resulting in a loss of depth information due to occlusion.

[0004] Patent Literature 1 discloses a technique for dealing with missing distance information. In this technique, in a depth map in which a value corresponding to a distance value is stored in each pixel, the value of a missing pixel is interpolated by weighting the value corresponding to the surrounding distance value based on the pixel value of an image captured by a camera and the inter-pixel distance between the missing pixel and the surrounding pixels.

[0005] Japanese Patent Application Laid-Open No. 2000-230809

[0006] Although Patent Document 1 discloses viewpoint conversion when the viewpoint from which distance information is acquired differs from the viewpoint from which an image is captured, it does not take into consideration loss of distance information caused by viewpoint conversion (parallax between viewpoints). Therefore, the technique of Patent Document 1 may result in a decrease in the accuracy of distance value interpolation depending on the cause of the loss.

[0007] The present invention has been made in view of the above circumstances, and provides a technique for interpolating the distance value of a missing area in a distance map using different methods depending on whether the missing area is caused by parallax or not.

[0008] In order to solve the above problem, the present invention provides a distance information processing device comprising: a detection means for detecting a missing area in a distance map of a second viewpoint generated based on distance information acquired by a ranging means for measuring the distance of one or more subjects at a first viewpoint, where the density of distance values ​​is less than a threshold; and an interpolation means for interpolating the distance values ​​of the missing area in different ways depending on whether the cause of the missing area is the parallax between the first viewpoint and the second viewpoint.

[0009] According to the present invention, it is possible to interpolate the distance value of a missing area in a distance map using different methods depending on whether the missing area is caused by parallax or not.

[0010] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiment of the present invention.

[0011] The accompanying drawings are included in the specification, constitute a part thereof, illustrate embodiments of the present invention, and are used to explain the principles of the present invention together with the description thereof. A block diagram showing an example configuration of a distance information processing device 100. A flowchart of distance information processing executed by a control unit 120. A conceptual diagram of a situation in which scene data is acquired using a distance measurement unit 101 and an image capture unit 102. A conceptual diagram of a depth map as seen from the image capture unit 102. A conceptual diagram of a depth map as seen from the distance measurement unit 101. A conceptual diagram of a reflectance map. A flowchart showing details of the processing in S205. A flowchart showing details of the processing in S206 (first interpolation method). A flowchart showing details of the processing in S207 (second interpolation method). A conceptual diagram of a method for setting a peripheral region of a pixel to be processed in a missing region. A conceptual diagram of a method for setting a peripheral region of a pixel to be processed in a missing region. A conceptual diagram of a depth map for explaining processing to exclude a portion of the peripheral region. A conceptual diagram of a depth map for explaining processing to exclude a portion of the peripheral region. A flowchart of processing to exclude a portion of the peripheral region.

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] 1 is a block diagram showing an example of the configuration of a distance information processing device 100. The distance information processing device 100 includes a distance measurement unit 101, an imaging unit 102, a nonvolatile memory 110, a system memory 111, and a control unit 120.

[0014] The distance measurement unit 101 is, for example, a LiDAR sensor that measures the three-dimensional shape of an object. The LiDAR sensor includes a light beam output unit that outputs laser light and irradiates the surface of the object with the light beam, and a receiving unit that receives the light beam reflected from the surface of the object. The distance measurement unit 101 calculates the distance to the object surface in the irradiation direction using the time it takes for the reflected light beam to return or the phase difference between the irradiated light beam and the reflected light beam. The distance measurement unit 101 acquires the three-dimensional position by combining the calculated distance with information about the irradiation direction. Information (distance information) stored as indicating the three-dimensional position can be information in a known format, such as a point cloud, voxel, polygon, mesh, implicit function representation, depth map, or parallax map. The distance measurement unit 101 also acquires information about the reflectance of the laser beam from each object.

[0015] The distance measuring unit 101 is not limited to the LiDAR sensor exemplified in this embodiment. For example, the distance measuring unit 101 may be a device that measures distance using electromagnetic waves, sound waves, or the like. Furthermore, when the distance measuring unit 101 uses light, this light is not limited to laser light, and any type of light can be used.

[0016] The imaging unit 102 is, for example, a camera that captures an image of an object (subject). The camera has a lens unit, an imaging element that converts an optical image into an electrical signal, and an A / D converter that converts an analog signal into a digital signal. The optical image is input to the imaging element via the lens unit, and the imaging element converts the optical image into an electrical signal, and then converts the electrical signal into a digital signal. In this way, an image is acquired.

[0017] The nonvolatile memory 110 has a distance information area 103, a system area 104, and an image area 105. The nonvolatile memory 110 is an electrically erasable and recordable memory, such as an EEPROM.

[0018] The distance information area 103 is an area for storing three-dimensional position information (distance information) and reflectance information acquired from the distance measurement unit 101 in a predetermined format. For example, the distance information is stored in a point cloud format, and the reflectance information is stored in a reflectance map format. The reflectance map is a map (information in an image format) in which a value corresponding to the reflectance is set as the pixel value for each pixel.

[0019] The system area 104 is an area for storing operation programs and operation constants for each block of the control unit 120. The operation programs here include programs for executing the processes of various flowcharts, which will be described later.

[0020] The image area 105 is an area for storing images acquired from the image capturing unit 102 .

[0021] The system memory 111 is a rewritable volatile memory such as a DRAM, and stores constants and variables used during operation of the control unit 120, as well as data read from the non-volatile memory 110, etc.

[0022] The control unit 120 is a control unit having at least one processor, and controls the entire distance information processing device 100. The control unit 120 also has a projection unit 106, a missing area determination unit 107, a missing area interpolation unit 108, and a full range interpolation unit 109. The control unit 120 executes the programs recorded in the system area 104 described above, thereby realizing various processes described below.

[0023] The projection unit 106 generates a depth map as seen from the imaging unit 102 based on the distance information recorded in the distance information area 103 and the relative positions of the ranging unit 101 and the imaging unit 102 recorded in the system area 104. The depth map is an example of a distance map, and is a map (image-format information) in which a value corresponding to the distance value to the subject (e.g., a value proportional to the distance value) is set as the pixel value for each pixel. Here, the value obtained by projecting the position vector of the subject as seen from the imaging unit 102 in the optical axis direction is used as the distance value to the subject, but the length of the position vector may also be used. Furthermore, a parallax map in which a value proportional to the reciprocal of the distance is stored as the distance value may also be used as the distance map. The following description will be given taking as an example a case in which the depth map is used as the distance map. Strictly speaking, lens distortion must be taken into consideration when projecting, but the following will assume an ideal situation in which there is no lens distortion.

[0024] In this embodiment, a pixel value of 0 is assigned to a distance value of 0 [mm], and a pixel value of 255 is assigned to a distance value equal to or greater than a certain threshold (for example, a distance value equal to or greater than 100,000 [mm]). A pixel value of 0 is stored for pixels that do not have a corresponding distance value. Hereinafter, pixel values ​​on the depth map are referred to as distance values ​​to distinguish them from pixel values ​​in an image captured by a camera. Distance values ​​on the depth map do not necessarily have to match physical distance values ​​(as in the example described above, a physical distance value of 100,000 [mm] may correspond to a distance value of 255 on the depth map).

[0025] The LiDAR sensor scans the laser light based on a certain angular resolution. Therefore, the obtained depth map stores values ​​corresponding to distance values ​​at specific intervals, and the remaining pixels are missing pixels storing a distance value of 0. Hereinafter, a depth map containing such missing pixels will be referred to as a sparse depth map. Note that the number of pixels in the sparse depth map may be different from that of the image acquired by the imaging unit 102. The sparse depth map is stored in the system area 104 and the system memory 111.

[0026] Missing distance values ​​also occur in occlusion areas caused by the parallax between the distance measuring unit 101 and the image capturing unit 102, and in areas of (part of) an object from which the distance measuring unit 101 cannot acquire information, such as an object with low reflectivity. In this case, a group of pixels with missing distance values ​​is distributed over a wider area than the interval at which the distance measuring unit 101 acquires distance values. Hereinafter, such a group of missing pixels will be referred to as a missing area. The missing area determination unit 107 identifies the location of the missing area and also identifies the cause of the missing area. A method for identifying (determining) the location and cause of the missing area will be described later.

[0027] The missing area interpolation unit 108 interpolates distance values ​​for the missing area based on the sparse depth map and reflectance map output by the projection unit 106 and the determination result by the missing area determination unit 107. Details of the processing by the missing area interpolation unit 108 will be described later.

[0028] The global interpolation unit 109 generates a depth map (hereinafter, referred to as a dense depth map) in which distance values ​​are stored in all pixels, based on the sparse depth map in which distance values ​​have been interpolated by the missing area interpolation unit 108 and the image acquired by the imaging unit 102. For example, the dense depth map can be generated by using a convolutional neural network (CNN) on a combination of the captured image and the corresponding sparse depth map. However, the method for generating the dense depth map is not limited to this, and other methods such as applying a filter to the distance image, super-resolution technology, and conditional random fields can also be used.

[0029] 2 is a flowchart of distance information processing executed by control unit 120. Unless otherwise specified, the processing of each step in this flowchart is realized by control unit 120 (or one or more blocks within control unit 120) executing a program stored in system area 104.

[0030] In S201, the control unit 120 acquires a captured image by capturing an image using the imaging unit 102. The acquired captured image is recorded in the image area 105.

[0031] In S202, the control unit 120 acquires distance information by measuring the distance using the distance measuring unit 101. The acquired distance information is recorded in the distance information area 103.

[0032] In S203, the projection unit 106 generates a sparse depth map as seen from the imaging unit 102, based on the distance information recorded in the distance information area 103 and the relative position information between the ranging unit 101 and the imaging unit 102 recorded in the system area 104. Hereinafter, unless otherwise specified, the sparse depth map will be simply referred to as a depth map.

[0033] Note that, although it is assumed here that relative position information between the distance measurement unit 101 and the image capture unit 102 is stored in advance in the system area 104 (i.e., the relative positions are known), this embodiment is not limited to this configuration. For example, the projection unit 106 may calculate the relative positions by matching feature points between the distance information (three-dimensional point cloud) and the captured image and using a known calibration method.

[0034] In S204, the missing area determination unit 107 detects (identifies) a missing area in the depth map generated in S203. The missing area determination unit 107 also determines (identifies) the cause of the detected missing area. The projection unit 106 is also used to determine the cause of the missing area. Causes of the missing area (causes of missing distance values) include the parallax between the viewpoint of the distance information used to generate the depth map (the viewpoint of the distance measurement unit 101) and the viewpoint of the depth map (the viewpoint of the imaging unit 102), and low reflectance of the subject. Details of the processing in S204 and the cause of the missing area will be described later.

[0035] In S205, the missing area interpolation unit 108 determines whether the missing area is caused by parallax. If the missing area is caused by parallax, the process proceeds to S206; if not, the process proceeds to S207.

[0036] In S206, the missing area interpolation unit 108 interpolates the distance value of the missing area using the first interpolation method. Details of the process in S206 will be described later.

[0037] In step S207, the missing area interpolation unit 108 interpolates the distance value of the missing area using the second interpolation method. Details of the process in step S207 will be described later.

[0038] If multiple missing areas are detected in S204, the processes of S205 to S207 are performed for each area. For example, consider a case where three missing areas are detected, two of which are caused by parallax, and one of which is caused by low reflectance. In this case, the first interpolation method is used for each of the two missing areas caused by parallax in S206, and the second interpolation method is used for the missing area caused by low reflectance in S207.

[0039] In S208, the full-range interpolation unit 109 generates a dense depth map based on the depth map (sparse depth map) in which the missing region has been interpolated and the captured image.

[0040] In the above description, it is assumed that the cause of the missing area is determined in S204. However, if there is only one candidate cause of the missing area, the determination of the cause of the missing area can be omitted. For example, if there is no parallax between the distance measurement unit 101 and the image capture unit 102, the missing area determination unit 107 does not need to determine the cause of the missing area. In this case, the missing area interpolation unit 108 can proceed to S207 without performing the processing of S205. As another example, when a scene consisting only of subjects with high reflectivity is captured, the missing area determination unit 107 does not need to determine the cause of the missing area. In this case, the missing area interpolation unit 108 can proceed to S206 without performing the processing of S205.

[0041] Alternatively, a configuration may be adopted in which the cause of the missing area is determined more precisely and the interpolation method is switched accordingly. For example, a configuration may be adopted in which the interpolation method is switched based on whether the object has low reflectivity or is a subject located very far away.

[0042] Furthermore, S208 may be omitted, and a configuration may be adopted in which processing such as object detection and segmentation is performed using a sparse depth map.

[0043] <Details of the Processing in S205 and Details of the Cause of the Missing Area> The details of the processing in S205 and details of the cause of the missing area will be described with reference to FIGS. 3A to 3D and 4.

[0044] 3A is a conceptual diagram of a situation in which scene data is acquired using the distance measurement unit 101 and the imaging unit 102. In Fig. 3A, 301 represents a wall, and 302 represents a subject placed in the scene. Area 303 represents an area of ​​the wall 301 that is blocked by the subject 302 when viewed from the distance measurement unit 101.

[0045] 3B is a conceptual diagram of the depth map as viewed from the image capturing unit 102, generated by the projection unit 106 in S203. A depth map 310 represents a portion of the depth map as viewed from the image capturing unit 102. For convenience of illustration, only the peripheral area of ​​the subject 302 is shown here. In the depth map 310, an area 320 is an area corresponding to the subject 302. A missing area 321 is a missing area that occurs to the right of the area 320 due to parallax.

[0046] 3C is a conceptual diagram of a depth map as seen from the distance measurement unit 101. The depth map 330 is obtained by converting the depth map 310 into a depth map as seen from the distance measurement unit 101 based on relative position information between the distance measurement unit 101 and the image capture unit 102. Here, the projection unit 106 converts the viewpoint position using the focal length and pixel pitch of the image capture unit 102. Therefore, the resolution of the depth map 310 and the depth map 330 as images is the same. Because the depth map 330 is distance information as seen from the viewpoint position of the distance measurement unit 101, no missing areas due to parallax occur.

[0047] 3D is a conceptual diagram of a reflectance map. The reflectance map 340 represents a reflectance map of a subject, acquired together with distance information by the distance measurement unit 101. In the reflectance map 340, brighter areas indicate higher reflectance. For simplicity of explanation, the depth map 330 and the reflectance map 340 have the same resolution, but they may have different resolutions.

[0048] 4 is a flowchart showing details of the process in S205. In S401, the defective area determination unit 107 detects (specifies) a defective area in the depth map generated in S203.

[0049] Here, the angular resolution when the LiDAR sensor of the distance measurement unit 101 scans the laser light is "φ", the focal length of the lens of the image capture unit 102 is "f", and the pixel pitch of the image capture element of the image capture unit 102 is "Δ". In the depth map 310, the interval at which distance values ​​exist when there are no defects is roughly given by D = fφ / Δ (this also applies to the depth map 330). Strictly speaking, a factor depending on the angle between the line connecting the center of the optical axis and the subject and the image capture element surface is multiplied, but this will be omitted here for simplicity of explanation. Also, it is assumed that the horizontal and vertical angular resolutions are the same. The density of distance values ​​when there are no defects is given by ρ = 1 / D ２ ρ is given by the following equation. Therefore, the defective area determination unit 107 references the depth map 310 and identifies a group of pixels whose density is lower than the density ρ as a defective area. Here, a predetermined constant c equal to or less than 1 is used to provide a margin for the determination criterion. That is, the defective area determination unit 107 identifies (detects) an area in the depth map 310 whose density is less than cρ as a defective area. The density of distance values ​​for a pixel is obtained by dividing the number of pixels having distance values ​​(the number of pixels having non-zero distance values) within a square aD × aD that includes the pixel by the area of ​​the square. Here, a is a value equal to or greater than 1, for example, 2. In the example of FIG. 3B, one defective area 321 is detected, but multiple defective areas may be detected depending on the configuration of the depth map. Also, there may be cases where no defective area is detected. In such cases, the processing of the flowchart of FIG. 4 ends, and the processing of S208 in FIG. 2 is subsequently performed.

[0050] In S402, the projection unit 106 generates a depth map 330 as seen from the ranging unit 101 by converting the viewpoint of the depth map 310 from the viewpoint of the imaging unit 102 to the viewpoint of the ranging unit 101 based on the relative position information between the ranging unit 101 and the imaging unit 102 recorded in the system area 104.

[0051] In S403, the missing region determination unit 107 sets, in the depth map 330, a determination target region that corresponds to one of the one or more missing regions detected in the depth map 310. The determination target region in the depth map 330 is set at the same position as the missing region in the depth map 310. Therefore, for the missing region 321, the determination target region is set at the same position in the depth map 330 that corresponds to the position of the missing region 321 in the depth map 310.

[0052] In S404, the missing area determination unit 107 compares the density ρ_i of distance values ​​in the determination target area of ​​the depth map 330 with the density ρ of distance values ​​in the case where there is no missing area (ρ×constant c). If ρ_i is equal to or greater than cρ (a threshold value or greater), the process proceeds to S405; otherwise, the process proceeds to S406.

[0053] In the above description, the threshold used in S401 and the threshold used in S404 are both the same (cρ), but the two thresholds may be different.

[0054] In S405, the defective area determination unit 107 determines that the cause of the defective area corresponding to the determination target area is the parallax between the viewpoint of the distance information used to generate the depth map (the viewpoint of the distance measurement unit 101) and the viewpoint of the depth map (the viewpoint of the imaging unit 102). The reason for this determination will be explained below.

[0055] When changing (converting) the viewpoint of the depth map, the amount of movement of the distance value position (coordinate) varies depending on the distance to the subject. If the distance measurement unit 101 and the image capture unit 102 are positioned horizontally apart by b [mm], a point on the subject at a distance d [mm] moves by bf / (Δd) [pix] on the image of the depth map due to parallax. Here, f is the focal length of the lens [mm], and Δ is the pixel pitch of the image capture element [mm]. As can be seen from this equation, the focal length of the lens and the number of pixels moved are proportional to each other. By converting the depth map 310 to the depth map 330 and returning the viewpoint of the depth map to that of the distance measurement unit 101, the missing area caused by parallax disappears. Therefore, if the density ρ_i in the determination target area of ​​the depth map 330 is equal to or greater than cρ (if the density ρ_i is similar to ρ), the missing area corresponding to the determination target area is considered to be caused by parallax. Conversely, if the density ρ_i in the determination target region is less than cρ, it is considered that low reflectance of the subject is the cause of a missing distance value in the depth map 330 at the viewpoint of the distance measurement unit 101. Therefore, it is considered that the missing region is not caused by parallax but by low reflectance of the subject (low reflectance of the subject).

[0056] Note that even when the subject is very far away, a lack of distance values ​​may occur in the depth map 330, and the density ρ_i in the determination target area may be less than cρ. Therefore, strictly speaking, when the cause of the missing area is not parallax (when the density ρ_i is less than cρ), the cause of the missing area may be low reflectance (low reflectance) of the subject, or the subject may be located very far away. However, in this embodiment, for the sake of simplicity, unless otherwise specified, no strict distinction is made between a subject with low reflectance and a subject that is so far away that a lack of distance values ​​occurs, and the latter subject is also treated as a type of low reflectance subject.

[0057] In the example of the depth map 330 shown in FIG. 3C, the density ρ_i of the determination target region corresponding to the missing region 321 is equal to or greater than cρ, and therefore it is determined that the cause is parallax.

[0058] In S406, the defective area determination unit 107 determines that the subject has low reflectance (the reflectance of the subject is low).

[0059] In S407, the defective area determination unit 107 determines whether processing has been completed for all defective areas. If processing has been completed for all defective areas, the processing of this flowchart ends. If there are any unprocessed defective areas remaining, the processing proceeds to S408.

[0060] In S408, the defect area determination unit 107 selects the next defect area, and then the cause of the defect is determined for the selected defect area through the processes of S403 to S406.

[0061] In S404, the defective area determination unit 107 may refer to pixel values ​​at the same positions in the reflectance map 340 as the determination target area, and if the reflectance indicated by the pixel value is lower than a threshold, determine that the cause of the defective area is low reflectance. Alternatively, the defective area determination unit 107 may segment the reflectance map 340 and determine whether the determination target area is included in a segmentation with a low average reflectance. When segmentation is performed, the defective area determination unit 107 may use information from the imaging unit 102, as will be described later in a second embodiment.

[0062] <Details of the Process in S206 (First Interpolation Method)> FIG. 5 is a flowchart showing details of the process in S206 (first interpolation method).

[0063] In S501, the missing area interpolation unit 108 selects a processing target pixel within the missing area. The processing target pixel is a pixel within the missing area that does not have a valid distance value (a pixel with a distance value of 0), and is selected at each sampling interval of the distance measurement unit 101. The processing order (selection order) may be an order opposite to the direction in which the missing area occurs due to parallax. For example, as in the depth map 310, if a missing area occurs on the right side of the subject, the processing target pixel is selected in order from right to left. This makes it possible to interpolate distance values ​​in order from right to left.

[0064] 3A to 3D, a case where a horizontal parallax exists between the distance measuring unit 101 and the imaging unit 102 will be described below. Similar processing is also possible when a vertical or diagonal parallax exists.

[0065] In S502, the missing area interpolation unit 108 sets a surrounding area including the pixel to be processed based on parallax information (relative position information between the distance measurement unit 101 and the image capture unit 102 recorded in the system area 104).

[0066] FIG. 7A is a conceptual diagram of a method for setting a peripheral region of a pixel to be processed within a defective region. Similar to FIG. 3B , the depth map 310 represents a portion of the depth map as viewed from the imaging unit 102. Reference numeral 701 denotes a pixel to be processed within the defective region 321, and reference numeral 702 denotes a peripheral region set for the pixel to be processed 701. The peripheral region 702 is set so that its sides in the parallax direction are longer. If the horizontal (parallax direction) and vertical (different from the parallax direction) sizes of the peripheral region 702 are a [pixels] and b [pixels], respectively, then, for example, a = 7 [pixels] and b = 3 [pixels]. In the example of FIG. 7A , the defective region 321 occurs on the right side of the subject, and therefore the peripheral region 702 is set to extend to the right of the pixel to be processed 701 (extending from the position of the pixel to be processed 701 in a direction from the viewpoint of the ranging unit 101 toward the viewpoint of the imaging unit 102). At this time, to prevent the distance value of another subject from being mixed into the peripheral region 702, the missing region interpolation unit 108 may set the length of the vertical side so that it is not too large compared with the sampling interval of the distance measurement unit 101. Note that the shape of the peripheral region is not limited to a rectangle, and may be another shape such as an ellipse. In this case as well, the major axis of the ellipse is assumed to be in the parallax direction.

[0067] In S503, the missing area interpolation unit 108 groups the multiple distance values ​​included in the surrounding area 702. Here, grouping means dividing the multiple distance values ​​into sets of distance values ​​that are within a certain threshold. For example, consider a case where the surrounding area 702 includes distance values ​​of 5.0, 5.5, 10.0, and 10.5 m, and the distance values ​​are divided so that the difference within each group is 1.0 m or less. In this case, these four distance values ​​are divided into two groups: "5.0, 5.5 m" and "10.0, 10.5 m." If the number of distance values ​​included in the surrounding area 702 is small, the number of groups may be one.

[0068] In S504, the missing area interpolation unit 108 assigns (sets) the representative distance value of the group corresponding to a relatively long distance to the target pixel 701. This is a measure to deal with cases where the peripheral region 702 includes multiple objects with different distance values, and takes advantage of the fact that distance values ​​of more distant objects are more likely to be stored in missing regions caused by parallax. For example, as shown in the depth map 311 of FIG. 7B , even if the peripheral region 702 includes a distance value (relatively small distance value) of the object 710, the distance value of the object 710 can be eliminated by using the representative distance value of the group corresponding to a relatively long distance value. The representative value of the distance value group can be the average, median, or maximum value of the distance values ​​within the group. Furthermore, if there is only one group, the missing area interpolation unit 108 assigns the representative distance value of that group to the target pixel 701. Note that, like the depth map 310 of FIG. 7A , the depth map 311 represents a portion of the depth map as seen from the imaging unit 102.

[0069] In S505, the missing area interpolation unit 108 determines whether processing of all pixels to be processed within the missing area has been completed. If processing of all pixels to be processed has been completed, the processing of this flowchart ends. If unprocessed pixels to be processed remain, the processing returns to S501. In this case, in S501, the next pixel to be processed is selected according to the processing order (selection order) described above, and the processing of S502 to S504 is performed on the selected pixel to be processed.

[0070] By the above process, distance values ​​are interpolated for missing areas caused by parallax.

[0071] <Details of the Process in S207 (Second Interpolation Method)> Figure 6 is a flowchart showing details of the process in S207 (second interpolation method). Here, an example will be described in which the distance value of a missing region 731 in the depth map 312 shown in Figure 7C is interpolated. Like the depth map 310 in Figure 7A, the depth map 312 represents a portion of the depth map as seen from the imaging unit 102. In the depth map 312, a region 730 corresponds to an object with low reflectance, and a missing region 731 is a missing region caused by low reflectance.

[0072] In S601, the missing area interpolation unit 108 selects a pixel to be processed within the missing area. The pixel to be processed is a pixel within the missing area that does not have a valid distance value (a pixel whose distance value is 0), and is selected at each sampling interval of the distance measurement unit 101. The order of processing (selection) can be from the upper left to the lower right of the missing area.

[0073] In S602, the missing area interpolation unit 108 sets a surrounding area including the target pixel. The surrounding area is set to include one or more pixels in the missing area for which distance values ​​are stored. In this case, unlike S502 in Fig. 5, the surrounding area does not need to be anisotropic. In Fig. 7C, 703 is the target pixel in the missing area 321, and 704 is the surrounding area set for the target pixel 703.

[0074] In S503, the missing area interpolation unit 108 groups a plurality of distance values ​​included in the surrounding area 704, similar to S503 in FIG.

[0075] In S603, the missing area interpolation unit 108 deletes all groups corresponding to relatively long distances. The purpose of S603 is to exclude distance values ​​at positions that are likely to be background included in the peripheral area 704. However, if different subjects can be separated using information from the reflectance map, or if different subjects can be separated using information from the imaging unit 102 as described in the second embodiment, S603 may be omitted.

[0076] In S604, the missing area interpolation unit 108 stores the representative value of the remaining group in the processing target pixel 703. The method for selecting the representative value is the same as in S504 of FIG. 5 . In this way, in S604, the distance value of the processing target pixel 703 is interpolated based on the distance value corresponding to the relatively closest distance among the multiple distance values ​​included in the surrounding area. The reason for performing the processing in S604 is that when distance information for part of the subject is missing, interpolation is performed using the distance value that is most likely to belong to the subject. Conversely, in the case of FIG. 5 , interpolation must be performed using a distance value that does not belong to the foreground subject. Therefore, the distance value interpolation methods differ between FIG. 5 and FIG. 6 .

[0077] In S605, the missing area interpolation unit 108 determines whether processing of all pixels to be processed within the missing area has been completed. If processing of all pixels to be processed has been completed, the processing of this flowchart ends. If unprocessed pixels to be processed remain, the processing returns to S601. In this case, in S601, the next pixel to be processed is selected according to the processing order (selection order) described above, and the processing of S602 to S604 is performed on the selected pixel to be processed.

[0078] Through the above process, distance values ​​are interpolated for missing areas caused by low reflectance of the subject.

[0079] Summary of First Embodiment As described above, according to the first embodiment, the distance information processing device 100 detects a missing area (e.g., missing area 321 or 731) where the density of distance values ​​is less than a threshold in a distance map (e.g., depth map 310 or 312) of a second viewpoint (e.g., the viewpoint of the imaging unit 102) generated based on distance information acquired by the distance measurement unit 101 that measures the distances of one or more subjects at a first viewpoint. Furthermore, the distance information processing device 100 interpolates the distance values ​​of the missing area using different methods depending on whether the missing area is caused by parallax between the first viewpoint and the second viewpoint.

[0080] As described above, according to the first embodiment, the distance value of the missing area is interpolated using different methods depending on whether the missing area is caused by parallax or not, thereby enabling the distance value of the missing area to be interpolated with high accuracy.

[0081] 2 and 4, the distance information processing apparatus 100 determines the cause of the missing area according to the density of distance values ​​in a region corresponding to the missing area (e.g., the determination target region set in S403) in a second distance map (e.g., the depth map 330 generated in S402) generated by converting the viewpoint of the distance map of the second viewpoint from the second viewpoint to the first viewpoint, and interpolates the distance value of the missing area using the first interpolation method or the second interpolation method according to the determination result. Therefore, when focusing on the density of distance values, it can be said that the distance information processing apparatus 100 is configured to interpolate the distance value of the missing area using different methods according to the density of distance values ​​in the region corresponding to the missing area in the second distance map.

[0082] Second Embodiment In the second embodiment, a configuration will be described in which the missing area interpolation unit 108 interpolates distance values ​​using information about the captured image acquired by the imaging unit 102. In the second embodiment, the basic configuration of the distance information processing device 100 ( FIG. 1 ) is the same as in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0083] In this embodiment, following the processing of S502 in FIG. 5 and S602 in FIG. 6 (processing for setting the surrounding area), processing for excluding part of the surrounding area is performed.

[0084] The process of excluding a portion of the peripheral region will be described with reference to Figures 8A-B and 9. Figures 8A-B are conceptual diagrams of depth maps for explaining the process of excluding a portion of the peripheral region. Figure 8A corresponds to a case where the cause of the missing region is parallax, and Figure 8B corresponds to a case where the cause of the missing region is low reflectance.

[0085] In Figures 8A and 8B, depth maps 801 and 802 are actually sparse depth maps, but for ease of understanding, they are illustrated using corresponding captured images. Reference numerals 810, 811, and 812 represent detection frames for subjects 302, 710, and 720 superimposed on depth maps 801 and 802, respectively. In Figure 8A, distance values ​​within missing region 321 are missing due to parallax, and in Figure 8B, distance values ​​for some pixels, including target pixel 703, are missing in the region of subject 720 with low reflectance. Surrounding region 702 is the surrounding region set in S502 of Figure 5, and surrounding region 704 is the surrounding region set in S602 of Figure 6.

[0086] In the following description, it is assumed that the number of pixels in the captured image and the depth map are the same, but they may be different. If the number of pixels is different, the pixel values ​​in the captured image corresponding to the pixels in the depth map are used.

[0087] Fig. 9 is a flowchart of the process of excluding a portion of the peripheral region. For a defective region caused by parallax, the process of Fig. 9 is performed following S502 of Fig. 5, and for a defective region caused by low reflectance, the process of Fig. 9 is performed following S602 of Fig. 6. Before the process of Fig. 9 starts, the detection of the defective region and the determination of the cause have been completed by the process of Fig. 4.

[0088] In step S901, subject detection is performed within a captured image. The subject detection process is performed by the image capture unit 102, but may be performed outside the image capture unit 102.

[0089] In S902, the missing area interpolation unit 108 determines whether a detection frame including the target pixel in the peripheral area exists. Here, if the cause of the missing area is parallax, the target pixel is the pixel selected in S501, and if the cause of the missing area is low reflectance, the target pixel is the pixel selected in S601. If a detection frame including the target pixel exists (hereinafter referred to as "detection frame A"), processing proceeds to S903; if not, processing proceeds to S904.

[0090] 8B , the processing target pixel 703 is included in the detection frame 812, so the detection frame 812 corresponds to detection frame A, and the process proceeds from S902 to S903. In S903, the missing area interpolation unit 108 excludes all areas included in any of the detection frames other than detection frame A from the surrounding area 704. In Fig. 8B , the area where the surrounding area 704 and detection frame 811 overlap is excluded from the surrounding area 704. The process then proceeds to S503.

[0091] In the case of Figure 8A, the target pixel 701 is not included in either detection frames 801 or 811, so the process proceeds from S902 to S904. In this case, the target pixel is likely to belong to the background. In S904, the missing area interpolation unit 108 excludes all areas included in either detection frame from the surrounding area 702. In Figure 8A, the union of the detection frames 801 and 811 is excluded from the surrounding area 702. Then, the process proceeds to S503.

[0092] If the entire surrounding area is excluded as a result of the processing in S903 or S904, the missing area interpolation unit 108 cancels the processing in S903 or S904. In other words, the missing area interpolation unit 108 uses the original surrounding area 702 or 704 as is.

[0093] Although the method of using the subject detection frame (subject detection result) has been described above, a method that simply utilizes differences in pixel values ​​of the image captured by the image capture unit 102 may also be employed. In this case, the missing area interpolation unit 108 compares the pixel value of the processing target pixel with the pixel value of another pixel in the surrounding area, and if the difference is less than a threshold, includes this other pixel in the surrounding area. This utilizes the property that the closer the pixel values ​​are, the more likely they are to belong to the same subject. Alternatively, the subject area may be identified using segmentation instead of subject detection. In this case, processing is performed in which the detection frame in FIG. 9 is replaced with the corresponding segmentation area. For example, processing is performed in which the detection frame 810 is replaced with the segmentation area corresponding to the subject 302.

[0094] As described above, according to the second embodiment, by also utilizing information about the captured image (pixel values ​​of the captured image, subject detection results in the captured image, or segmentation results of the captured image), it is possible to interpolate distance values ​​with higher accuracy.

[0095] [Third Embodiment] In the third embodiment, a case will be described in which the distance information processing device 100 includes a distance measurement mechanism for acquiring distance information to a subject in addition to the distance measurement unit 101, or in which distance information acquired at a different time is utilized. In the third embodiment, the basic configuration of the distance information processing device 100 ( FIG. 1 ) is the same as in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0096] Possible methods for acquiring distance information include using two or more distance measuring units, estimating distance information from information from two or more imaging units 102 using stereo matching or multi-view stereo technology, estimating distance information from information from one or more imaging units 102 using a neural network, and converting phase difference into the distance to the subject using an imaging surface phase difference method.

[0097] Here, the imaging surface phase difference method is a method in which the image sensor in the imaging unit 102 holds multiple photodiodes (photoelectric conversion units) per microlens, and the light beam is separated by the microlens to form an image. A signal obtained by adding signals from two photodiodes is used as the imaging signal. An image shift amount p is obtained from a pair of signals read out from each photodiode. The control unit 120 converts the image shift amount p into a defocus amount d using a conversion coefficient K that depends on the incident angle, F-number, and optical axis position of the imaging optical system, and then converts the defocus value into a distance value using a known method such as the lens formula.

[0098] 2, if it is determined that the cause of the missing area is parallax, the control unit 120 interpolates the distance value using information from another ranging unit or another imaging unit that has parallax in a direction different from the parallax between the imaging unit 102 and the ranging unit 101. For example, if the imaging unit 102 and ranging unit 101 are arranged horizontally as shown in FIGS. 3A to 3D, it is possible to utilize information from a ranging unit or imaging unit that has parallax in the vertical direction. On the other hand, with the imaging surface phase difference method, a depth map from the same viewpoint as the imaging unit 102 is obtained, so interpolation is possible as is.

[0099] If it is determined in S205 of Figure 2 that the cause of the missing area is low reflectivity, it is difficult to obtain a distance value from the subject using the distance measurement unit 101, so it is possible to complement the distance value using multi-view stereo or a depth map obtained using an image plane phase difference method.

[0100] Although the above describes an example in which distance values ​​are interpolated using a distance measurement mechanism other than the distance measurement unit 101, distance values ​​may be interpolated using distance values ​​from the distance measurement unit 101 at other times. In this case, distance values ​​can be appropriately interpolated, particularly for missing areas caused by parallax, when the background and subject are stationary.

[0101] As described above, according to the third embodiment, by utilizing distance information acquired by other distance measuring mechanisms or at other times, it becomes possible to perform interpolation of distance values ​​with higher accuracy.

[0102] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0103] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0104] This application claims priority based on Japanese Patent Application No. 2024-074919, filed May 2, 2024, the entire contents of which are incorporated herein by reference.

[0105] 100... distance information processing device, 101... distance measuring unit, 102... imaging unit, 110... non-volatile memory, 111... system memory, 120... control unit

Claims

1. A distance information processing device comprising: a detection means for detecting a missing area where the density of distance values ​​is less than a threshold in a distance map of a second viewpoint generated based on distance information acquired by a distance measurement means that measures the distance of one or more subjects at a first viewpoint; and an interpolation means for interpolating the distance values ​​of the missing area using different methods depending on whether the missing area is caused by parallax between the first viewpoint and the second viewpoint.

2. The distance information processing device according to claim 1, further comprising a determination means for determining whether the missing area is caused by the parallax.

3. A distance information processing device as described in claim 1 or 2, characterized in that the distance measuring means acquires the distance information using at least one of light, electromagnetic waves, and sound waves, and the missing area occurs due to the parallax or low reflectance of the subject corresponding to the missing area.

4. A distance information processing device according to any one of claims 1 to 3, characterized in that the interpolation means sets a surrounding area including the pixel to be processed in the missing area using a different method depending on whether the cause of the missing area is the parallax or not, and interpolates the distance value of the pixel to be processed based on the distance value included in the surrounding area.

5. The distance information processing device according to claim 4, characterized in that, when the missing area is caused by the parallax, the interpolation means sets the surrounding area so that the size of the surrounding area in the direction of the parallax is larger than the size of the surrounding area in a direction different from the direction of the parallax.

6. The distance information processing device according to claim 5, characterized in that, when the cause of the missing area is the parallax, the interpolation means sets the surrounding area so that the surrounding area extends from the position of the pixel to be processed in a direction from the first viewpoint to the second viewpoint.

7. A distance information processing device according to any one of claims 4 to 6, characterized in that, when the cause of the missing area is the parallax, the interpolation means interpolates the distance value of the pixel to be processed based on a distance value corresponding to a relatively long distance among multiple distance values ​​included in the surrounding area.

8. The distance information processing device according to claim 4, characterized in that, when the cause of the missing area is not the parallax, the interpolation means sets the surrounding area so that the surrounding area includes one or more pixels having a distance value within the missing area.

9. A distance information processing device according to claim 4 or 8, characterized in that, when the missing area is not caused by the parallax, the interpolation means interpolates the distance value of the pixel to be processed based on a distance value corresponding to a relatively close distance among multiple distance values ​​included in the surrounding area.

10. A distance information processing device according to any one of claims 4 to 9, characterized in that the interpolation means sets the surrounding area based on pixel values ​​of the captured image at the second viewpoint, subject detection results in the captured image, or segmentation results of the captured image.

11. The distance information processing device according to claim 2, characterized in that the determination means generates a second distance map by converting the viewpoint of the distance map from the second viewpoint to the first viewpoint, and determines whether the missing area is caused by the parallax based on the density of distance values ​​in an area in the second distance map corresponding to the missing area.

12. A distance information processing device according to any one of claims 1 to 11, characterized in that the distance information is information in the form of a point cloud, voxel, polygon, mesh, implicit function representation, depth map, or disparity map, and the distance map is a depth map or a disparity map.

13. A distance information processing device according to any one of claims 1 to 12, further comprising: the distance measurement means; and a generation means for generating the distance map based on the distance information acquired by the distance measurement means.

14. A distance information processing device comprising: a detection means for detecting a missing area where the density of distance values ​​is less than a first threshold in a distance map of a second viewpoint generated based on distance information acquired by a distance measurement means for measuring the distance of one or more subjects at a first viewpoint; and an interpolation means for interpolating the distance values ​​of the missing area using different methods depending on the density of distance values ​​in an area corresponding to the missing area in a second distance map generated by converting the viewpoint of the distance map from the second viewpoint to the first viewpoint.

15. The distance information processing device according to claim 14, characterized in that the interpolation means sets a peripheral region including the pixel to be processed in the missing region in a manner that differs depending on the density of distance values ​​in the region corresponding to the missing region in the second distance map, and when the density of distance values ​​in the region corresponding to the missing region in the second distance map is equal to or greater than a second threshold, the interpolation means sets the peripheral region so that the size of the peripheral region in the direction of the parallax between the first viewpoint and the second viewpoint is larger than the size of the peripheral region in a direction different from the direction of the parallax, and so that the peripheral region extends from the position of the pixel to be processed in a direction from the first viewpoint to the second viewpoint, and interpolates the distance value of the pixel to be processed based on a distance value corresponding to a relatively long distance out of multiple distance values ​​included in the peripheral region.

16. The distance information processing device according to claim 15, characterized in that, when the density of distance values ​​in an area corresponding to the missing area in the second distance map is less than the second threshold, the interpolation means sets the surrounding area so that the surrounding area includes one or more pixels having a distance value within the missing area, and interpolates the distance value of the pixel to be processed based on, of the multiple distance values ​​included in the surrounding area, a distance value that corresponds to a relatively closer distance than when the density of distance values ​​in the area corresponding to the missing area is equal to or greater than the second threshold.

17. An imaging device comprising: a distance information processing device according to any one of claims 1 to 16; and imaging means for generating an image captured at the second viewpoint.

18. A distance information processing method executed by a distance information processing device, comprising: a detection step of detecting a missing area where the density of distance values ​​is less than a threshold in a distance map of a second viewpoint generated based on distance information acquired by a distance measurement means that measures the distance of one or more subjects at a first viewpoint; and an interpolation step of interpolating the distance values ​​of the missing area using different methods depending on whether the missing area is caused by parallax between the first viewpoint and the second viewpoint.

19. A distance information processing method executed by a distance information processing device, comprising: a detection step of detecting a missing area where the density of distance values ​​is less than a threshold in a distance map of a second viewpoint generated based on distance information acquired by a distance measurement means that measures the distance of one or more subjects at a first viewpoint; and an interpolation step of interpolating the distance values ​​of the missing area using different methods depending on the density of distance values ​​in an area corresponding to the missing area in a second distance map generated by converting the viewpoint of the distance map from the second viewpoint to the first viewpoint.

20. A program for causing a computer to function as each means of the distance information processing device according to any one of claims 1 to 16.

Images