Distance information processing device, imaging device, distance information processing method, and program

Abstract

To provide a technique for interpolating a distance value of a deficient area by a method that differs depending on whether the cause of the deficient area in a distance map is a parallax error.SOLUTION: A distance information processing device includes detection means for detecting a deficient area in which a density of a distance value is less than a threshold in a distance map of a second viewpoint, which is generated on the basis of distance information acquired by distance measuring means for measuring a distance of one or more subjects at a first viewpoint, and interpolation means for interpolating the distance value of the deficient area by a method that differs depending on whether the cause of the deficient area is a parallax error between the first viewpoint and the second viewpoint.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a distance information processing device, an imaging device, a distance information processing method, and a program. [Background technology]

[0002] In recent years, high-precision 3D information (distance information) has become increasingly important for map generation in autonomous driving, object recognition for robots, video production, game production, and other applications. 3D distance measurement devices such as LiDAR sensors are commonly used to obtain high-precision 3D information. LiDAR sensors obtain distance information to an object by emitting laser light and measuring the time it takes for the light to be reflected back from the object. Therefore, it may not be possible to obtain distance information for objects with low reflectivity or that are very far away.

[0003] In addition, to increase the density or rendering of distance information, there are increasing opportunities to capture images with a separate camera and process the images and distance information together. In such cases, if the viewpoints of the camera and the LiDAR sensor are different, the occlusion relationship of the subject will differ, resulting in a loss of distance information due to occlusion.

[0004] Patent Document 1 discloses a technique for dealing with missing distance information. In this technique, in a depth map in which a value corresponding to a distance value is stored in each pixel, the value of a missing pixel is interpolated by weighting the value corresponding to the surrounding distance value based on the pixel value of the image captured by the camera and the inter-pixel distance between the missing pixel and the surrounding pixels. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2000-230809 Summary of the Invention [Problem to be solved by the invention]

[0006] Although Patent Document 1 discloses viewpoint conversion when the viewpoint from which distance information is acquired differs from the viewpoint from which an image is captured, it does not take into consideration loss of distance information caused by viewpoint conversion (parallax between viewpoints). Therefore, the technique of Patent Document 1 may result in a decrease in the accuracy of interpolation of distance values ​​depending on the cause of the loss.

[0007] The present invention has been made in consideration of this situation, and aims to provide a technology for interpolating the distance value of a missing area in a distance map using different methods depending on whether the missing area is caused by parallax or not. [Means for solving the problem]

[0008] In order to solve the above problem, the present invention provides a distance information processing device characterized by comprising: a detection means for detecting a missing area in a distance map of a second viewpoint generated based on distance information acquired by a ranging means for measuring the distance of one or more subjects at a first viewpoint, where the density of distance values ​​is less than a threshold; and an interpolation means for interpolating the distance values ​​of the missing area in different ways depending on whether the cause of the missing area is the parallax between the first viewpoint and the second viewpoint. [Effects of the Invention]

[0009] According to the present invention, it is possible to interpolate the distance value of a missing area in a distance map using different methods depending on whether the missing area is caused by parallax or not.

[0010] Other features and advantages of the present invention will become more apparent from the accompanying drawings and the following detailed description of the preferred embodiment of the present invention. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a distance information processing device 100. [Figure 2] 10 is a flowchart of distance information processing executed by a control unit 120. [Figure 3](a) Conceptual diagram of a situation in which scene data is acquired using a distance measuring unit 101 and an imaging unit 102. (b) Conceptual diagram of a depth map as seen from the imaging unit 102. (c) Conceptual diagram of a depth map as seen from the distance measuring unit 101. (d) Conceptual diagram of a reflectance map. [Figure 4] 10 is a flowchart showing details of the process in S205. [Figure 5] 10 is a flowchart showing details of the process (first interpolation method) in S206. [Figure 6] 10 is a flowchart showing details of the process (second interpolation method) in S207. [Figure 7] 10 is a conceptual diagram of a method for setting a peripheral area of ​​a pixel to be processed in a defective area. [Figure 8] FIG. 10 is a conceptual diagram of a depth map for explaining a process of excluding part of a peripheral region. [Figure 9] 10 is a flowchart of a process for excluding a part of the surrounding area. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [First embodiment] <Configuration of distance information processing device 100> 1 is a block diagram showing an example of the configuration of a distance information processing device 100. The distance information processing device 100 includes a distance measuring unit 101, an imaging unit 102, a nonvolatile memory 110, a system memory 111, and a control unit 120.

[0014] The distance measurement unit 101 is, for example, a LiDAR sensor, and measures the three-dimensional shape of an object. The LiDAR sensor includes a light beam output unit that outputs laser light and irradiates the light beam onto the surface of the object, and a receiving unit that receives the light beam reflected from the surface of the object. The distance measurement unit 101 calculates the distance to the object surface in the irradiation direction using the time it takes for the reflected light beam to return or the phase difference between the irradiated light beam and the reflected light beam. The distance measurement unit 101 acquires the three-dimensional position by combining the calculated distance with information about the irradiation direction. Information (distance information) stored as indicating the three-dimensional position can be information in a known format, such as a point cloud, voxel, polygon, mesh, implicit function representation, depth map, or parallax map. The distance measurement unit 101 also acquires information about the reflectance of the laser beam from each object.

[0015] The distance measuring unit 101 is not limited to the LiDAR sensor exemplified in this embodiment. For example, the distance measuring unit 101 may be a device that measures distance using electromagnetic waves or sound waves. Furthermore, when the distance measuring unit 101 uses light, this light is not limited to laser light, and any type of light can be used.

[0016] The imaging unit 102 is, for example, a camera, and captures an image of an object (subject). The camera has a lens unit, an imaging element that converts an optical image into an electrical signal, and an A / D converter that converts an analog signal into a digital signal. The optical image is input to the imaging element via the lens unit, and the imaging element converts the optical image into an electrical signal, and then converts the electrical signal into a digital signal. In this way, an image is acquired.

[0017] The nonvolatile memory 110 has a distance information area 103, a system area 104, and an image area 105. The nonvolatile memory 110 is an electrically erasable and recordable memory, such as an EEPROM.

[0018] The distance information area 103 is an area for storing three-dimensional position information (distance information) and reflectance information acquired from the distance measurement unit 101 in a predetermined format. For example, the distance information is stored in a point cloud format, and the reflectance information is stored in a reflectance map format. The reflectance map is a map (information in an image format) in which a value corresponding to the reflectance is set as the pixel value for each pixel.

[0019] The system area 104 is an area for storing operation programs and operation constants for each block of the control unit 120. The operation programs here include programs for executing the processes of various flowcharts described later.

[0020] The image area 105 is an area for storing images acquired from the image capturing unit 102 .

[0021] The system memory 111 is a rewritable volatile memory such as a DRAM, etc. The system memory 111 stores constants and variables used during operation of the control unit 120, as well as data read from the nonvolatile memory 110, etc.

[0022] The control unit 120 is a control unit having at least one processor, and controls the entire distance information processing device 100. The control unit 120 also has a projection unit 106, a missing area determination unit 107, a missing area interpolation unit 108, and a full range interpolation unit 109. The control unit 120 executes the programs recorded in the system area 104 described above, thereby realizing various processes described below.

[0023] The projection unit 106 generates a depth map as seen from the imaging unit 102 based on the distance information recorded in the distance information area 103 and the relative positions of the ranging unit 101 and the imaging unit 102 recorded in the system area 104. The depth map is an example of a distance map, and is a map (image-format information) in which a value corresponding to the distance value to the subject (e.g., a value proportional to the distance value) is set as the pixel value for each pixel. Here, the value obtained by projecting the position vector of the subject as seen from the imaging unit 102 in the optical axis direction is used as the distance value to the subject, but the length of the position vector may also be used. Alternatively, a parallax map in which a value proportional to the reciprocal of the distance is stored as the distance value may also be used as the distance map. The following description will be given taking as an example a case in which the depth map is used as the distance map. Strictly speaking, lens distortion must be taken into consideration when projecting, but the following will assume an ideal situation in which there is no lens distortion.

[0024] In this embodiment, a pixel value of 0 is assigned to a distance value of 0 [mm], and a pixel value of 255 is assigned to a distance value equal to or greater than a certain threshold (for example, a distance value equal to or greater than 100,000 [mm]). Pixels for which no corresponding distance value exists are stored with a pixel value of 0. Hereinafter, pixel values ​​on the depth map are referred to as distance values ​​to distinguish them from pixel values ​​in an image captured by a camera. Distance values ​​on the depth map do not necessarily have to match physical distance values ​​(as in the example described above, a physical distance value of 100,000 [mm] may correspond to a distance value of 255 on the depth map).

[0025] The LiDAR sensor scans with laser light based on a certain angular resolution. Therefore, the resulting depth map stores values ​​corresponding to distance values ​​at specific intervals, and the remaining pixels are missing pixels storing a distance value of 0. Hereinafter, a depth map containing such missing pixels will be referred to as a sparse depth map. Note that the number of pixels in the sparse depth map may differ from that of the image acquired by the imaging unit 102. The sparse depth map is stored in the system area 104 and the system memory 111.

[0026] Missing distance values ​​also occur in occlusion areas caused by the parallax between the distance measuring unit 101 and the imaging unit 102, and in areas of (part of) the subject from which the distance measuring unit 101 cannot acquire information, such as subjects with low reflectivity. In this case, the group of pixels with missing distance values ​​is distributed over a wider area than the interval at which the distance measuring unit 101 acquires distance values. Hereinafter, such a group of missing pixels will be referred to as a missing area. The missing area determination unit 107 identifies the location of the missing area and also identifies the cause of the missing area. A method for identifying (determining) the location and cause of the missing area will be described later.

[0027] The missing area interpolation unit 108 interpolates distance values ​​for the missing area based on the sparse depth map and reflectance map output by the projection unit 106 and the determination result by the missing area determination unit 107. Details of the processing by the missing area interpolation unit 108 will be described later.

[0028] The global interpolation unit 109 generates a depth map (hereinafter referred to as a dense depth map) in which distance values ​​are stored in all pixels, based on the sparse depth map in which distance values ​​have been interpolated by the missing area interpolation unit 108 and the image acquired by the imaging unit 102. For example, the dense depth map can be generated by using a convolutional neural network (CNN) on a combination of the captured image and the corresponding sparse depth map. However, the method for generating the dense depth map is not limited to this, and other methods such as applying a filter to the distance image, super-resolution technology, and conditional random fields can also be used.

[0029] <Distance information processing> 2 is a flowchart of distance information processing executed by control unit 120. Unless otherwise specified, the processing of each step in this flowchart is realized by control unit 120 (or one or more blocks within control unit 120) executing a program stored in system area 104.

[0030] In S201, the control unit 120 acquires a captured image by capturing an image using the imaging unit 102. The acquired captured image is recorded in the image area 105.

[0031] In S202, the control unit 120 acquires distance information by measuring the distance using the distance measuring unit 101. The acquired distance information is recorded in the distance information area 103.

[0032] In S203, the projection unit 106 generates a sparse depth map as seen from the imaging unit 102, based on the distance information recorded in the distance information area 103 and the relative position information between the distance measurement unit 101 and the imaging unit 102 recorded in the system area 104. Hereinafter, unless otherwise specified, the sparse depth map will be simply referred to as a depth map.

[0033] It should be noted that, although it is assumed here that relative position information between the distance measuring unit 101 and the image capturing unit 102 is stored in advance in the system area 104 (i.e., the relative positions are known), this embodiment is not limited to this configuration. For example, the projection unit 106 may calculate the relative positions by matching feature points between the distance information (three-dimensional point cloud) and the captured image and using a known calibration method.

[0034] In S204, the defective area determination unit 107 detects (identifies) a defective area in the depth map generated in S203. The defective area determination unit 107 also determines (identifies) the cause of the detected defective area. The projection unit 106 is also used to determine the cause of the defective area. Causes of the defective area (causes of missing distance values) include the parallax between the viewpoint of the distance information used to generate the depth map (the viewpoint of the distance measurement unit 101) and the viewpoint of the depth map (the viewpoint of the imaging unit 102), and low reflectance of the subject. Details of the processing in S204 and the cause of the defective area will be described later.

[0035] In S205, the missing area interpolation unit 108 determines whether the missing area is caused by parallax. If the missing area is caused by parallax, the process proceeds to S206, and if not, the process proceeds to S207.

[0036] In S206, the missing area interpolation unit 108 interpolates the distance value of the missing area using the first interpolation method. Details of the processing in S206 will be described later.

[0037] In S207, the missing area interpolation unit 108 interpolates the distance value of the missing area using the second interpolation method. Details of the processing in S207 will be described later.

[0038] If multiple missing areas are detected in S204, the processes of S205 to S207 are performed for each area. For example, consider a case where three missing areas are detected, two of which are caused by parallax, and one of which is caused by low reflectance. In this case, interpolation processing using the first interpolation method is performed in S206 for each of the two missing areas caused by parallax, and interpolation processing using the second interpolation method is performed in S207 for the one missing area caused by low reflectance.

[0039] In S208, the full-range interpolation unit 109 generates a dense depth map based on the depth map (sparse depth map) in which the missing region has been interpolated, and the captured image.

[0040] In the above description, it is assumed that the cause of the missing area is determined in S204. However, if there is only one candidate cause of the missing area, the determination of the cause of the missing area can be omitted. For example, if there is no parallax between the distance measurement unit 101 and the imaging unit 102, the missing area determination unit 107 does not need to determine the cause of the missing area. In this case, the missing area interpolation unit 108 can proceed to S207 without performing the process of S205. As another example, when a scene consisting only of subjects with high reflectivity is captured, the missing area determination unit 107 does not need to determine the cause of the missing area. In this case, the missing area interpolation unit 108 can proceed to S206 without performing the process of S205.

[0041] Also, a configuration may be adopted that further subdivides the determination of the cause of the missing area and switches the interpolation method. For example, a configuration may be adopted that determines whether the object has a low reflectivity or is a subject located very far away, and switches the interpolation method.

[0042] Also, S208 may be omitted, and a configuration may be adopted that performs processes such as object detection and segmentation using a sparse depth map.

[0043] <Details of the processing in S205 and details of the cause of the missing area> Referring to FIGS. 3 and 4, the details of the processing in S205 and the details of the cause of the missing area will be described.

[0044] FIG. 3(a) is a conceptual diagram of a situation where data of a scene is acquired using the distance measurement unit 101 and the imaging unit 102. In FIG. 3(a), 301 represents a wall, and 302 represents a subject placed in the scene. The area 303 represents an area of the wall 301 that is shielded by the subject 302 when viewed from the distance measurement unit 101.

[0045] FIG. 3(b) is a conceptual diagram of a depth map as viewed from the imaging unit 102, which is generated by the projection unit 106 in S203. The depth map 310 represents a part of the depth map as viewed from the imaging unit 102. Here, for the sake of illustration, only the peripheral part of the subject 302 is shown. In the depth map 310, the area 320 corresponds to the subject 302. The missing area 321 is a missing area that occurred on the right side of the area 320 due to parallax.

[0046] 3(c) is a conceptual diagram of a depth map as seen from the distance measurement unit 101. The depth map 330 is obtained by converting the depth map 310 into a depth map as seen from the distance measurement unit 101 based on relative position information between the distance measurement unit 101 and the image capture unit 102. Here, the projection unit 106 converts the viewpoint position using the focal length and pixel pitch of the image capture unit 102. Therefore, the resolution of the depth map 310 and the depth map 330 as images is the same. Because the depth map 330 is distance information as seen from the viewpoint position of the distance measurement unit 101, no missing areas due to parallax occur.

[0047] 3(d) is a conceptual diagram of a reflectance map. The reflectance map 340 represents a reflectance map of a subject, acquired together with distance information by the distance measurement unit 101. In the reflectance map 340, brighter areas indicate higher reflectance. For simplicity of explanation, the depth map 330 and the reflectance map 340 have the same resolution, but they may also have different resolutions.

[0048] 4 is a flowchart showing details of the process in S205. In S401, the defective area determination unit 107 detects (specifies) a defective area in the depth map generated in S203.

[0049] Here, the angular resolution when the LiDAR sensor of the distance measurement unit 101 scans the laser light is "φ", the focal length of the lens of the image capture unit 102 is "f", and the pixel pitch of the image capture element of the image capture unit 102 is "Δ". In the depth map 310, the interval at which distance values ​​exist when there are no defects is roughly given by D=fφ / Δ (this also applies to the depth map 330). Strictly speaking, a factor depending on the angle between the line connecting the center of the optical axis and the subject and the image capture element surface is multiplied, but this is omitted here for simplicity of explanation. Also, it is assumed that the horizontal and vertical angular resolutions are the same. The density of distance values ​​when there are no defects is given by ρ=1 / D 2The missing area determination unit 107 then references the depth map 310 and identifies a group of pixels whose density is lower than the density ρ as a missing area. Here, a predetermined constant c equal to or less than 1 is used to provide a margin for the determination criteria. That is, the missing area determination unit 107 identifies (detects) an area in the depth map 310 whose density is less than cρ as a missing area. The density of distance values ​​for a pixel is obtained by dividing the number of pixels with distance values ​​(the number of pixels with non-zero distance values) within a square aD × aD that includes the pixel by the area of ​​the square. Here, a is a value equal to or greater than 1, e.g., 2. In the example of FIG. 3(b), one missing area 321 is detected, but multiple missing areas may be detected depending on the configuration of the depth map. In addition, there may be cases where no missing area is detected. In such cases, the processing of the flowchart in FIG. 4 ends, and the processing of S208 in FIG. 2 is subsequently performed.

[0050] In S402, the projection unit 106 generates a depth map 330 as seen from the ranging unit 101 by converting the viewpoint of the depth map 310 from the viewpoint of the imaging unit 102 to the viewpoint of the ranging unit 101 based on the relative position information between the ranging unit 101 and the imaging unit 102 recorded in the system area 104.

[0051] In S403, the defective region determination unit 107 sets, in the depth map 330, a determination target region that corresponds to one of the one or more defective regions detected in the depth map 310. The determination target region in the depth map 330 is set at the same position as the defective region in the depth map 310. Therefore, for the defective region 321, the determination target region is set at the same position in the depth map 330 that corresponds to the position of the defective region 321 in the depth map 310.

[0052] In S404, the missing area determination unit 107 compares the density ρ_i of distance values ​​in the determination target area of ​​the depth map 330 with the density ρ of distance values ​​in the case where there is no missing area × constant c. If ρ_i is equal to or greater than cρ (a threshold value or greater), the process proceeds to S405; otherwise, the process proceeds to S406.

[0053] In the above description, the threshold used in S401 and the threshold used in S404 are both the same (cρ), but the two thresholds may be different.

[0054] In S405, the defective area determination unit 107 determines that the cause of the defective area corresponding to the determination target area is the parallax between the viewpoint of the distance information used to generate the depth map (the viewpoint of the distance measurement unit 101) and the viewpoint of the depth map (the viewpoint of the imaging unit 102). The reason for this determination will be explained below.

[0055] When the viewpoint of the depth map is changed (transformed), the amount of movement of the distance value position (coordinate) varies depending on the distance to the subject. If the distance measurement unit 101 and the image capture unit 102 are arranged horizontally apart by b [mm], a point on the subject at a distance d [mm] moves by bf / (Δd) [pix] on the image of the depth map due to parallax. Here, f is the focal length of the lens [mm], and Δ is the pixel pitch of the image capture element [mm]. As can be seen from this equation, the focal length of the lens and the number of pixels moved are proportional to each other. When the viewpoint of the depth map is returned to the viewpoint of the distance measurement unit 101 by converting the depth map 310 to the depth map 330, the missing area caused by parallax disappears. Therefore, if the density ρ_i in the determination target area of ​​the depth map 330 is equal to or greater than cρ (if the density ρ_i is similar to ρ), the missing area corresponding to the determination target area is considered to be caused by parallax. Conversely, if the density ρ_i in the determination target region is less than cρ, it is considered that low reflectance of the subject is the cause of a missing distance value in the depth map 330 at the viewpoint of the distance measuring unit 101. Therefore, it is considered that the missing region is not caused by parallax but by low reflectance of the subject (low reflectance of the subject).

[0056] Note that even when a subject is very far away, a lack of distance values ​​may occur in the depth map 330, and the density ρ_i in the determination target area may be less than cρ. Therefore, strictly speaking, when the cause of the missing area is not parallax (when the density ρ_i is less than cρ), the cause of the missing area may be low reflectance (low reflectance) of the subject, or the subject may be very far away. However, in this embodiment, for the sake of simplicity, unless otherwise specified, no strict distinction is made between a subject with low reflectance and a subject that is so far away that a lack of distance values ​​occurs, and the latter subject is also treated as a type of low reflectance subject.

[0057] In the example of the depth map 330 shown in FIG. 3(c), the density ρ_i of the determination target region corresponding to the missing region 321 is equal to or greater than cρ, and therefore it is determined that the cause is disparity.

[0058] In S406, the defective area determination unit 107 determines that the reflectance of the subject is low (the reflectance of the subject is low).

[0059] In S407, the defective area determination unit 107 determines whether processing has been completed for all defective areas. If processing has been completed for all defective areas, the processing of this flowchart ends. If there are any unprocessed defective areas remaining, the processing proceeds to S408.

[0060] In S408, the defective area determination unit 107 selects the next defective area, and then the cause of the selected defective area is determined by the processes of S403 to S406.

[0061] In S404, the defect area determination unit 107 may refer to the pixel value at the same position as the determination target area in the reflectance map 340, and determine that the cause of the defect area is low reflectance when the reflectance indicated by the pixel value is lower than the threshold. Further, the defect area determination unit 107 may segment the reflectance map 340 and determine whether the determination target area is included in the segmentation with a low average reflectance. When performing segmentation, as will be described later in the second embodiment, the defect area determination unit 107 may use the information of the imaging unit 102.

[0062] <Details of the process in S206 (first interpolation method)> FIG. 5 is a flowchart showing the details of the process in S206 (first interpolation method).

[0063] In S501, the defect area interpolation unit 108 selects a processing target pixel in the defect area. The processing target pixel is a pixel in the defect area where no valid distance value exists (pixel with a distance value of 0), and is selected at each sampling interval of the distance measurement unit 101. Also, as the processing order (selection order), an order opposite to the direction in which the defect area occurs due to parallax can be adopted. For example, when a defect area occurs on the right side of the subject as in the depth map 310, the processing target pixels are sequentially selected from the right side to the left side. This enables the distance values to be interpolated in order from right to left.

[0064] Hereinafter, as described in FIG. 3, the case where a horizontal parallax exists between the distance measurement unit 101 and the imaging unit 102 will be described. The same processing is possible even when parallax exists in the vertical direction or obliquely.

[0065] In S502, the defect area interpolation unit 108 sets a peripheral area including the processing target pixel based on the parallax information (relative position information between the distance measurement unit 101 and the imaging unit 102 recorded in the system area 104).

[0066] FIG. 7(a) is a conceptual diagram of a method for setting a peripheral region of a pixel to be processed within a defective region. Similar to FIG. 3(b), the depth map 310 represents a portion of the depth map as viewed from the imaging unit 102. Reference numeral 701 denotes a pixel to be processed within the defective region 321, and reference numeral 702 denotes a peripheral region set for the pixel to be processed 701. The peripheral region 702 is set so that its sides in the parallax direction are longer. If the horizontal (parallax direction) and vertical (different from the parallax direction) sizes of the peripheral region 702 are a [pixels] and b [pixels], respectively, then, for example, a = 7 [pixels] and b = 3 [pixels]. In the example of FIG. 7(a), the defective region 321 occurs on the right side of the subject, and therefore the peripheral region 702 is set to extend to the right of the pixel to be processed 701 (extending from the position of the pixel to be processed 701 in a direction from the viewpoint of the ranging unit 101 toward the viewpoint of the imaging unit 102). At this time, to prevent the distance value of another subject from being mixed into the peripheral area 702, the missing area interpolation unit 108 may set the length of the vertical side so that it is not too large compared to the sampling interval of the distance measurement unit 101. Note that the shape of the peripheral area is not limited to a rectangle, and may be another shape such as an ellipse. In this case as well, the major axis of the ellipse is assumed to be in the parallax direction.

[0067] In S503, the missing area interpolation unit 108 groups the distance values ​​contained in the surrounding area 702. Here, grouping refers to dividing the distance values ​​into sets of distance values ​​that fall within a certain threshold. For example, consider a case where the distance values ​​5.0, 5.5, 10.0, and 10.5 m are contained in the surrounding area 702 and the distance values ​​are divided so that the difference within each group is 1.0 m or less. In this case, these four distance values ​​are divided into two groups: "5.0, 5.5 m" and "10.0, 10.5 m." If the number of distance values ​​contained in the surrounding area 702 is small, the number of groups may be one.

[0068] In S504, the missing area interpolation unit 108 substitutes (sets) the representative distance value of the group corresponding to a relatively far distance into the processing target pixel 701. This is a countermeasure for the case where a plurality of subjects with different distance values are included in the peripheral area 702, and utilizes the property that a distance value of a farther subject is likely to be stored in a missing area caused by parallax. For example, as shown in the depth map 311 of FIG. 7(b), even when the distance value (relatively small distance value) of the subject 710 is included in the peripheral area 702, the distance value of the subject 710 can be excluded by using the representative distance value of the group corresponding to a relatively far value. As the representative value in the group of distance values, the average value, median value, maximum value, or the like within the group can be used. Also, when there is one group, the missing area interpolation unit 108 substitutes the representative distance value of that group into the processing target pixel 701. Note that, similar to the depth map 310 of FIG. 7(a), the depth map 311 represents a part of the depth map seen from the imaging unit 102.

[0069] In S505, the missing area interpolation unit 108 determines whether the processing of all the processing target pixels in the missing area has been completed. If the processing of all the processing target pixels has been completed, the processing of this flowchart ends. If unprocessed processing target pixels remain, the processing returns to S501. In this case, in S501, the next processing target pixel is selected according to the above-described processing order (selection order), and the processing of S502 to S504 is performed on the selected processing target pixel.

[0070] Through the above processing, a distance value is interpolated for a missing area caused by parallax.

[0071] <Details of the processing in S207 (second interpolation method)> Fig. 6 is a flowchart showing details of the process (second interpolation method) in S207. Here, an example will be described in which distance values ​​of a missing region 731 in the depth map 312 shown in Fig. 7(c) are interpolated. Similar to the depth map 310 in Fig. 7(a), the depth map 312 represents a portion of the depth map as seen from the imaging unit 102. In the depth map 312, the region 730 corresponds to an object with low reflectance, and the missing region 731 is a missing region caused by the low reflectance.

[0072] In S601, the defective area interpolation unit 108 selects a pixel to be processed in the defective area. The pixel to be processed is a pixel in the defective area that does not have a valid distance value (a pixel whose distance value is 0), and is selected at each sampling interval of the distance measurement unit 101. The processing order (selection order) can be from the top left to the bottom right of the defective area.

[0073] In S602, the missing area interpolation unit 108 sets a surrounding area including the pixel to be processed. The surrounding area is set to include one or more pixels in the missing area that store distance values. At this time, unlike S502 in FIG. 5, the surrounding area does not need to be anisotropic. In FIG. 7(c), 703 is the pixel to be processed in the missing area 321, and 704 is the surrounding area set for the pixel to be processed 703.

[0074] In S503, the missing area interpolation unit 108 groups a plurality of distance values ​​included in the surrounding area 704, similar to S503 in FIG.

[0075] In S603, the missing area interpolation unit 108 deletes all groups corresponding to relatively long distances. The purpose of S603 is to exclude distance values ​​at positions that are likely to be background included in the surrounding area 704. However, if different subjects can be separated using information from the reflectance map, or if different subjects can be separated using information from the imaging unit 102 as described in the second embodiment, S603 may be omitted.

[0076] In S604, the missing area interpolation unit 108 stores the representative value of the remaining group in the processing target pixel 703. The method for selecting the representative value is the same as in S504 of FIG. 5. Thus, in S604, the distance value of the processing target pixel 703 is interpolated based on the distance value corresponding to the relatively closest distance among the multiple distance values ​​included in the surrounding area. The reason for performing the processing as in S604 is that when distance information for part of the subject is missing, interpolation is performed using the distance value that is most likely to belong to the subject. Conversely, in the case of FIG. 5, interpolation must be performed using a distance value that does not belong to the foreground subject. Therefore, the distance value interpolation methods differ between FIG. 5 and FIG. 6.

[0077] In S605, the missing area interpolation unit 108 determines whether or not processing of all pixels to be processed in the missing area has been completed. If processing of all pixels to be processed has been completed, the processing of this flowchart ends. If unprocessed pixels to be processed remain, the processing returns to S601. In this case, in S601, the next pixel to be processed is selected according to the processing order (selection order) described above, and the processing of S602 to S604 is performed on the selected pixel to be processed.

[0078] Through the above process, distance values ​​are interpolated for missing areas caused by low reflectance of the subject.

[0079] <Summary of the First Embodiment> As described above, according to the first embodiment, the distance information processing device 100 detects a missing area (e.g., missing area 321 or 731) where the density of distance values ​​is less than a threshold in a distance map (e.g., depth map 310 or 312) of a second viewpoint (e.g., the viewpoint of the imaging unit 102) generated based on distance information acquired by the distance measurement unit 101 that measures the distance of one or more subjects at a first viewpoint. Furthermore, the distance information processing device 100 interpolates the distance values ​​of the missing area using different methods depending on whether the missing area is caused by parallax between the first viewpoint and the second viewpoint.

[0080] As described above, according to the first embodiment, the distance value of the missing area is interpolated using different methods depending on whether the missing area is caused by parallax or not, which makes it possible to interpolate the distance value of the missing area with high accuracy.

[0081] 2 and 4, the distance information processing device 100 determines the cause of the missing area according to the density of distance values ​​in a region (e.g., a determination target region set in S403) corresponding to the missing area in a second distance map (e.g., the depth map 330 generated in S402) generated by converting the viewpoint of the distance map of the second viewpoint from the second viewpoint to the first viewpoint, and interpolates the distance value of the missing area using the first interpolation method or the second interpolation method according to the determination result. Therefore, when focusing on the density of distance values, it can be said that the distance information processing device 100 is configured to interpolate the distance value of the missing area using different methods according to the density of distance values ​​in the region corresponding to the missing area in the second distance map.

[0082] [Second embodiment] In the second embodiment, a configuration will be described in which the missing area interpolation unit 108 interpolates distance values ​​using information on the captured image acquired by the imaging unit 102. In the second embodiment, the basic configuration of the distance information processing device 100 (FIG. 1) is the same as in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0083] In this embodiment, following the processing of S502 in FIG. 5 and S602 in FIG. 6 (processing for setting the surrounding area), processing for excluding part of the surrounding area is performed.

[0084] The process of excluding a part of the surrounding area will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a conceptual diagram of a depth map for explaining the process of excluding a part of the surrounding area. Fig. 8(a) corresponds to the case where the cause of the missing area is parallax, and Fig. 8(b) corresponds to the case where the cause of the missing area is low reflectance.

[0085] In Figure 8, depth maps 801 and 802 are actually sparse depth maps, but for ease of understanding, they are illustrated using corresponding captured images. 810, 811, and 812 are detection frames for objects 302, 710, and 720 superimposed on depth maps 801 and 802, respectively. In Figure 8(a), distance values ​​within missing region 321 are missing due to parallax, and in Figure 8(b), distance values ​​are missing for some pixels, including target pixel 703, in the region of object 720 with low reflectivity. Surrounding region 702 is the surrounding region set in S502 of Figure 5, and surrounding region 704 is the surrounding region set in S602 of Figure 6.

[0086] In the following description, it is assumed that the number of pixels in the captured image and the depth map are the same, but they may be different. If the number of pixels is different, the pixel values ​​in the captured image corresponding to the pixels in the depth map are used.

[0087] Fig. 9 is a flowchart of the process of excluding part of the peripheral area. For a missing area caused by parallax, the process of Fig. 9 is performed following S502 in Fig. 5, and for a missing area caused by low reflectance, the process of Fig. 9 is performed following S602 in Fig. 6. Before the process of Fig. 9 starts, the detection of the missing area and the determination of the cause have been completed by the process of Fig. 4.

[0088] In step S901, subject detection within a captured image is performed. The subject detection process is performed by the image capture unit 102, but may be performed outside the image capture unit 102.

[0089] In S902, the missing area interpolation unit 108 determines whether a detection frame including the target pixel in the surrounding area exists. Here, if the cause of the missing area is parallax, the target pixel is the pixel selected in S501, and if the cause of the missing area is low reflectance, the target pixel is the pixel selected in S601. If a detection frame including the target pixel exists (hereinafter referred to as "detection frame A"), processing proceeds to S903; if not, processing proceeds to S904.

[0090] In the case of FIG. 8(b), the processing target pixel 703 is included in the detection frame 812, so the detection frame 812 corresponds to detection frame A, and the process proceeds from S902 to S903. In S903, the missing area interpolation unit 108 excludes from the surrounding area 704 all areas that are included in any of the detection frames other than detection frame A. In FIG. 8(b), the area where the surrounding area 704 and detection frame 811 overlap is excluded from the surrounding area 704. The process then proceeds to S503.

[0091] In the case of FIG. 8(a), the processing target pixel 701 is not included in either detection frames 801 or 811, so the process proceeds from S902 to S904. In this case, the processing target pixel is likely to belong to the background. In S904, the missing area interpolation unit 108 excludes all areas included in either detection frame from the surrounding area 702. In FIG. 8(a), the union of the detection frames 801 and 811 is excluded from the surrounding area 702. Then, the process proceeds to S503.

[0092] If the entire surrounding area is excluded as a result of the processing in S903 or S904, the missing area interpolation unit 108 cancels the processing in S903 or S904. In other words, the missing area interpolation unit 108 uses the original surrounding area 702 or 704 as is.

[0093] Although the method of using the subject detection frame (subject detection result) has been described above, a method of simply using the difference in pixel values ​​of the image captured by the image capture unit 102 may also be employed. In this case, the missing area interpolation unit 108 compares the pixel value of the pixel to be processed with the pixel value of another pixel in the surrounding area, and if the difference is less than a threshold, includes this other pixel in the surrounding area. This utilizes the property that the closer the pixel values ​​are, the more likely they are to belong to the same subject. Alternatively, the subject area may be identified using segmentation instead of subject detection. In this case, processing is performed in which the detection frame in FIG. 9 is replaced with the corresponding segmentation area. For example, processing is performed in which the detection frame 810 is replaced with the segmentation area corresponding to the subject 302.

[0094] As described above, according to the second embodiment, by utilizing information about the captured image (pixel values ​​of the captured image, subject detection results in the captured image, or segmentation results of the captured image), it is possible to interpolate distance values ​​with higher accuracy.

[0095] [Third embodiment] In the third embodiment, a case will be described in which the distance information processing device 100 includes a distance measurement mechanism for acquiring distance information to a subject other than the distance measurement unit 101, or a case in which distance information acquired at a different time is utilized. In the third embodiment, the basic configuration of the distance information processing device 100 (FIG. 1) is the same as in the first embodiment. Below, differences from the first embodiment will be mainly described.

[0096] Possible methods for obtaining distance information include using two or more distance measuring units, estimating distance information from information from two or more imaging units 102 using stereo matching or multi-view stereo technology, estimating distance information from information from one or more imaging units 102 using a neural network, and converting phase difference into the distance to the subject using an imaging surface phase difference method.

[0097] Here, the imaging surface phase difference method is a method in which the imaging element in the imaging unit 102 has multiple photodiodes (photoelectric conversion units) for each microlens, and the light beam is separated by the microlens to form an image. A signal obtained by adding signals from two photodiodes is used as an imaging signal. An image shift amount p is obtained from a pair of signals read out from each photodiode. The control unit 120 converts the image shift amount p into a defocus amount d using a conversion coefficient K that depends on the incident angle, F-number, and optical axis position of the imaging optical system, and then converts the defocus value into a distance value using a known method such as the lens formula.

[0098] 2, if it is determined that the cause of the missing area is parallax, the control unit 120 interpolates the distance value using information from another ranging unit or another imaging unit that has parallax in a direction different from the parallax between the imaging unit 102 and the ranging unit 101. For example, if the imaging unit 102 and ranging unit 101 are arranged horizontally as shown in FIG. 3, it is possible to utilize information from a ranging unit or imaging unit that has parallax in the vertical direction. On the other hand, with the imaging plane phase difference method, a depth map from the same viewpoint as the imaging unit 102 is obtained, so interpolation is possible as is.

[0099] In S205 of Figure 2, if it is determined that the cause of the missing area is low reflectivity, it is a subject for which it is difficult to obtain a distance value using the distance measurement unit 101, so it is possible to complement the distance value using multi-view stereo or a depth map obtained using the image plane phase difference method.

[0100] Although the above shows an example in which distance values ​​are interpolated using a distance measurement mechanism other than the distance measurement unit 101, distance values ​​may be interpolated using distance values ​​from the distance measurement unit 101 at other times. In this case, distance values ​​can be appropriately interpolated, particularly for missing areas caused by parallax, when the background and subject are stationary.

[0101] As described above, according to the third embodiment, by utilizing distance information acquired by other distance measuring mechanisms or at other times, it becomes possible to perform interpolation of distance values ​​with higher accuracy.

[0102] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0103] [summary] The above-described embodiment discloses at least the inventions shown in the following items, but is not limited to these inventions. (Item 1) a detection means for detecting a missing area where the density of distance values ​​is less than a threshold in a distance map of a second viewpoint generated based on distance information acquired by a distance measurement means for measuring the distance of one or more objects at a first viewpoint; an interpolation means for interpolating a distance value of the missing area using different methods depending on whether the missing area is caused by a parallax between the first viewpoint and the second viewpoint; A distance information processing device comprising: (Item 2) further comprising a determination means for determining whether the missing area is caused by the parallax. 2. The distance information processing device according to item 1, (Item 3) the distance measuring means acquires the distance information using at least one of light, electromagnetic waves, and sound waves; The defective area occurs due to the parallax or low reflectance of the subject corresponding to the defective area. 3. The distance information processing device according to item 1 or 2. (Item 4) The interpolation means a peripheral area including a pixel to be processed in the defective area is set by a method that differs depending on whether the cause of the defective area is the parallax; interpolating the distance value of the pixel to be processed based on the distance values ​​included in the surrounding area; 4. The distance information processing device according to any one of items 1 to 3. (Item 5) When the missing area is caused by the parallax, the interpolation means sets the peripheral area so that the size of the peripheral area in the direction of the parallax is larger than the size of the peripheral area in a direction different from the direction of the parallax. 5. The distance information processing device according to item 4. (Item 6) When the cause of the missing area is the parallax, the interpolation means sets the surrounding area so that the surrounding area extends from the position of the pixel to be processed in a direction from the first viewpoint toward the second viewpoint. 6. The distance information processing device according to item 5, (Item 7) When the cause of the missing area is the parallax, the interpolation means interpolates the distance value of the processing target pixel based on a distance value corresponding to a relatively long distance among a plurality of distance values ​​included in the peripheral area. 7. The distance information processing device according to any one of items 4 to 6, (Item 8) If the cause of the missing area is not the disparity, the interpolation means sets the surrounding area so that the surrounding area includes one or more pixels having a distance value within the missing area. 5. The distance information processing device according to item 4. (Item 9) When the missing area is not caused by the parallax, the interpolation means interpolates the distance value of the processing target pixel based on a distance value corresponding to a relatively short distance among a plurality of distance values ​​included in the peripheral area. 9. The distance information processing device according to item 4 or 8, characterized in that (Item 10) the interpolation means sets the surrounding area based on pixel values ​​of the captured image at the second viewpoint, a subject detection result in the captured image, or a segmentation result of the captured image. 10. The distance information processing device according to any one of items 4 to 9, characterized in that (Item 11) The determination means generating a second distance map by transforming a viewpoint of the distance map from the second viewpoint to the first viewpoint; determining whether the missing region is caused by the disparity based on a density of distance values ​​in a region corresponding to the missing region in the second distance map; 3. The distance information processing device according to item 2. (Item 12) the distance information is in the form of a point cloud, a voxel, a polygon, a mesh, an implicit function representation, a depth map, or a disparity map; The distance map is a depth map or a disparity map. 12. The distance information processing device according to any one of items 1 to 11, characterized in that: (Item 13) the distance measuring means; a generating means for generating the distance map based on the distance information acquired by the distance measuring means; 13. The distance information processing device according to any one of items 1 to 12, further comprising: (Item 14) a detection means for detecting a missing area where the density of distance values ​​is less than a first threshold in a distance map of a second viewpoint generated based on distance information acquired by a distance measurement means for measuring the distance of one or more objects at a first viewpoint; an interpolation means for interpolating distance values ​​of the missing region by different methods depending on the density of distance values ​​in a region corresponding to the missing region in a second distance map generated by converting a viewpoint of the distance map from the second viewpoint to the first viewpoint; A distance information processing device comprising: (Item 15) the interpolation means sets a peripheral area including a pixel to be processed in the missing area using a method that differs depending on a density of distance values ​​in an area corresponding to the missing area in the second distance map; When the density of distance values ​​in the region corresponding to the missing region in the second distance map is equal to or greater than a second threshold, the interpolation means setting the peripheral region so that a size of the peripheral region in a direction of parallax between the first viewpoint and the second viewpoint is larger than a size of the peripheral region in a direction different from the direction of parallax, and so that the peripheral region extends from the position of the pixel to be processed in a direction from the first viewpoint to the second viewpoint; interpolating the distance value of the pixel to be processed based on a distance value corresponding to a relatively long distance among a plurality of distance values ​​included in the peripheral region; 15. A distance information processing device according to item 14. (Item 16) When the density of distance values ​​in the region corresponding to the missing region in the second distance map is less than the second threshold, the interpolation means setting the surrounding area so that the surrounding area includes one or more pixels having a distance value within the defective area; interpolating the distance value of the pixel to be processed based on a distance value corresponding to a relatively closer distance than when the density of distance values ​​in an area corresponding to the missing area is equal to or greater than the second threshold value, among the plurality of distance values ​​included in the surrounding area; Item 16. A distance information processing device according to item 15. (Item 17) A distance information processing device according to any one of items 1 to 16, an imaging means for generating a captured image at the second viewpoint; An imaging device comprising: (Item 18) A distance information processing method executed by a distance information processing device, a detection step of detecting a missing area in a distance map of a second viewpoint generated based on distance information acquired by a distance measuring means that measures the distance of one or more objects at a first viewpoint, where the density of distance values ​​is less than a threshold; an interpolation step of interpolating a distance value of the missing area using different methods depending on whether the missing area is caused by a parallax between the first viewpoint and the second viewpoint; A distance information processing method comprising: (Item 19) A distance information processing method executed by a distance information processing device, a detection step of detecting a missing area in a distance map of a second viewpoint generated based on distance information acquired by a distance measuring means that measures the distance of one or more objects at a first viewpoint, where the density of distance values ​​is less than a threshold; an interpolation step of interpolating distance values ​​of the missing region using different methods depending on the density of distance values ​​in a region corresponding to the missing region in a second distance map generated by converting a viewpoint of the distance map from the second viewpoint to the first viewpoint; A distance information processing method comprising: (Item 20) A program for causing a computer to function as each of the means of the distance information processing device according to any one of items 1 to 16.

[0104] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0105] 100... distance information processing device, 101... distance measuring unit, 102... imaging unit, 110... nonvolatile memory, 111... system memory, 120... control unit

Claims

1. a detection means for detecting a missing area where the density of distance values ​​is less than a threshold in a distance map of a second viewpoint generated based on distance information acquired by a distance measurement means for measuring the distance of one or more objects at a first viewpoint; an interpolation means for interpolating a distance value of the missing area using different methods depending on whether the missing area is caused by a parallax between the first viewpoint and the second viewpoint; A distance information processing device comprising:

2. further comprising a determination means for determining whether the missing area is caused by the parallax.

2. The distance information processing device according to claim 1,

3. the distance measuring means acquires the distance information using at least one of light, electromagnetic waves, and sound waves; The defective area occurs due to the parallax or low reflectance of the subject corresponding to the defective area.

2. The distance information processing device according to claim 1,

4. The interpolation means a peripheral area including a pixel to be processed in the defective area is set by a method that differs depending on whether the cause of the defective area is the parallax; interpolating the distance value of the pixel to be processed based on the distance values ​​included in the surrounding area; 2. The distance information processing device according to claim 1,

5. When the missing area is caused by the parallax, the interpolation means sets the peripheral area so that the size of the peripheral area in the direction of the parallax is larger than the size of the peripheral area in a direction different from the direction of the parallax.

5. The distance information processing device according to claim 4.

6. When the cause of the missing area is the parallax, the interpolation means sets the surrounding area so that the surrounding area extends from the position of the pixel to be processed in a direction from the first viewpoint toward the second viewpoint.

6. The distance information processing device according to claim 5,

7. When the cause of the missing area is the parallax, the interpolation means interpolates the distance value of the processing target pixel based on a distance value corresponding to a relatively long distance among a plurality of distance values ​​included in the peripheral area.

5. The distance information processing device according to claim 4.

8. If the cause of the missing area is not the disparity, the interpolation means sets the surrounding area so that the surrounding area includes one or more pixels having a distance value within the missing area.

5. The distance information processing device according to claim 4.

9. When the cause of the missing area is not the parallax, the interpolation means interpolates the distance value of the processing target pixel based on a distance value corresponding to a relatively close distance among a plurality of distance values ​​included in the peripheral area.

5. The distance information processing device according to claim 4.

10. the interpolation means sets the surrounding area based on pixel values ​​of the captured image at the second viewpoint, a subject detection result in the captured image, or a segmentation result of the captured image.

5. The distance information processing device according to claim 4.

11. The determination means generating a second distance map by transforming a viewpoint of the distance map from the second viewpoint to the first viewpoint; determining whether the missing region is caused by the parallax based on a density of distance values ​​in a region corresponding to the missing region in the second distance map; 3. The distance information processing device according to claim 2, wherein:

12. the distance information is in the form of a point cloud, a voxel, a polygon, a mesh, an implicit function representation, a depth map, or a disparity map; The distance map is a depth map or a disparity map.

2. The distance information processing device according to claim 1,

13. the distance measuring means; a generating means for generating the distance map based on the distance information acquired by the distance measuring means; 2. The distance information processing device according to claim 1, further comprising:

14. a detection means for detecting a missing area in a distance map of a second viewpoint, the distance map being generated based on distance information acquired by a distance measurement means for measuring the distance of one or more objects at a first viewpoint, where the density of distance values ​​is less than a first threshold; an interpolation means for interpolating distance values ​​of the missing region by different methods depending on the density of distance values ​​in a region corresponding to the missing region in a second distance map generated by converting a viewpoint of the distance map from the second viewpoint to the first viewpoint; A distance information processing device comprising:

15. the interpolation means sets a peripheral area including a pixel to be processed in the missing area using a method that differs depending on a density of distance values ​​in an area corresponding to the missing area in the second distance map; When the density of distance values ​​in the region corresponding to the missing region in the second distance map is equal to or greater than a second threshold, the interpolation means setting the peripheral region so that a size of the peripheral region in a direction of parallax between the first viewpoint and the second viewpoint is larger than a size of the peripheral region in a direction different from the direction of parallax, and so that the peripheral region extends from the position of the pixel to be processed in a direction from the first viewpoint to the second viewpoint; interpolating the distance value of the pixel to be processed based on a distance value corresponding to a relatively long distance among a plurality of distance values ​​included in the peripheral region; 15. The distance information processing device according to claim 14.

16. When the density of distance values ​​in the region corresponding to the missing region in the second distance map is less than the second threshold, the interpolation means setting the surrounding area so that the surrounding area includes one or more pixels having a distance value within the defective area; interpolating the distance value of the processing target pixel based on a distance value corresponding to a relatively closer distance than when the density of distance values ​​in an area corresponding to the missing area is equal to or greater than the second threshold value, among the plurality of distance values ​​included in the surrounding area; 16. The distance information processing device according to claim 15,

17. a distance information processing device according to any one of claims 1 to 16; an imaging means for generating a captured image at the second viewpoint; An imaging device comprising:

18. A distance information processing method executed by a distance information processing device, a detection step of detecting a missing area in a distance map of a second viewpoint generated based on distance information acquired by a distance measuring means that measures the distance of one or more objects at a first viewpoint, where the density of distance values ​​is less than a threshold; an interpolation step of interpolating a distance value of the missing area using a different method depending on whether the missing area is caused by a parallax between the first viewpoint and the second viewpoint; A distance information processing method comprising:

19. A distance information processing method executed by a distance information processing device, a detection step of detecting a missing area in a distance map of a second viewpoint generated based on distance information acquired by a distance measuring means that measures the distance of one or more objects at a first viewpoint, where the density of distance values ​​is less than a threshold; an interpolation step of interpolating distance values ​​of the missing region using different methods depending on the density of distance values ​​in a region corresponding to the missing region in a second distance map generated by converting a viewpoint of the distance map from the second viewpoint to the first viewpoint; A distance information processing method comprising:

20. A program for causing a computer to function as each of the means of the distance information processing device according to any one of claims 1 to 16.

Images