Information processing equipment, information processing methods, camera equipment, mobile equipment, computer-readable storage media, and computer program products
By acquiring and processing distance maps from camera equipment and LiDAR, and combining them with deep learning to generate dense and accurate synthetic distance maps, the problem of insufficient information density and accuracy in existing technologies is solved, thereby improving the reliability and real-time performance of vehicle driving control.
Patent Information
- Application Number
- CN202610654174.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2025-05-15
- Filing Date
- 2026-05-13
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies struggle to generate dense and accurate distance maps corresponding to captured images, especially when using a single camera device and LiDAR, where there are issues with insufficient information density and accuracy.
By acquiring the first and second distance maps from a single camera device, data extraction, coefficient calculation, and generation are performed using information processing equipment to generate a third distance map. Combining deep learning and LiDAR data, a dense and accurate synthetic distance map is generated.
It enables the generation of dense and accurate distance maps from images captured at the same viewpoint and time, improving the reliability and real-time performance of vehicle driving control while reducing processing load and time.
Smart Images

Figure CN123123734A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to information processing equipment, information processing methods, camera equipment, mobile devices, computer-readable storage media, and computer program products, and particularly to the technical field of generating distance information of a subject from a captured image. Background Technology
[0002] There is a known process that involves acquiring distance information of a subject from an image captured by a camera device, and using a distance map that indicates distance information corresponding to the position in the captured image to detect objects in front of a vehicle or to blur the background area of the main subject. Various methods, such as stereo ranging and time-of-flight (ToF) methods, can be used to acquire distance information, with the stereo ranging method employing multiple camera devices. However, if a single method is used, it is difficult to acquire dense and accurate distance information corresponding to the captured image. On the other hand, patent documents propose a method that acquires a dense and accurate distance map by combining depth information estimated from the captured image using machine learning, such as deep learning using a trained artificial intelligence (AI) model, and distance information acquired from LiDAR.
[0003] Existing technical literature: J. Qiu, et al., "DeepLiDAR: Deep Surface Normal GuidedDepth Prediction for Outdoor Scene From Sparse LiDAR Data and Single ColorImage", Computer Vision and Pattern Recognition, 2019
[0004] As described in existing technical literature, when the camera and LiDAR are positioned in different locations and the camera and LiDAR have parallax, it is sometimes impossible to generate a dense and accurate distance map corresponding to the captured image. Summary of the Invention
[0005] This disclosure has taken into account the above-mentioned problems and provides technical advantages for generating dense and accurate distance maps.
[0006] To address the aforementioned problems, this disclosure relates to an information processing device, comprising: an acquisition unit for acquiring a first distance map and a second distance map, wherein the first distance map indicates first distance information corresponding to a position in a captured image, and the second distance map indicates second distance information corresponding to a position in the captured image; and a generation unit for generating a third distance map based on the first distance information of the first distance map and the second distance information of the second distance map.
[0007] To address the aforementioned problems, this disclosure relates to a camera device, comprising: an information processing device; and a camera unit for generating captured images, wherein the information processing device comprises: an acquisition unit for acquiring a first distance map and a second distance map, the first distance map indicating first distance information corresponding to a position in the captured image, and the second distance map indicating second distance information corresponding to a position in the captured image; and a generation unit for generating a third distance map based on the first distance information of the first distance map and the second distance information of the second distance map.
[0008] To address the aforementioned problems, this disclosure relates to a mobile device, comprising: an information processing device; a camera device for generating captured images; and a control device for controlling the mobile device based on a third distance map generated by the information processing device. The information processing device includes: an acquisition unit for acquiring a first distance map and a second distance map, wherein the first distance map indicates first distance information corresponding to a position in the captured image, and the second distance map indicates second distance information corresponding to a position in the captured image; and a generation unit for generating a third distance map based on the first distance information of the first distance map and the second distance information of the second distance map.
[0009] To address the aforementioned problems, this disclosure relates to an information processing method executed by an information processing device. The information processing method includes: a step of acquiring a first distance map and a second distance map, wherein the first distance map indicates first distance information corresponding to a position in a captured image, and the second distance map indicates second distance information corresponding to a position in the captured image; and a step of generating a third distance map based on the first distance information and the second distance information.
[0010] To address the aforementioned problems, this disclosure relates to a computer-readable storage medium that stores a program for enabling a computer to function as at least the information processing device described above.
[0011] To address the aforementioned problems, this disclosure relates to a computer program product comprising a program for enabling a computer to function at least as an information processing device as described above.
[0012] According to this disclosure, dense and accurate distance maps can be generated.
[0013] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of embodiments is given by way of example. Attached Figure Description
[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the embodiments.
[0015] Figure 1 This is a diagram illustrating the configuration of the system according to the first embodiment;
[0016] Figure 2 This is a block diagram illustrating the configuration of a camera device according to the first embodiment;
[0017] Figure 3 This is a block diagram illustrating the configuration of an information processing device according to the first embodiment;
[0018] Figures 4A to 4D This is a flowchart illustrating the processing of the extraction unit, calculation unit, and generation unit of the information processing apparatus according to the first embodiment;
[0019] Figures 5A to 5D This is a flowchart illustrating the detailed processing of the extraction unit and the calculation unit of the information processing device according to the first embodiment;
[0020] Figure 6A and Figure 6B This is an explanation Figure 5A The diagram shows the effective area setting process;
[0021] Figure 7 This is an explanation Figure 5B The diagram shows the calculation and processing of local variations;
[0022] Figures 8A to 8D This is an explanation Figure 5C The diagram shows the background value region search process and the data extraction process used for background value region exclusion.
[0023] Figure 9 This is a block diagram illustrating the configuration of a camera device according to the second embodiment;
[0024] Figure 10 This is a diagram illustrating the configuration of the camera unit according to the second embodiment;
[0025] Figure 11 This is a cross-sectional view illustrating the pixel configuration of the camera unit according to the second embodiment;
[0026] Figures 12A to 12EThis is a diagram illustrating the camera surface phase difference detection method according to the second embodiment;
[0027] Figure 13 This is a flowchart illustrating the data extraction and processing of the depth map according to the second embodiment; and
[0028] Figure 14 This is a diagram illustrating the main subject region search processing according to the second embodiment. Detailed Implementation
[0029] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are required, and several such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions thereof are omitted. [First Embodiment]
[0030] In a first embodiment, a system will be described in which a camera device and an information processing device are installed in a vehicle, the camera device captures an image of the front side of the vehicle, and the information processing device generates distance information to the subject based on the captured image. <System Configuration>
[0031] Figure 1 This is a diagram illustrating the configuration of the system according to the first embodiment.
[0032] Vehicle 100 refers to mobile devices such as cars. However, this is not limited to cars, and can include various public transportation vehicles such as trains and airplanes, small mobilities, or various robots such as automated guided vehicles (AGVs).
[0033] The imaging device 102 is an onboard camera positioned near the center of the upper end of the windshield of the vehicle 100 and capturing images of a predetermined area (hereinafter referred to as the cameraable range) in front of the vehicle 100. Based on images acquired at the same viewpoint and time by capturing images including the surrounding environment of the road on which the vehicle 100 is traveling, the imaging device 102 generates a distance map indicating the distance from the imaging device 102 to the subject and a depth map indicating the position of the subject in the depth direction in the captured images, and outputs these to the information processing device 103.
[0034] The information processing device 103 extracts reliable data from the distance map (first distance map) and depth map (second distance map) obtained from the same viewpoint and the same time captured by the self-camera device 102, and synthesizes the data to generate a dense and accurate composite distance map (third distance map) and outputs the composite distance map.
[0035] The vehicle control device 104 uses a composite distance map generated by the information processing device 103 to perform driving control of the vehicle 100 or object detection notification to the driver 101.
[0036] The camera device 102 is a digital camera or digital video camera. The information processing device 103 is an on-board sensor device or a robot vision sensor device. <Configuration of Camera Equipment 102>
[0037] Next, we will refer to Figure 2 Describe the configuration of camera device 102.
[0038] The imaging optical system 10 forms an optical image of the subject on the imaging surface of the imaging unit 11. The imaging optical system 10 includes a plurality of lenses arranged along the optical axis 12. The plurality of lenses includes a focusing lens 15 configured to adjust the focal length of the imaging optical system 10. The focusing lens 15 is movable along the optical axis. The control unit 200 drives the focusing lens 15 according to the defocus amount generated by the processing unit 201.
[0039] The imaging optical system 10 includes an aperture 14 capable of adjusting its f-number (aperture amount). The control unit 200 controls the f-number of the aperture 14 based on imaging conditions, for example, determined by automatic exposure control (AE). The exit pupil 13 is the view from the imaging unit 11 of the image of the imaging optical system 10 with its open aperture, and... Figure 2 The position of the projection pupil 13 is shown in the example.
[0040] The imaging unit 11 includes an image sensor, such as a CCD or CMOS, which includes a color filter, for example, a primary color Bayer array. The imaging unit 11 includes a pixel array in which multiple pixels are arranged in a two-dimensional manner and peripheral circuitry configured to read signals from the pixels. Each pixel accumulates a charge corresponding to the amount of incident light through photoelectric conversion. A signal having a voltage corresponding to the amount of charge accumulated during the exposure period is read from each pixel, thereby acquiring a group of pixel signals (analog image signals) representing the image of the subject formed on the imaging surface by the imaging optical system 10.
[0041] The pixels in the camera unit 11 include multiple photoelectric conversion regions or photoelectric conversion elements, and can generate parallax image pairs through a single image capture. Based on the parallax image pairs, autofocus control (phase difference AF) using autofocus detection via a phase difference method can be performed, or distance information can be generated. Details will be described later in the second embodiment.
[0042] The control unit 200 includes one or more processors, such as a CPU capable of executing programs. The processor of the control unit 200 loads programs stored in, for example, ROM 203 into RAM 202, executes the programs, thereby controlling the constituent elements of the camera device 102 and thus realizing the various functions of the camera device 102.
[0043] The processing unit 201 applies predetermined image processing to the analog image signal read from the imaging unit 11, thereby generating signal or image data corresponding to the application purpose, or acquiring and / or generating various information. The processing unit 201 is, for example, a dedicated hardware circuit such as an application-specific integrated circuit (ASIC) designed to implement specific functions. Alternatively, in the processing unit 201, the specific functions can be implemented by software executed by a processor such as a digital signal processor (DSP) or a graphics processing unit (GPU). The processing unit 201 outputs the acquired or generated information to the control unit 200 or stores the acquired or generated information in RAM 202, depending on the application purpose.
[0044] The processing unit 201 can also read the data stored in the storage unit 204 and process the data.
[0045] The processing unit 201 can perform processes such as preprocessing, color interpolation, correction, data processing, special effects processing, distance map generation, and depth map generation.
[0046] Preprocessing includes, for example, A / D conversion, signal amplification, reference level adjustment, and defective pixel correction.
[0047] Color interpolation is a process performed when the image sensor includes a color filter, and it involves interpolating the values of color components that are not included in the individual pixel data of the image data. Color interpolation is also known as demosaicing.
[0048] Correction processing includes, for example, white balance adjustment, tone correction, correction of image degradation caused by optical aberrations of the camera optical system 10 (image restoration), correction of the vignetting effect of the camera optical system 10, and color correction processing.
[0049] Data processing includes, for example, region cropping, compositing, scaling, encoding and decoding, header information generation (data file generation), and processing to generate and display image data or record image data.
[0050] Special effects processing includes, for example, adding blur effects, changing hue, and relighting.
[0051] In the distance map generation process, a distance map is generated from image pairs with parallax acquired by the camera unit 11.
[0052] In depth map generation processing, inference processing is performed to estimate depth maps from captured images using machine learning, such as deep learning with trained artificial intelligence (AI) models. AI models are formed, for example, by convolutional neural networks (CNNs), but are not limited to CNNs, and can also be formed by neural networks such as Transformers.
[0053] Inference processing, for example, can be performed using a GPU, but a Field Programmable Gate Array (FPGA) can be used instead. Furthermore, inference processing can be performed through collaboration between a CPU and a GPU or DSP, or it can be performed by either a GPU or a DSP.
[0054] Note that the above processing is merely an example of the processing that processing unit 201 can apply, and is not intended to limit the processing that processing unit 201 can apply.
[0055] RAM 202 serves as working memory or buffer memory. The working memory holds programs to be executed by the CPU of control unit 200 or the processor of processing unit 201, as well as data necessary during program execution. The buffer memory is configured to store image data acquired by camera unit 11 or image data processed by processing unit 201.
[0056] ROM 203 is, for example, a rewritable non-volatile memory that stores programs, settings, and graphical user interfaces (GUIs) that can be executed by the CPU of control unit 200.
[0057] Storage unit 204 is a recording medium configured to record data files containing image data acquired by camera unit 11. Storage unit 204 may be, for example, a combination of a memory card and its reader / writer. Storage unit 204 may be able to handle multiple recording media.
[0058] The communication unit 205 is an interface for communicating with external devices. The control unit 200 can communicate with external devices via the communication unit 205 in accordance with one or more wired standards. Configuration of Information Processing Equipment 103
[0059] Next, we will refer to Figure 3 The configuration of the information processing device 103 according to this embodiment is described.
[0060] Extraction unit 300 extracts reliable ranging data from the distance map acquired by camera device 102. Additionally, extraction unit 300 extracts reliable depth data from the depth map acquired by camera device 102. Here, "reliable" indicates that the reliability of the object being determined is not less than a predetermined threshold. This also applies to the following description.
[0061] The coefficient calculation unit 301 calculates coefficients used to match the scale between the reliable distance values extracted by the extraction unit 300 and the reliable depth values.
[0062] The generation unit 302 uses the coefficients obtained by the coefficient calculation unit 301 to perform scaling on the depth values of the depth map to convert the scale and match the scale of the distance values of the distance map, and then synthesizes the scaled depth map (data at dense locations) with the distance map (data at sparse locations) to generate a composite distance map.
[0063] The extraction unit 300, coefficient calculation unit 301, and generation unit 302 can each be, for example, a dedicated hardware circuit such as an application-specific integrated circuit (ASIC) designed to perform a specific function. Alternatively, in each of the extraction unit 300, coefficient calculation unit 301, and generation unit 302, the specific function can be implemented by a processor such as a digital signal processor (DSP) or a graphics processing unit (GPU) executing software.
[0064] The extraction unit 300, the coefficient calculation unit 301, and the generation unit 302 each store the acquired or generated information or data in RAM 303 or the like according to the application purpose.
[0065] ROM 304 is a memory similar to ROM 203 incorporated in the camera device 102. Furthermore, RAM 303 is used as working memory by the extraction unit 300, coefficient calculation unit 301, and generation unit 302, etc.
[0066] Storage unit 305 is a recording medium configured to record data files containing image data acquired by camera unit 11. Storage unit 305 may be, for example, a combination of a memory card and its reader / writer. Storage unit 305 may be able to handle multiple recording media. Distance map and depth map
[0067] The distance map and depth map according to this embodiment will be described.
[0068] The distance map is generated using captured images and contains distance values indicating distance information of subjects at sparse locations in the image. Furthermore, the distance information indicates the actual distance from the camera device 102 to the subject, or parallax information or defocus information equivalent to the actual distance. Additionally, the depth map is generated using the same captured images as the distance map and contains depth values indicating depth information of subjects at dense locations in the image. Note that the distance information shows the distance relationships (positional relationships in the depth direction) of subjects in the image, and only needs to be relative distances rather than actual distances.
[0069] As mentioned above, the resolution of a depth map is higher than that of a distance map. Furthermore, the scale of a distance map differs from that of a depth map. For example, in a distance map, the unit of distance is [m]. In a depth map, the unit of distance is [au (astronomical units)]. Note that distance maps and depth maps do not always need to have different scales, and they can have the same dimensions if they were acquired from the same captured image using different methods. <Processing by Information Processing Equipment 103>
[0070] Next, we will refer to Figures 4A to 4D and Figures 5A to 5D The process performed by the information processing device 103 according to this embodiment is described.
[0071] Figures 4A to 4D This is a flowchart illustrating the processing of the extraction unit, calculation unit, and generation unit of the information processing device 103 according to the first embodiment. Figures 5A to 5D This is a flowchart illustrating the detailed processing of the extraction unit and the calculation unit of the information processing device 103 according to the first embodiment.
[0072] Whenever the camera device 102 acquires captured images and distance information, it executes... Figures 4A to 4D The processing shown.
[0073] Will first refer to Figure 4A The processing described is carried out by the information processing device 103. (Processing of extraction unit 300)
[0074] In step S300, the extraction unit 300 extracts reliable ranging data from the distance map and depth map acquired by the camera device 102, and stores the extracted data in RAM 303.
[0075] Figure 4B This is an example Figure 4A The flowchart shows the detailed processing of step S300.
[0076] In step S400, the extraction unit 300 extracts the region with a reliable ranging value to be used in the subsequent step S301 from the distance map acquired by the camera device 102.
[0077] Figure 5A This is an example Figure 4B The flowchart shows the detailed processing of step S400.
[0078] In step S500, the extraction unit 300 considers the region including the effective range value corresponding to the position of the camera device 102 and the region with high reliability range value based on the camera optical system 10, sets the region with high reliability in the distance map, and extracts the range value of the set region.
[0079] This will be referenced Figure 6A and Figure 6B describe Figure 5A The effective area setting process in step S500.
[0080] Figure 6A The captured image 600 shown is an image acquired by the camera device 102, and captures the sky area 601, the vehicle hood area 602, and the road area 603. In this case, the sky area 601 and the vehicle hood area 602 are invalid areas because proper distance measurement cannot be performed there, and the remaining area 603, including the road, is set as the valid area 604 (X-1). Figure 6B Examples are shown in Figure 6A The state of the effective area set in the captured image 600 shown.
[0081] Next, a region with high reliability from the camera optical system 10 derived from the camera device 102 is defined. More specifically, ranging accuracy decreases in the vignetting region of the camera optical system 10, or in the peripheral portion of the image when the optical system is wide-angle. Therefore, using the image height of the captured image as a parameter, the center of the image is set as an effective region (X-2) where a highly reliable ranging value can be obtained.
[0082] Note that there are no restrictions on the effective area setting method, and it can be used to exclude subjects that are too close, subjects that are driving next to vehicle 100, or subjects that are floating in the air.
[0083] exist Figure 5A In step S501, the extraction unit 300 extracts the distance values from the effective area (X-1 and X-2) set in step S500 from the distance map of the camera device 102, and outputs the extracted effective area extraction distance map. Afterwards, the process proceeds to... Figure 4B Step S401 in the process.
[0084] In step S401, the extraction unit 300 extracts a distance map for the effective area extracted in step S400 and extracts the distance values in the effective distance measurement range.
[0085] Figure 5B This is an example Figure 4B The flowchart shows the detailed processing of step S401.
[0086] In step S502, the extraction unit 300 evaluates whether the distance values in the effective area extraction distance map extracted in step S400 fall within the effective distance range, extracts more reliable distance value data, and outputs the extracted effective distance extraction distance map. The effective distance range is set by the camera device 102 or the distance accuracy. Furthermore, the effective distance range can be adjusted according to the surrounding environment during filming (e.g., changes in the brightness of the vehicle's exterior or the temperature inside the vehicle during vehicle movement).
[0087] In step S503, the extraction unit 300 calculates the local variation of the ranging values in the effective ranging extraction distance map extracted in step S502. In areas where the variation of the ranging values is less than a predetermined range, the ranging error is small, and for local variations of ranging values with high reliability (except in areas where distant and near subjects coexist), a pixel of interest and a surrounding area including the pixel of interest are set in the distance map, and the standard deviation is used for calculation. For example, setting as follows... Figure 7 In the distance map shown, for the pixel of interest 700 and the surrounding area (hereinafter referred to as the standard deviation calculation area 701), the standard deviation of the distance values in the pixel area is calculated, and the obtained value is set as the local variation. Note that the standard deviation calculation area 701 used to calculate the local variation is not limited, and it is sufficient to include the area of the pixel of interest 700. Furthermore, the method for calculating the local variation is not limited, and it is sufficient to calculate the variation as long as it can be either the standard deviation or the variance.
[0088] In step S504, the extraction unit 300 performs evaluation processing on the local variation data calculated in step S503, extracts data with high reliability ranging values whose reliability is not less than a predetermined threshold from the effective ranging extraction distance map obtained in step S502, and outputs the extracted distance map (hereinafter referred to as the extracted distance map). This threshold is set by the ranging characteristics of the camera device 102.
[0089] and Figure 4B The processes of steps S400 and S401 are performed in parallel, and the process of step S402 is executed.
[0090] In step S402, the extraction unit 300 extracts the region with a reliable ranging value to be used in the depth map acquired by the camera device 102.
[0091] Figure 5C This is an example Figure 4B The flowchart shows the detailed processing of step S402.
[0092] In step S505, the extraction unit 300 searches for regions with depth values that can be considered as the background of the shooting scene from the depth value data of the depth map, and sets the background value region. The background value region corresponds to the region of values that are identified as background values through inference processing.
[0093] This will be referenced Figures 8A to 8C Describe the background value region search method.
[0094] As a characteristic of the estimated depth value of the depth map obtained through inference processing, it can be considered as a region of background. Figure 8A The background area 800 indicated by the shadow in the image shows a predetermined value. Figure 8B It shows Figure 8A The estimated depth values in the depth map of the captured image 600 shown are represented as an example of a histogram. From Figure 8B It can be seen that the estimated depth value of the region that can be considered background is output as a predetermined value of 801. This suggests that the estimated depth value of the region that can be considered background has low reliability and is unnecessary in subsequent processing stages. Therefore, Figure 8C The area surrounded by the dotted line is set as the background value area 802.
[0095] In step S506, the extraction unit 300 uses the background value region 802 set in step S505 to extract the depth map data and outputs the extracted depth map (hereinafter referred to as the extracted depth map).
[0096] In the data extraction process for excluding background value regions, the background value region 802 set in step S505 is excluded, and data in regions with values larger than those in the background value region 802 (hereinafter referred to as exclusion region 803) is excluded (see [link]). Figure 8D Regions with values larger than those in the background value region have depth values greater than the background values, and therefore these depth values are considered outliers in the depth map. For this reason, in the data extraction process for excluding background value regions in step S506, exclusion region 803 is excluded, thereby extracting data.
[0097] Notice, Figure 4A The data to be extracted in step S300 is unrestricted, and more reliable data can be used from at least one of the distance map and depth map for extraction. Furthermore, the timing of the extraction process in step S300 is unrestricted, and can be performed at any time. Figure 4B More reliable data extraction may be performed in at least one of steps S400, S401, and S402.
[0098] exist Figure 4BAfter the processing of steps S400, S401 and S402, the process proceeds to step S403.
[0099] In step S403, the extraction unit 300 extracts data of the common region (hereinafter referred to as common extraction data) between the extraction distance map obtained in step S401 and the extraction depth map obtained in step S402.
[0100] Note that for the extraction of jointly extracted data, it can be obtained through logical operations such as AND between the extracted distance map and the extracted depth map. Also note that the method for extracting jointly extracted data is unrestricted and can be based on OR operations between either the extracted distance map or the extracted depth map, or on the ranging reliability of the extracted distance map and the extracted depth map.
[0101] Ranging reliability is a quantified value representing the reliability of ranging values within a distance map. Ranging reliability is set for each pixel position in the distance map and is calculated and set using data such as the contrast, brightness, illuminance, exposure, luminance, temperature, and humidity of the captured image. Ranging reliability is stored in RAM 202 and read out as needed during processing.
[0102] exist Figure 4B After the processing in step S403, in Figure 4A In step S301, the coefficient calculation unit 301 performs the processing. (Processing of coefficient calculation unit 301)
[0103] In step S301, the coefficient calculation unit 301 calculates coefficients for scaling between the reliable distance map data and the reliable depth map data obtained in step S300.
[0104] Figure 4C This is an example Figure 4A The flowchart shows the detailed processing of step S301.
[0105] In step S404, the coefficient calculation unit 301 uses the common extracted data obtained in step S300 to fit and output the obtained coefficients.
[0106] Figure 5D This is an example Figure 4C The flowchart shows the detailed processing of step S404.
[0107] In step S507, the coefficient calculation unit 301 determines the reasonableness of the common extraction data obtained in step S300 and removes the data determined to be unreasonable. The coefficient calculation unit 301 performs outlier removal processing on the common extraction data obtained in step S300 and outputs the common extraction data with outliers removed (hereinafter referred to as outlier-removed common extraction data). To remove outliers from the common extraction data, Random Sample Consensus (hereinafter referred to as RANSAC) is used. Note that the outlier removal method is not limited and can use random sampling, systematic sampling, stratified sampling, or cluster sampling, etc.
[0108] In step S507, the coefficient calculation unit 301 removes outliers from the distance and depth maps that cannot be removed by the extraction process in step S300. Outliers in a state of insufficient data extraction are removed during coefficient calculation in the preprocessing of step S300 or during intermediate processing. Because the distance and depth maps acquired by the camera device 102 contain errors, it is difficult to complete outlier removal solely in step S507 for outlier outputs in wide areas. For this reason, the same effect as step S507 cannot be achieved before and after the processing in step S300. Therefore, it is effective to perform outlier removal in the co-extracted data after the processing in step S300 as the timing for outlier removal.
[0109] In step S508, the coefficient calculation unit 301 calculates the fitting coefficients of the approximate expression from the common extracted data after outlier removal.
[0110] An appropriate function is used to fit the depth values of the depth map acquired by the camera device 102. For example, if the output is a relative value proportional to the inverse of the distance as the depth value of the depth map, a linear function Y=AX+B is used for fitting, and coefficients A and B are obtained as fitting coefficients. Note that the function expression used for fitting is not limited, and a function expression corresponding to the depth values of the depth map acquired by the camera device 102 is used.
[0111] exist Figure 4C After the processing in step S404, the process proceeds to step S405.
[0112] In step S405, the coefficient calculation unit 301 uses the fitting coefficients calculated in step S404 and a threshold value for the fitting coefficients to determine the reliability of the fitting coefficients, and outputs the calculated coefficients as those with high reliability. When determining the reliability of the fitting coefficients, coefficients within the threshold range are directly output. When a coefficient is outside the threshold range, the process returns to step S404. Step S301 is repeated until coefficients falling within the threshold range are calculated.
[0113] The threshold for the fitting coefficients is set based on the distance map and depth map acquired by the camera device 102. For the threshold of the fitting coefficients, highly reliable coefficients are set from the coefficients processed when scaling the depth map in subsequent stages, based on indicators such as the distance measurement value of the nearest obstacle, the vanishing point on the road surface, or the slope of the road surface.
[0114] exist Figure 4B After the processing in step S405, in Figure 4A In step S302, the generation unit 302 is processed. (Processing of generation unit 302)
[0115] In step S302, the generation unit 302 uses the distance map and depth map acquired by the camera device 102 and the coefficients (coefficient A and coefficient B) calculated in step S301 to generate a composite distance map.
[0116] Figure 4D This is an example Figure 4A The flowchart shows the detailed processing of step S302.
[0117] In step S406, the generation unit 302 uses the depth map acquired by the camera device 102 and the coefficients calculated in step S301 to scale the depth map and outputs the scaled depth map with the depth value data.
[0118] In step S407, generation unit 302 uses the scaled data of the depth map obtained in step S406 to generate a synthetic distance map. Note that the data used to generate the synthetic distance map is not limited, and in addition to the scaled depth map in step S406, multiple data such as the distance map and ranging reliability can be used to generate the synthetic distance map.
[0119] For the generation of the composite distance map, the scaled depth values in step S406 are converted into map format data, thereby generating and outputting the composite distance map. Note that the method for generating the composite distance map is not limited, and methods can be used such as those for dividing the distance map and scaled depth map into regions based on ranging reliability, methods for changing the correction ratio of the distance map and scaled depth map based on ranging reliability, or methods for correcting missing data areas in the distance map using the scaled depth map, etc.
[0120] As described above, according to the first embodiment, highly reliable data is extracted from distance and depth information acquired from the same images taken at the same viewpoint and time, and the strengths of this data are utilized while its weaknesses are compensated for, thereby generating an accurate and dense distance map. Therefore, the accuracy of the synthetic distance map generated by the information processing device 103 can be improved, and the reliability of vehicle driving control can be enhanced. Furthermore, when the synthetic distance map output from the information processing device 103 is input to the vehicle control device 104, the vehicle 100 can be controlled or obstacles can be notified to the driver 101.
[0121] On the other hand, in existing technical literature, images captured by camera devices and distance images from LiDAR suffer from time lag and parallax, and the time axis direction and ranging position are not synchronized. For this reason, it is sometimes impossible to generate dense and accurate distance information corresponding to the captured images.
[0122] Similarly, in real-world vehicles, distance information needs to be calculated in real time during driving. When combining distance information obtained from parallax images with depth information estimated from captured images, the processing load is excessive because the time axis direction and ranging position need to be synchronized.
[0123] However, in this embodiment, since two or more distance and depth information obtained from the same captured images from the same viewpoint and at the same time are combined, it is not necessary to synchronize the time axis direction and the ranging position, thereby reducing the amount of processing and shortening the processing time.
[0124] In this embodiment, since the ranging value obtained from the parallax image pair and the captured image are data acquired by a single camera process (shutter release), reliable data from the same viewpoint and at the same time can be obtained. In this case, by matching pixel positions, the depth value estimated from the captured image can be easily aligned with the ranging value.
[0125] Note that this embodiment has described an example of ranging using an image sensor including phase difference detection pixels on the camera surface, but this disclosure is not limited thereto. This embodiment can also be applied to the case of using a stereo camera formed by two or more camera devices. In this case, a depth map can be estimated based on an image captured by any of the camera devices of the stereo camera, and distance information and depth information can be synthesized as in the embodiment.
[0126] Furthermore, this embodiment has described an example of estimating a depth map from a captured image using inference processing of an AI model, but this disclosure is not limited thereto. For example, methods can be used that utilize contrast (bokeh) in the captured image as a key, methods can utilize vanishing points estimated from lines such as roads, methods can utilize objects with known dimensions such as signs, or methods can be used to estimate the depth map by analyzing the captured image. [Second Embodiment]
[0127] In the second embodiment, a configuration example in which the camera device 102 and the information processing device 103 according to the first embodiment are integrated into the camera device 901 will be described.
[0128] Note that in the second embodiment, the same reference numerals as in the first embodiment denote the same or similar components in the second embodiment, and their repeated descriptions will be omitted.
[0129] The camera device 901 is a digital camera or digital video camera. <Device Configuration>
[0130] Figure 9 This is a block diagram illustrating the configuration of the camera device 901 according to this embodiment.
[0131] The camera optical system 10, camera unit 11, control unit 200, RAM 202, ROM 203, storage unit 204, and communication unit 205 are the same as those in the first embodiment. Figure 2 The same as in.
[0132] In addition to the processing according to the first embodiment, the processing unit 902 also performs edge detection processing, virtual camera position setting processing, virtual viewpoint image generation processing, and virtual viewpoint motion image generation processing. The processing unit 902 also includes the processing of the extraction unit 300, the coefficient calculation unit 301, and the generation unit 302 described in the first embodiment. That is, the processing unit 902 has… Figure 3The functions of the extraction unit 300, coefficient calculation unit 301, and generation unit 302 are illustrated. Each of the extraction unit 300, coefficient calculation unit 301, and generation unit 302 can be, for example, a dedicated hardware circuit such as an application-specific integrated circuit (ASIC) designed to implement a specific function. Alternatively, in each of the extraction unit 300, coefficient calculation unit 301, and generation unit 302, the specific function can be implemented by software executed by a processor such as a digital signal processor (DSP) or a graphics processing unit (GPU).
[0133] Input unit 206 is an input device such as a dial, button, switch, or touch panel that is installed on the camera device 901 and can be operated by the user. Operation of input unit 206 is monitored by control unit 200. When an operation is detected on input unit 206, control unit 200 executes an operation corresponding to the function and operation assigned to the operated input device.
[0134] Display unit 207 is a display device such as a liquid crystal display (LCD) or an organic EL display. When continuously capturing moving images and displaying the captured moving images on display unit 207, display unit 207 can be used as an electronic viewfinder (EVF). The operation of using display unit 207 as an electronic viewfinder (EVF) is sometimes referred to as live view display or through-the-lens display. Furthermore, the images displayed on display unit 207 via live view display or through-the-lens display are sometimes referred to as live view images or through-the-lens images.
[0135] Display unit 207 may be a touch display. If display unit 207 is a touch display, soft keys can be implemented by combining a touch panel displayed on display unit 207 with a GUI. Control unit 200 handles soft keys such as input devices included in input unit 206. <Camera Unit Configuration>
[0136] Figure 10 This is a plan view of the pixel array of camera unit 11 as viewed from the camera side.
[0137] Set up primary color Bayer color filters in the pixel array. Any of the red (R), green (G), and blue (B) color filters are arranged on each pixel, such that a 2-row × 2-column pixel group 16 is arranged repeatedly and periodically. Note that color filters with arrangements other than the primary color Bayer arrangement can be set.
[0138] Figure 11 Is with Figure 10 The vertical cross-sectional view of one pixel corresponding to the structure of the I-I' section in the image.
[0139] Each pixel includes a light-guiding layer 19 and a light-receiving layer 20. The light-guiding layer 19 includes a microlens 17 and a color filter 18. The light-receiving layer 20 includes a first photoelectric conversion unit 21 and a second photoelectric conversion unit 22.
[0140] The microlens 17 is configured to efficiently guide the light beam that has entered the pixel to the first photoelectric conversion unit 21 and the second photoelectric conversion unit 22. Furthermore, the color filter 18 corresponds to one of the R filter, G filter, and B filter.
[0141] The first photoelectric conversion unit 21 and the second photoelectric conversion unit 22 each generate a charge corresponding to the amount of incident light. The imaging unit 11 can selectively read signals from one or both of the first photoelectric conversion unit 21 and the second photoelectric conversion unit 22 of each pixel. In this embodiment, the signal obtained from the first photoelectric conversion unit 21 is defined as signal A, the signal obtained from the second photoelectric conversion unit 22 is defined as signal B, and the signal obtained from both the first photoelectric conversion unit 21 and the second photoelectric conversion unit 22 is defined as (A+B) signal.
[0142] Since the first photoelectric conversion unit 21 and the second photoelectric conversion unit 22 view the exit pupil 13 from different viewpoints, the image formed by signal A and the image formed by signal B, read from the same pixel area, form a parallax image pair. By using signals A and B, the defocus amount can be obtained according to the phase difference detection method. Therefore, it can be said that signals A and B are focus detection signals.
[0143] On the other hand, the (A+B) signal corresponds to the signal acquired when a pixel includes a photoelectric conversion unit. Therefore, when acquiring the (A+B) signal from each pixel, an analog image signal can be obtained.
[0144] Note that signal A can also be obtained by subtracting signal B from signal (A+B). Similarly, signal B can be obtained by subtracting signal A from signal (A+B). Therefore, when reading out signals (A+B) and either A or B from each pixel, signals A, B, and (A+B) can be obtained. The type of signal to be read out from each pixel is controlled by control unit 200.
[0145] Note that in Figure 10 and Figure 11In the example shown, each pixel includes two photoelectric conversion units 21 and 22 arranged in the horizontal direction, but this disclosure is not limited thereto. For example, a total of four photoelectric conversion units, including two photoelectric conversion units in the horizontal direction and two photoelectric conversion units in the vertical direction, can be arranged. Furthermore, multiple pairs of pixels, each including a dedicated pixel for generating signal A and a dedicated pixel for generating signal B, can be distributed in the pixel array. The imaging unit 11 can have a known configuration and an arbitrary configuration corresponding to the phase difference AF. <Method for Detecting Phase Difference in Camera Surface>
[0146] Figures 12A to 12E This is a diagram illustrating the camera surface phase difference detection method according to this embodiment.
[0147] The camera phase difference detection method uses A and B signals to calculate the defocus amount.
[0148] Figure 12A This is a schematic diagram showing the relationship between the exit pupil 13 of the camera optical system 10 and the light beam entering the first photoelectric conversion unit 21 of the pixel. Figure 12B This is a schematic diagram showing the relationship between the exit pupil 13 and the light beam entering the second photoelectric conversion unit 22 of the same pixel.
[0149] Note that in this embodiment, the direction parallel to the optical axis of the camera optical system 10 is defined as the z-direction or defocus direction, the direction orthogonal to the optical axis and parallel to the horizontal direction of the imaging surface is defined as the x-direction, and the direction orthogonal to the optical axis and parallel to the vertical direction of the imaging surface is defined as the y-direction.
[0150] Microlens 17 is arranged such that the exit pupil 13 and the light-receiving layer 20 maintain an optical conjugate relationship. The light beam that has passed through the exit pupil 13 of the imaging optical system 10 is focused by microlens 17 and enters either the first photoelectric conversion unit 21 or the second photoelectric conversion unit 22. In this case, as... Figure 12A and Figure 12B As shown, the light beams that have passed through different regions of the exit pupil 13 mainly enter the first photoelectric conversion unit 21 and the second photoelectric conversion unit 22. More specifically, the light beams that have passed through the first pupil region 1210 enter the first photoelectric conversion unit 21, and the light beams that have passed through the second pupil region 1220 enter the second photoelectric conversion unit 22.
[0151] When A and B signals are acquired from each of a plurality of pixels arranged in a horizontal direction with respect to the pixel of interest as the center, the relative positional deviation (phase difference or disparity) between the image signal based on the A signal column (A image) and the image signal based on the B signal column (B image) has a magnitude corresponding to the amount of defocusing of the pixel of interest.
[0152] exist Figures 12C to 12E In the figure, reference numeral 1211 indicates a first light beam passing through a first pupil region 1210; and 1221 indicates a second light beam passing through a second pupil region 1220.
[0153] Figure 12C The focused state is shown, and the first beam 1211 and the second beam 1221 converge on the imaging surface. In this case, the phase difference or parallax between image A and image B is 0.
[0154] exist Figure 12D In this configuration, the first beam 1211 and the second beam 1221 converge on the object side (the negative z-axis side) of the imaging surface. In this case, the phase difference or parallax between image A and image B has a negative value (<0).
[0155] exist Figure 12E In this configuration, when viewed from the object side, the first beam 1211 and the second beam 1221 converge on the far side of the imaging surface (the positive z-axis side). In this case, the phase difference or parallax between image A and image B has a positive value (>0).
[0156] As described above, the phase difference or disparity between image A and image B has a sign corresponding to the relationship between the camera surface and position where the first beam 1211 and the second beam 1221 converge, and a magnitude corresponding to the magnitude of the defocusing amount. The correlation quantity is calculated while the relative offset between image A and image B is made, and the phase difference or disparity between image A and image B can be obtained as a deviation quantity that maximizes the correlation quantity. <Distance Map Acquisition and Processing>
[0157] The control unit 200 can acquire the distance image generated by the processing unit 902 as a distance map to be used for calculating the three-dimensional shape. Alternatively, the control unit 200 and the storage unit 204 can acquire the distance map by reading a pre-recorded distance image.
[0158] The distance map generation process performed by the processing unit 902 includes processing for calculating disparity, processing for converting disparity to defocus, processing for converting defocus to distance values, processing for generating a distance map from the distance values, and processing for generating ranging reliability from the captured image with disparity. The distance map is limited to an image-based distance map. The image-based distance map is generated by phase difference detection pixels on the imaging surface provided in the imaging device 901. Note that the method for calculating disparity is not limited, and a stereo camera can be used. <Depth Map Acquisition and Processing>
[0159] The depth map is estimated from the captured image through the depth map acquisition processing of the processing unit 902. The depth map estimation method is not particularly limited as described in the first embodiment, and can use: methods utilizing neural networks, the contrast or vanishing point of the captured image, or a subject with a known size. <Processing of extraction unit 300, coefficient calculation unit 301 and generation unit 302>
[0160] Reference Figures 4A to 4D , Figures 5A to 5D and Figure 13 The processing of the extraction unit 300, the coefficient calculation unit 301, and the generation unit 302, which are executed in the processing unit 902, are described.
[0161] Note that the following will mainly describe the process that differs from the first embodiment.
[0162] When the camera device 901 generates the distance map and depth map, it begins... Figures 4A to 4D The processing shown. (Processing of extraction unit 300)
[0163] exist Figure 4A In step S300, as in the first embodiment, highly reliable data is extracted from the distance map and depth map.
[0164] Figure 4B This is an example in Figure 4A The flowchart shows the process performed in step S300. Figure 5A This is an example in Figure 4B The flowchart shows the process performed in step S400.
[0165] In step S500, unlike the first embodiment, the extraction unit 300 only sets the area with high reliability originating from the camera optical system 10 of the camera device 901. This is because the camera device 901 captures various camera scenes, and therefore cannot set assumptions for specific camera scenes.
[0166] Next, in Figure 4B In step S401, as in the first embodiment, the extraction unit 300 extracts a distance map for the effective area extracted in step S400 and extracts the distance values within the effective distance measurement range. Unlike the first embodiment, in this case, the effective distance measurement range is set by an AF evaluation value, such as focal length, calculated when the camera device 901 is recording. Furthermore, the effective distance measurement range can be adjusted according to the surrounding environment during recording (e.g., the brightness of the captured image or the temperature during recording).
[0167] Figure 13 This is an example that differs from the first embodiment. Figure 4BThe flowchart for step S402 in the process.
[0168] In step S509, the extraction unit 300 searches for regions with depth values that can be recognized as the main subject from the depth value data of the depth map, and sets the main subject region.
[0169] Figure 14 This is an explanation Figure 13 The diagram shows the method for setting the main subject area in step S509.
[0170] Using a main subject region search method, semantic segmentation is employed to detect the main subject region 1401 from the captured image 1400. Figure 14 (The shadow area in the image) and set the main subject area to 1401. Note that the main subject area detection method is not limited.
[0171] In step S510, the extraction unit 300 extracts the data of the main subject area set in step S509 from the depth map and outputs the extracted data as an extracted depth map.
[0172] Next, in Figure 4B In step S403, the same processing as in the first embodiment is performed. (Processing of coefficient calculation unit 301)
[0173] exist Figure 4A In step S301, the same processing as in the first embodiment is performed.
[0174] Figure 4C Examples are shown in Figure 4A The processing performed in step S301.
[0175] In step S405, as in the first embodiment, the coefficient calculation unit 301 uses the fitting coefficients calculated in step S404 and the threshold of the fitting coefficients to determine the reliability of the fitting coefficients. After processing, the calculated coefficients are output as coefficients with high reliability. The method for setting the threshold of the fitting coefficients used in this case is different from the method in the first embodiment, and the threshold is set using the distance map and depth map acquired by the camera device 901. For the threshold of the fitting coefficients, the coefficients with high reliability are set among the coefficients processed when scaling the depth map in subsequent stages, based on indicators such as the AF evaluation value (e.g., focal length) calculated when the camera device 901 is taking pictures, or the distance value of the nearest subject. (Processing of generation unit 302)
[0176] exist Figure 4A In step S302, the same processing as in the first embodiment is performed.
[0177] As described above, according to the second embodiment, a dense and accurate distance map can be obtained. Therefore, the distance map can be used to blur the background of the main subject in a captured image or to generate a 3D model, and various applications are possible. [Other Embodiments]
[0178] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.
[0179] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.
Claims
1. An information processing device, comprising: The acquisition unit is used to acquire a first distance map and a second distance map, wherein the first distance map indicates first distance information corresponding to a position in the captured image, and the second distance map indicates second distance information corresponding to a position in the captured image; as well as The generation unit is configured to generate a third distance map based on the first distance information of the first distance map and the second distance information of the second distance map.
2. The information processing device according to claim 1, wherein, The first distance map is generated based on distance information obtained from image pairs with parallax.
3. The information processing device according to claim 1, wherein, The first distance map includes distance information at sparse locations in the captured image, and The second distance map includes distance information for dense locations in the captured image.
4. The information processing device according to claim 1, further comprising: The extraction unit is configured to extract the first distance information from the first distance map and extract the second distance information from the second distance map; as well as The calculation unit is used to calculate coefficients that match the scale of the first distance information extracted by the extraction unit with the scale of the second distance information. The generation unit scales the first distance information and the second distance information based on the coefficients, and generates the third distance map.
5. The information processing device according to claim 4, wherein, The extraction unit extracts the first distance information and the second distance information, each with a reliability of not less than a predetermined threshold, from the first distance map and the second distance map.
6. The information processing device according to claim 4, wherein, The calculation unit uses at least one of the first distance information and the second distance information extracted by the extraction unit to fit the first distance information and the second distance information, thereby calculating the coefficient.
7. The information processing device according to claim 6, wherein, The calculation unit uses a threshold to determine the reliability of the coefficient, and repeats the calculation of the coefficient until the coefficient falls within the range of the threshold.
8. The information processing device according to claim 1, wherein, The second distance map is generated by reasoning through a trained artificial intelligence model, or AI model.
9. The information processing device according to claim 1, wherein, The second distance map is generated by analyzing at least the contrast, vanishing point, and size of the subject in the captured image.
10. The information processing device according to claim 4, wherein, The extraction unit extracts the first distance information of the effective region in the captured image from the first distance map.
11. The information processing device according to claim 4, wherein, The extraction unit extracts the first distance information from the effective ranging range in the first distance map.
12. The information processing device according to claim 4, wherein, The extraction unit extracts distance information where the local variation of the first distance information is less than a predetermined range.
13. The information processing device according to claim 4, wherein, The extraction unit extracts the second distance information of the regions in the captured image other than the background region from the second distance map.
14. The information processing device according to claim 4, wherein, The extraction unit extracts the second distance information of the main subject region in the captured image from the second distance map.
15. The information processing device according to claim 4, wherein, The extraction unit extracts information about the common region in the captured image from the first distance information and the second distance information.
16. The information processing device according to claim 15, wherein, The extraction unit extracts information about the common region through logical operations on the first distance information and the second distance information.
17. The information processing apparatus according to claim 15, wherein, The calculation unit determines the reasonableness of the information in the common region extracted by the extraction unit, and uses the information in the common region from which information determined to be unreasonable has been removed to calculate the coefficient.
18. The information processing apparatus according to claim 17, wherein, The reasonableness is determined using a threshold set based on distance information to the subject in the captured image.
19. The information processing apparatus according to claim 17, wherein, The reasonableness is determined using a threshold set based on an evaluation value of the autofocus control of the subject in the captured image.
20. The information processing device according to claim 1, wherein, The first distance map and the second distance map were obtained from the same images taken at the same viewpoint and at the same time.
21. A camera device, comprising: Information processing equipment; as well as The camera unit is used to generate captured images. The information processing device includes: The acquisition unit is configured to acquire a first distance map and a second distance map, wherein the first distance map indicates first distance information corresponding to a position in the captured image, and the second distance map indicates second distance information corresponding to a position in the captured image; and The generation unit is configured to generate a third distance map based on the first distance information of the first distance map and the second distance information of the second distance map.
22. A mobile device, comprising: Information processing equipment; Camera equipment used to generate captured images; as well as A control device for controlling the mobile device based on a third distance map generated by the information processing device. The information processing device includes: The acquisition unit is configured to acquire a first distance map and a second distance map, wherein the first distance map indicates first distance information corresponding to a position in the captured image, and the second distance map indicates second distance information corresponding to a position in the captured image; and The generation unit is configured to generate a third distance map based on the first distance information of the first distance map and the second distance information of the second distance map.
23. An information processing method, executed by an information processing device, the information processing method comprising: The steps for obtaining a first distance map and a second distance map are as follows: the first distance map indicates first distance information corresponding to a position in the captured image, and the second distance map indicates second distance information corresponding to a position in the captured image; as well as The step of generating a third distance map based on the first distance information and the second distance information.
24. A computer-readable storage medium storing a program that, when executed by a computer, causes the computer to implement the information processing method according to claim 23.
25. A computer program product comprising a program that, when executed by a computer, causes the computer to perform the information processing method according to claim 23.
