A depth map generation method and device, electronic equipment and storage medium

Abstract

This application provides a depth map generation method, apparatus, electronic device, and storage medium. It acquires a first depth map from a time-of-flight module and a second depth map from a binocular module. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information describes the depth of pixels in the visible light image. By using a preset coordinate system transformation, the first depth information is mapped onto the visible light image to obtain third depth information of pixels in the visible light image. The second and third depth information of pixels in the visible light image are then fused to obtain a fused depth map. This method combines the advantages of both a time-of-flight module and a binocular module, avoiding the shortcomings of using only one module and improving the effectiveness and reliability of the depth information.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a depth map generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Depth maps are crucial in computer vision and image processing, providing distance information about objects in a scene. They are invaluable for many applications, such as 3D reconstruction, virtual reality, autonomous driving, medical imaging, layer segmentation, and face recognition. The following sections briefly describe these applications in different fields.

[0003] 1. 3D Reconstruction: Depth maps play a crucial role in 3D reconstruction. By capturing the depth information of each pixel in a scene, a 3D model of the scene can be generated. Applications include building scanning, cultural heritage preservation, and virtual museums.

[0004] 2. Virtual Reality: In virtual reality, depth maps are used to create realistic virtual environments and interactive experiences. They are applied in games, virtual tourism, training simulators, etc.

[0005] 3. Autonomous Driving: Depth maps are used for environmental perception and navigation in autonomous driving. They can be used for obstacle detection, path planning, lane keeping, etc.

[0006] 4. Medical Imaging: In medical imaging, depth maps are used to obtain three-dimensional information about the internal structures of the human body. They can be applied to surgical navigation, tumor detection, bone scanning, and more.

[0007] 5. Layer Segmentation: Depth maps are used in layer segmentation to divide an image into different levels or regions. Applications include background replacement, foreground extraction, and image editing.

[0008] 6. Facial Recognition: In facial recognition, depth maps are used to capture the three-dimensional structure of a face, improving the accuracy and robustness of recognition. It is applied in security monitoring, identity verification, social media, and other fields.

[0009] Depth maps, acquired through various sensors or computer vision algorithms, help computers understand the spatial structure of a scene and the positional relationships of objects. Currently, depth maps are typically acquired using TOF (Time of Flight) cameras or stereo depth cameras. TOF cameras perform well in recognizing the depth information of white objects, but their performance is easily affected by surface reflection, strong light, and device power. For example, TOF cameras cannot obtain effective depth information for black objects that are slightly further away within their effective range due to insufficient reflected light. To address this, stereo depth cameras employ projected speckle, which provides better depth information recognition for black objects. While projected speckle improves the depth acquisition of black areas, stereo depth cameras cannot obtain effective depth information for white objects that are slightly further away within their effective range. Furthermore, stereo depth cameras cannot estimate depth in repetitive areas, weak textures, or low-light conditions. Summary of the Invention

[0010] The purpose of this application is to provide a depth map generation method, apparatus, electronic device, and storage medium to solve the problem of poor depth information measurement results. The specific technical solution is as follows:

[0011] Firstly, in order to achieve the above objectives, embodiments of this application provide a depth map generation method, the method comprising:

[0012] The first depth map acquired by the time-of-flight module and the second depth map acquired by the binocular module are obtained. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information is used to describe the depth of pixels in the visible light image.

[0013] By using a preset coordinate system transformation relationship, the first depth information is mapped onto the visible light image to obtain the third depth information of the pixels in the visible light image;

[0014] The second and third depth information of pixels in the visible light image are fused to obtain a fused depth map.

[0015] The depth map generation method provided in this application acquires a first depth map collected by a time-of-flight module and a second depth map collected by a binocular module. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information describes the depth of pixels in the visible light image. By using a preset coordinate system transformation relationship, the first depth information is mapped onto the visible light image to obtain third depth information of pixels in the visible light image. The second and third depth information of pixels in the visible light image are then fused to obtain a fused depth map. This method combines the advantages of both a time-of-flight module and a binocular module, avoiding the shortcomings of using only one module and improving the effectiveness and reliability of the depth information.

[0016] In one possible implementation, the method further includes:

[0017] Upsample the first depth map;

[0018] Invalid values ​​are filled into the upsampled first depth map to obtain a filled first depth map, wherein the filled first depth map has the same resolution as the second depth map.

[0019] In this embodiment, the first depth map is upsampled; invalid values ​​are then filled into the upsampled first depth map to obtain a filled first depth map, wherein the filled first depth map has the same resolution as the second depth map. This ensures that each pixel in the second depth map has corresponding first depth information, facilitating subsequent depth information fusion.

[0020] In one possible implementation, fusing the second and third depth information of pixels in the visible light image to obtain a fused depth map includes:

[0021] For any pixel in a visible light image, if the value of the third depth information of that pixel is invalid, then the value of the second depth information of that pixel is used as the depth value of that pixel in the fused depth map.

[0022] In this embodiment, for any pixel in a visible light image, if one of the two depth maps has an invalid value for that pixel, then a valid value is used to fill the fusion depth of that pixel. This ensures that the pixel is not invalidated.

[0023] In one possible implementation, fusing the second and third depth information of pixels in the visible light image to obtain a fused depth map includes:

[0024] For any pixel in a visible light image, if the values ​​of the second depth information and the third depth information of the pixel are both valid values ​​and the difference is less than a preset value, the confidence level of the pixel is determined.

[0025] Based on the confidence level, the values ​​of the second depth information and the third depth information are weighted to obtain the depth value of the pixel in the fused depth map.

[0026] In this embodiment, for any pixel in a visible light image, if both the second and third depth information values ​​of the pixel are valid and the difference is less than a preset value, the confidence level of the pixel is determined. Based on the confidence level, the values ​​of the second and third depth information are weighted to obtain the depth value of the pixel in the fused depth map. This makes the value of the second or third depth information more accurate at the pixel, allowing the depth value obtained by fusing the second and third depth information to better reflect the true depth of the pixel and improving the reliability of the depth information.

[0027] In one possible implementation, determining the confidence level of the pixel includes:

[0028] Pixels within a preset range centered on the pixel are obtained from the second depth information and the third depth information respectively, to obtain the first neighborhood and the second neighborhood;

[0029] Obtain the semantic segmentation map corresponding to the visible light image;

[0030] Based on the different pixel values ​​corresponding to different semantic categories in the semantic segmentation map, the depth values ​​of pixels in the first neighborhood and the second neighborhood that are not of the same category as the pixel are set to invalid values.

[0031] The effective value rate and average effective value of the first neighborhood and the second neighborhood are calculated respectively. The effective value rate represents the number of pixels with effective depth values ​​in the neighborhood, and the average effective value represents the average depth value of pixels with effective depth values ​​in the neighborhood.

[0032] The confidence level of a pixel is determined based on the difference between the effective value rate, the average effective value, and the depth value of the pixel. The confidence level is directly proportional to the effective value rate and inversely proportional to the difference. The depth value of the pixel includes the value of the second depth information and the value of the third depth information of the pixel.

[0033] In this embodiment, pixels within a preset range centered on the pixel are obtained from the second depth information and the third depth information to obtain a first neighborhood and a second neighborhood; a semantic segmentation map corresponding to the visible light image is obtained; based on different pixel values ​​corresponding to different semantic categories in the semantic segmentation map, the depth values ​​of pixels in the first neighborhood and the second neighborhood that are not of the same category as the pixel are set to invalid values; the effective value rate and average effective value of the first neighborhood and the second neighborhood are calculated respectively, where the effective value rate represents the number of pixels with effective depth values ​​in the neighborhood, and the average effective value represents the average depth value of pixels with effective depth values ​​in the neighborhood; based on the difference between the effective value rate, the average effective value, and the depth value of the pixel, the confidence level of the pixel is determined, where the confidence level is directly proportional to the effective value rate and inversely proportional to the difference, wherein the depth value of the pixel includes the value of the second depth information and the value of the third depth information of the pixel. The confidence level of a pixel is obtained by combining the confidence level with that of its surrounding pixels. Since semantic segmentation information is introduced in the confidence level calculation process, the jump and trailing problems that occur in subsequent depth interpolation are well avoided.

[0034] In one possible implementation, fusing the second and third depth information of pixels in the visible light image to obtain a fused depth map includes:

[0035] For any pixel in a visible light image, if the values ​​of the second depth information and the third depth information of the pixel are both valid values ​​and the difference is not less than a preset value;

[0036] In the second depth information, pixels within a preset range centered on the pixel are obtained to form the first neighborhood.

[0037] Based on the depth values ​​of the pixels in the first neighborhood, determine the average depth value and variance of the first neighborhood;

[0038] If the average depth value is less than the preset depth value, then the value of the second depth information of the pixel is used as the depth value of the pixel in the fused depth map.

[0039] If the variance is less than the preset variance value, then the value of the third depth information of the pixel is used as the depth value of the pixel in the fused depth map.

[0040] In this embodiment, for any pixel in a visible light image, if both the second and third depth information values ​​of the pixel are valid and the difference is not less than a preset value; in the second depth information, pixels within a preset range centered on the pixel are obtained to form a first neighborhood; based on the depth values ​​of the pixels in the first neighborhood, the average depth value and variance of the first neighborhood are determined; if the average depth value is less than a preset depth value, the value of the second depth information of the pixel is used as the depth value of the pixel in the fused depth map; if the variance is less than a preset variance value, the value of the third depth information of the pixel is used as the depth value of the pixel in the fused depth map. Utilizing the average depth value and variance of the neighborhood to assist in pixel depth fusion ensures that different depth information is used for different situations, resulting in a more accurate fused depth value.

[0041] In one possible implementation, acquiring the first depth map acquired by the time-of-flight module and the second depth map acquired by the binocular module includes:

[0042] The system acquires a first depth map under low-light conditions collected by the time-of-flight module and a second depth map under low-light conditions collected by the binocular module. The first depth map and the second depth map include white vehicles and black vehicles.

[0043] The method further includes:

[0044] The distance information of black / white vehicles is calculated based on the fused depth map.

[0045] The method provided in this application embodiment can achieve better depth measurement results for black / white vehicles under low-light conditions, overcoming the shortcomings of using only one sensor.

[0046] Secondly, embodiments of this application provide a depth map generation apparatus, the apparatus comprising:

[0047] The acquisition module is used to acquire a first depth map collected by the time-of-flight module and a second depth map collected by the binocular module. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information is used to describe the depth of pixels in the visible light image.

[0048] The mapping module is used to map the first depth information onto the visible light image through a preset coordinate system transformation relationship, so as to obtain the third depth information of the pixels in the visible light image;

[0049] The fusion module is used to fuse the second depth information and the third depth information of the pixels in the visible light image to obtain a fused depth map.

[0050] Thirdly, embodiments of this application also provide an electronic device, including:

[0051] One or more processors and memory;

[0052] The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, which the one or more processors call to cause the electronic device to execute the depth map generation method described in any of the first aspects above.

[0053] Fourthly, embodiments of this application also provide a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform any of the depth map generation methods described in the first aspect.

[0054] The depth map generation method provided in this application acquires a first depth map collected by a time-of-flight module and a second depth map collected by a binocular module. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information describes the depth of pixels in the visible light image. By using a preset coordinate system transformation relationship, the first depth information is mapped onto the visible light image to obtain third depth information of pixels in the visible light image. The second and third depth information of pixels in the visible light image are then fused to obtain a fused depth map. This method combines the advantages of both a time-of-flight module and a binocular module, avoiding the shortcomings of using only one module and improving the effectiveness and reliability of the depth information. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1a A depth map acquired by a TOF sensor;

[0057] Figure 1b A depth map acquired by a binocular stereo depth camera;

[0058] Figure 1c A hardware structure diagram of an electronic device provided in an embodiment of this application;

[0059] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0060] Figure 3 A schematic flowchart of a depth map generation method provided in an embodiment of this application;

[0061] Figure 4a The first depth map obtained by the TOF sensor;

[0062] Figure 4b The second depth map obtained by the binocular stereo depth camera;

[0063] Figure 4c A fused depth map obtained by applying the depth map generation method provided in the embodiments of this application;

[0064] Figure 5 A schematic diagram illustrating the calculation of effective value rate and average effective value provided in the embodiments of this application;

[0065] Figure 6 It is a map of a real-world environment;

[0066] Figure 7 A camera calibration method provided in this application embodiment;

[0067] Figure 8 A method for fusing depth maps provided in this application embodiment. Detailed Implementation

[0068] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0069] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. For example, "first instruction" and "second instruction" are used to distinguish different user instructions and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0070] It should be noted that, in this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0071] Depth maps are crucial in computer vision and image processing, providing distance information about objects in a scene. They are invaluable for many applications, such as 3D reconstruction, virtual reality, autonomous driving, medical imaging, layer segmentation, and face recognition. The following sections briefly describe these applications in different fields.

[0072] 1. 3D Reconstruction: Depth maps play a crucial role in 3D reconstruction. By capturing the depth information of each pixel in a scene, a 3D model of the scene can be generated. Applications include building scanning, cultural heritage preservation, and virtual museums.

[0073] 2. Virtual Reality: In virtual reality, depth maps are used to create realistic virtual environments and interactive experiences. They are applied in games, virtual tourism, training simulators, etc.

[0074] 3. Autonomous Driving: Depth maps are used for environmental perception and navigation in autonomous driving. They can be used for obstacle detection, path planning, lane keeping, etc.

[0075] 4. Medical Imaging: In medical imaging, depth maps are used to obtain three-dimensional information about the internal structures of the human body. They can be applied to surgical navigation, tumor detection, bone scanning, and more.

[0076] 5. Layer Segmentation: Depth maps are used in layer segmentation to divide an image into different levels or regions. Applications include background replacement, foreground extraction, and image editing.

[0077] 6. Facial Recognition: In facial recognition, depth maps are used to capture the three-dimensional structure of a face, improving the accuracy and robustness of recognition. It is applied in security monitoring, identity verification, social media, and other fields.

[0078] Depth maps, acquired through various sensors or computer vision algorithms, help computers understand the spatial structure of a scene and the positional relationships of objects. Currently, depth maps are typically acquired using TOF (Time of Flight) cameras or stereo depth cameras. TOF cameras generally perform well in recognizing the depth information of white objects, but their performance is easily affected by surface reflection, strong light, and device power. For example, TOF cameras may fail to obtain effective depth information for black objects at a slightly greater distance within their effective range due to insufficient reflected light. (See [link to documentation]). Figure 1a To address this issue, binocular stereo depth cameras employ projected speckle, which provides good depth information recognition for black objects. While projected speckle improves depth acquisition in black areas, binocular stereo depth cameras cannot obtain effective depth information for white objects at a slightly greater distance within their effective range. (See [link to article]). Figure 1b Furthermore, binocular stereo depth cameras are unable to estimate depth in situations involving repetitive areas, weak textures, or low light.

[0079] Based on this, embodiments of this application provide a depth map generation method, which can be applied to electronic devices. These electronic devices are those with image acquisition capabilities, such as smartphones, tablets, and smartwatches. The solution provided in this application includes: acquiring a first depth map acquired by a time-of-flight module and a second depth map acquired by a binocular module, wherein the first depth map contains first depth information, and the second depth map contains a visible light image and second depth information, the second depth information being used to describe the depth of pixels in the visible light image; mapping the first depth information onto the visible light image through a preset coordinate system transformation relationship to obtain third depth information of pixels in the visible light image; and fusing the second and third depth information of pixels in the visible light image to obtain a fused depth map.

[0080] The structure of the electronic device used in the above depth map generation method is described below.

[0081] For example, Figure 1c A structural diagram of electronic device 100 is shown. Electronic device 100 may include a processor 110, a display screen 120, a camera 130, internal memory 140, a SIM (Subscriber Identification Module) card interface 150, a USB (Universal Serial Bus) interface 160, a charging management module 170, a battery management module 171, a battery 172 with battery cells and battery protection devices, a sensor module 180, a mobile communication module 190, a wireless communication module 200, antenna 1, and antenna 2, etc. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, etc.

[0082] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0083] Processor 110 may include one or more processing units, such as a CPU (Central Processing Unit), AP (Application Processor), modem processor, GPU (Graphics Processing Unit), ISP (Image Signal Processor), controller, video codec, DSP (Digital Signal Processor), baseband processor, and / or NPU (Neural-network Processing Unit). Different processing units may be independent components or integrated into one or more processors. In some embodiments, electronic device 100 may also include one or more processors 110. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. In other embodiments, processor 110 may also include a memory for storing instructions and data. For example, the memory in processor 110 may be a cache memory. This memory can store instructions or data that processor 110 has just used or is repeatedly used. If processor 110 needs to reuse the instruction or data, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100 in processing data or executing instructions.

[0084] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an I2C (Inter-Integrated Circuit) interface, an I2S (Inter-Integrated Circuit Sound) interface, a PCM (Pulse Code Modulation) interface, a UART (Universal Asynchronous Receiver / Transmitter) interface, a MIPI (Mobile Industry Processor Interface) interface, a GPIO (General-Purpose Input / Output) interface, a SIM card interface, and / or a USB interface. The USB interface 160 is a USB standard-compliant interface, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface 160 can be used to connect a charger to charge the electronic device 100, and can also be used for data transfer between the electronic device 100 and peripheral devices. The USB interface 160 can also be used to connect headphones for audio playback.

[0085] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are for illustrative purposes only and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0086] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 190, wireless communication module 200, modem processor and baseband processor, etc.

[0087] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0088] Electronic device 100 implements display functions through a GPU, display screen 120, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0089] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel can be an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), Active Matrix Organic Light-Emitting Diode, AMOLED (Active-Matrix Organic Light-Emitting Diode), FLED (Flexible Light-Emitting Diode), MiniLED, MicroLED, Micro-OLED, QLED (Quantum Dot Light-Emitting Diodes), etc. In some embodiments, the electronic device 100 may include one or more display screens 120.

[0090] In some embodiments of this application, when the display panel uses materials such as OLED, AMOLED, and FLED, the above-mentioned Figure 1c The display screen 120 can be bent. Here, "the display screen 120 can be bent" means that the display screen can be bent to any angle at any part and can maintain that angle. For example, the display screen 120 can be folded from the middle left to right. It can also be folded from the middle up and down.

[0091] The display screen 120 of electronic device 100 can be a flexible screen. Currently, flexible screens are attracting much attention due to their unique characteristics and enormous potential. Compared to traditional screens, flexible screens are highly flexible and bendable, providing users with new interaction methods based on their bendability and meeting more user needs for electronic devices. For electronic devices equipped with foldable displays, the foldable display can switch between a small screen in a folded state and a large screen in an unfolded state at any time. Therefore, users are increasingly using split-screen functionality on electronic devices equipped with foldable displays.

[0092] Electronic device 100 can perform shooting functions through ISP, camera 130, video codec, GPU, display 120 and application processor, wherein camera 130 includes a front camera and a rear camera.

[0093] The ISP is used to process data fed back from the camera 130. For example, during shooting, when the shutter is opened, light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 130.

[0094] Camera 130 is used to capture photos or videos. An object is projected onto a photosensitive element through a lens, generating an optical image. The photosensitive element can be a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard RGB (Red, Green, Blue) or YUV (a color encoding method) formats. In some embodiments, the electronic device 100 may include one or N cameras 130, where N is a positive integer greater than 1.

[0095] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0096] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as MPEG (Moving Picture Experts Group) 1, MPEG2, MPEG3, and MPEG4.

[0097] An NPU (Neural Processing Unit) is a neural network computing processor that, by drawing inspiration from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, rapidly processes input information and can continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0098] Internal memory 140 can be used to store one or more computer programs, which include instructions. Processor 110 can execute the aforementioned instructions stored in internal memory 140, thereby causing electronic device 100 to perform the depth map generation method provided in some embodiments of this application, as well as various applications and data processing. Internal memory 140 may include a program storage area and a data storage area. The program storage area may store the operating system; the program storage area may also store one or more applications (such as a gallery, contacts, etc.). The data storage area may store data created during the use of electronic device 100 (such as photos, contacts, videos, etc.). In addition, internal memory 140 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage components, flash memory components, general-purpose flash memory, etc. In some embodiments, processor 110 can execute instructions stored in internal memory 140 and / or instructions stored in memory disposed in processor 110, thereby causing electronic device 100 to perform the depth map generation method provided in embodiments of this application, as well as other applications and data processing.

[0099] The internal memory 140 can be used to store the relevant program of the depth map generation method provided in the embodiments of this application. The processor 110 can be used to call the relevant program of the depth map generation method stored in the internal memory 140 when displaying information, and execute the depth map generation method of the embodiments of this application.

[0100] The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, etc.

[0101] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 120. Pressure sensor 180A can be of many types, such as resistive pressure sensor, inductive pressure sensor, or capacitive pressure sensor. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes, and electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 120, electronic device 100 detects the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed; when a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0102] The fingerprint sensor 180B is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to perform functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0103] Touch sensor 180C, also known as a touch device, can be disposed on display screen 120. The touch sensor 180C and display screen 120 together form a touchscreen, also known as a touch display. Touch sensor 180C is used to detect touch operations applied to or near it. Touch sensor 180C can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 120. In other embodiments, touch sensor 180C may also be disposed on the surface of electronic device 100 and in a different location from display screen 120.

[0104] The ambient light sensor 180D is used to sense the ambient light intensity. The electronic device 100 can adaptively adjust the brightness of the display screen 120 based on the sensed ambient light intensity. The ambient light sensor 180D can also be used to automatically adjust the white balance during shooting. The ambient light sensor 180D can also transmit environmental information about the device's location to the GPU.

[0105] The ambient light sensor 180D is also used to acquire the brightness, light ratio, color temperature, and other parameters of the environment in which the camera 130 captures images.

[0106] Figure 2This is a software architecture block diagram for an electronic device to which embodiments of this application apply. The software system of the electronic device may adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture.

[0107] The following explanation uses the software system of an electronic device, divided into hardware, driver, data link, and application layers, as an example. The hardware layer includes a depth sensor, which is used for depth measurement. The hardware layer is responsible for two main parts: extracting depth data and processing image signals. The depth sensor sends the extracted sensor data to the driver layer, which includes depth sensor drivers, such as ToF sensors and stereo depth cameras. The depth sensor driver sends the raw depth data and images to the data link layer, which includes image algorithm libraries and depth fusion algorithms. The depth fusion algorithm performs a depth fusion operation based on the received raw depth data, images, and algorithms from the image algorithm library, and sends the fused depth data to the application layer. The application layer includes various algorithm applications, such as autonomous driving algorithms, VR (Virtual Reality), and 3D reconstruction.

[0108] Understandable, Figure 2 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0109] It is understood that, in order to implement the depth map generation method provided in the embodiments of this application, the electronic device includes hardware and / or software modules that perform various functions. Based on the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0110] The depth map generation method provided in this application will be described in detail below through specific embodiments.

[0111] See Figure 3 This application provides a depth map generation method, the method comprising:

[0112] S201. Acquire the first depth map collected by the time-of-flight module and the second depth map collected by the binocular module.

[0113] The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information is used to describe the depth of pixels in the visible light image.

[0114] The time-of-flight module, integrated into the time-of-flight sensor, acquires infrared images and depth maps, while the binocular module, integrated into the binocular stereo depth camera, acquires visible light images and depth maps. Both the time-of-flight sensor and the binocular stereo depth camera were pre-calibrated before use.

[0115] The calibration process generally includes the following steps: TOF sensor data acquisition: Using a TOF sensor to acquire distance data of objects in the environment. Binocular stereo depth camera image acquisition: Simultaneously using a binocular stereo depth camera to acquire stereo images of the environment. Data fusion and calibration: Fusing the distance data measured by the TOF sensor with the images from the binocular stereo depth camera, performing time synchronization and spatial calibration to ensure the consistency and accuracy of the data. Specifically, this includes: After fixing the relative positions of the TOF sensor and the binocular stereo depth camera, calibrating the intrinsic parameters of the TOF sensor and the binocular stereo depth camera, as well as the coordinate system transformation relationship between them. Simultaneously acquiring calibration data from the TOF sensor and the binocular stereo depth camera, upsampling the infrared image from the TOF sensor and filling in invalid values ​​to align its resolution with the depth map acquired by the binocular stereo depth camera. Matching the synchronously acquired calibration data from the TOF sensor and the binocular stereo depth camera, calibrating the TOF sensor and the binocular stereo depth camera using camera intrinsic parameter calibration methods and stereo calibration methods.

[0116] Camera intrinsic parameter calibration includes: preparing a calibration board: using a calibration board, such as a checkerboard, ensuring it contains enough corner points for detection and calculation. Image capture: simultaneously capturing images of the calibration board using a ToF sensor and a stereo depth camera at different angles and positions. Corner detection: using a calibration module from an image processing library (such as OpenCV), detecting the corner points on the calibration board in each image. Intrinsic parameter calculation: based on the pixel coordinates of the corner points and the known dimensions of the calibration board, calculating the intrinsic parameters of the ToF sensor, the left stereo depth camera, and the right stereo depth camera, such as the camera's focal length, principal point position, and lens distortion parameters.

[0117] The stereo calibration method includes: acquiring stereo images: using a TOF sensor, a left-eye stereo depth camera, and a right-eye stereo depth camera to acquire stereo images containing the scene; matching feature points: extracting and matching feature points from the three images to determine their correspondence; calculating extrinsic parameters: based on known intrinsic parameters and the stereo matching results of the feature points, calculating the relative poses between the TOF sensor, the left-eye stereo depth camera, and the right-eye stereo depth camera, including rotation matrices and translation vectors.

[0118] S202. By using a preset coordinate system transformation relationship, the first depth information is mapped onto the visible light image to obtain the third depth information of the pixels in the visible light image.

[0119] Both the time-of-flight sensor and the stereo depth camera were pre-calibrated before use. This is because the time-of-flight sensor and the stereo depth camera may use different coordinate systems. In order to use the depth maps acquired by the time-of-flight sensor and the stereo depth camera simultaneously, these two depth maps need to be transformed into the same coordinate system for processing.

[0120] After acquiring the first depth map from the time-of-flight module and the second depth map from the binocular module, the first depth information is mapped onto the visible light image through a preset coordinate system transformation relationship. Since this application combines the first depth map acquired by the time-of-flight module and the second depth map acquired by the binocular module to obtain a new depth map, it is necessary to map the first depth information onto the second depth information, so that each pixel in the visible light image corresponds to two depth values. Then, a new depth value can be obtained by fusing the two depth values.

[0121] S203. The second depth information and the third depth information of the pixels in the visible light image are fused to obtain a fused depth map.

[0122] For each pixel in the visible light image, different fusion strategies are adopted according to different situations to fuse the second depth information and the third depth information of the pixel to obtain a fused depth map.

[0123] The depth map generation method provided in this application acquires a first depth map collected by a time-of-flight module and a second depth map collected by a binocular module. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information describes the depth of pixels in the visible light image. By using a preset coordinate system transformation relationship, the first depth information is mapped onto the visible light image to obtain third depth information of pixels in the visible light image. The second and third depth information of pixels in the visible light image are then fused to obtain a fused depth map. This method combines the advantages of both a time-of-flight module and a binocular module, avoiding the shortcomings of using only one module and improving the effectiveness and reliability of the depth information.

[0124] In one example, see Figure 4a This is the depth map obtained by the TOF sensor. The depth map obtained by the TOF sensor has low resolution and is difficult to obtain effective depth for low reflective targets. Figure 4b This is a depth map obtained from a stereo depth camera. Stereo depth cameras are not effective at measuring depth in textureless or repetitive patterned areas. By applying the depth map generation method provided in this application to fuse the depth map generated by the TOF sensor and the depth map generated by the stereo depth camera, a fused image is obtained. This combines the advantages of both methods, avoids the shortcomings of using only one, and improves the effectiveness and reliability of the depth information. See [link to relevant documentation]. Figure 4c .

[0125] In one example, the method further includes:

[0126] Upsample the first depth map;

[0127] Invalid values ​​are filled into the upsampled first depth map to obtain a filled first depth map, wherein the filled first depth map has the same resolution as the second depth map.

[0128] Upsampling refers to the process of converting an image from low resolution to high resolution. In image processing, upsampling typically involves interpolation operations to increase the number of pixels and details in an image, thereby improving image sharpness and quality. During upsampling, new pixel values ​​may be generated, and these areas need to be filled with invalid values; the original image may also contain invalid values ​​or areas with uncertain depth, which also need to be filled with invalid values; to ensure that the first and second depth maps have the same resolution, some areas also need to be filled with invalid values ​​so that these areas can be identified and processed in subsequent processing.

[0129] In depth image processing, invalid values ​​are typically used to represent regions in a depth image where no valid depth information can be obtained. These regions may include:

[0130] Areas outside the effective depth range: For example, a depth camera may not be able to measure depth correctly at a distance or at a specific angle, resulting in invalid depth values ​​in these areas.

[0131] Occluded area: The area where depth information cannot be obtained because it is obscured by objects or obstacles.

[0132] Areas that the sensor failed to detect correctly: For example, a TOF sensor may have inaccurate or invalid depth information in some areas due to issues such as ambient light and reflection.

[0133] Invalid values ​​can typically be represented in the following way.

[0134] Distance value: Sets a specific distance value, such as 0 or a very large number, indicating that pixels within that distance range are considered invalid. Flag bit: Uses an additional channel or flag bit to mark invalid pixels. Typically, in a grayscale image of a depth map, the RGB channels are preserved, and an additional channel represents depth information.

[0135] After invalid values ​​are filled into the first depth map after upsampling, so that the resolution of the first depth map after filling is the same as that of the second depth map, the above step S202 can be performed.

[0136] In this embodiment, the first depth map is upsampled; invalid values ​​are then filled into the upsampled first depth map to obtain a filled first depth map, wherein the filled first depth map has the same resolution as the second depth map. This ensures that each pixel in the second depth map has corresponding first depth information, facilitating subsequent depth information fusion.

[0137] In one example, step S203 above may specifically include:

[0138] For any pixel in a visible light image, if the value of the third depth information of that pixel is invalid, then the value of the second depth information of that pixel is used as the depth value of that pixel in the fused depth map.

[0139] Since invalid values ​​were filled into the first depth map, after mapping the first depth information onto the visible light image, each pixel in the visible light image needs to be judged one by one. If the value of the third depth information of the pixel is invalid, then the value of the second depth information of the pixel is used as the depth value of the pixel in the fused depth map.

[0140] In addition, when using a depth camera, the following situations may occur that cause invalid values ​​or abnormalities to appear in the depth map.

[0141] Occlusion issue: When an object is occluded by other objects, the depth camera may not be able to accurately calculate the depth information of the occluded object. These areas are usually marked as invalid values.

[0142] Insufficient texture or reflective surfaces: In areas with insufficient texture or on reflective surfaces, depth cameras may have difficulty accurately calculating depth information, resulting in unstable or invalid depth values ​​in these areas.

[0143] Depth range limitation: When measuring objects at a distance or very close distance, depth cameras may exceed their effective depth measurement range, causing depth values ​​in these areas to be unavailable or marked as invalid.

[0144] Lighting conditions: Strong light or shadow areas may affect the performance of the camera sensor, thus affecting the acquisition of depth information.

[0145] Camera calibration issues: If the depth camera (such as a stereo depth camera) is not properly calibrated, or if the camera parameters (such as baseline length, focal length, etc.) are set incorrectly, it may cause distortion or inaccurate depth information in the depth image.

[0146] Environmental factors: Adverse environmental conditions, such as strong light reflection, highly dynamic scenes, or complex background structures, may make it difficult or inaccurate to obtain depth information.

[0147] Unmatched pixel pairs: Binocular cameras typically obtain depth information by calculating the parallax between the two cameras. If some pixels do not have a corresponding pixel between the two cameras (usually due to occlusion, reflection, or low contrast), these pixels may be marked as invalid values ​​or undefined depth.

[0148] Depth sensor issues: The depth sensor itself may be faulty or fail to function properly under certain conditions, which may result in invalid depth values ​​in the depth map.

[0149] Therefore, in other words, in the depth map acquired by the TOF sensor or the stereo depth camera, there are cases where the pixel values ​​are invalid.

[0150] For any pixel in a visible light image, if the value of the second depth information of that pixel is invalid, then the value of the third depth information of that pixel is used as the depth value of that pixel in the fused depth map.

[0151] In this embodiment, for any pixel in a visible light image, if one of the two depth maps has an invalid value for that pixel, then a valid value is used to fill the fusion depth of that pixel. This ensures that the pixel is not invalidated.

[0152] In one example, step S203 above may specifically include:

[0153] S301. For any pixel in a visible light image, if the values ​​of the second depth information and the third depth information of the pixel are both valid values ​​and the difference is less than a preset value, determine the confidence level of the pixel.

[0154] For any pixel in a visible light image, if the second and third depth information corresponding to the pixel are both values ​​that normally represent depth, and the difference between the two is less than a preset value, which can be a value set according to the actual situation, such as 1, 2, 3, etc., then the confidence level of the pixel is determined.

[0155] S302. According to the confidence level, the values ​​of the second depth information and the third depth information are weighted to obtain the depth value of the pixel in the fused depth map.

[0156] In the fields of depth cameras and computer vision, confidence level generally refers to the assessment of the degree of credibility or certainty of a measurement or calculation result. Specifically, confidence level can be represented as a value between 0 and 1, where 0 represents completely unreliable or uncertain, and 1 represents completely reliable or certain. In depth cameras, confidence level is typically used to describe the accuracy and reliability of each pixel or each depth value. For example, in a depth image calculated using stereo vision, each pixel's depth value may be accompanied by a confidence level value to indicate the reliability of that depth value. A higher confidence level means the depth value is more reliable, while a lower confidence level indicates that there may be inaccurate measurement or uncertain depth estimation.

[0157] Therefore, after determining the confidence level of a pixel, the values ​​of the second depth information and the third depth information are weighted using the confidence level, so that the values ​​of the second depth information or the third depth information are closer to the real situation, ensuring that the depth value obtained by fusion is more reliable.

[0158] In this embodiment, for any pixel in a visible light image, if both the second and third depth information values ​​of the pixel are valid and the difference is less than a preset value, the confidence level of the pixel is determined. Based on the confidence level, the values ​​of the second and third depth information are weighted to obtain the depth value of the pixel in the fused depth map. This makes the value of the second or third depth information more accurate at the pixel, allowing the depth value obtained by fusing the second and third depth information to better reflect the true depth of the pixel and improving the reliability of the depth information.

[0159] In one example, the confidence level of a pixel is determined as follows:

[0160] Step 1: Obtain the pixels within a preset range centered on the pixel point from the second depth information and the third depth information respectively, to obtain the first neighborhood and the second neighborhood;

[0161] If both the second and third depth information corresponding to the pixel are normal depth values, and the difference between them is less than a preset value, then the neighborhood of the two depth maps is extracted within a set range. The set range can be 5 (pixels) x 5 (pixels), 6 (pixels) x 6 (pixels), etc., depending on the actual situation. The first and second neighborhoods have the same size.

[0162] Step 2: Obtain the semantic segmentation map corresponding to the visible light image;

[0163] Semantic segmentation is a commonly used image processing technique in computer vision. Its main goal is to assign each pixel in an image to a predefined semantic category. Unlike ordinary image segmentation, semantic segmentation not only focuses on segmenting objects but also requires distinguishing the specific category to which each pixel belongs, such as people, vehicles, roads, and buildings. Semantic segmentation assigns a category label to each pixel, and these labels are usually drawn from a predefined set of categories, such as people, vehicles, and trees. Unlike object detection or semantic segmentation, semantic segmentation requires prediction across the entire image to generate a category label for each pixel.

[0164] In autonomous driving, semantic segmentation is used to understand the location and boundaries of roads, pedestrians, vehicles, etc., to help vehicles make decisions. In medical image analysis, semantic segmentation can be used to identify and segment specific types of tissues or lesions, such as tumors or organs. In agriculture or environmental monitoring, semantic segmentation can help identify different vegetation types or land use patterns. In video understanding, semantic segmentation can help track and analyze the movement and interaction of different objects in dynamic scenes. In virtual reality and augmented reality applications, semantic segmentation can help virtual objects interact accurately with real-world scenes. Semantic segmentation maps have wide applications in many fields, and their development and application have promoted the advancement of computer vision technology in complex scene understanding and automated decision-making.

[0165] Step 3: Based on the different pixel values ​​corresponding to different semantic categories in the semantic segmentation map, set the depth values ​​of pixels in the first neighborhood and the second neighborhood that are not of the same category as the pixel to invalid values.

[0166] In semantic image segmentation, pixel values ​​of different categories refer to the significant differences in numerical values ​​between pixels assigned to different semantic categories. Specifically, each pixel in a semantic segmentation task is assigned to a predefined category, such as person, car, road, etc. Pixel values ​​of different categories are typically numerically different because they represent different object or scene attributes. For example, in a grayscale image, pixels of different categories may have different grayscale levels (pixel values). In a color image, pixels of different categories may have different color combinations (component values ​​in RGB or other color spaces). Therefore, pixel values ​​of different categories refer to the significant differences in their numerical representation in the image based on their semantic category.

[0167] For each pixel in the first and second neighborhoods other than the center pixel, it is necessary to determine whether it belongs to the same category as the center pixel. Specifically, this can be done by comparing the pixel value of the pixel with that of the center pixel. For example, for the semantic category "car", the corresponding pixel value range is 10 to 100. The center pixel has a pixel value of 50. A pixel with a pixel value of 10 belongs to the same category as the center pixel (car). A pixel with a pixel value of 110 does not belong to the same category as the center pixel. Therefore, the depth value of the pixel that does not belong to the same category as the center pixel will be set to an invalid value.

[0168] Step 4: Calculate the effective value rate and average effective value of the first neighborhood and the second neighborhood respectively. The effective value rate represents the number of pixels with effective depth values ​​in the neighborhood, and the average effective value represents the average depth value of pixels with effective depth values ​​in the neighborhood.

[0169] refer to Figure 5 The diagram shows the calculation of the effective value rate and the average effective value. At this time, the effective value rate is 14 / 25, and the average effective value is (153+151+153+147+151+155+154+154+150+149+153+152+152+155) / 14.

[0170] Step 5: Determine the confidence level of the pixel based on the difference between the effective value rate, the average effective value, and the depth value of the pixel. The confidence level is directly proportional to the effective value rate and inversely proportional to the difference. The depth value of the pixel includes the value of the second depth information and the value of the third depth information of the pixel.

[0171] For the first neighborhood, the effective value rate and the average effective value of the first neighborhood are calculated. Based on the difference between the effective value rate, the average effective value, and the second depth information of the pixel, the confidence level of the pixel is determined. Since the confidence level is directly proportional to the effective value rate and inversely proportional to the difference, the product of the inverse proportion of the effective value rate and the difference can be used as the confidence level to obtain the first confidence level.

[0172] For the second neighborhood, the effective value rate and the average effective value of the second neighborhood are calculated. Based on the difference between the effective value rate, the average effective value, and the third depth information of the pixel, the confidence level of the pixel is determined. Since the confidence level is directly proportional to the effective value rate and inversely proportional to the difference, the product of the inverse proportion of the effective value rate and the difference can be used as the confidence level to obtain the second confidence level.

[0173] Finally, the first confidence level can be used as the weight of the second depth information, and the second confidence level can be used as the weight of the third depth information. The values ​​of the second depth information and the third depth information are weighted to obtain the depth value of the pixel in the fused depth map.

[0174] In this embodiment, pixels within a preset range centered on the pixel are obtained from the second depth information and the third depth information to obtain a first neighborhood and a second neighborhood; a semantic segmentation map corresponding to the visible light image is obtained; based on different pixel values ​​corresponding to different semantic categories in the semantic segmentation map, the depth values ​​of pixels in the first neighborhood and the second neighborhood that are not of the same category as the pixel are set to invalid values; the effective value rate and average effective value of the first neighborhood and the second neighborhood are calculated respectively, where the effective value rate represents the number of pixels with effective depth values ​​in the neighborhood, and the average effective value represents the average depth value of pixels with effective depth values ​​in the neighborhood; based on the difference between the effective value rate, the average effective value, and the depth value of the pixel, the confidence level of the pixel is determined, where the confidence level is directly proportional to the effective value rate and inversely proportional to the difference, wherein the depth value of the pixel includes the value of the second depth information and the value of the third depth information of the pixel. The confidence level of a pixel is obtained by combining the confidence level with that of its surrounding pixels. Since semantic segmentation information is introduced in the confidence level calculation process, the jump and trailing problems that occur in subsequent depth interpolation are well avoided.

[0175] In one example, step S203 above may specifically include:

[0176] S401. For any pixel in a visible light image, if the values ​​of the second depth information and the third depth information of the pixel are both valid values ​​and the difference is not less than a preset value.

[0177] For any pixel in a visible light image, if the second and third depth information corresponding to the pixel are both values ​​that normally represent depth, and the difference between the two is not less than a preset value, which can be 1, 2, 3, etc., set according to the actual situation, then the following depth fusion strategy is executed.

[0178] S402. In the second depth information, obtain the pixels within a preset range centered on the pixel point to obtain the first neighborhood.

[0179] If the second and third depth information corresponding to the pixel are both values ​​that normally represent depth, and the difference between them is not less than a preset value, then the neighborhood of the second depth map is extracted within a set range. The set range can be 5 (pixels) x 5 (pixels), 6 (pixels) x 6 (pixels), etc., depending on the actual situation.

[0180] S403. Determine the average depth value and variance of the first neighborhood based on the depth values ​​of the pixels in the first neighborhood.

[0181] The average depth value of the first neighborhood is obtained by comparing the depth values ​​of all pixels in the first neighborhood with the number of pixels. If the average depth value is less than a preset depth value, it indicates that the corresponding object is black.

[0182] Variance is a metric used in statistics to measure the dispersion of data. In probability theory and statistics, variance represents the degree of deviation between a random variable and its expected value (mean). The variance of the first neighborhood is the average of the squared differences between the pixel values ​​of each pixel and the average depth value. The larger the variance, the greater the deviation of the data points from the mean; the smaller the variance, the more concentrated the data points are around the mean. If the variance of the first neighborhood is less than a preset variance value, it indicates that the corresponding texture is repetitive.

[0183] S404. If the average depth value is less than the preset depth value, then the value of the second depth information of the pixel is used as the depth value of the pixel in the fused depth map.

[0184] If the average depth value is less than the preset depth value, it indicates that the corresponding object is black. Since the binocular module has a better recognition effect on the depth information of black objects, the second depth information is selected as the depth value of that pixel.

[0185] S405. If the variance is less than the preset variance value, then the value of the third depth information of the pixel is used as the depth value of the pixel in the fused depth map.

[0186] If the variance is less than the preset variance value, it indicates that the corresponding texture is repeated. Since the time-of-flight module has a good recognition effect on the depth information of repeated textures, the three-dimensional information is selected as the depth value of the pixel.

[0187] In this embodiment, for any pixel in a visible light image, if both the second and third depth information values ​​of the pixel are valid and the difference is not less than a preset value; in the second depth information, pixels within a preset range centered on the pixel are obtained to form a first neighborhood; based on the depth values ​​of the pixels in the first neighborhood, the average depth value and variance of the first neighborhood are determined; if the average depth value is less than a preset depth value, the value of the second depth information of the pixel is used as the depth value of the pixel in the fused depth map; if the variance is less than a preset variance value, the value of the third depth information of the pixel is used as the depth value of the pixel in the fused depth map. Utilizing the average depth value and variance of the neighborhood to assist in pixel depth fusion ensures that different depth information is used for different situations, resulting in a more accurate fused depth value.

[0188] In one example, a first depth map under low-light conditions is acquired by a time-of-flight module and a second depth map under low-light conditions is acquired by a binocular module. The first depth map and the second depth map include white vehicles and black vehicles.

[0189] Depth maps are used for environmental perception and navigation in autonomous driving, and can be used for obstacle detection, path planning, lane keeping, etc. Therefore, when obstacle detection, path planning, or lane keeping is required, the depth map generation method provided in this application is triggered to obtain the first depth map collected by the time-of-flight module and the second depth map collected by the binocular module.

[0190] Because TOF cameras are ineffective at measuring the depth of black objects, and stereo cameras are ineffective at measuring the depth of white objects, and both are susceptible to lighting conditions, the depth map generation method provided in this application can be used to obtain a first depth map under low-light conditions acquired by the time-of-flight module and a second depth map under low-light conditions acquired by the stereo module. Both the first and second depth maps include white and black vehicles. Then, the specific steps of the aforementioned depth map generation method are executed.

[0191] See Figure 6 Imagine a white sedan and a black truck traveling towards each other. In the real world, from closest to farthest, the images are the white sedan, the black truck, and a house. For an autonomous vehicle, it needs to identify the true depth information of these three objects in the image to make subsequent decisions. Therefore, for this scenario, the depth map generation method provided in this application can be applied to obtain a first depth map and a second depth map containing the white sedan and the black truck, ultimately generating a fused depth map containing the depth information of both the white sedan and the black truck.

[0192] The distance information of black / white vehicles is calculated based on the fused depth map.

[0193] The depth map contains distance information of objects. The depth map generation method provided in this application can combine the advantages of both ToF camera and stereo depth camera. Therefore, the distance information of black / white vehicles can be accurately calculated based on the fused depth map.

[0194] The method provided in this application embodiment can achieve better depth measurement results for black / white vehicles under low-light conditions, overcoming the shortcomings of using only one sensor.

[0195] In one example, see Figure 7 This application provides a camera calibration method. It includes: fixing the relative positions of the ToF sensor and the stereo camera; synchronously acquiring calibration data; upsampling the infrared image of the ToF and filling it with invalid values ​​to achieve the same resolution as the stereo camera; extracting pixels and world coordinate system calibration points from the infrared image of the ToF and the two images of the stereo camera; calibrating the intrinsic parameters of each camera using a camera intrinsic parameter calibration method; calibrating the coordinate system transformation relationship from the right image of the stereo camera and from the ToF to the left image of the stereo camera using a stereo calibration method; completing the calibration to obtain the intrinsic parameters of each camera and the coordinate system transformation relationship between the cameras.

[0196] In one example, see Figure 8 This application provides a depth map fusion method. It includes: upsampling the depth map obtained from a ToF sensor and filling it with invalid values ​​to maintain a consistent resolution with the stereo depth; projecting the ToF depth map onto the stereo depth viewpoint using a calibrated coordinate system transformation relationship; and traversing each pixel of the stereo depth, using different fusion methods depending on the situation, which are divided into the following three cases.

[0197] Scenario 1: If one of the two depth maps has an invalid value for this pixel, use an invalid value to fill in the fusion depth of this pixel, thus obtaining the fusion depth of this pixel.

[0198] Scenario 2: If both depth maps have valid values ​​at this pixel and the difference is less than the set value d, extract the neighborhood of the two depth maps using the set Size. Extract the semantic segmentation map from the left image of the stereo camera. Classify the pixel values ​​in the two extracted neighborhoods that are not of the same category as the center point as invalid values. Calculate the valid value rate and average valid value in the two neighborhoods respectively. Combine the valid value rate and the difference between the average valid value and the center value to obtain the confidence of the two depth maps at this pixel. Calculate the weighted average of the two depth maps at this pixel according to the confidence, and obtain the fusion depth of this pixel.

[0199] Scenario 3: If both depth maps have valid values ​​for this pixel and the difference is greater than the set value d, extract the neighborhood from the left image of the stereo camera. Detect whether there is a duplicate texture and a black object based on the neighborhood mean and variance. If there is a duplicate texture, fill the blending depth with TOF depth; if there is a black object, fill the blending depth with stereo depth, thus obtaining the blending depth for that pixel.

[0200] On the other hand, embodiments of this application also provide a depth map generation apparatus, the apparatus comprising:

[0201] The acquisition module is used to acquire a first depth map collected by the time-of-flight module and a second depth map collected by the binocular module. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information is used to describe the depth of pixels in the visible light image.

[0202] The mapping module is used to map the first depth information onto the visible light image through a preset coordinate system transformation relationship, so as to obtain the third depth information of the pixels in the visible light image;

[0203] The fusion module is used to fuse the second depth information and the third depth information of the pixels in the visible light image to obtain a fused depth map.

[0204] This application also provides an electronic device including one or more processors and a memory; the memory is coupled to one or more processors and is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform some or all of the steps in the above method embodiments.

[0205] This application also provides a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform some or all of the steps described in the method embodiments.

[0206] The aforementioned storage media can be magnetic disks, optical disks, read-only memory (ROM), or random access memory (RAM), etc.

[0207] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0208] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a Digital Signal Processor (DSP), a microcontroller, an Application Specific Integrated Circuit (ASIC), or a microprocessor.

[0209] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0210] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, compact disc read-only memory (CD-ROMs), magneto-optical disks, read-only memory, random access memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media includes any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0211] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0212] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0213] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0214] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. A method for generating depth maps, characterized in that, The method includes: The first depth map acquired by the time-of-flight module and the second depth map acquired by the binocular module are obtained. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information is used to describe the depth of pixels in the visible light image. The first depth map is upsampled; invalid values ​​are filled into the upsampled first depth map to obtain a filled first depth map, wherein the filled first depth map has the same resolution as the second depth map; By using a preset coordinate system transformation relationship, the first depth information is mapped onto the visible light image to obtain the third depth information of the pixels in the visible light image; The second and third depth information of pixels in the visible light image are fused to obtain a fused depth map, including: For any pixel in a visible light image, if both the second and third depth information values ​​of the pixel are valid and the difference is not less than a preset value; in the second depth information, pixels within a preset range centered on the pixel are obtained to form a first neighborhood; based on the depth values ​​of the pixels in the first neighborhood, the average depth value and variance of the first neighborhood are determined; if the average depth value is less than a preset depth value, the value of the second depth information of the pixel is used as the depth value of the pixel in the fused depth map; if the variance is less than a preset variance value, the value of the third depth information of the pixel is used as the depth value of the pixel in the fused depth map.

2. The method of claim 1, wherein, The process of fusing the second and third depth information of pixels in the visible light image to obtain a fused depth map includes: For any pixel in a visible light image, if the value of the third depth information of that pixel is invalid, then the value of the second depth information of that pixel is used as the depth value of that pixel in the fused depth map.

3. The method according to claim 1, characterized in that, The process of fusing the second and third depth information of pixels in the visible light image to obtain a fused depth map includes: For any pixel in a visible light image, if the values ​​of the second depth information and the third depth information of the pixel are both valid values ​​and the difference is less than a preset value, the confidence level of the pixel is determined. Based on the confidence level, the values ​​of the second depth information and the third depth information are weighted to obtain the depth value of the pixel in the fused depth map.

4. The method of claim 3, wherein, Determining the confidence level of the pixel includes: Pixels within a preset range centered on the pixel are obtained from the second depth information and the third depth information respectively, to obtain the first neighborhood and the second neighborhood; Obtain the semantic segmentation map corresponding to the visible light image; Based on the different pixel values ​​corresponding to different semantic categories in the semantic segmentation map, the depth values ​​of pixels in the first neighborhood and the second neighborhood that are not of the same category as the pixel are set to invalid values. The effective value rate and average effective value of the first neighborhood and the second neighborhood are calculated respectively. The effective value rate represents the number of pixels with effective depth values ​​in the neighborhood, and the average effective value represents the average depth value of pixels with effective depth values ​​in the neighborhood. The confidence level of a pixel is determined based on the difference between the effective value rate, the average effective value, and the depth value of the pixel. The confidence level is directly proportional to the effective value rate and inversely proportional to the difference. The depth value of the pixel includes the value of the second depth information and the value of the third depth information of the pixel.

5. The method of claim 1, wherein, The acquisition of the first depth map acquired by the time-of-flight module and the second depth map acquired by the binocular module includes: The system acquires a first depth map under low-light conditions collected by the time-of-flight module and a second depth map under low-light conditions collected by the binocular module. The first depth map and the second depth map include white vehicles and black vehicles. The method further includes: The distance information of black / white vehicles is calculated based on the fused depth map.

6. A depth map generating apparatus characterized by comprising: The device includes: The acquisition module is used to acquire a first depth map collected by the time-of-flight module and a second depth map collected by the binocular module. The first depth map contains first depth information, and the second depth map contains a visible light image and second depth information. The second depth information is used to describe the depth of pixels in the visible light image. The mapping module is used to upsample the first depth map; fill the upsampled first depth map with invalid values ​​to obtain a filled first depth map, wherein the filled first depth map has the same resolution as the second depth map; and map the first depth information onto the visible light image through a preset coordinate system transformation relationship to obtain the third depth information of the pixels in the visible light image. The fusion module is used to, for any pixel in a visible light image, if the values ​​of the second depth information and the third depth information of the pixel are both valid and the difference is not less than a preset value; in the second depth information, obtain pixels within a preset range centered on the pixel to obtain a first neighborhood; determine the average depth value and variance of the first neighborhood based on the depth values ​​of the pixels in the first neighborhood; if the average depth value is less than a preset depth value, then the value of the second depth information of the pixel is used as the depth value of the pixel in the fused depth map; if the variance is less than a preset variance value, then the value of the third depth information of the pixel is used as the depth value of the pixel in the fused depth map.

7. An electronic device, comprising: include: One or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Includes a computer program that, when run on an electronic device, causes the electronic device to perform the method according to any one of claims 1-5.

Images