Depth calculation method, tof depth camera, and computer-readable storage medium

By calculating the original phase image and PSF model of the TOF depth camera, scattering interference compensation is performed, and the problem of depth error in the TOF depth camera is solved, achieving more accurate and efficient depth measurement.

WO2024239618A1PCT designated stage expired Publication Date: 2024-11-28SHENZHEN ORBBEC CO LTD +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/138946
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-23
Filing Date
2023-12-14
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

In practical applications, TOF depth cameras cause depth errors due to indirectly reflected light, which affects the accuracy of depth measurement.

Method used

By acquiring the original phase image acquired by the TOF depth camera and the pre-stored PSF model, the scattered interference map is calculated and compensated to obtain an ideal phase image, thereby obtaining the depth information of the target area based on the principle of time of flight, reducing the calculation amount and improving accuracy.

Benefits of technology

It effectively alleviates the problem of depth error, improves the accuracy and computing efficiency of depth measurement, and significantly improves the performance of TOF depth cameras in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023138946_28112024_PF_FP_ABST
    Figure CN2023138946_28112024_PF_FP_ABST
Patent Text Reader

Abstract

A depth calculation method (500), a time-of-flight (ToF) depth camera (900), and a computer-readable storage medium. The depth calculation method (500) is applied to the ToF depth camera (900), and comprises: acquiring an original phase image which is collected by the ToF depth camera (900) and comprises a target region, and a pre-stored PSF model corresponding to the ToF depth camera (900) (S510); obtaining a scattering interference map by using the original phase image and the PSF model (S520); adjusting the scattering interference map on the basis of the energy of at least some of pixels in the original phase image to obtain a scattering compensation map (S530); compensating for the original phase image by using the scattering compensation map to obtain an ideal phase image (S540); and processing the ideal phase image on the basis of a ToF principle to obtain depth information of the target region (S550). Therefore, depth errors can be relieved.
Need to check novelty before this filing date? Find Prior Art

Description

Depth calculation method, TOF depth camera and computer-readable storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 23, 2023, with application number 202310591393.7 and titled “Depth Calculation Method, TOF Depth Camera and Computer-Readable Storage Medium,” the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of image processing, and in particular to a depth calculation method, a TOF depth camera, and a computer-readable storage medium. Background Art

[0003] In the actual application environment of the depth camera, in addition to receiving the reflected pulse light directly reflected back from the object to be measured, the depth camera also receives some other indirect reflected light. After these indirect reflected lights are received by some pixels, they will cause internal scattering and lead to depth errors of these pixels.

[0004] Summary of the Invention

[0005] Embodiments of the present application provide a depth calculation method, a TOF depth camera, and a computer-readable storage medium, which can alleviate depth errors.

[0006] In a first aspect, a depth calculation method is provided, which obtains an original phase image of a target area captured by a TOF depth camera and a pre-stored PSF model corresponding to the TOF depth camera; uses the original phase image and the PSF model to obtain a scattering interference map, where the scattering interference map includes a scattering interference amount indicating the degree to which the energy of at least some pixels in the original phase image interferes with the energy of other surrounding pixels; adjusts the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map; uses the scattering compensation map to compensate the original phase image to obtain an ideal phase image; and processes the ideal phase image based on the time-of-flight principle to obtain depth information of the target area.

[0007] In the second aspect, a TOF depth camera is provided, including a transmitter, a receiver and a processor, wherein the transmitter is used to project a periodically modulated transmitted light signal to a target area, the receiver is used to collect the received light signal transmitted back from the target area to generate an original phase image, and the processor is used to process the original phase image using a depth calculation method such as in the first aspect and any possible implementation method of the first aspect thereof to obtain depth information of the target area.

[0008] In a third aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the depth calculation method in the first aspect and any possible implementation of the first aspect.

[0009] In a fourth aspect, a chip is provided, comprising a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes a depth calculation method as in the first aspect and any possible implementation of the first aspect.

[0010] In a fifth aspect, a computer program is provided, which enables a computer to execute the depth calculation method in the first aspect and any possible implementation of the first aspect.

[0011] In the sixth aspect, a program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, the processor in the electronic device executes the depth calculation method as in the first aspect and any possible implementation of the first aspect thereof.

[0012] Based on the above technical solution, the original phase image of the target area captured by the TOF depth camera and the pre-stored PSF model corresponding to the TOF depth camera are first obtained. Secondly, the original phase image and the PSF model are used to obtain a scattering interference map, and the scattering interference map is adjusted based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map. Finally, the original phase image is compensated with the scattering compensation map to obtain an ideal phase image, and the ideal phase image is processed based on the time-of-flight principle to obtain the depth information of the target area. The ideal phase image obtained by compensating the original phase image with the scattering compensation map can reduce the amount of calculation compared to the ideal phase image obtained by directly deconvolving the ideal phase image * PSF model = original phase image, and can also more accurately obtain the original phase image under ideal conditions, thereby achieving the purpose of alleviating depth error. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG1 shows a schematic structural diagram of an iTOF depth camera.

[0014] FIG2 shows a schematic diagram of an application of an iTOF depth camera according to an embodiment of the present application.

[0015] FIG3 shows a schematic block diagram of a depth calculation method according to an embodiment of the present application.

[0016] FIG4 shows a pixel distribution diagram of an original phase image and a scattering interference map according to an embodiment of the present application.

[0017] FIG5 shows another schematic block diagram of the depth calculation method according to an embodiment of the present application.

[0018] FIG. 6 shows a pixel distribution diagram of a scattering compensation map according to an embodiment of the present application.

[0019] FIG. 7 shows a pixel distribution diagram of an amplitude diagram corresponding to an original phase image according to an embodiment of the present application.

[0020] FIG8 shows a pixel distribution diagram of an ideal phase image according to an embodiment of the present application.

[0021] FIG9 shows a schematic block diagram of a method for obtaining a PSF model of a TOF depth camera according to an embodiment of the present application.

[0022] FIG10 and FIG11 are comparative diagrams showing the effects of depth error using the traditional solution and the technical solution provided by the embodiment of the present application, respectively.

[0023] FIG12 shows a schematic block diagram of a depth calculation device according to an embodiment of the present application.

[0024] FIG13 shows another schematic block diagram of a depth calculation device according to an embodiment of the present application.

[0025] FIG14 shows a schematic block diagram of an apparatus for obtaining a PSF model of a TOF depth camera according to an embodiment of the present application.

[0026] FIG15 shows a schematic block diagram of a TOF depth camera according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0028] A Time-of-Flight (TOF) depth camera, or TOF depth camera for short, works by continuously sending light pulses to a target object, then using a sensor to receive the light returning from the target object. The distance to the target object is determined by detecting the round-trip (flight) time of the light pulses. TOF depth cameras typically include direct-Time-of-Flight (DTOF) depth cameras and indirect-Time-of-Flight (indirect-Time-of-Flight) depth cameras. The basic principle of an iTOF depth camera is to calculate the time of flight by measuring the phase difference between the transmitted and received light signals. Specifically, as shown in Figure 1, the iTOF depth camera 100 includes a transmitter 101, a receiver 102, and a processor 103. The transmitter 101 transmits a periodically modulated light signal to the surface of the object under test in the target area, and the receiver 102 collects the received light signal reflected from the surface of the object under test. The received light signal reflected from the surface of the object under test generates a time delay relative to the transmitted light signal in the timing sequence, which is manifested as an additional phase delay on the periodically modulated transmitted light signal. The processor 103 is used to read the phase delay and obtain the flight time of the light signal between the iTOF depth camera 100 and the object under test based on the phase delay. The distance between the object under test and the iTOF depth camera can be expressed as: Where D is the depth between the object to be measured and the iTOF depth camera, That is, the phase delay between the transmitted light signal and the reflected light signal, f is the frequency of the transmitted light signal, and c is the speed of light.

[0029] In one embodiment, the transmitting end 101 includes a light source, a lens and / or a patterned optical element, wherein the light source includes a single vertical-cavity surface-emitting laser (VCSEL) or a VCSEL array, which is used to emit a light beam to the lens; the lens includes a single lens or a lens group, which is used to collimate the light beam; the patterned optical element includes any one of a diffractive optical element (DOE), a microlens array (MLA), and a mask, which is used to modulate the light beam so that the light beam passing through the patterned optical element is projected toward the target area in any form such as speckle or flood light, which is not limited here.

[0030] In one embodiment, the receiving end 102 includes a TOF image sensor, which receives the reflected light beam and generates a raw phase image. Preferably, the TOF image sensor includes a plurality of pixels, each pixel includes at least one tap for storing charge, each tap is used to accumulate the amount of charge generated by the corresponding pixel collecting the light signal reflected from the object to be measured within a preset exposure time, and output a raw phase image containing the depth information of the object to be measured, wherein the raw phase image is the raw data obtained by the TOF image sensor converting the collected light signal into a digital signal. Furthermore, the receiving end 102 also includes a lens, which is provided on the light incident side of the TOF image sensor, for focusing the reflected light beam on the corresponding pixel of the TOF image sensor.

[0031] In one embodiment, each pixel in the TOF image sensor includes three or more taps for storing and reading or discharging the charge signal generated by the reflected light pulse under the control of the corresponding electrodes. When each pixel includes multiple taps, the taps are switched in sequence in a certain order within a single frame period T (or a single exposure time) to collect electrons generated by the pixel receiving the light signal reflected back by the target object to form a charge signal.

[0032] In one embodiment, the processor 103 can be an independent dedicated circuit, such as a dedicated system-on-chip (SOC) chip, a field programmable gate array (FPGA) chip, an application-specific integrated circuit (ASIC) chip, etc., which includes a central processing unit (CPU), a memory, a bus, etc. It can also include a general-purpose processing circuit. For example, when the depth camera is integrated into a smart terminal such as a mobile phone, a television, or a computer, the processing circuit in the terminal can serve as at least a part of the processor 103. In some embodiments, the processor 103 is used to provide the modulation signal (transmission signal) required for the transmitting end 101 to transmit an optical signal. The transmitting end 101 transmits a light beam to the object to be measured under the control of the modulation signal. In addition, the processor 103 also provides a demodulation signal (acquisition signal) of the tap in each pixel of the image sensor in the receiving end 102. The tap collects the charge signal generated by the light beam containing the reflected pulse light beam reflected back from the object to be measured under the control of the demodulation signal. The processor 103 can also provide auxiliary monitoring signals, such as temperature sensing, overcurrent, overvoltage protection, and shedding protection. The processor 103 can also be used to save the raw data collected by each tap in the image sensor and perform corresponding processing to obtain the specific position information of the object to be measured.

[0033] In some embodiments, the iTOF depth camera may also include a driver circuit, power supply, color camera, infrared camera, inertial measurement unit (IMU), and other components. These components are not shown in the figure, but the combination of these components can achieve more extensive functions, such as 3D texture modeling, infrared face recognition, and simultaneous localization and mapping (SLAM). The iTOF depth camera can be embedded in electronic products such as mobile phones, tablets, and computers.

[0034] In the actual application environment of iTOF depth cameras, in addition to receiving the reflected pulse light beam directly reflected back by the object to be measured, the iTOF depth camera also receives some other indirect reflected light. After these indirect reflected lights are received by some pixels, they will cause inward scattering, resulting in depth errors for these pixels. For example, as shown in Figure 2, when the iTOF depth camera 100 is mounted on a sweeping robot 200 for obstacle avoidance application, it only needs to determine whether there is an obstacle 300 on the ground within a certain distance range directly in front. It needs to have a large field of view in the horizontal and vertical directions. However, since the iTOF depth camera 100 mounted on the sweeping robot 200 is relatively close to the ground 400, which is a high reflectivity area, the light beam emitted by the transmitting end 101 of the iTOF depth camera 100 is easily reflected by the ground 400 and enters the receiving end 102, causing inward scattering at the receiving end 102. That is, the light beam reflected back from the high reflectivity area will reflect back and forth between the receiving lens and the pixel plane of the image sensor, and part of the light beam will be received by other pixels, thereby generating depth errors.

[0035] In view of this, an embodiment of the present application provides a depth calculation method that can alleviate depth errors.

[0036] FIG3 shows a schematic block diagram of a depth calculation method 500 according to an embodiment of the present application. Optionally, the depth calculation method 500 can be applied to a TOF depth camera. For example, the depth calculation method 500 can be applied to the iTOF depth camera 100 shown in FIG1 , and the depth calculation method 500 can be executed by the processor 103 in the iTOF depth camera 100. Specifically, as shown in FIG3 , the depth calculation method 500 can include some or all of the following contents.

[0037] S510: Acquire an original phase image of a target area captured by a TOF depth camera and a pre-stored PSF model corresponding to the TOF depth camera.

[0038] S520 , obtaining a scattering interference map using the original phase image and the PSF model, wherein the scattering interference map includes a scattering interference amount indicating the degree to which the energy of at least some pixels in the original phase image interferes with the energy of other surrounding pixels.

[0039] S530: Adjust the scattering interference map based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map.

[0040] S540 , compensating the original phase image using the scattering compensation map to obtain an ideal phase image.

[0041] S550 , processing the ideal phase image based on the time-of-flight principle to obtain depth information of the target area.

[0042] Typically, when there is an interfering object between the TOF depth camera and the target area, under ideal conditions, the transmitter transmits a light beam to the target area and the interfering object and is received by the receiver. There is no interference between the light beam reflected by the interfering object and the light beam reflected back from the target area, which means that it will not affect the receiver's acquisition of the light beam reflected back from the target area, so that the original phase image collected by the TOF depth camera is an original phase image without depth error. However, if the interfering object between the TOF depth camera and the target area is a highly reflective object, since the highly reflective object has strong reflective energy and is located between the depth camera and the target area, the light beam reflected from the surface of the highly reflective object will diffuse and be received by the receiver through diffraction, reflection, etc. and enter the pixels of the image sensor, thereby affecting the receiver's reception of the weaker energy light beam reflected back from the target area. When the receiver receives the weaker energy light beam reflected back from the target area, it also receives the light beam reflected from the surface of the highly reflective object, causing the corresponding pixels in the image sensor that receive this part of the light beam to have a depth error. At this time, the original phase image collected by the TOF depth camera is an original phase image with depth error. Typically, a TOF depth camera has a maximum detection range and a minimum detection range, and the target area is the area between the minimum detection range and the maximum detection range where depth information needs to be obtained.

[0043] It should be understood that the original phase image in the embodiment of the present application refers to the image captured by the TOF depth camera without any computational processing. The value of the pixel in the original phase image is the energy information, which can be expressed as the amount of charge collected by each tap on the pixel in the TOF image sensor of the depth camera.

[0044] It should be noted that the pixel positions in the original phase image correspond to the pixel positions in the scattering interference map. In other words, the original phase image and the scattering interference map contain the same number of pixel data. For example, as shown in Figure 4, the pixel distribution in both the original phase image and the scattering interference map is 4*4, where the pixels in the original phase image are A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, and A16. The pixels in the scattering interference map are B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14, B15, and B16. A1 and B1 correspond to the same pixel, A2 and B2 correspond to the same pixel, ..., A16 and B16 correspond to the same pixel, and so on.

[0045] Similarly, the pixel positions in the scatter compensation map, the original phase image, and the scatter interference map also correspond, and the pixel positions in the scatter compensation map, the original phase image, and the ideal phase image also correspond.

[0046] Generally, the three-dimensional light intensity distribution of the image formed by a point light source on an object surface through an optical system is called a point spread function (PSF). In an embodiment of the present application, the PSF model may include energy data, which represents the influence of the amplitude of a single pixel on the amplitudes of other surrounding pixels. Generally, the PSF model corresponding to the TOF depth camera refers to the initial PSF model obtained by optical fiber or other means before the TOF depth camera leaves the factory, and is stored inside the TOF depth camera for subsequent use. In some other embodiments, the initial PSF model obtained may be further processed before the TOF depth camera leaves the factory. For example, the initial PSF model may be normalized or non-normalized. And the PSF model obtained after processing may be further stored inside the TOF depth camera for subsequent use. That is, in S510, the pre-stored PSF model corresponding to the TOF depth camera obtained may be the initial PSF model obtained before the TOF depth camera leaves the factory, or it may be a PSF model obtained after processing the initial PSF model. In other words, in S520, the scattering interference map can be obtained using the original phase image and the pre-stored initial PSF model of the depth camera, or the scattering interference map can be obtained using the original phase image and the pre-stored PSF model of the depth camera after processing.

[0047] Theoretically, the ideal phase image can be obtained by deconvolving the ideal phase image * PSF model = the original phase image. However, because there are multiple pixels in the image, the PSF model corresponding to each pixel may be different, that is, the deconvolution calculation process may require multiple PSF models, resulting in excessive computational complexity. In an embodiment of the present application, the original phase image of the target area captured by the TOF depth camera and the pre-stored PSF model corresponding to the TOF depth camera are first obtained. Secondly, the original phase image and the PSF model are used to obtain a scattering interference map, and the scattering interference map is adjusted based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map. Finally, the original phase image is compensated with the scattering compensation map to obtain an ideal phase image, and the ideal phase image is processed based on the time-of-flight principle to obtain the depth information of the target area. The ideal phase image obtained by compensating the original phase image with the scattering compensation map can reduce the computational complexity compared to the ideal phase image obtained by directly deconvolving the ideal phase image * PSF model = the original phase image, and can also more accurately obtain the original phase image under ideal conditions, thereby achieving the purpose of alleviating the depth error.

[0048] In some embodiments, as shown in FIG5 , S530 , ie, adjusting the scattering interference map based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map, may include some or all of the following.

[0049] S531, determining the relationship between the energy of the pixels in the original phase image and a preset first threshold.

[0050] S532: If the energy of a certain pixel in the original phase image is less than a first threshold, the scattering interference amount generated by the corresponding pixel on surrounding pixels in the scattering interference map is set to 0 to obtain a scattering compensation map.

[0051] Optionally, in one embodiment, the scattering interference map may include the scattering interference amount corresponding to all pixels in the original phase image. In other embodiments, the scattering interference map may also include the scattering interference amount corresponding to some pixels in the original phase image. The scattering interference amount of each pixel can be determined based on the energy of each pixel in the original phase image and the PSF model of the corresponding pixel in the PSF model corresponding to the depth camera. For example, the value of pixel A1 in the original phase image is A1, and the PSF model corresponding to the depth camera is a non-normalized PSF model, and the matrix corresponding to pixel A1 can be obtained from the non-normalized PSF model as X1, that is, the scattering interference amount of pixel A1 on the surrounding pixels can be obtained as A1*X1, and so on, the scattering interference amount of each pixel in the original phase image on the surrounding pixels can be obtained.

[0052] After obtaining the scattering interference map, a scattering compensation map can be further obtained based on the relationship between the energy of the pixels in the original phase image and the first threshold. For example, assuming that the value of the left half of the pixel position in the original phase image is less than the preset first threshold, it means that the interference caused by the current pixel to the surrounding pixels is small and can be ignored. In other words, the value of the left half of the pixel position in the scattering interference map can be set to 0, thereby obtaining the scattering compensation map shown in Figure 6.

[0053] In this embodiment, after obtaining the scattering interference map, the relationship between the energy of the pixel in the original phase image and the preset first threshold value can be further determined. If the energy of a certain pixel in the original phase image is less than the first threshold value, the scattering interference amount of the corresponding pixel in the scattering interference map is set to 0, thereby obtaining a scattering compensation map. Since the energy of a certain pixel in the original phase image is weak, it will not affect the depth effect of other pixels, thereby reducing the computational complexity without affecting the depth error correction.

[0054] Optionally, the first threshold may be a preset value set based on experience. For example, the first threshold may be a threshold selected based on factors such as application scenario and exposure time.

[0055] In other embodiments, as shown in FIG6 , S530 , i.e., adjusting the scattering interference map based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map, may include some or all of the following contents.

[0056] S533: Process the energy of at least some pixels in the original phase image to obtain an amplitude map corresponding to the original phase image. In the embodiment of the present application, the image calculated based on the original phase image refers to an image including amplitude information.

[0057] S534, determining a magnitude relationship between the amplitude values ​​of at least some pixels in the amplitude map corresponding to the original phase map and a preset second threshold;

[0058] S535: If the amplitude values ​​of at least some pixels in the amplitude map corresponding to the original phase map are greater than a preset second threshold, the scattering interference amount of the corresponding pixels in the scattering interference map is set to 0 to obtain a scattering compensation map.

[0059] Optionally, after processing the energy of at least part of the pixels in the original phase image in FIG4 , an amplitude map corresponding to the original phase image as shown in FIG7 can be obtained. Similarly, the correspondence between the original phase image and the amplitude map can refer to the correspondence between pixel positions. Processing the energy of at least part of the pixels in the original phase image to obtain the amplitude map specifically includes accumulating a number of pixels in the original phase image and subtracting the ambient noise to obtain the pixel value corresponding to a pixel on the amplitude map. For example, the pixel distribution of the amplitude map in FIG7 is also 4*4, and the pixels in the amplitude map are C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, and C16, respectively. The value of pixel C1 in the amplitude map can be obtained by adding the value obtained by pixel A1 within the sampling time and subtracting the ambient noise, and the same applies to the remaining pixels in the amplitude map.

[0060] Optionally, when the PSF model is a non-normalized PSF model, S520 , ie, obtaining a scattering interference map using the original phase image and the PSF model: multiplying the original phase image by the PSF model to obtain a scattering interference map.

[0061] For example, the scattering interference pattern can be obtained by the following formula:

[0062] Original phase image × non-normalized PSF model = scattering interference pattern.

[0063] In this embodiment, a non-normalized PSF model is multiplied by the original phase image to obtain a scattering interference map, and further, an ideal phase image can be obtained based on the scattering compensation map obtained from the scattering interference map and the original phase image. The result is similar to the result of deconvolution of the original phase image and the initial PSF model, and can also greatly reduce the amount of calculation.

[0064] In some embodiments, S540, ie, compensating the original phase image using the scattering compensation map to obtain the ideal phase image, includes: subtracting the original phase image from the scattering compensation map to obtain the ideal phase image.

[0065] For example, by subtracting the values ​​of the corresponding pixel positions of the original phase image in FIG4 and the scatter compensation image in FIG6, an ideal phase image can be obtained. The ideal phase image can be shown in FIG8. That is, the values ​​of the pixels included in the ideal phase image are A1, A2, A3-B3, A4-B4, A5, A6, A7-B7, A8-B8, A9, A10, A11-B11, A12-B12, A13, A14, A15-B15, and A16-B16. Among them, ..., A3 and A3-B3 correspond to the same pixel, A4 and A4-B4 correspond to the same pixel, ..., A7 and A7-B7 correspond to the same pixel, A8 and A8-B8 correspond to the same pixel, ..., A15 and A15-B15 correspond to the same pixel, and A16 and A16-B16 correspond to the same pixel.

[0066] Optionally, in an embodiment of the present application, the depth calculation method 500 may further include: according to the pixel position in the original phase image, obtaining the PSF model of the corresponding pixel of the original phase image from the pre-stored PSF model of the TOF depth camera by looking up the table.

[0067] In some embodiments, the PSF model of the TOF depth camera can be constructed by optical fiber or other means before the depth camera leaves the factory and stored inside the depth camera. After the TOF depth camera leaves the factory and during actual use, the PSF model stored inside the TOF depth camera can be called by looking up the table according to the pixel position in the original phase image. For example, the PSF model corresponding to all pixels in the original phase image can be obtained by looking up the table. For another example, the PSF model corresponding to some pixels in the original phase image can be obtained by looking up the table.

[0068] As mentioned above, the PSF model of the TOF depth camera is pre-stored inside the TOF depth camera, but before leaving the factory, the PSF model of the TOF depth camera can be obtained in advance. The embodiment of the present application also provides a method for obtaining the PSF model of the TOF depth camera.

[0069] Specifically, referring to FIG. 9 , the method 600 for obtaining the PSF model of the TOF depth camera may include part or all of the following contents.

[0070] S610, controlling the TOF depth camera to acquire a first PSF amplitude image using a first exposure time;

[0071] S620: Control the TOF depth camera to acquire a second PSF amplitude image using a second exposure time, where the second exposure time is less than the second exposure time.

[0072] S630: Integrate the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map.

[0073] S640: Obtain a PSF model of the TOF depth camera according to the integrated amplitude map.

[0074] Optionally, in one embodiment, the PSF model of the TOF depth camera can be obtained by means of optical fiber. Optionally, one end of the optical fiber is connected to the transmitting end of the depth camera, and the other end is aligned with the receiving end. Since the outlet of the optical fiber is very narrow, it can be ensured that 50% of its energy is concentrated on a single pixel. The initial PSF model of the TOF depth camera can be constructed in the form of high and low exposure. High and low exposure refers to the integration time of the TOF depth camera. For example, the integration time of high exposure is long, which can also be called a long exposure time; the integration time of low exposure is short, which can also be called a short exposure time. The initial PSF model is a model obtained by integrating a long exposure time model and a short exposure time model through a scaling relationship. It should be understood that the initial PSF model is the integrated amplitude diagram in the embodiment of the present application.

[0075] It should be noted that each pixel needs to obtain the corresponding PSF model using the above-mentioned high and low exposure forms. After obtaining the PSF model corresponding to each pixel, the initial PSF model can be formed.

[0076] In another embodiment, a PSF amplitude map with high exposure (i.e., long exposure time) and a PSF amplitude map with low exposure (i.e., short exposure time) are collected. Usually, the exposure time and amplitude are in an approximately linear relationship. In order to obtain the influence of the amplitude of a single pixel on other surrounding pixels, that is, the PSF amplitude map, very high energy is required, which can be achieved through a long exposure time. However, high exposure will bring a problem. The pixel area at the energy center will be overexposed, and this will require a PSF amplitude map with a short exposure time. The two PSF amplitude maps are integrated. For example, the overexposed part in the high-exposure PSF amplitude map can be replaced by a low-exposure local PSF amplitude map, and the two PSF amplitude maps are integrated to obtain an integrated amplitude map.

[0077] In this embodiment, the initial PSF model of the depth camera is constructed in the form of high and low exposure. Compared with the initial PSF model obtained by a single exposure time, it can more accurately represent the influence of the amplitude of a single pixel on the amplitudes of other surrounding pixels, thereby improving the accuracy of correcting the depth error.

[0078] In some embodiments, the obtained integrated amplitude map (i.e., the initial PSF model) can be directly stored within the depth camera. In actual use, the initial PSF model of the depth camera can be called by table lookup based on the pixel position in the original phase image. The original phase image and the initial PSF model of the original phase image obtained by table lookup can be further processed to obtain the ideal phase image. For example, the following formula can be deconvolved: Ideal phase image ★ Initial PSF model = Original phase image.

[0079] Among them, (★) represents convolution.

[0080] Optionally, S630, i.e., integrating the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map, includes: replacing the amplitude of a part of the pixel area in the first PSF amplitude map with the amplitude of a part of the pixel area in the second PSF amplitude map according to the proportional relationship between the first exposure time and the second exposure time, so as to obtain the integrated amplitude map.

[0081] Furthermore, optionally, based on the proportional relationship between the first exposure time and the second exposure time, the amplitudes of a portion of the pixel area in the first PSF amplitude map can be replaced with the amplitudes of a portion of the pixel area in the second function PSF amplitude map to obtain the integrated amplitude map. This can include replacing the amplitudes of a portion of the pixel area in the first PSF amplitude map with the product of the amplitudes of a portion of the pixel area in the second PSF amplitude map and the proportional relationship. Optionally, the portion of the pixel area in the first PSF amplitude map can be the central pixel area of ​​the first PSF amplitude map, and the portion of the pixel area in the second PSF amplitude map can also be the central pixel area of ​​the second PSF amplitude map.

[0082] For example, the long exposure time may be 1000 microseconds, the short exposure time may be 10 microseconds, and the long exposure time is 100 times the short exposure time. The overexposed area in the PSF amplitude diagram of the long exposure time may be replaced by multiplying the amplitude of the central pixel area of ​​the PSF amplitude diagram of the short exposure time by 100 times.

[0083] It should be understood that obtaining an integrated amplitude map based on the first and second PSF amplitude maps may not depend on the proportional relationship between the long exposure time and the short exposure time. For example, the overexposed area in the PSF amplitude map with a long exposure time can be replaced by multiplying the amplitude of the central pixel area of ​​the PSF amplitude map with a short exposure time by a preset value. It should also be understood that the replaced pixel area does not necessarily have to be the central pixel area of ​​the first and / or second PSF amplitude maps. In other words, the partial pixel area of ​​the first PSF amplitude map depends on the overexposed area of ​​the first PSF amplitude map. If the overexposed area of ​​the first PSF amplitude map is in the left half of the pixel area, the amplitude of the left half of the pixel area of ​​the first PSF amplitude map is replaced by the amplitude of the partial pixel area of ​​the second PSF amplitude map. The partial pixel area of ​​the second PSF amplitude map may also not be located in the central pixel area of ​​the second PSF amplitude map. For example, the amplitude of the overexposed area of ​​the first PSF amplitude map can be replaced by the amplitude of the right half of the pixel area of ​​the second PSF amplitude map. This embodiment of the present application is not limited to this.

[0084] Optionally, in some embodiments, the point spread function PSF model is a non-normalized point spread function PSF model, S640, that is, obtaining the PSF model corresponding to the depth camera based on the integrated amplitude map, including: dividing the amplitude of each pixel in the integrated amplitude map by the sum of the amplitudes of all pixels in the integrated amplitude map to obtain the non-normalized PSF model of the depth camera.

[0085] As mentioned above, the PSF model may include an initial PSF model obtained by using optical fiber or the like. In another embodiment, the PSF model may also include a PSF model obtained by further processing the obtained initial PSF model. In one embodiment, the integrated amplitude map may be non-normalized. For example, the amplitude of each pixel in the integrated amplitude map may be divided by the sum of the amplitudes of all pixels in the integrated amplitude map, so that the corresponding value of each pixel is between 0 and a value less than 1. Since there are multiple pixels in each original phase image, the initial PSF model corresponding to each pixel may be different. Multiple initial PSF models may be required during the deconvolution calculation process, resulting in an excessive amount of calculation. However, the solution of obtaining a scattering interference map based on the original phase image and the non-normalized PSF model corresponding to the original phase image, and then obtaining a scattering compensation map based on the original phase image and the scattering interference map can greatly reduce the amount of calculation.

[0086] In another embodiment, the integrated amplitude map may be normalized. For example, the amplitude of each pixel in the integrated amplitude map may be divided by the maximum amplitude in the integrated amplitude map, so that the corresponding value of each pixel is between 0 and 1.

[0087] Optionally, in an embodiment of the present application, an initial PSF model can be obtained before the depth camera leaves the factory and stored inside the depth camera; and / or a non-normalized PSF model can be obtained before the depth camera leaves the factory and stored inside the depth camera. After the depth camera leaves the factory, the PSF model of the original phase image can be obtained from the initial PSF model and / or the non-normalized PSF model stored inside the depth camera as needed.

[0088] Usually, due to the reasons of the optical system, the non-normalized PSF model corresponding to each pixel is different. In order to compensate for this difference, different non-normalized PSF models can be uniformly constructed at different pixel positions, such as uniformly taking different non-normalized PSF models at different positions of the 240*320 pixel image (such as 20*20 pixels corresponding to one position). Specifically, in the application field of sweepers, because the scattering of sweepers from the ground is more serious, it is distributed in the 80 rows of pixels below the image pixel 240*320, that is, 80*320. Based on this, taking 20*20 pixels as a position in the image, a 4*16 PSF model at the bottom of the image can be constructed. It should be noted that the non-normalized PSF model is constructed based on the image, and the arrangement of its number depends on the application scenario.

[0089] In one embodiment, the non-normalized PSF model is pre-constructed, and after construction is completed, it is associated with the pixel position relationship and stored (that is, the pixel position is associated with the non-normalized PSF model) and is obtained through a table lookup method according to the position relationship of the current pixel during actual use. Specifically, a storage method of the non-normalized PSF model corresponding to a single pixel can be adopted, such as down-sampling the original phase image and the PSF model by 5*5 at the same time, obtaining a scattering interference map after processing, and then up-sampling the original phase image by 5*5 to reduce the amount of calculation. It is also possible to use non-normalized PSF models corresponding to multiple pixels for storage. For example, based on the non-normalized PSF model, the influence of 5*5 pixels on other surrounding pixels is calculated, and a lookup table is constructed. In actual applications, the lookup table is called based on the average value of the 5*5 pixel values. In this way, the influence of 25 pixels instead of one pixel can be found at a time, thereby reducing the amount of calculation.

[0090] In some embodiments, S550, i.e., processing the ideal phase image based on the time-of-flight principle to obtain depth information of the target area, includes: obtaining, based on the ideal phase image, the amount of charge collected by each tap of each pixel in the TOF image sensor of the TOF depth camera; determining a depth value of each pixel based on the amount of charge collected by each tap of each pixel; and obtaining the depth information of the target area based on the depth value of each pixel.

[0091] Specifically, the depth calculation of the ideal phase image can be performed based on the time of flight principle. For example, taking the example of each pixel in the TOF image sensor containing three taps collecting light signals three times in sequence, the charge amount collected by each tap is C1, C2 and C3 based on the ideal phase image, and the pulse width T of the pulse collection signal of each tap is h , the flight time of a single pixel can be calculated using the following formula:

[0092] The depth value of a single pixel can then be obtained according to d=c*t / 2, where c represents the speed of light. The depth value of each pixel is obtained by traversing all pixels, thereby obtaining the depth information of the target area.

[0093] Figures 10 and 11 show a comparison of the effects of a conventional technical solution and the technical solution provided by the embodiments of this application. It is clear from the figures that in Figure 10, due to an error in the ground scattering depth, the entire wall surface appears incomplete and discontinuous. In Figure 11, however, the wall surface is clearly visible.

[0094] The depth calculation method according to the embodiment of the present application is described in detail above. The depth calculation device according to the embodiment of the present application will be described below in combination with Figures 12 to 13. The technical features described in the method embodiment are applicable to the following device embodiments.

[0095] Fig. 12 shows a schematic block diagram of a depth calculation device 700 according to an embodiment of the present application. As shown in Fig. 12, the depth calculation device 700 may include some or all of the following contents.

[0096] The first acquiring unit 710 is configured to acquire an original phase image of a target area captured by a TOF depth camera and a pre-stored PSF model corresponding to the TOF depth camera.

[0097] The second acquisition unit 720 is configured to obtain a scattering interference map using the original phase image and the PSF model. The scattering interference map includes a scattering interference amount indicating the degree to which the energy of at least some pixels in the original phase image interferes with the amplitudes of other surrounding pixels.

[0098] A third acquiring unit 730 is configured to adjust the scattering interference map based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map;

[0099] The fourth acquiring unit 740 is configured to compensate the original phase image using the scattering compensation map to obtain an ideal phase image.

[0100] The fifth acquiring unit 750 is configured to process the ideal phase image based on the time-of-flight principle to obtain depth information of the target area.

[0101] Optionally, in an embodiment of the present application, the third acquisition unit 730 is specifically used to: determine the relationship between the energy of a pixel in the original phase image and a preset first threshold; if the energy of a pixel in the original phase image is less than the first threshold, the amount of scattering interference generated by the corresponding pixel in the scattering interference map on the surrounding pixels is set to 0 to obtain a scattering compensation map.

[0102] Optionally, in an embodiment of the present application, the third acquisition unit 730 is specifically used to: process the energy of at least part of the pixels in the original phase image to obtain an amplitude map corresponding to the original phase image; determine the relationship between the amplitude values ​​of at least part of the pixels in the amplitude map corresponding to the original phase map and a preset second threshold; if the amplitude values ​​of at least part of the pixels in the amplitude map corresponding to the original phase map are greater than the preset second threshold, then set the amount of scattering interference received by the corresponding pixels in the scattering interference map to 0 to obtain a scattering compensation map.

[0103] Optionally, in the embodiment of the present application, the second acquisition unit 720 is specifically configured to: multiply the original phase image by the PSF model to obtain a scattering interference map.

[0104] Optionally, in the embodiment of the present application, the fourth acquiring unit 740 is specifically configured to: subtract the original phase image from the scattering compensation image to acquire an ideal phase image.

[0105] Optionally, as shown in Figure 13, the depth calculation device 700 also includes: a sixth acquisition unit 760, which is used to obtain the PSF model of the corresponding pixel of the original phase image from the pre-stored PSF model of the TOF depth camera by looking up the table according to the pixel position in the original phase image.

[0106] Optionally, an embodiment of the present application also provides a device 800 for obtaining a PSF model corresponding to a TOF depth camera. In this embodiment, the PSF model corresponding to the TOF depth camera is pre-acquired. As shown in Figure 14, the device 800 includes: a control unit 810, used to control the TOF depth camera to use a first exposure time to obtain a first PSF amplitude map, and control the TOF depth camera to use a second exposure time to obtain a second PSF amplitude map, where the second exposure time is less than the second exposure time; an integration unit 820, used to integrate the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map; an acquisition unit 830, used to obtain the PSF model corresponding to the TOF depth camera based on the integrated amplitude map.

[0107] Optionally, in an embodiment of the present application, the integration unit 820 is specifically used to replace the amplitude of a part of the pixel area in the first PSF amplitude map with the product of the amplitude of a part of the pixel area in the second PSF amplitude map and the proportional relationship between the first exposure time and the second exposure time to obtain an integrated amplitude map.

[0108] Optionally, in the embodiment of the present application, the partial pixel area of ​​the first PSF amplitude map is the central pixel area of ​​the first PSF amplitude map, and the partial pixel area of ​​the second PSF amplitude map is the central pixel area of ​​the second PSF amplitude map.

[0109] Optionally, in an embodiment of the present application, the PSF model is a non-normalized PSF model, and the acquisition unit 830 is specifically used to: divide the amplitude of each pixel in the integrated amplitude map by the sum of the amplitudes of all pixels in the integrated amplitude map to obtain the non-normalized PSF model of the depth camera.

[0110] It should be understood that the depth calculation device 700 according to the embodiment of the present application may correspond to the execution entity in the embodiment of the depth calculation method 500 according to the embodiment of the present application, and the device 800 for obtaining the PSF model corresponding to the TOF depth camera according to the embodiment of the present application may correspond to the execution entity in the embodiment of the method 600 for obtaining the PSF model corresponding to the TOF depth camera according to the embodiment of the present application, and the above-mentioned and other operations and / or functions of each unit in the depth calculation device 700 are respectively for implementing the corresponding processes in the methods of Figures 1 to 8, and the above-mentioned and other operations and / or functions of each unit in the device 800 are respectively for implementing the corresponding processes in the method shown in Figure 9. For the sake of brevity, they will not be repeated here.

[0111] Optionally, as shown in FIG15 , an embodiment of the present application further provides a TOF depth camera 900. As shown in FIG15 , the TOF depth camera 900 may include a transmitter 910, a receiver 920, and a processor 930, wherein the transmitter 910 is used to project a periodically modulated transmitted light signal to the target area, the receiver 920 is used to collect the received light signal transmitted back from the target area to generate an original phase image, and the processor 930 is used to call and run a program from a memory to implement the depth calculation method provided in the embodiment of the present application. For example, the processor 930 can control the TOF receiver 920 to collect the original phase image, the processor 930 can also obtain a pre-stored PSF model corresponding to the TOF depth camera from the memory, and the processor 930 is also used to obtain a scattering interference map using the original phase image and the PSF model. The processor 930 can also adjust the scattering interference map based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map. The processor 930 can also use the scattering compensation map to compensate the original phase image to obtain an ideal phase image. The processor 930 may also process the ideal phase image based on the time-of-flight principle to obtain depth information of the target area. Optionally, the processor 930 may correspond to the depth calculation device 700 shown in FIG. 12 and FIG. 13 .

[0112] Optionally, as shown in FIG15 , the TOF depth camera 900 may further include a memory 940 , wherein the processor 930 may call and run a depth calculation program from the memory 940 to implement the method in the embodiment of the present application.

[0113] The memory 940 may be a separate device independent of the processor 930 , or may be integrated into the processor 930 .

[0114] Optionally, in an embodiment of the present application, the TOF depth camera 900 is an iTOF depth camera.

[0115] Optionally, an embodiment of the present application further provides a chip, including a processor, which can call and run a computer program from a memory to implement the method in the embodiment of the present application.

[0116] Optionally, an embodiment of the present application further provides a computer-readable storage medium for storing a computer program, which enables a computer to execute the method in the embodiment of the present application.

[0117] Optionally, an embodiment of the present application further provides a computer program product, including computer program instructions, which enable a computer to execute the method in the embodiment of the present application.

[0118] Optionally, the embodiments of the present application further provide a computer program that enables a computer to execute the method in the embodiments of the present application.

[0119] It is understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0120] Although the present application has been described with reference to preferred embodiments, various modifications may be made thereto and components may be substituted with equivalents without departing from the scope of the present application. In particular, the various technical features described in the various embodiments may be combined in any manner as long as there are no structural conflicts. The present application is not limited to the specific embodiments disclosed herein, but encompasses all technical solutions within the scope of the claims.

Claims

1. A depth calculation method applied to TOF depth camera, It is characterized in that include: Acquire an original phase image including a target area captured by the TOF depth camera and a pre-stored PSF model corresponding to the TOF depth camera; Obtaining a scattering interference map using the original phase image and the PSF model, wherein the scattering interference map includes a scattering interference amount indicating the energy of at least part of the pixels in the original phase image on other surrounding pixels; Adjusting the scattering interference map based on energy of at least part of pixels in the original phase image to obtain a scattering compensation map; Compensating the original phase image using the scattering compensation map to obtain an ideal phase image; The ideal phase image is processed based on the time-of-flight principle to obtain the depth information of the target area.

2. The depth calculation method according to claim 1, It is characterized in that The step of adjusting the scattering interference map based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map includes: Determine the relationship between the energy of the pixel in the original phase image and a preset first threshold; If the energy of a certain pixel in the original phase image is less than the first threshold, the scattering interference amount generated by the corresponding pixel on the surrounding pixels in the scattering interference map is set to 0 to obtain the scattering compensation map.

3. The depth calculation method according to claim 1, It is characterized in that The step of adjusting the scattering interference map based on the energy of at least part of the pixels in the original phase image to obtain a scattering compensation map includes: Processing the energy of at least part of the pixels in the original phase image to obtain an amplitude map corresponding to the original phase image; Determine the magnitude relationship between the amplitude value of at least some pixels in the amplitude map corresponding to the original phase image and a preset second threshold; If the amplitude values ​​of at least some pixels in the amplitude map corresponding to the original phase image are greater than the preset second threshold, the scattering interference amount of the corresponding pixels in the scattering interference map is set to 0 to obtain the scattering compensation map.

4. The depth calculation method according to claim 1, It is characterized in that The step of obtaining a scattering interference map by using the original phase image and the PSF model includes: The original phase image is multiplied by the PSF model to obtain the scattering interference map.

5. The depth calculation method according to claim 1, It is characterized in that The method of compensating the original phase image by using the scattering compensation map to obtain an ideal phase image includes: The original phase image is subtracted from the scatter compensation image to obtain the ideal phase image.

6. The depth calculation method according to claim 1, It is characterized in that The depth calculation method further includes: According to the pixel position in the original phase image, the PSF model of the corresponding pixel of the original phase image is obtained from the pre-stored PSF model of the TOF depth camera by table lookup.

7. The depth calculation method according to claim 6, It is characterized in that The PSF model of the TOF depth camera is pre-acquired, and the acquiring of the PSF model of the depth camera includes: Controlling the TOF depth camera to acquire a first PSF amplitude image using a first exposure time; Controlling the TOF depth camera to acquire a second PSF amplitude map using a second exposure time, wherein the second exposure time is less than the second exposure time; Integrate the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map; A PSF model of the TOF depth camera is obtained according to the integrated amplitude map.

8. The depth calculation method according to claim 7, It is characterized in that The step of integrating the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map includes: The amplitude of a part of the pixel area in the first PSF amplitude map is replaced by the product of the amplitude of a part of the pixel area in the second PSF amplitude map and the proportional relationship between the first exposure time and the second exposure time to obtain the integrated amplitude map.

9. The depth calculation method according to claim 8, It is characterized in that The partial pixel area of ​​the first PSF amplitude map is the central pixel area of ​​the first PSF amplitude map, and the partial pixel area of ​​the second PSF amplitude map is the central pixel area of ​​the second PSF amplitude map.

10. The depth calculation method according to claim 7, It is characterized in that The PSF model is a non-normalized PSF model, and obtaining the PSF model of the depth camera according to the integrated amplitude map includes: The amplitude of each pixel in the integrated amplitude map is divided by the sum of the amplitudes of all pixels in the integrated amplitude map to obtain a non-normalized PSF model of the depth camera.

11. A TOF depth camera, It is characterized in that It includes a transmitter, a receiver and a processor, wherein the transmitter is used to project a periodically modulated transmission light signal to a target area, the receiver is used to collect a reception light signal transmitted back from the target area to generate an original phase image, and the processor is used to process the original phase image using the depth calculation method described in any one of claims 1 to 10 to obtain the depth information of the target area.

12. A computer-readable storage medium, It is characterized in that Used to store a computer program, wherein the computer program enables a computer to execute the depth calculation method according to any one of claims 1 to 10.

Citation Information

Patent Citations

  • Method for denoising based on pulse type ToF depth image

    CN110097513A

  • Depth calculation method, TOF depth camera and computer readable storage medium

    CN116381729A

  • Light time-of-flight camera

    DE102019110329A1

  • Generating an improved depth map using a multi-aperture imaging system

    US20160255334A1

  • System for 3D image filtering

    US20180211398A1