Depth calculation method, TOF depth camera, and computer-readable storage medium

Abstract

Embodiments of the present application provide a depth calculation method, a TOF depth camera, and a computer-readable storage medium. The depth calculation method is applied to a TOF depth camera and includes: obtaining an original phase image including a target area collected by the TOF depth camera and a PSF model corresponding to the TOF depth camera stored in advance; obtaining a scattering interference map by using the original phase image and the PSF model; adjusting the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map; compensating the original phase image by using the scattering compensation map to obtain an ideal phase image; and processing the ideal phase image based on the time-of-flight principle to obtain depth information of the target area. The depth calculation method, the TOF depth camera, and the computer-readable storage medium according to the embodiments of the present application can alleviate depth errors.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular, to a depth calculation method, a TOF depth camera, and a computer-readable storage medium. Background Art

[0002] In the actual application environment of a depth camera, in addition to receiving the reflected pulsed light directly reflected by the object to be measured, there are also some other non-directly reflected lights. After these non-directly reflected lights are received by some pixels, an internal scattering phenomenon will occur, resulting in depth errors for these pixels. Summary of the Invention

[0003] The embodiments of the present application provide a depth calculation method, a TOF depth camera, and a computer-readable storage medium, which can alleviate depth errors.

[0004] In a first aspect, a depth calculation method is provided. An original phase image including a target area collected by a TOF depth camera and a PSF model corresponding to the TOF depth camera stored in advance are obtained; a scattering interference map is obtained by using the original phase image and the PSF model, and the scattering interference map includes a scattering interference amount for indicating the interference degree of the energy of at least some pixels in the original phase image on the energy of other surrounding pixels; the scattering interference map is adjusted based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map; the original phase image is compensated by using the scattering compensation map to obtain an ideal phase image; and the depth information of the target area is obtained by processing the ideal phase image based on the time-of-flight principle.

[0005] In a second aspect, a TOF depth camera is provided, including a transmitting end, a receiving end, and a processor. The transmitting end is configured to project a periodically modulated transmitted light signal to a target area, the receiving end is configured to collect a received light signal reflected back by the target area to generate an original phase image, and the processor is configured to process the original phase image by using the depth calculation method in the first aspect and any one of the possible implementation manners in the first aspect to obtain the depth information of the target area.

[0006] In a third aspect, a computer-readable storage medium is provided for storing a computer program, and the computer program causes a computer to execute the depth calculation method in the first aspect and any one of the possible implementation manners in the first aspect.

[0007] In a fourth aspect, a chip is provided, including a processor for calling and running a computer program from a memory, so that a device installed with the chip executes the depth calculation method in the first aspect and any one of the possible implementation manners in the first aspect.

[0008] Fifth aspect, a computer program is provided, which enables a computer to execute the depth calculation method in any possible implementation manner of the first aspect and its first aspect.

[0009] Sixth aspect, a program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the depth calculation method in any possible implementation manner of the first aspect and its first aspect.

[0010] Based on the above technical solutions, first, obtain the original phase image including the target area collected by the TOF depth camera and the PSF model corresponding to the TOF depth camera stored in advance. Secondly, use the original phase image and the PSF model to obtain the scattering interference map, and adjust the scattering interference map based on the energy of at least some pixels in the original phase image to obtain the scattering compensation map. Finally, compensate the original phase image with the scattering compensation map to obtain the ideal phase image, and process the ideal phase image based on the time-of-flight principle to obtain the depth information of the target area. The ideal phase image obtained by compensating the original phase image with the scattering compensation map can reduce the calculation amount compared with the ideal phase image obtained by directly performing deconvolution on the ideal phase image * PSF model = the original phase image, and can also more accurately obtain the original phase image in the ideal situation, achieving the purpose of alleviating the depth error. Description of the Drawings

[0011] Figure 1 Shows a schematic structural diagram of an iTOF depth camera.

[0012] Figure 2 Shows an application schematic diagram of the iTOF depth camera according to an embodiment of the present application.

[0013] Figure 3 Shows a schematic block diagram of the depth calculation method according to an embodiment of the present application.

[0014] Figure 4 Shows a pixel distribution diagram of the original phase image and the scattering interference map according to an embodiment of the present application.

[0015] Figure 5 Shows another schematic block diagram of the depth calculation method according to an embodiment of the present application.

[0016] Figure 6 Shows a pixel distribution diagram of the scattering compensation map according to an embodiment of the present application.

[0017] Figure 7 Shows a pixel distribution diagram of the amplitude map corresponding to the original phase image according to an embodiment of the present application.

[0018] Figure 8 Shows the pixel distribution diagram of the ideal phase image of the embodiment of the present application.

[0019] Figure 9 Shows a schematic block diagram of a method for obtaining a PSF model of a TOF depth camera according to an embodiment of the present application.

[0020] Figure 10 and Figure 11 Shows a comparison diagram of the effects of depth errors using the traditional scheme and the technical scheme provided by the embodiment of the present application respectively.

[0021] Figure 12 Shows a schematic block diagram of a depth calculation device according to an embodiment of the present application.

[0022] Figure 13 Shows another schematic block diagram of a depth calculation device according to an embodiment of the present application.

[0023] Figure 14 Shows a schematic block diagram of a device for obtaining a PSF model of a TOF depth camera according to an embodiment of the present application.

[0024] Figure 15 Shows a schematic block diagram of a TOF depth camera according to an embodiment of the present application. Detailed implementation manners

[0025] Next, the technical solutions in the embodiments of the present application will be described in conjunction with the accompanying drawings.

[0026] A Time-Of-Flight (TOF) depth camera, abbreviated as TOF depth camera, whose basic principle is to continuously send light pulses to a target object, and then use a sensor to receive the light returned from the target object, and obtain the distance of the target object by detecting the flight (round-trip) time of the light pulse. TOF depth cameras generally include direct-Time-of-Flight (DTOF) depth cameras and indirect-Time-Of-Flight depth cameras. Among them, the basic principle of an iTOF depth camera is to calculate the flight time by measuring the phase difference between the emitted light signal and the received light signal. Specifically, as Figure 1As shown in the figure, the iTOF depth camera 100 includes a transmitting end 101, a receiving end 102, and a processor 103. Among them, the transmitting end 101 emits a transmitting optical signal modulated periodically in time series to the surface of the object to be measured in the target area. The receiving end 102 collects the received optical signal reflected from the surface of the object to be measured. The received optical signal reflected from the surface of the object to be measured generates a time delay relative to the transmitting optical signal in terms of time sequence, which is manifested as an additional phase delay on the periodically modulated transmitting optical signal. The processor 103 is used to read out the phase delay and obtain the flight time of the optical signal traveling back and forth between the iTOF depth camera 100 and the object to be measured based on the phase delay. Furthermore, the distance between the object to be measured and the iTOF depth camera can be expressed as: where D is the depth between the object to be measured and the iTOF depth camera, is the phase delay between the transmitting optical signal and the reflected optical signal, f is the frequency of the transmitting optical signal, and c is the speed of light.

[0027] In one embodiment, the transmitting end 101 includes a light source, a lens, and / or a patterned optical element. Among them, the light source includes a single vertical-cavity surface-emitting laser (VCSEL) or a VCSEL array for emitting a light beam to the lens. The lens includes a single lens or a lens group for collimating the light beam. The patterned optical element includes any one of diffractive optical elements (DOE), a microlens array (MLA), and a mask, for modulating the light beam so that the light beam passing through the patterned optical element is projected onto the target area in any form such as speckle or floodlight, without limitation here.

[0028] In one embodiment, the receiving end 102 includes a TOF image sensor. The TOF image sensor receives the reflected light beam and generates a raw phase image. Preferably, the TOF image sensor includes a plurality of pixels, and each pixel includes at least one tap for storing charge. Each tap is used to accumulate the electric charge generated by the light signal reflected from the object to be measured collected by the corresponding pixel within a preset exposure time, and outputs a raw phase image containing the depth information of the object to be measured. Among them, the raw phase image is the raw data obtained by the TOF image sensor converting the collected light signal into a digital signal. Further, the receiving end 102 further includes a lens disposed on the light incident side of the TOF image sensor for focusing the reflected light beam onto the corresponding pixels of the TOF image sensor.

[0029] In one embodiment, each pixel in the TOF image sensor includes three or more taps for storing, reading, or discharging the charge signal generated by the reflected light pulse under the control of the corresponding electrode. When each pixel includes multiple taps, the taps are sequentially switched in a certain order within a single frame period T (or within a single exposure time) to collect the electrons generated by the pixel receiving the light signal reflected by the target object, thereby forming a charge signal.

[0030] In one embodiment, the processor 103 can be an independent dedicated circuit, such as a dedicated system-on-chip (SOC) chip, a field programmable gate array (FPGA) chip, an application specific integrated circuit (ASIC) chip, etc. that includes a central processing unit (CPU), a memory, a bus, etc., or it can also include a general processing circuit. For example, when the depth camera is integrated into intelligent terminals such as mobile phones, TVs, and computers, the processing circuit in the terminal can be at least a part of the processor 103. In some embodiments, the processor 103 is used to provide the modulation signal (transmission signal) required when the transmitter 101 emits a light signal, and the transmitter 101 emits a light beam to the object to be measured under the control of the modulation signal; in addition, the processor 103 also provides the demodulation signal (acquisition signal) for each tap in the image sensor of the receiver 102, and the tap collects the charge signal generated by the light beam including the reflected pulse light beam reflected by the object to be measured under the control of the demodulation signal; the processor 103 can also provide auxiliary monitoring signals, such as temperature sensing, overcurrent, overvoltage protection, disconnection protection, etc.; the processor 103 can also be used to save and process the raw data collected by each tap in the image sensor to obtain the specific position information of the object to be measured.

[0031] In some embodiments, the iTOF depth camera can also include devices such as a driving circuit, a power supply, a color camera, an infrared camera, an inertial measurement unit (IMU), etc. These are not shown in the figure, but the combination with these devices can achieve more abundant functions, such as 3D texture modeling, infrared face recognition, simultaneous localization and mapping (SLAM), etc. The iTOF depth camera can be embedded in electronic products such as mobile phones, tablet computers, and computers.

[0032] In the actual application environment of an iTOF depth camera, in addition to receiving the reflected pulse light beam directly reflected by the object to be measured, there is also some other non-directly reflected light. After this non-directly reflected light is received by some pixels, it will cause an internal scattering phenomenon, resulting in depth errors for these pixels. For example, as Figure 2 shown, when the iTOF depth camera 100 is mounted on a sweeping robot 200 for obstacle avoidance applications, it only needs to determine whether there are obstacles 300 on the ground within a certain distance directly in front. It requires a relatively large field of view in the horizontal and vertical directions. Since the iTOF depth camera 100 mounted on the sweeping robot 200 is relatively close to the ground 400, and the ground 400 is a high-reflectivity area, the light beam emitted by the transmitter 101 in the iTOF depth camera 100 is easily reflected back by the ground 400 and enters the receiver 102, causing an internal scattering phenomenon in the receiver 102. That is, the light beam reflected back by the high-reflectivity area will reflect back and forth between the receiving lens and the pixel plane of the image sensor. A part of the light beam will be received by other pixels, thus generating a depth error.

[0033] In view of this, the embodiments of the present application provide a depth calculation method that can alleviate depth errors.

[0034] Figure 3 shows a schematic block diagram of the depth calculation method 500 according to the embodiments of the present application. Optionally, the depth calculation method 500 can be applied to a TOF depth camera. For example, the depth calculation method 500 can be applied to an iTOF depth camera 100 as Figure 1 shown. The depth calculation method 500 can be executed by the processor 103 in the iTOF depth camera 100. Specifically, as Figure 3 shown, the depth calculation method 500 can include the following parts or all of the content.

[0035] S510, obtain the original phase image including the target area collected by the TOF depth camera and the PSF model corresponding to the pre-stored TOF depth camera.

[0036] S520, use the original phase image and the PSF model to obtain a scattering interference map. The scattering interference map includes scattering interference amounts for indicating the interference degree of the energy of at least some pixels in the original phase image on the energy of other surrounding pixels.

[0037] S530, adjust the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map.

[0038] S540, use the scattering compensation map to compensate the original phase image to obtain an ideal phase image.

[0039] The S550 processes the ideal phase image based on the time-of-flight principle to obtain the depth information of the target area.

[0040] Generally, when there is an interfering object between the TOF depth camera and the target area, ideally, the transmitting end emits light beams to the target area and the interfering object and the receiving end receives them. The light beams reflected by the interfering object and the light beams reflected back from the target area do not interfere with each other, that is, it does not affect the receiving end from collecting the light beams reflected back from the target area, so that the original phase image collected by the TOF depth camera is an original phase image without depth error. However, if the interfering object between the TOF depth camera and the target area is a highly reflective object, because the highly reflective object has strong reflection energy and is located between the depth camera and the target area, the light beams reflected from the surface of the highly reflective object will spread and be received by the receiving end through diffraction, reflection and other ways and enter the pixels of the image sensor, thus affecting the receiving end from receiving the light beams with weaker energy reflected back from the target area. When the receiving end receives the light beams with weaker energy reflected back from the target area, it also receives the light beams reflected from the surface of the highly reflective object, so that the corresponding pixels in the image sensor that receive this part of the light beams generate depth errors. At this time, the original phase image collected by the TOF depth camera is an original phase image with depth errors. Generally, the TOF depth camera has a maximum detection range and a minimum detection range, and the target area is the area between the minimum detection range and the maximum detection range where depth information needs to be obtained.

[0041] It should be understood that the original phase image in the embodiments of the present application refers to the image collected by the TOF depth camera without any calculation and processing. The value of the pixel in the original phase image is the energy information, and the energy information can be expressed as the electric charge collected by each tap on the pixel in the TOF image sensor of the depth camera.

[0042] It should be noted that the pixel positions in the original phase image correspond to the pixel positions in the scattering interference map. That is to say, the original phase image and the scattering interference map include the same number of pixel data. For example, as Figure 4 shown, the pixel distributions in the original phase image and the scattering interference map are both 4*4. Among them, the pixels in the original phase image are A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16 respectively. The pixels in the scattering interference map are B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14, B15, B16 respectively. A1 corresponds to the same pixel as B1, A2 corresponds to the same pixel as B2,..., A16 corresponds to the same pixel as B16, etc.

[0043] Similarly, the pixel positions in the scattering compensation map, the original phase image, and the scattering interference map also correspond to each other, and the pixel positions in the scattering compensation map, the original phase image, and the ideal phase image also correspond to each other. This will be described in detail below with reference to the accompanying drawings.

[0044] Generally, the three-dimensional light intensity distribution of the image formed by a point light source of an object point through an optical system is called the Point Spread Function (PSF). In the embodiments of the present application, the PSF model may include energy data, which represents the influence of the amplitude of a single pixel on the amplitudes of other surrounding pixels. Generally, the PSF model corresponding to the TOF depth camera refers to the initial PSF model obtained by means of optical fibers or the like before the TOF depth camera leaves the factory and is stored inside the TOF depth camera for subsequent use. In some other embodiments, further processing of the obtained initial PSF model may also be included before the TOF depth camera leaves the factory. For example, normalizing or non-normalizing the initial PSF model. And further, the processed PSF model can be stored inside the TOF depth camera for subsequent use. That is to say, in S510, the pre-stored PSF model corresponding to the TOF depth camera obtained may be the initial PSF model obtained before the TOF depth camera leaves the factory, or the PSF model obtained after processing the initial PSF model. Or in other words, in S520, the scattering interference map can be obtained by using the original phase image and the initial PSF model of the pre-stored depth camera, or the scattering interference map can be obtained by using the original phase image and the processed PSF model of the pre-stored depth camera.

[0045] Theoretically, the ideal phase image can be obtained by deconvolving the ideal phase image *PSF model = the original phase image. However, since there are multiple pixels in the image, the PSF models corresponding to each pixel may be different, that is, the deconvolution calculation process may require multiple PSF models, resulting in a large amount of computation. In the embodiments of the present application, first, the original phase image including the target area collected by the TOF depth camera and the PSF model corresponding to the TOF depth camera stored in advance are obtained. Secondly, the scattering interference map is obtained by using the original phase image and the PSF model, and the scattering interference map is adjusted based on the energy of at least some pixels in the original phase image to obtain the scattering compensation map. Finally, the original phase image is compensated with the scattering compensation map to obtain the ideal phase image, and the depth information of the target area is obtained by processing the ideal phase image based on the time-of-flight principle. The ideal phase image obtained by compensating the original phase image with the scattering compensation map can reduce the amount of computation compared with the ideal phase image obtained by directly deconvolving the ideal phase image *PSF model = the original phase image, and can also more accurately obtain the original phase image in the ideal situation, achieving the purpose of alleviating the depth error.

[0046] In some embodiments, as Figure 5 shown, S530, that is, adjusting the scattering interference map based on the energy of at least some pixels in the original phase image to obtain the scattering compensation map, may include the following parts or all of the content.

[0047] S531, determining the magnitude relationship between the energy of the pixels in the original phase image and a preset first threshold.

[0048] S532, if the energy of a certain pixel in the original phase image is less than the first threshold, then set the scattering interference amount generated by the corresponding pixel in the scattering interference map on the surrounding pixels to 0 to obtain the scattering compensation map.

[0049] Optionally, in one embodiment, the scattering interference map may include the scattering interference amounts corresponding to all the pixels in the original phase image. In other embodiments, the scattering interference map may also include the scattering interference amounts corresponding to some pixels in the original phase image. The scattering interference amount of each pixel may be determined based on the energy of each pixel in the original phase image and the PSF model of the corresponding pixel in the PSF model corresponding to the depth camera. For example, the value of pixel A1 in the original phase image is A1, and the PSF model corresponding to the depth camera is a non-normalized PSF model, and the matrix X1 corresponding to pixel A1 can be obtained from this non-normalized PSF model, that is, the scattering interference amount of pixel A1 on the surrounding pixels can be obtained as A1*X1, and so on, the scattering interference amounts of each pixel in the original phase image on the surrounding pixels can be obtained.

[0050] After obtaining the scattering interference map, further, according to the relationship between the energy of the pixels in the original phase image and the first threshold, a scattering compensation map can be obtained. For example, assuming that the values of the pixel positions in the left half of the original phase image are less than the preset first threshold, it indicates that the interference generated by the current pixel to the surrounding pixels is small and can be ignored. That is, the values of the pixel positions in the left half of the scattering interference map can be set to 0, so as to obtain the scattering compensation map as shown in Figure 6 shown.

[0051] In this embodiment, after obtaining the scattering interference map, it is also possible to further determine the relationship between the energy of the pixels in the original phase image and the preset first threshold. If the energy of a certain pixel in the original phase image is less than the first threshold, the scattering interference amount of the corresponding pixel in the scattering interference map is set to 0, so as to obtain the scattering compensation map. Since the energy of a certain pixel in the original phase image is weak, it will not affect the depth effect of other pixels, so that the calculation complexity can be reduced while not affecting the depth error correction.

[0052] Optionally, the first threshold can be a preset value set according to experience. For example, the first threshold can be a threshold selected according to factors such as the application scenario and exposure time.

[0053] In some other embodiments, as shown in Figure 6 S530, that is, adjusting the scattering interference map based on the energy of at least some pixels in the original phase image to obtain the scattering compensation map, may include the following part or all of the content.

[0054] S533, processing the energy of at least some pixels in the original phase image to obtain the amplitude map corresponding to the original phase image. In the embodiments of the present application, the image calculated based on the original phase image refers to an image including amplitude information.

[0055] S534, determining the relationship between the amplitude values of at least some pixels in the amplitude map corresponding to the original phase map and the preset second threshold;

[0056] S535, if the amplitude values of at least some pixels in the amplitude map corresponding to the original phase map are greater than the preset second threshold, setting the scattering interference amount received by the corresponding pixels in the scattering interference map to 0 to obtain the scattering compensation map.

[0057] Optionally, after processing the energy of at least some pixels in the original phase image in Figure 4 , an image as shown in Figure 7The amplitude map corresponding to the original phase image shown. Similarly, the correspondence between the original phase image and the amplitude map can refer to the correspondence between pixel positions. Processing the energy of at least some of the pixels in the original phase image to obtain the amplitude map specifically includes accumulating several pixels in the original phase image and subtracting the ambient noise to obtain the pixel value corresponding to a pixel in the amplitude map, and obtaining the pixel value corresponding to a pixel in the amplitude map. For example, Figure 7 The pixel distribution of the amplitude map in [reference] is also 4*4. The pixels in this amplitude map are C1, C2, C3, C4, C5, C6, C7, C8, C9, C10, C11, C12, C13, C14, C15, C16 respectively. Adding the values obtained by pixel A1 within the sampling time and subtracting the ambient noise can obtain the value of pixel C1 in the amplitude map, and so on for the remaining pixels in the amplitude map.

[0058] Optionally, in the case where the PSF model is a non-normalized PSF model, in S520, that is, obtaining the scattering interference map by using the original phase image and the PSF model: multiplying the original phase image by the PSF model to obtain the scattering interference map.

[0059] For example, the scattering interference map can be obtained through the following formula:

[0060] Original phase image × Non-normalized PSF model = Scattering interference map.

[0061] In this embodiment, multiplying the non-normalized PSF model by the original phase image can obtain the scattering interference map, and further, an ideal phase image can be obtained based on the scattering compensation map obtained from the scattering interference map and the original phase image. The result is approximately the same as the result of deconvolving the original phase image and the initial PSF model, and it can also greatly reduce the computational amount.

[0062] In some embodiments, in S540, that is, compensating the original phase image with the scattering compensation map to obtain the ideal phase image, includes: subtracting the original phase image from the scattering compensation map to obtain the ideal phase image.

[0063] For example, subtracting Figure 4 the values of the corresponding pixel positions of the original phase image in [reference] and Figure 6 the scattering compensation map in [reference] can obtain the ideal phase image. This ideal phase image can be as shown in Figure 8As shown. That is, the values of the pixels included in the ideal phase image are A1, A2, A3 - B3, A4 - B4, A5, A6, A7 - B7, A8 - B8, A9, A10, A11 - B11, A12 - B12, A13, A14, A15 - B15, A16 - B16 respectively. Among them,..., A3 and A3 - B3 correspond to the same pixel, A4 and A4 - B4 correspond to the same pixel,..., A7 and A7 - B7 correspond to the same pixel, A8 and A8 - B8 correspond to the same pixel,..., A15 and A15 - B15 correspond to the same pixel, A16 and A16 - B16 correspond to the same pixel.

[0064] Optionally, in the embodiments of the present application, the depth calculation method 500 may further include: obtaining the PSF model of the corresponding pixel of the original phase image from the pre - stored PSF model of the TOF depth camera by looking up a table according to the pixel position in the original phase image.

[0065] In some embodiments, before the depth camera leaves the factory, the PSF model of the TOF depth camera can be constructed by means of optical fibers, etc. and stored inside the depth camera. After the TOF depth camera leaves the factory and during actual use, according to the pixel position in the original phase image, the PSF model stored inside the TOF depth camera can be called by looking up a table. For example, according to all the pixels in the original phase image, the PSF models corresponding to all these pixels can be obtained by looking up a table. For another example, according to some of the pixels in the original phase image, the PSF models corresponding to these part of the pixels can be obtained by looking up a table.

[0066] As mentioned above, the PSF model of the TOF depth camera is pre - stored inside the TOF depth camera, but before leaving the factory, the PSF model of this TOF depth camera can be obtained in advance. The embodiments of the present application also provide a method for obtaining the PSF model of the TOF depth camera.

[0067] Specifically, it can be seen that Figure 9 the method 600 for obtaining the PSF model of the TOF depth camera may include the following part or all of the content.

[0068] S610, controlling the TOF depth camera to obtain the first PSF amplitude map with the first exposure time;

[0069] S620, controlling the TOF depth camera to obtain the second PSF amplitude map with the second exposure time, and the second exposure time is less than the first exposure time.

[0070] S630, integrating the first PSF amplitude map and the second PSF amplitude map to obtain the integrated amplitude map.

[0071] S640. Obtain the PSF model of the TOF depth camera according to the integrated amplitude map.

[0072] Optionally, in one embodiment, the PSF model of the TOF depth camera can be obtained by using an optical fiber. Optionally, one end of the optical fiber is connected to the transmitting end of the depth camera, and the other end is aligned with the receiving end. Since the outlet of the optical fiber is very narrow, 50% of its energy can be ensured to be concentrated on a single pixel. The initial PSF model of the TOF depth camera can be constructed in the form of high and low exposure. High and low exposure refer to the integration time of the TOF depth camera. For example, the integration time of high exposure is long, which can also be referred to as a long exposure time; the integration time of low exposure is short, which can also be referred to as a short exposure time. The initial PSF model is a model obtained by integrating a long exposure time model and a short exposure time model through a scaling ratio relationship. It should be understood that this initial PSF model is the integrated amplitude map in the embodiments of the present application.

[0073] It should be noted that each pixel needs to obtain the corresponding PSF model in the above form of high and low exposure. After obtaining the PSF model corresponding to each pixel, the initial PSF model can be formed.

[0074] In another embodiment, collect a PSF amplitude map of high exposure (i.e., long exposure time) and a PSF amplitude map of low exposure (i.e., short exposure time). Generally, the exposure time and the amplitude are in an approximately linear relationship. To obtain the influence of the amplitude of a single pixel on other surrounding pixels, that is, the PSF amplitude map, high energy is required, which can be achieved through a long exposure time. However, high exposure will bring a problem that the pixel area at the energy center will be overexposed, and this requires a PSF amplitude map with a short exposure time. Integrate these two PSF amplitude maps. For example, the overexposed part in the PSF amplitude map of high exposure can be replaced with the local PSF amplitude map of low exposure, and the two PSF amplitude maps are integrated to obtain the integrated amplitude map.

[0075] In this embodiment, the initial PSF model of the depth camera is constructed in the form of high and low exposure. Compared with the initial PSF model obtained by using a single exposure time, it can more accurately represent the influence of the amplitude of a single pixel on the amplitudes of other surrounding pixels, thereby improving the accuracy of correcting the depth error.

[0076] In some embodiments, the obtained integrated amplitude map (i.e., the initial PSF model) can be directly stored inside the depth camera. During actual use, according to the pixel position in the original phase image, the initial PSF model of the depth camera can be called by looking up a table, and the original phase image and the initial PSF model of the original phase image obtained by looking up the table can be further processed to obtain an ideal phase image. For example, the following formula can be deconvolved:

[0077] Ideal phase image ★ Initial PSF model = Original phase image.

[0078] Among them, (★) represents convolution.

[0079] Optionally, in S630, that is, integrating the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map, including: replacing the amplitudes of some pixel regions in the first PSF amplitude map with the amplitudes of some pixel regions in the second PSF amplitude map according to the proportional relationship between the first exposure time and the second exposure time, so as to obtain the integrated amplitude map.

[0080] Furthermore, optionally, according to the proportional relationship between the first exposure time and the second exposure time, the amplitudes of some pixel regions in the first PSF amplitude map can be replaced with the amplitudes of some pixel regions in the second function PSF amplitude map to obtain the integrated amplitude map, which may include: replacing the amplitudes of some pixel regions in the first PSF amplitude map with the product of the amplitudes of some pixel regions in the second PSF amplitude map and the proportional relationship. Optionally, some pixel regions of the first PSF amplitude map can be the central pixel region of the first PSF amplitude map, and some pixel regions of the second PSF amplitude map can also be the central pixel region of the second PSF amplitude map.

[0081] For example, the long exposure time can be 1000 microseconds, the short exposure time can be 10 microseconds, and the long exposure time is 100 times the short exposure time. The amplitude of the central pixel region of the PSF amplitude map with the short exposure time can be multiplied by 100 times to replace the overexposed region in the PSF amplitude map with the long exposure time.

[0082] It should be understood that obtaining the integrated amplitude map according to the first PSF amplitude map and the second PSF amplitude map may not depend on the proportional relationship between the long exposure time and the short exposure time. For example, the amplitude of the central pixel region of the PSF amplitude map with the short exposure time can be multiplied by a preset value to replace the overexposed region in the PSF amplitude map with the long exposure time. It should also be understood that the replaced pixel region may not necessarily be the central pixel region in the first PSF amplitude map and / or the second PSF amplitude map. That is to say, some pixel regions of the first PSF amplitude map depend on the overexposed region of the first PSF amplitude map. If the overexposed region of the first PSF amplitude map is in the left half pixel region, the amplitude of the left half pixel region of the first PSF amplitude map is replaced by the amplitude of some pixel regions of the second PSF amplitude map. Some pixel regions of the second PSF amplitude map may not be located in the central pixel region of the second PSF amplitude map. For example, the amplitude of the right half pixel region of the second PSF amplitude map can be used to replace the amplitude of the overexposed region of the first PSF amplitude map. The embodiments of the present application do not limit this.

[0083] Optionally, in some embodiments, the point spread function (PSF) model is a non-normalized PSF model. In S640, that is, according to the integrated amplitude map, the PSF model corresponding to the depth camera is obtained, including: dividing the amplitude of each pixel in the integrated amplitude map by the sum of the amplitudes of all pixels in the integrated amplitude map to obtain the non-normalized PSF model of the depth camera.

[0084] As mentioned above, the PSF model may include an initial PSF model obtained by means such as optical fibers. In another embodiment, the PSF model may further include a PSF model obtained by further processing the obtained initial PSF model. In one embodiment, the integrated amplitude map may be non-normalized. For example, the amplitude of each pixel in the integrated amplitude map may be divided by the sum of the amplitudes of all pixels in the integrated amplitude map respectively, so that the value corresponding to each pixel finally lies between 0 and less than 1. Since there are multiple pixels in each original phase image, the initial PSF models corresponding to each pixel may be different. Multiple initial PSF models may be required in the deconvolution calculation process, resulting in a large amount of calculation. However, the scheme of obtaining the scattering interference map based on the original phase image and the non-normalized PSF model corresponding to the original phase image, and then obtaining the scattering compensation map based on the original phase image and the scattering interference map is beneficial to greatly reducing the calculation amount.

[0085] In another embodiment, the integrated amplitude map may also be normalized. For example, the amplitude of each pixel in the integrated amplitude map may be divided by the maximum amplitude in the integrated amplitude map respectively, so that the value corresponding to each pixel finally lies between 0 and 1.

[0086] Optionally, in the embodiments of the present application, the initial PSF model may be obtained and stored inside the depth camera before the depth camera leaves the factory; and / or, the non-normalized PSF model may be obtained and stored inside the depth camera before the depth camera leaves the factory. After the depth camera leaves the factory, the PSF model of the original phase image may be obtained according to the requirements from the initial PSF model and / or the non-normalized PSF model stored inside the depth camera.

[0087] Generally, due to the reasons of the optical system, the non-normalized PSF models corresponding to each pixel are different. To make up for this difference, different non-normalized PSF models can be uniformly constructed at different pixel positions. For example, different non-normalized PSF models are uniformly taken at different positions in an image of 240 * 320 pixels (for example, a 20 * 20 pixel corresponds to one position). Specifically, in the application field of a sweeping robot, since the scattering from the ground of the sweeping robot is relatively serious, it is distributed in the lower 80 rows of pixels of the 240 * 320 image pixels, that is, 80 * 320. Based on this, taking 20 * 20 pixels as one position in the image, 4 * 16 PSF models can be constructed at the bottom of the image. It should be noted that the non-normalized PSF model is constructed based on the image, and the arrangement of its quantity depends on the application scenario.

[0088] In one embodiment, the non-normalized PSF model is pre-constructed. After construction, it is associated and stored with the pixel position relationship (that is, the pixel position is associated with the non-normalized PSF model) and obtained by the look-up table method according to the current pixel position relationship during actual use. Specifically, the storage method of the non-normalized PSF model corresponding to a single pixel can be adopted. For example, the original phase image and the PSF model are both downsampled by 5 * 5, and after processing, a scattering interference map is obtained. Then, the original phase image is upsampled by 5 * 5 to reduce the calculation amount. The storage of the non-normalized PSF models corresponding to multiple pixels can also be adopted. For example, based on the non-normalized PSF model, the influence of 5 * 5 pixels on other surrounding pixels is obtained, and a look-up table is constructed. In actual application, the look-up table is called based on the average value of the values of 5 * 5 pixels. In this way, the influence of 25 pixels can be found at one time instead of one pixel, thereby reducing the operation amount.

[0089] In some embodiments, S550, that is, processing the ideal phase image based on the time-of-flight principle to obtain the depth information of the target area, includes: according to the ideal phase image, obtaining the charge amount collected by each tap of each pixel in the TOF image sensor in the TOF depth camera; determining the depth value of each pixel according to the charge amount collected by each tap of each pixel. According to the depth value of each pixel, the depth information of the target area is obtained.

[0090] Specifically, the depth of the ideal phase image can be calculated according to the time-of-flight principle. For example, taking the case where each pixel in the TOF image sensor includes 3 taps and sequentially collects 3 optical signals as an example, based on the ideal phase image, the charge amounts collected by each tap are C1, C2, and C3, and the pulse widths T of the pulse acquisition signals of each tap h , from which the time of flight of a single pixel can be calculated using the following formula:

[0091]

[0092] Furthermore, the depth value of a single pixel can be obtained according to d = c * t / 2, where c represents the speed of light. By traversing all pixels, the depth values of each pixel can be obtained, thereby obtaining the depth information of the target area.

[0093] Figure 10 and Figure 11 shows a comparison diagram of the effects of adopting the traditional technical solution and the technical solution provided by the embodiment of the present application. It can be clearly seen from the figure that Figure 10 due to the error in the scattering depth in the ground, the entire wall surface shows a phenomenon of fault and incompleteness. While in Figure 11 its wall surface is clearly visible.

[0094] The depth calculation method according to the embodiment of the present application has been described in detail above. Next, in combination with Figures 12 to 13 it will be described the depth calculation device according to the embodiment of the present application. The technical features described in the method embodiment are applicable to the following device embodiment.

[0095] Figure 12 shows a schematic block diagram of the depth calculation device 700 according to the embodiment of the present application. As Figure 12 shown, the depth calculation device 700 may include the following parts or all of the content.

[0096] The first acquisition unit 710 is configured to acquire the original phase image including the target area collected by the TOF depth camera and the PSF model corresponding to the pre-stored TOF depth camera.

[0097] The second acquisition unit 720 is configured to obtain a scattering interference map by using the original phase image and the PSF model. The scattering interference map includes a scattering interference amount for indicating the interference degree of the energy of at least some pixels in the original phase image on the amplitude of other surrounding pixels.

[0098] The third acquisition unit 730 is configured to adjust the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map;

[0099] The fourth acquisition unit 740 is configured to compensate the original phase image by using the scattering compensation map to obtain an ideal phase image.

[0100] The fifth acquisition unit 750 is configured to process the ideal phase image based on the time-of-flight principle to obtain the depth information of the target area.

[0101] Optionally, in the embodiment of the present application, the third acquisition unit 730 is specifically configured to: determine the magnitude relationship between the energy of the pixels in the original phase image and a preset first threshold; if the energy of a certain pixel in the original phase image is less than the first threshold, then set the scattering interference amount generated by the corresponding pixel in the scattering interference map on the surrounding pixels to 0 to obtain the scattering compensation map.

[0102] Optionally, in the embodiments of the present application, the third obtaining unit 730 is specifically configured to: process the energy of at least some pixels in the original phase image to obtain an amplitude map corresponding to the original phase image; determine the magnitude relationship between the amplitude values of at least some pixels in the amplitude map corresponding to the original phase map and a preset second threshold; if the amplitude values of at least some pixels in the amplitude map corresponding to the original phase map are greater than the preset second threshold, set the amount of scattering interference received by the corresponding pixels in the scattering interference map to 0 to obtain a scattering compensation map.

[0103] Optionally, in the embodiments of the present application, the second obtaining unit 720 is specifically configured to: multiply the original phase image by the PSF model to obtain a scattering interference map.

[0104] Optionally, in the embodiments of the present application, the fourth obtaining unit 740 is specifically configured to: subtract the original phase image from the scattering compensation map to obtain an ideal phase image.

[0105] Optionally, as Figure 13 shown, the depth calculation device 700 further includes: a sixth obtaining unit 760, configured to obtain the PSF model corresponding to the pixel of the original phase image from the pre-stored PSF model of the TOF depth camera by looking up a table according to the pixel position in the original phase image.

[0106] Optionally, the embodiments of the present application further provide a device 800 for obtaining the PSF model corresponding to a TOF depth camera. In this embodiment, the PSF model corresponding to the TOF depth camera is pre-obtained. As Figure 14 shown, the device 800 includes: a control unit 810, configured to control the TOF depth camera to obtain a first PSF amplitude map using a first exposure time, and control the TOF depth camera to obtain a second PSF amplitude map using a second exposure time, where the second exposure time is less than the second exposure time; an integration unit 820, configured to integrate the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map; an obtaining unit 830, configured to obtain the PSF model corresponding to the TOF depth camera according to the integrated amplitude map.

[0107] Optionally, in the embodiments of the present application, the integration unit 820 is specifically configured to: replace the amplitude of a partial pixel region in the first PSF amplitude map with the product of the amplitude of a partial pixel region in the second PSF amplitude map and the ratio of the first exposure time to the second exposure time to obtain an integrated amplitude map.

[0108] Optionally, in the embodiments of the present application, the partial pixel region of the first PSF amplitude map is the central pixel region of the first PSF amplitude map, and the partial pixel region of the second PSF amplitude map is the central pixel region of the second PSF amplitude map.

[0109] Optionally, in the embodiments of the present application, the PSF model is a non-normalized PSF model. The obtaining unit 830 is specifically configured to divide the amplitude of each pixel in the integrated amplitude map by the sum of the amplitudes of all pixels in the integrated amplitude map to obtain the non-normalized PSF model of the depth camera.

[0110] It should be understood that the depth calculation device 700 according to the embodiments of the present application may correspond to the execution subject in the embodiments of the depth calculation method 500 according to the embodiments of the present application. The device 800 for obtaining the PSF model corresponding to the TOF depth camera according to the embodiments of the present application may correspond to the execution subject in the embodiments of the method 600 for obtaining the PSF model corresponding to the TOF depth camera according to the embodiments of the present application. And the above and other operations and / or functions of each unit in the depth calculation device 700 are respectively for implementing Figures 1 to 8 the corresponding processes in the respective methods, and the above and other operations and / or functions of each unit in the device 800 are respectively for implementing Figure 9 the corresponding processes in the methods shown. For the sake of brevity, they will not be described in detail here.

[0111] Optionally, as Figure 15 shown, the embodiments of the present application further provide a TOF depth camera 900. As Figure 15 shown, the TOF depth camera 900 may include a transmitting end 910, a receiving end 920, and a processor 930. Among them, the transmitting end 910 is configured to project a periodically modulated transmitted optical signal to a target area. The receiving end 920 is configured to collect the received optical signal reflected back from the target area to generate an original phase image. The processor 930 is configured to call and run a program from a memory to implement the depth calculation method provided by the embodiments of the present application. For example, the processor 930 may control the TOF receiving end 920 to collect the original phase image. The processor 930 may also obtain the PSF model corresponding to the TOF depth camera pre-stored in the memory. The processor 930 is further configured to obtain a scattering interference map by using the original phase image and the PSF model. The processor 930 may also adjust the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map. The processor 930 may also compensate the original phase image by using the scattering compensation map to obtain an ideal phase image. The processor 930 may also process the ideal phase image based on the time-of-flight principle to obtain the depth information of the target area. Optionally, the processor 930 may correspond to Figure 12 and Figure 13 the depth calculation device 700 shown.

[0112] Optionally, as Figure 15As shown, the TOF depth camera 900 may further include a memory 940. Among them, the processor 930 may call and run a depth calculation program from the memory 940 to implement the method in the embodiments of the present application.

[0113] Among them, the memory 940 may be a separate device independent of the processor 930 or integrated in the processor 930.

[0114] Optionally, in the embodiments of the present application, the TOF depth camera 900 is an iTOF depth camera.

[0115] Optionally, the embodiments of the present application further provide a chip, including a processor, and the processor may call and run a computer program from a memory to implement the method in the embodiments of the present application.

[0116] Optionally, the embodiments of the present application further provide a computer-readable storage medium for storing a computer program, and the computer program enables a computer to execute the method in the embodiments of the present application.

[0117] Optionally, the embodiments of the present application further provide a computer program product, including computer program instructions, and the computer program instructions enable a computer to execute the method in the embodiments of the present application.

[0118] Optionally, the embodiments of the present application further provide a computer program. The computer program enables a computer to execute the method in the embodiments of the present application.

[0119] It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include but not be limited to these and any other suitable types of memory.

[0120] Although the present application has been described with reference to the preferred embodiments, various modifications can be made to it and components therein can be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in each embodiment can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A depth calculation method, applied to a TOF depth camera, characterized in that Including: Obtaining an original phase image including a target area collected by the TOF depth camera and a PSF model corresponding to the TOF depth camera stored in advance; Using the original phase image and the PSF model to obtain a scattering interference map, where the scattering interference map includes the amount of scattering interference of the energy of at least some pixels in the original phase image on other surrounding pixels; Adjusting the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map; Compensating the original phase image with the scattering compensation map to obtain an ideal phase image; Processing the ideal phase image based on the time-of-flight principle to obtain the depth information of the target area.

2. The depth calculation method according to claim 1, characterized in that, The adjusting the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map includes: Judging the magnitude relationship between the energy of the pixels in the original phase image and a preset first threshold; If the energy of a certain pixel in the original phase image is less than the first threshold, setting the amount of scattering interference generated by the corresponding pixel in the scattering interference map on the surrounding pixels to 0 to obtain the scattering compensation map.

3. The depth calculation method according to claim 1, characterized in that The adjusting the scattering interference map based on the energy of at least some pixels in the original phase image to obtain a scattering compensation map includes: Processing the energy of at least some pixels in the original phase image to obtain an amplitude map corresponding to the original phase image; Judging the magnitude relationship between the amplitude values of at least some pixels in the amplitude map corresponding to the original phase image and a preset second threshold; If the amplitude values of at least some pixels in the amplitude map corresponding to the original phase image are greater than the preset second threshold, setting the amount of scattering interference received by the corresponding pixels in the scattering interference map to 0 to obtain the scattering compensation map.

4. The depth calculation method according to claim 1, characterized in that The using the original phase image and the PSF model to obtain a scattering interference map includes: Multiplying the original phase image by the PSF model to obtain the scattering interference map.

5. The depth calculation method according to claim 1, wherein The compensating the original phase image with the scattering compensation map to obtain an ideal phase image includes: Subtracting the original phase image from the scattering compensation map to obtain the ideal phase image.

6. The depth calculation method according to claim 1, wherein The depth calculation method further includes: According to the pixel positions in the original phase image, obtaining the PSF model corresponding to the pixels of the original phase image from the PSF model of the TOF depth camera stored in advance by looking up a table.

7. The depth calculation method according to claim 6, wherein The PSF model of the TOF depth camera is obtained in advance, and the obtaining the PSF model of the depth camera includes: Controlling the TOF depth camera to obtain a first PSF amplitude map with a first exposure time; Controlling the TOF depth camera to obtain a second PSF amplitude map with a second exposure time, where the second exposure time is less than the second exposure time; Integrating the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map; According to the integrated amplitude map, obtaining the PSF model of the TOF depth camera.

8. The depth calculation method according to claim 7, wherein Integrating the first PSF amplitude map and the second PSF amplitude map to obtain an integrated amplitude map includes: Replacing the amplitudes of some pixel regions in the first PSF amplitude map with the product of the amplitudes of some pixel regions in the second PSF amplitude map and the ratio of the first exposure time to the second exposure time to obtain the integrated amplitude map.

9. The depth calculation method according to claim 8, characterized in that, The part of the pixel region of the first PSF amplitude map is the central pixel region of the first PSF amplitude map, and the part of the pixel region of the second PSF amplitude map is the central pixel region of the second PSF amplitude map.

10. The depth calculation method according to claim 7, characterized in that, The PSF model is a non-normalized PSF model. Obtaining the PSF model of the depth camera according to the integrated amplitude map includes: Dividing the amplitude of each pixel in the integrated amplitude map by the sum of the amplitudes of all pixels in the integrated amplitude map to obtain the non-normalized PSF model of the depth camera.

11. A TOF depth camera, characterized in that, It includes a transmitting end, a receiving end and a processor. The transmitting end is used to project a periodically modulated transmitted optical signal to the target area. The receiving end is used to collect the received optical signal reflected back by the target area to generate an original phase image. The processor is used to process the original phase image by using the depth calculation method according to any one of claims 1 to 10 to obtain the depth information of the target area.

12. A computer-readable storage medium, characterized in that, For storing a computer program, the computer program causes a computer to execute the depth calculation method according to any one of claims 1 to 10.

Images