Depth diagram resolving method for indirect time-of-flight image sensor

Abstract

The invention relates to a depth diagram resolving method for an indirect time-of-flight image sensor, which comprises the following steps of: utilizing all original data frames for two times in a resolving process except a first original data frame and a last original data frame in a calculation process; all the other depth data frames, which are the same as the original data frame, except the first depth data frame and the last depth data frame are utilized twice in the calculation process. Therefore, for N original data frames sent by a sensor in the same time, only N / 2 depth image frames can be obtained in the traditional method, and the method can obtain N-1 depth data frames and N-2 depth image frames through calculation, so that the number of output images in the same time can be increased, and the output frame rate is improved.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional imaging, and aims to improve the utilization rate of depth data by proposing a depth map solution method for an indirect time-of-flight image sensor, so as to achieve the output of more depth images in the same time under the condition of the same depth data input, compared with the traditional scheme, thereby achieving an improvement in frame rate. Background Art

[0002] With the rapid advancement of science and technology, depth image sensor technology has become a bridge between the real and digital worlds, and its importance is becoming increasingly prominent. In the wave of digital transformation, whether it is smart manufacturing, smart home, or smart transportation system, higher requirements are placed on the accuracy of spatial perception. By providing depth information, depth image sensors give machine vision a "stereoscopic vision", enabling it to better understand and interact with the physical environment. In the fields of medical diagnosis, cultural relics protection, topographic mapping, etc., its value is self-evident, opening up new perspectives for scientific research and cultural heritage protection.

[0003] In this technical framework, high frame rate depth image sensors are particularly critical. They show their unique advantages in processing continuously changing motion scenes, bringing revolutionary improvements to the fields of real-time interaction and object tracking. In virtual reality (VR) and augmented reality (AR) experiences, such sensors can capture user gestures and body movements in real time. Higher frame rates bring a smoother operating experience, greatly enhancing the sense of immersion and naturalness. In the field of smart medical care, high frame rate and high precision depth image sensors can generate high-definition, low-latency depth images of organs and tissues in patients in real time, providing surgeons with a visual experience like personal exploration. This unprecedented level of detail and instant feedback capabilities greatly improve the accuracy and safety of surgery, especially in high-risk operations such as cardiovascular interventional treatment and neurosurgery, helping doctors accurately identify key anatomical structures and reduce damage to surrounding tissues. On the other hand, higher frame rates make it possible for sensors to use multi-frame synthesis technology. By merging multiple instant images captured at high frame rates, this technology can not only eliminate motion blur and improve picture clarity, but also significantly improve image quality in low-light environments. This feature is particularly important in film and television special effects production, night monitoring systems and advanced driver assistance systems (ADAS). It enables the system to maintain delicate perception and analysis of the surrounding environment while moving quickly, providing strong technical support for safe driving and night security monitoring. In summary, high frame rate depth image sensors are the core driving force for many high-tech application scenarios to develop to a deeper level.

[0004] At present, there are multiple technical routes for depth image sensors, mainly including structured light technology, binocular stereo vision, and time of flight (ToF) technology. Structured light technology calculates depth information by projecting a specific pattern and analyzing its deformation; binocular stereo vision imitates human eyes and estimates distance through parallax; and ToF sensors calculate distance by measuring the time from the emission of light pulses to the reflection back to the sensor, which is divided into direct time of flight (d-ToF) and indirect time of flight (i-ToF). Compared with other three-dimensional imaging technologies, ToF sensors have shown unique advantages. First, ToF sensors can work stably under various lighting conditions and are less dependent on ambient light; second, they can achieve faster response speeds and higher depth resolution, which are particularly suitable for scenes that require real-time depth perception; third, ToF technology has a high degree of integration and is easy to miniaturize, making it very suitable for integration into mobile devices such as smartphones and tablets. ToF sensors are divided into indirect time of flight (i-ToF) sensors and direct time of flight (d-ToF) sensors. Although d-ToF has extremely high sensitivity and fast response capabilities in theory, its implementation relies on high-end components such as single-photon avalanche diodes (SPADs), which are costly and have extremely high hardware requirements, making it difficult to be widely used in ordinary consumer products. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a depth map solution method for an indirect time-of-flight image sensor. The problem of low utilization of inter-frame information in traditional inventions is improved, and a new depth image solution method is proposed to improve the output image frame rate without adding additional hardware. In addition, the present invention also processes the data after the solution, reducing the edge blur problem caused by the movement of objects in the depth image.

[0006] The present invention solves the technical problem by adopting the following technical inventions:

[0007] A depth map calculation method for an indirect time-of-flight image sensor comprises the following steps:

[0008] Step 1: Send a start signal to the sensor to start receiving raw data;

[0009] Step 2: Read a raw data frame from the sensor control system and write it into the raw data cache. W1 The value of adds a mark to the frame, and then N W1 Increase by 1;

[0010] Step 3: Determine N R1 Is it greater than N? W1 +1, if true, go to step 4, otherwise return to step 2;

[0011] Step 4: Determine N R1 Check whether the remainder of 2 is 0. If yes, go to step 5, otherwise go to step 6;

[0012] Step 5: Read the Nth R1 and Nth R1 +1 frame data, the Nth R1 The data of the frame is recorded as A 24 , Nth R1 +1 frame data is recorded as A 13 After reading, N R1 Increase by 1;

[0013] Step 6: Read the Nth R1 and Nth R1 +1 frame data, the Nth R1 The data of the frame is recorded as A 13 , Nth R1 +1 frame data is recorded as A 24 After reading, N R1 Increase by 1;

[0014] Step 7: Solve the original data to obtain the depth data and write it into the depth data cache, marking it as the Nth W2 frame, then N W2 Increase by 1;

[0015] Step 8: Determine N R2 Is it greater than N? W2 +1, if true, go to step 9, otherwise go to step 2;

[0016] Step 9: Calculate the depth image;

[0017] Step 10: Write the calculated depth image into the cache.

[0018] Moreover, the N W1 、N R1 、N W2 、N R2 is a counting variable, and its initial value is 1. Whenever it increases to its respective counting upper limit, its value will be reset to 1 when it increases again; counting variable N W1 , counting variable N R1 The upper limit of the count is the same as the number of frames that the original data buffer can accommodate. The count variable N W2 , counting variable N R2 The upper limit of the count is the same as the number of frames that the depth data buffer can accommodate.

[0019] Furthermore, in step 5 or 6, A 24 and A13 The specific calculation method is:

[0020]

[0021] The sampling point is at time t 1 ,t 2 ,t 3 ,t 4 , the corresponding sampling signal strength is A 1 , A 2 , A 3 , A 4 , A is the amplitude of the modulated light, C is the amplitude of the background light, f m is the modulation frequency.

[0022] Moreover, the specific implementation method of step 7 is:

[0023]

[0024] Where d is the distance between the object to be measured and the camera, f m is the modulation frequency.

[0025] Moreover, the specific implementation method of step 9 is: R2 The data of the frame is recorded as A D1 , Nth W1 +1 frame data is recorded as A D2 , using formula A D3 =C 1 A D1 +C 2 A D2 Calculate the depth image, where C 1 , C 2 satisfy C 2 =1-C 1 .

[0026] The advantages and positive effects of the present invention are:

[0027] 1. In the calculation process of the present invention, except for the first and last original data frames, all original data frames will be used twice in the calculation process; similar to the original data frames, except for the first and last depth data frames, all other depth data frames will be used twice in the calculation process. Therefore, for N original data frames sent by the sensor in the same time, the traditional invention can only obtain N / 2 depth image frames, while the present invention can obtain N-1 depth data frames and N-2 depth image frames, which can increase the number of output images in the same time, thereby increasing the output frame rate.

[0028] 2. The present invention further utilizes the solved depth data frame and uses different weights to calculate two adjacent groups of depth data frames, thereby suppressing the edge blur problem caused by the movement of the object and achieving clearer depth image output. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a work flow chart of the present invention;

[0030] Figure 2 It is a schematic diagram of the working process of the present invention. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below with reference to the accompanying drawings.

[0032] The present invention is based on the improvement of the traditional 2-taps i-ToF sensor solution method. In the present invention, the original data frame (for a sensor with a resolution of W×H, each group of W×H data is called an original data frame) is solved using a formula to obtain a depth data frame. This process is called a depth data solution process; two adjacent groups of data frames are further calculated to obtain a depth image. This process is called a depth image calculation process. The process of this invention requires 4 counting variables N W1 、N R1 、N W2 、N R2 And 3 independent cache spaces (raw data cache, depth data cache, output image cache) are used to control the correct operation of the entire workflow. W1 、N R1 Used to operate the original data cache, N W2 、N R2 Used for depth data caching. In this invention, raw data reading and depth image output are implemented by separate systems: the sensor sends raw data to the sensor control system and caches it, and the image output system reads and outputs images from the output image cache at a set frame rate. Figure 2 The working process of this invention is shown, and the part within the dotted frame is the part for which the present invention is responsible.

[0033] A depth map calculation method for an indirect time-of-flight image sensor, such as Figure 1 As shown, the following steps are included:

[0034] Step 1: Send a start signal to the sensor to start receiving raw data.

[0035] Step 2: Read a raw data frame from the sensor control system and write it into the raw data cache. W1 The value of adds a mark to the frame, and then N W1 Increase by 1.

[0036] Step 3: Determine N R1 Is it greater than N? W1 +1, if true, go to step 4, otherwise return to step 2.

[0037] Step 4: Determine N R1 Check whether the remainder of 2 is 0. If so, go to step 5; otherwise, go to step 6.

[0038] Step 5: Read the Nth R1 and Nth R1 +1 frame data, the Nth R1 The data of the frame is recorded as A 24 , Nth R1 +1 frame data is recorded as A 13 After reading, N R1 Increase by 1.

[0039] Step 6: Read the Nth R1 and Nth R1 +1 frame data, the Nth R1 The data of the frame is recorded as A 13 , Nth R1 +1 frame data is recorded as A 24 After reading, N R1 Increase by 1.

[0040] A 24 and A 13 The specific calculation method is:

[0041]

[0042] The sampling point is at time t 1 ,t 2 ,t 3 ,t 4 , the corresponding sampling signal strength is A 1 , A 2 , A 3 , A 4 , A is the amplitude of the modulated light, C is the amplitude of the background light, f m is the modulation frequency.

[0043] Step 7: Solve the original data to obtain the depth data and write it into the depth data cache, marking it as the Nth W2 frame, then N W2 Increase by 1.

[0044] The specific implementation method of step 7 is:

[0045]

[0046] Where d is the distance between the object to be measured and the camera, f m is the modulation frequency.

[0047] Step 8: Determine N R2 Is it greater than N? W2 +1, if true, go to step 9, otherwise go to step 2.

[0048] Step 9: Calculate the depth image.

[0049] The specific implementation method of step 9 is: R2 The data of the frame is recorded as A D1 , Nth W1 +1 frame data is recorded as A D2 , using formula A D3 =C 1 A D1 +C 2 A D2 Calculate the depth image, where C 1 , C 2 satisfy C 2 =1-C 1 .

[0050] Step 10: Write the calculated depth image into the cache.

[0051] The above N W1 、N R1 、N W2 、N R2 is a counting variable, and its initial value is 1. Whenever it increases to its respective counting upper limit, its value will be reset to 1 when it increases again; counting variable N W1 , counting variable N R1 The upper limit of the count is the same as the number of frames that the original data buffer can accommodate. The count variable N W2 , counting variable N R2 The upper limit of the count is the same as the number of frames that the depth data buffer can accommodate.

[0052] The present invention is applied to the host computer system of the existing 2-taps i-ToF sensor. By using the process proposed in this solution to write a program between the existing raw data receiving program and the depth image output program, a depth image output with a higher frame rate and lower motion blur can be achieved compared to the traditional solution. In the actual software implementation process, the size of the cache space is limited, and it should be adjusted appropriately according to the hardware resources and computing speed of the system, but there should be space to cache at least 4 data frames. In order to recycle the cache, the cache space occupied by the frames that have been used twice can be released. The judgment logic is that the frame mark is less than N RxAll frames (N Rx The frame count variable corresponding to the cache space where the frame is located. When releasing the original data cache, all frames marked less than N can be released. R1 of frames).

[0053] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical inventions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A depth map calculation method for an indirect time-of-flight image sensor, characterized in that: The following steps are involved: Step 1: Send a start signal to the sensor to start receiving raw data; Step 2: Read a raw data frame from the sensor control system and write it into the raw data cache. W1 The value of adds a mark to the frame, and then N W1 Increase by 1; Step 3: Determine N R1 Is it greater than N? W1 +1, if true, go to step 4, otherwise return to step 2; Step 4: Determine N R1 Check whether the remainder of 2 is 0. If yes, go to step 5, otherwise go to step 6; Step 5: Read the Nth R1 and Nth R1 +1 frame data, the Nth R1 The data of the frame is recorded as A 24 , Nth R1 +1 frame data is recorded as A 13 After reading, N R1 Increase by 1; Step 6: Read the Nth R1 and Nth R1 +1 frame data, the Nth R1 The data of the frame is recorded as A 13 , Nth R1 +1 frame data is recorded as A 24 After reading, N R1 Increase by 1; Step 7: Solve the original data to obtain the depth data and write it into the depth data cache, marking it as the Nth W2 frame, then N W2 Increase by 1; Step 8: Determine N R2 Is it greater than N? W2 +1, if true, go to step 9, otherwise go to step 2; Step 9: Calculate the depth image; Step 10: Write the calculated depth image into the cache.

2. The depth map calculation method for an indirect time-of-flight image sensor according to claim 1, characterized in that: The N W1 、N R1 、N W2 、N R2 is a counting variable, and its initial value is 1. Whenever it increases to its respective counting upper limit, its value will be reset to 1 when it increases again; counting variable N W1 , counting variable N R1 The upper limit of the count is the same as the number of frames that the original data buffer can accommodate. The count variable N W2 , counting variable N R2 The upper limit of the count is the same as the number of frames that the depth data buffer can accommodate.

3. The depth map calculation method for an indirect time-of-flight image sensor according to claim 1, characterized in that: In step 5 or 6, A 24 and A 13 The specific calculation method is: The sampling points are at t1, t2, t3, and t4, and the corresponding sampling signal intensities are A1, A2, A3, and A4, where A is the amplitude of the modulated light, C is the amplitude of the background light, and f m is the modulation frequency.

4. The depth map calculation method for an indirect time-of-flight image sensor according to claim 3, characterized in that: The specific implementation method of step 7 is: Where d is the distance between the object to be measured and the camera, f m is the modulation frequency.

5. The depth map calculation method for an indirect time-of-flight image sensor according to claim 4, characterized in that: The specific implementation method of step 9 is: R2 The data of the frame is recorded as A D1 , Nth W1 +1 frame data is recorded as A D2 , using formula A D3 =C1A D1 +C2A D2 Calculate the depth image, where C1 and C2 satisfy

Images