A method for depth map calculation for indirect time-of-flight image sensors

Abstract

This invention relates to a depth map calculation method for an indirect time-of-flight image sensor. During the calculation process, all original data frames, except for the first and last ones, are used twice. Similarly, all depth data frames, except for the first and last ones, are used twice during the calculation. Therefore, for N original data frames sent by the sensor within the same time period, conventional methods can only obtain N / 2 depth image frames, while this invention can calculate N-1 depth data frames and N-2 depth image frames, increasing the number of output images within the same time period and thus improving the output frame rate.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional imaging. Its purpose is to improve the utilization rate of depth data by proposing a depth map calculation method for indirect time-of-flight image sensors. Under the same depth data input, it can output more depth images in the same time compared with traditional methods, thereby improving the frame rate. Background Technology

[0002] With the rapid advancement of technology, depth image sensor technology has become a bridge connecting the real and digital worlds, and its importance is increasingly prominent. In the wave of digital transformation, whether it's smart manufacturing, smart homes, or intelligent transportation systems, higher demands are being placed on the accuracy of spatial perception. Depth image sensors, by providing depth information, endow machine vision with a "stereoscopic view," enabling it to better understand and interact with the physical environment. In fields such as medical diagnosis, cultural relic preservation, and topographic mapping, its value is self-evident, opening up new perspectives for scientific research and cultural heritage protection.

[0003] Within this technological framework, high frame rate depth image sensors are particularly crucial, demonstrating unique advantages in handling continuously changing motion scenes and bringing revolutionary improvements to real-time interaction and object tracking. In virtual reality (VR) and augmented reality (AR) experiences, these sensors can capture users' gestures and body movements in real time. The higher frame rate brings a smoother operating experience, greatly enhancing immersion and naturalness. In the field of smart healthcare, high frame rate, high-precision depth image sensors can generate high-definition, low-latency depth images of organs and tissues inside the patient's body in real time, providing surgeons with a visual experience akin to being there in person. This unprecedented level of detail and instant feedback capability greatly improves the accuracy and safety of surgery, especially during high-risk procedures such as cardiovascular interventional therapy and neurosurgery, helping doctors accurately identify key anatomical structures and reduce damage to surrounding tissues. On the other hand, higher frame rates enable the use of multi-frame synthesis technology with sensors. This technology, by merging multiple instantaneous images captured at high frame rates, not only eliminates motion blur and improves image clarity but also significantly improves image quality in low-light environments. This characteristic is particularly important in film and television special effects production, nighttime surveillance systems, and advanced driver assistance systems (ADAS). It enables the system to maintain a detailed perception and analysis of its surroundings even while moving at high speeds, providing strong technical support for safe driving and nighttime security monitoring. In conclusion, high frame rate depth image sensors are the core driving force propelling many high-tech applications to greater depths.

[0004] Currently, depth image sensor technologies are diverse, primarily including structured light technology, binocular stereo vision, and time-of-flight (ToF) technology. Structured light technology calculates depth information by projecting a specific pattern and analyzing its deformation; binocular stereo vision mimics human eyes, estimating distance through parallax; while ToF sensors calculate distance by measuring the time it takes for a light pulse to travel from emission to reflection back to the sensor, and are divided into two types: direct time-of-flight (d-ToF) and indirect time-of-flight (i-ToF). Compared to other 3D imaging technologies, ToF sensors exhibit unique advantages. First, ToF sensors can operate stably under various lighting conditions, with less dependence on ambient light; second, they can achieve faster response times and higher depth resolution, making them particularly suitable for scenarios requiring real-time depth perception; third, ToF technology has high integration and is easy to miniaturize, making it ideal for integration into mobile devices such as smartphones and tablets. ToF sensors are divided into indirect time-of-flight (i-ToF) sensors and direct time-of-flight (d-ToF) sensors. While d-ToF theoretically possesses extremely high sensitivity and fast response capabilities, its implementation relies on high-end components such as single-photon avalanche diodes (SPADs), which are costly and have extremely high hardware requirements, making it difficult to widely apply in ordinary consumer products. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a depth map calculation method for indirect time-of-flight image sensors. It improves upon the problem of low utilization of inter-frame information in traditional inventions by proposing a new depth image calculation method that increases the output image frame rate without adding additional hardware. Furthermore, this invention processes the calculated data to reduce edge blurring caused by object motion in the depth image.

[0006] The technical problem solved by this invention is achieved through the following technical invention:

[0007] A depth map calculation method for an indirect time-of-flight image sensor includes the following steps:

[0008] Step 1: Send a start signal to the sensor to begin receiving raw data;

[0009] Step 2: Read one raw data frame from the sensor control system and write it into the raw data buffer. After buffering is complete, use N... W1 The value is used to add a marker to the frame, then N W1 Increase by 1;

[0010] Step 3, determine N R1 Is it greater than N? W1 +1, if true, proceed to step 4; otherwise return to step 2.

[0011] Step 4: Determine N R1 Check if the remainder when divided by 2 is 0. If it is, proceed to step 5; otherwise, proceed to step 6.

[0012] Step 5: Read the Nth data from the original data cache. R1 and the Nth R1 +1 frame of data, the Nth frame R1 The data of the frame is denoted as A. 24 , Nth R1 The data of +1 frame is denoted as A. 13 After reading N R1 Increase by 1;

[0013] Step 6: Read the Nth data from the original data cache. R1 and the Nth R1 +1 frame of data, the Nth frame R1 The data of the frame is denoted as A. 13 , Nth R1 The data of +1 frame is denoted as A. 24 After reading N R1 Increase by 1;

[0014] Step 7: Solve the raw data to obtain depth data and write it into the depth data cache, marking it as the Nth depth data. W2 Frames, then N W2 Increase by 1;

[0015] Step 8: Determine N R2 Is it greater than N? W2 +1, if true, proceed to step 9; otherwise, proceed to step 2.

[0016] Step 9: Calculate the depth image;

[0017] Step 10: Write the calculated depth image into the cache.

[0018] Moreover, the N W1 N R1 N W2 N R2 These are counting variables, each initially set to 1. Whenever they reach their respective count limits, their values ​​are reset to 1 upon further increments. The counting variable N... W1 Count variable N R1 The upper limit of the count is the same as the number of frames that the original data buffer can hold, and the count variable N W2 Count variable N R2 The upper limit of the count is the same as the number of frames that the depth data buffer can hold.

[0019] Moreover, in step 5 or 6, A 24 and A13 The specific calculation method is as follows:

[0020]

[0021] Where the sampling times are t1, t2, t3, and t4, and the corresponding sampling signal intensities are A1, A2, A3, and A4, respectively, where A is the amplitude of the modulated light, C is the amplitude of the background light, and f is the amplitude of the background light. m The modulation frequency.

[0022] Furthermore, the specific implementation method of step 7 is as follows:

[0023]

[0024] Where d is the distance between the object being measured and the camera, and f m The modulation frequency.

[0025] Furthermore, the specific implementation method of step 9 is as follows: The Nth... R2 The data of the frame is denoted as A. D1 , Nth W1 The data of +1 frame is denoted as A. D2 Using formula A D3 =C1A D1 +C2A D2 Calculate the depth image, where C1 and C2 satisfy... C2 = 1 - C1.

[0026] The advantages and positive effects of this invention are:

[0027] 1. In the calculation process of this invention, all original data frames except the first and last ones are used twice during the calculation. Similarly, all depth data frames except the first and last ones are used twice during the calculation. Therefore, for N original data frames sent by the sensor at the same time, traditional inventions can only obtain N / 2 depth image frames, while this invention can calculate N-1 depth data frames and N-2 depth image frames, which can increase the number of output images in the same time, thereby increasing the output frame rate.

[0028] 2. This invention further utilizes the calculated depth data frames by using different weights to calculate two adjacent sets of depth data frames, thereby suppressing the edge blurring problem caused by the motion of objects and achieving a clearer depth image output. Attached Figure Description

[0029] Figure 1 This is a flowchart of the process of the present invention;

[0030] Figure 2 This is a schematic diagram of the working process of the present invention. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to the accompanying drawings.

[0032] This invention improves upon the traditional 2-taps i-ToF sensor's calculation method. In this invention, the original data frames (for a W×H resolution sensor, each set of W×H data points is called an original data frame) are solved using a formula to obtain depth data frames; this process is called the depth data calculation process. Further calculations are performed on adjacent sets of data frames to obtain a depth image; this process is called the depth image calculation process. This invention requires four counting variables N. W1 N R1 N W2 N R2 It also includes three independent cache spaces (raw data cache, depth data cache, and output image cache) to ensure the correct operation of the entire workflow. W1 N R1 Used for operations on the raw data cache, N W2 N R2 For depth data caching. In this invention, raw data reading and depth image output are implemented by a separate system: the sensor sends raw data to the sensor control system for caching, and the image output system reads and outputs the image from the output image cache according to a set frame rate. Figure 2 The working process of this invention is shown, with the part within the dashed box representing the part that this invention is responsible for.

[0033] A depth map calculation method for indirect time-of-flight image sensors, such as Figure 1 As shown, it includes the following steps:

[0034] Step 1: Send a start signal to the sensor to begin receiving raw data.

[0035] Step 2: Read one raw data frame from the sensor control system and write it into the raw data buffer. After buffering is complete, use N... W1 The value is used to add a marker to the frame, then N W1 Increase by 1.

[0036] Step 3, determine N R1 Is it greater than N? W1 +1. If true, proceed to step 4; otherwise, return to step 2.

[0037] Step 4: Determine N R1 Check if the remainder when divided by 2 is 0. If it is, proceed to step 5; otherwise, proceed to step 6.

[0038] Step 5: Read the Nth data from the original data cache. R1 and the Nth R1 +1 frame of data, the Nth frame R1 The data of the frame is denoted as A. 24 , Nth R1 The data of +1 frame is denoted as A. 13 After reading N R1 Increase by 1.

[0039] Step 6: Read the Nth data from the original data cache. R1 and the Nth R1 +1 frame of data, the Nth frame R1 The data of the frame is denoted as A. 13 , Nth R1 The data of +1 frame is denoted as A. 24 After reading N R1 Increase by 1.

[0040] A 24 and A 13 The specific calculation method is as follows:

[0041]

[0042] Where the sampling times are t1, t2, t3, and t4, and the corresponding sampling signal intensities are A1, A2, A3, and A4, respectively, where A is the amplitude of the modulated light, C is the amplitude of the background light, and f is the amplitude of the background light. m The modulation frequency.

[0043] Step 7: Solve the raw data to obtain depth data and write it into the depth data cache, marking it as the Nth depth data. W2 Frames, then N W2 Increase by 1.

[0044] The specific implementation method for step 7 is as follows:

[0045]

[0046] Where d is the distance between the object being measured and the camera, and f m The modulation frequency.

[0047] Step 8: Determine N R2 Is it greater than N? W2 +1. If true, proceed to step 9; otherwise, proceed to step 2.

[0048] Step 9: Calculate the depth image.

[0049] The specific implementation method of step 9 is as follows: [The Nth...] R2 The data of the frame is denoted as A. D1 , Nth W1 The data of +1 frame is denoted as A.D2 Using formula A D3 =C1A D1 +C2A D2 Calculate the depth image, where C1 and C2 satisfy... C2 = 1 - C1.

[0050] Step 10: Write the calculated depth image into the cache.

[0051] The above N W1 N R1 N W2 N R2 These are counting variables, each initially set to 1. Whenever they reach their respective count limits, their values ​​are reset to 1 upon further increments. The counting variable N... W1 Count variable N R1 The upper limit of the count is the same as the number of frames that the original data buffer can hold, and the count variable N W2 Count variable N R2 The upper limit of the count is the same as the number of frames that the depth data buffer can hold.

[0052] This invention is applied to the host computer system of an existing 2-taps i-ToF sensor. By using the proposed flow in this solution to program between the existing raw data receiving program and depth image output program, a higher frame rate and lower motion blur depth image output can be achieved compared to traditional solutions. In the actual software implementation, the size of the buffer space is limited and should be adjusted appropriately according to the system's hardware resources and computing speed, but it should have at least enough space to buffer four data frames. To ensure buffer reuse, the buffer space occupied by frames that have been used twice can be released. The judgment logic is that the frame mark is less than N. Rx All frames (N) Rx This is the frame count variable corresponding to the buffer space where the frame resides. For example, when releasing the original data buffer, all frames with a flag less than N can be released. R1 (frames).

[0053] It should be emphasized that the embodiments described in this invention are illustrative rather than limiting. Therefore, this invention includes, but is not limited to, the embodiments described in the specific implementation. Any other implementations derived by those skilled in the art based on the technical invention of this invention are also within the scope of protection of this invention.

Claims

1. A depth map calculation method for an indirect time-of-flight image sensor, characterized in that: Includes the following steps: Step 1: Send a start signal to the sensor to begin receiving raw data; Step 2: Read one raw data frame from the sensor control system and write it into the raw data buffer. After buffering is complete, use N... W1 The value is used to add a marker to the frame, then N W1 Increase by 1; Step 3, determine N R1 Is it greater than N? W1 +1, if true, proceed to step 4; otherwise return to step 2. Step 4: Determine N R1 Check if the remainder when divided by 2 is 0. If it is, proceed to step 5; otherwise, proceed to step 6. Step 5: Read the Nth data from the original data cache. R1 and the Nth R1 +1 frame of data, the Nth frame R1 The data of the frame is denoted as A. 24 , Nth R1 The data of +1 frame is denoted as A. 13 After reading N R1 Increase by 1; Step 6: Read the Nth data from the original data cache. R1 and the Nth R1 +1 frame of data, the Nth frame R1 The data of the frame is denoted as A. 13 , Nth R1 The data of +1 frame is denoted as A. 24 After reading N R1 Increase by 1; A 24 and A 13 The specific calculation method is as follows: Where the sampling times are t1, t2, t3, and t4, and the corresponding sampling signal intensities are A1, A2, A3, and A4, respectively, where A is the amplitude of the modulated light, C is the amplitude of the background light, and f is the amplitude of the background light. m The modulation frequency; Step 7: Solve the raw data to obtain depth data and write it into the depth data cache, marking it as the Nth depth data. W2 Frames, then N W2 Increase by 1; The specific implementation method for step 7 is as follows: Where d is the distance between the object being measured and the camera, and f m Modulation frequency Step 8: Determine N R2 Is it greater than N? W2 +1, if true, proceed to step 9; otherwise, proceed to step 2. Step 9: Calculate the depth image; Step 10: Write the calculated depth image into the cache.

2. The depth map calculation method for an indirect time-of-flight image sensor according to claim 1, characterized in that: The N W1 N R1 N W2 N R2 These are counting variables, each initially set to 1. Whenever they reach their respective count limits, their values ​​are reset to 1 upon further increments. The counting variable N... W1 Count variable N R1 The upper limit of the count is the same as the number of frames that the original data buffer can hold, and the count variable N W2 Count variable N R2 The upper limit of the count is the same as the number of frames that the depth data buffer can hold.

Images