Depth image imaging method and device

Abstract

The present invention provides a depth image imaging method, comprising the following steps: presetting according to a calibration correction parameter of a sensor; after initializing the sensor, collecting depth image data; processing the depth image data in parallel to obtain grayscale values ​​and depth values; and outputting the processed data for digital image imaging. A depth image imaging device is also provided, which uses a parallel processing method with programmable logic characteristics to realize depth calculation, improves processing speed, and can achieve higher resolution and lower latency.

Description

Technical Field

[0001] The present invention relates to three-dimensional imaging technology, and in particular to an imaging method and an imaging device involving an image with depth properties. Background Art

[0002] Among the existing three-dimensional imaging technologies, ToF technology (Time of Flight) has been widely used in consumer electronics, virtual reality, face recognition, somatosensory interaction, new retail, smart security, automotive electronics, industrial automation, medical electronics, and many other life and industrial fields in recent years because of its multiple advantages such as being not easily disturbed by external light, compact size, fast response speed, and high recognition accuracy. Although ToF technology has become the preferred technical solution for three-dimensional images and has obvious good development prospects, the existing ToF technology cannot meet the high requirements actually encountered in applications, such as higher resolution, lower latency, etc., and has the following defects.

[0003] First of all, existing ToF technologies all require dedicated processing chips, such as the RK series or HiSilicon series, etc. Dedicated chips are not only expensive, but also lag behind in technological development.

[0004] Secondly, there is a contradiction between frame rate and resolution. When the frame rate is high, the depth map pixel resolution is relatively low, such as 180*240, 240*320, 240*480. If a higher resolution is required, the frame rate processing capability will be reduced (for example, the frame rate is less than 10fps).

[0005] In addition, the processing mode of existing computing software all uses frames as units, which will fundamentally affect the latency, resulting in a large computing delay at the millisecond level.

[0006] Based on the above-mentioned defects, the present invention proposes to improve and optimize the depth image imaging method and device of the prior art. Summary of the invention

[0007] A major advantage of the present invention is that it provides a depth image imaging method and device thereof, which uses a parallel processing method with programmable logic characteristics to realize depth calculation, thereby improving the processing speed while achieving higher resolution and lower latency.

[0008] Another advantage of the present invention is to provide a depth image imaging method and device thereof, which calculate and process the depth value and the grayscale value in parallel, thereby avoiding a high delay in the calculation and processing.

[0009] Another advantage of the present invention is to provide a depth image imaging method and device thereof, which utilizes grayscale values ​​in parallel to perform filtering processing on depth values, preferably performs adaptive filtering processing to ensure the processing effectiveness of the depth values.

[0010] Another advantage of the present invention is to provide a depth image imaging method and device, which can store and process high-resolution phase data and can implement subsequent calibration and correction to ensure the output of high-resolution images.

[0011] Another advantage of the present invention is that it provides a depth image imaging method and device, which, in addition to adopting a parallel computing processing strategy, also provides an optimization strategy for memory call processing, so that the overall computing operation is reliable.

[0012] Another advantage of the present invention is that it provides a depth image imaging method and device, and the calculated data can be transmitted to a computer or mobile terminal and the like by wired or wireless means after digital image processing, and has high environmental adaptability.

[0013] Another advantage of the present invention is to provide a depth image imaging method and device, which can achieve the following effects: high resolution, up to 640*480; faster frame rate, up to 30fps, and frame processing can be adaptively changed with the input frame rate; lower latency, reaching the subtle us level.

[0014] Other advantages and features of the present invention will become more apparent from the following detailed description and may be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0015] According to one aspect of the present invention, a depth image imaging method of the present invention that can achieve the above-mentioned objects and other objects and advantages includes the following steps:

[0016] Presetting according to calibration parameters of a sensor;

[0017] After initializing the sensor, collecting depth image data;

[0018] Processing the depth image data in parallel to obtain grayscale values ​​and depth values; and

[0019] Output the processed data for digital imaging.

[0020] According to one embodiment of the present invention, the presetting step further comprises the steps of:

[0021] Obtaining calibration correction parameters of the sensor; and

[0022] The calibration parameters are stored in a memory for use in data processing.

[0023] According to an embodiment of the present invention, the memory is a Flash memory.

[0024] According to one embodiment of the present invention, the step of collecting depth image data further comprises the steps of:

[0025] Transmits 940nm narrowband optical signal after 120Mh sine wave debugging;

[0026] When the light signal hits an object and is reflected, receiving the reflected light signal; and

[0027] Sampling is performed at the set period.

[0028] According to one embodiment of the present invention, the received reflected optical signal is quantized.

[0029] According to one embodiment of the present invention, the step of processing the depth image data in parallel further comprises the steps of:

[0030] Converting the information data format of the collected optical signal;

[0031] Arranging N phase data for storage in the memory in a certain data format;

[0032] Calculating in parallel to obtain depth values ​​and grayscale values; and

[0033] Output depth and grayscale values ​​for digital image processing.

[0034] According to an embodiment of the present invention, the memory is a DDR SDRAM memory (Double Data Rate Synchronous Dynamic Random Access Memory).

[0035] According to one embodiment of the present invention, the parallel computing step further comprises the steps of:

[0036] Calculate the phase difference;

[0037] Calibrate and correct the phase difference value according to the calibration correction parameter;

[0038] Calculate depth distance;

[0039] The depth distance is filtered to obtain a depth value.

[0040] According to one embodiment of the present invention, the calibration correction phase difference value is selected from at least one of the following methods: temperature calibration correction, gradient error calibration correction, and cyclic error calibration correction.

[0041] According to an embodiment of the present invention, the filtering process for the depth distance is selected from at least one of the following methods: confidence filtering process and adaptive filtering process.

[0042] According to another aspect of the present invention, the present invention further provides a depth image imaging device, comprising:

[0043] A parallel processor, at least one memory, a sensor and an interface component, wherein the memory, the sensor and the interface component are all communicatively connected to the parallel processor, wherein the calibration correction parameters of the sensor are preset in the memory for use by the parallel processor, wherein the sensor collects depth image data by emitting light signals and receiving reflected light signals, wherein the parallel processor processes the depth image data in parallel and performs calibration correction according to the calibration correction parameters to obtain grayscale values ​​and depth values, so that the interface component outputs the processed data for digital image imaging.

[0044] According to an embodiment of the present invention, the calibration correction parameters of the sensor are obtained by a terminal and stored in the memory after being processed.

[0045] According to an embodiment of the present invention, the calibration correction parameters of the sensor are obtained by a computer terminal through an I2C interface (Inter-Integrated Circuit, two-wire serial bus) and stored in the memory after conversion.

[0046] According to an embodiment of the present invention, the sensor emits a 940nm narrowband optical signal after being adjusted by a 120Mh sine wave. When the optical signal hits an object and is reflected, the sensor receives the reflected optical signal and samples it according to a set period.

[0047] According to one embodiment of the present invention, the parallel processor converts the information data format of the collected optical signal, organizes N phase data, and stores them in the memory in a certain data format, calculates in parallel to obtain depth values ​​and grayscale values, and outputs the depth values ​​and grayscale values ​​through the interface component for digital image processing.

[0048] According to an embodiment of the present invention, the memory is a DDR SDRAM memory.

[0049] According to an embodiment of the present invention, the DDR SDRAM memory is defined with processing priorities, from high to low: calibration correction parameter write operation, image data write operation, calibration correction parameter read operation, image data read operation.

[0050] According to one embodiment of the present invention, the parallel processor obtains image data from the memory to calculate the phase difference, calibrates and corrects the phase difference value according to the calibration correction parameters, calculates the depth distance, and filters the depth distance to obtain the depth value.

[0051] According to one embodiment of the present invention, the parallel processor uses the calibration correction parameters from the memory to calibrate and correct the phase difference, and is selected from at least one of the following methods: temperature calibration correction, gradient error calibration correction, and cyclic error calibration correction.

[0052] According to an embodiment of the present invention, the parallel processor performs filtering processing on the depth distance in at least one of the following ways: confidence filtering processing and adaptive filtering processing.

[0053] Further objects and advantages of the present invention will be fully apparent from an understanding of the following description and the accompanying drawings.

[0054] These and other objects, features and advantages of the present invention will be more fully understood from the following detailed description, accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a flowchart of a depth image imaging method and an apparatus thereof according to a preferred embodiment of the present invention.

[0056] Figure 2 is a block diagram of a depth image imaging device according to the above preferred embodiment of the present invention.

[0057] Figure 3 It is a specific flow chart of the depth image imaging method and device thereof according to the above preferred embodiment of the present invention, explaining the preset steps.

[0058] Figure 4 It is a specific flow chart of the depth image imaging method and device thereof according to the above preferred embodiment of the present invention, explaining the acquisition steps.

[0059] Figure 5 It is a specific flow chart of the depth image imaging method and device thereof according to the above preferred embodiment of the present invention, illustrating the steps of data parallel processing.

[0060] Figure 6 It is a specific flow chart of the depth image imaging method and device thereof according to the above preferred embodiment of the present invention, and explains the specific depth value processing steps.

[0061] Figure 7 It is a specific flow chart of the depth image imaging method and device thereof according to the above preferred embodiment of the present invention.

[0062] Figure 8 Detailed circuit diagram of the depth image imaging device according to the above preferred embodiment of the present invention.

[0063] Fig. 9 It is a schematic diagram of the data format of optical signal information converted by the depth image imaging method and device according to the above preferred embodiment of the present invention.

[0064] Fig.10 It is a waveform diagram of optical signals collected by the depth image imaging method and device thereof according to the above preferred embodiment of the present invention.

[0065] Figures 11 to 14 It is a block diagram of the depth image imaging method and the device thereof according to the above preferred embodiment of the present invention, which illustrates the three-level state machine of the specific DDR priority definition.

[0066] Figure 15 to Figure 16 It is a block diagram of a depth image imaging method and a device thereof according to the above preferred embodiment of the present invention, illustrating the implementation of adaptive filtering. DETAILED DESCRIPTION

[0067] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0068] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0069] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the element may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0070] The present invention provides an imaging method and an imaging device for depth images based on ToF technology (Time of Flight). Those skilled in the art understand that for the same programming idea or concept, different types of programmable processors and corresponding electronic components can be used to complete it, which is an equivalent means of executing the same method and is still within the scope of the invention.

[0071] In order to facilitate the explanation of the present invention, the preferred embodiment is provided. Figures 1 to 16 As shown. The specific electronic component models mentioned in this preferred embodiment can be selected and replaced in the same device category that achieves the same function. For example, the stereo camera, depth camera, three-dimensional image sensor, etc. involved in this preferred embodiment can be completed by the acquisition device represented by the ToF camera module or other equivalent devices to perform the image acquisition function with depth-related data.

[0072] The depth image imaging method provided by this preferred embodiment is as follows: Figure 1 and Figure 2 , suitable for obtaining a depth image of an object 80, comprising the following steps:

[0073] After initialization, depth image data is collected; the depth image data is processed in parallel to obtain grayscale values ​​and depth values; and the processed data is output for digital imaging.

[0074] Specifically, the imaging method includes the steps of: presetting relevant parameters and methods for subsequent data processing, including data format setting, calibration correction setting, calculation and filtering sequence, etc. before the acquisition step. That is, after initialization according to the preset setting, the depth image data of the object 80 can be directly processed according to the preset parameters, so that the subsequent processing speed can be improved.

[0075] The depth image imaging device provided in this preferred embodiment includes:

[0076] A parallel processor 10, at least one memory 20, a sensor 30 and an interface component 40, wherein the memory 20, the sensor 30 and the interface component 40 are all communicatively connected to the parallel processor 10. The depth image imaging device is preferably in the form of an integrated circuit as the main body, and more preferably in the form of an integrated integrated module. In this preferred embodiment, the sensor 30 is preferably implemented as a ToF camera module.

[0077] The sensor 30 further includes: a laser emission module and a photosensitive sensor module. The sensor 30 can be controlled to emit a light signal in the field of view. When the light signal encounters the object 80 and is reflected, the sensor 30 can receive the reflected light signal. Preferably, the sensor 30 also includes a filter to ensure the validity of the received light signal.

[0078] In this preferred embodiment, the sensor 30 emits a 940nm laser signal, modulated into a 120Mhz sine wave optical signal, and receives the reflected light signal after passing through a narrow-band filter to ensure that only the 940nm optical signal is received to obtain the final optical signal for imaging. This process is to collect depth image data in this preferred embodiment.

[0079] In order to achieve high resolution, faster frame rate, and lower latency imaging effects. The specific technical means of this preferred embodiment are as follows: Figures 3 to 6 shown.

[0080] like Figure 3 As shown, the presetting step further includes the steps of:

[0081] Obtaining calibration parameters of the sensor 30; and

[0082] The calibration parameters are stored in the memory 20 for use in data processing.

[0083] Preferably, the presetting step is implemented in the parallel processor 10 and a computer terminal 90 through the interface component 40 .

[0084] Preferably, the memory 20 storing the calibration correction parameters is a Flash memory.

[0085] Furthermore, the calibration correction parameters obtained include: gradient error compensation parameters, cyclic error compensation parameters, and temperature drift coefficients. By preprocessing the calibration correction parameters, the processing speed of the subsequent calibration process of the image data can be improved.

[0086] More specifically, the calibration correction parameters of the sensor 30 are obtained by a computer terminal 90 through the I2C interface in the interface component 40 and stored in the memory 20 after conversion. That is, the computer terminal reads the calibration data of the ToF camera module through the I2C interface module, and converts it into data that can be directly used by the parallel processor 10 of the depth image imaging device in the host computer. The computer terminal stores the calibration data in the Flash memory through the I2C interface module.

[0087] In particular, the depth image imaging device further includes a plurality of different types of memories 20, such as a Flash memory and a DDR SDRAM memory. During the initialization process, the parallel processor 10 reads the calibration correction parameters from the Flash memory and caches them in the DDR SDRAM memory. Of course, it can also be cached from the Flash memory to the static cache, or directly called from the Flash memory / DDR SDRAM memory.

[0088] like Figure 4 As shown, the step of collecting depth image data further includes the steps of:

[0089] Transmits 940nm narrowband optical signal after 120Mh sine wave debugging;

[0090] When the light signal hits the object 80 and is reflected, receiving the reflected light signal; and

[0091] Sampling is performed at the set period.

[0092] It is worth mentioning that the numerical performance of the optical signal is only used as an example here for illustration and is not limited.

[0093] Preferably, the sensor 30 samples a set of image data every quarter period. Image data is data that can be converted from optical signals and can be digitally imaged. The image data in this embodiment all have depth properties, that is, a three-dimensional image can be obtained by processing the image data. More preferably, the image data also carries temperature information.

[0094] Preferably, the sensor 30 can quantify the received reflected light signal and obtain image data by statistics.

[0095] The sensor 30 transmits the image data to the parallel processor 10 through the interface component 40 for processing. Preferably, the image data is stored in the memory 20 through the interface component 40. The memory 20 is preferably a DDR SDRAM memory.

[0096] More, such as Figure 5 As shown, the step of processing the depth image data in parallel further comprises the steps of:

[0097] Converting the information data format of the collected optical signal;

[0098] Arrange N phase data for storage in the memory in a certain data format, where N is an integer greater than or equal to 1;

[0099] Calculating in parallel to obtain depth values ​​and grayscale values; and

[0100] Output depth and grayscale values ​​for digital image processing.

[0101] More specifically, in this preferred embodiment, four phase image data with a resolution of 640x480 are sorted out and stored in the memory 20 according to a certain data format. After a complete frame of data is stored in the memory 20, data is read from the memory 20 in row units for subsequent parallel calculation of phase difference values ​​and grayscale values.

[0102] like Figure 6 As shown, the parallel calculation step of this preferred embodiment further includes the steps of:

[0103] Calculate the phase difference;

[0104] Calibrate and correct the phase difference value according to the calibration correction parameter;

[0105] Calculate depth distance;

[0106] The depth distance is filtered to obtain a depth value.

[0107] It is worth mentioning that in this preferred embodiment, the phase difference is not directly calculated to output the depth distance, but the phase difference is calibrated and corrected so that a specific correction is obtained before further processing, thereby ensuring the reliability of the subsequent depth distance calculation. In addition, before the depth distance is calculated to obtain the depth value, the depth distance is filtered to improve the accuracy of the depth value.

[0108] In this preferred embodiment, the calibration correction phase difference is selected from at least one of the following methods: temperature calibration correction, gradient error calibration correction, and cyclic error calibration correction. In the temperature calibration correction, the calibration correction parameters used are from the image data collected by the sensor 30 .

[0109] Furthermore, the filtering process for the depth distance is selected from at least one of the following methods: confidence filtering process and adaptive filtering process. In the confidence filtering process, the gray value is used to assist in filtering the depth distance. Specifically, the gray value corresponding to the depth distance is judged. If it is greater than a certain threshold, the depth distance of the point is considered to be valid, that is, it can be used as a depth value; if it is less than or equal to the threshold, the depth distance of the point is considered to be invalid.

[0110] Through parallel processing, the depth value and the grayscale value are processed and transmitted to the terminal device 90 through the interface component 40. The terminal device 90 is, for example, a computer or a portable intelligent terminal, such as an Android system terminal.

[0111] In this preferred embodiment, the parallel processor 10 of the depth image imaging device adopts an FPGA processor (Field Programmable Gate Array). The memory 20 adopts a combination of Flash memory and DDR SDRAM memory. The interface component 40 adopts a combination of I2C interface, SPI interface (Serial Peripheral Interface) and MIPI interface (Mobile Industry Processor Interface). A depth image imaging method is roughly operated in the depth image imaging device as follows Figure 7 shown.

[0112] A feasible circuit diagram of the depth image imaging device is as follows: Figure 8 As shown. That is to say, the depth image imaging device can be implemented by means of a control circuit. Of course, in a feasible manner, the depth image imaging device can be integrated into a separate component, which is not limited by the present invention. In addition, the necessary electronic components required for the FPGA processor can be connected in an existing manner, with reference to the expression diagram of the Clock. The interface component 40 serves as the input and output port of the parallel processor 10, which can be multiple and can be selected according to different specific program settings.

[0113] In this embodiment, the FPGA processor as the parallel processor 10 is communicatively connected to the Flash memory of the memory 20 through the SPI interface of the interface component 40, is communicatively connected to the sensor 30 through the I2C interface and the MIPI interface of the interface component 40, and is connected to the terminal 90 through the I2C interface and the MIPI interface of the interface component 40. The terminal 90 can be further connected to the parallel processor 10 through USB and / or another chip.

[0114] Specifically, the specific scheme of the depth image imaging method of this embodiment using the FPGA processor as the parallel processor 10 as the core and the ToF camera module as the sensor 30 as the technical basis is as follows: Figures 8 to 16 shown.

[0115] In the presetting step, calibration parameters of the sensor 30 are obtained and stored in the memory 20 .

[0116] The computer terminal 90 reads the calibration correction parameters of the sensor 30 through the I2C interface of the interface component 40, and converts them into a data format that can be directly used by the FPGA module. The computer terminal 90 stores the converted calibration correction parameters into the Flash memory through the I2C interface of the interface component 40. By preprocessing the calibration data, the processing speed of the calibration process can be improved.

[0117] During the initialization process, the parallel processor 10 reads calibration correction data from the Flash memory and caches it in the static cache of the FPGA or the DDR SDRAM memory. The parallel processor 10 performs initialization settings on the sensor 30 for subsequent acquisition of depth image data.

[0118] After initialization, in the step of collecting the depth image, the optical signal is a 940nm laser signal modulated into a 120Mhz sine wave optical signal. The laser emission module of the sensor 30 will emit the modulated optical signal to the target object. After the optical signal hits the object, it is reflected by the object and received by the photosensitive sensing module of the sensor 30. The received optical signal is quantified and counted, wherein the sensor 30 samples a set of data every quarter cycle, and finally the sensor 30 sends the data to the parallel processor 10 through the MIPI interface of the interface component 40.

[0119] Since the parallel processor 10, FPGA processor, and internal submodules process in units of lines, that is, after the previous module processes a line of data, the next module can initiate the operation, and the waiting time between each module is 1 line of data, while the existing software processes data in units of frames, that is, each process waits for N lines (such as 480 lines). Therefore, compared with the existing software calculated in units of frames, the delay is greatly reduced, and the waiting time of N-1 lines is advanced.

[0120] For example, the format of the image data sent to the parallel processor 10 via the MIPI interface is as follows: Fig. 9 As shown, the image data contains 4 phase data, the Effective pixel area represents the data information of one phase, the resolution is 1280*480, and every 2 pixels represent A and B respectively. According to the depth image detection technology, AB represents 1 phase value, and the resolution becomes 640*480 after conversion.

[0121] In particular, image data carries temperature information, such as Fig. 9Each Phase data in the data has 3 lines of EmbeddedData Line data at the end. The length of each line of data is 128 bytes. After parsing according to the data definition, the real-time temperature information can be obtained. The temperature value is used for temperature calibration.

[0122] Before parallel computing, the parallel processor 10 sorts out the four-phase data with a resolution of 640x480 and stores it in the DDR SDRAM memory of the memory 20 according to a certain data format. After a complete frame of data is stored in the DDR SDRAM memory, the data is read from the DDR SDRAM memory in row units, and then the phase difference value and grayscale value are calculated in parallel.

[0123] For calculating the depth value, the specific steps include:

[0124] Step 1: According to the setting, the ToF module performs exposure every quarter cycle to obtain the light signal values ​​of 4 phases, such as Fig.10 , which are respectively denoted as S0, S1, S2, and S3;

[0125] Step 2: Calculate the phase difference using the following formula:

[0126]

[0127] Step 3: By taking the antiderivative, the phase difference can be calculated

[0128]

[0129] Step 4: Calculate the depth distance value D

[0130]

[0131] Wherein, c represents the speed of light, T represents the transmission modulation period, and f represents the transmission modulation frequency.

[0132] In addition, for calculating the gray value, the specific steps include:

[0133] According to the 4-phase data, the formula for calculating the gray value G is:

[0134] G=|I|+|Q|————————————————Formula 4

[0135] Where I = SO-S2 Q = S3-S1

[0136] It is worth mentioning that after step 3 and before step 4, the phase difference Perform calibration correction.

[0137] The specific methods in this preferred embodiment include temperature calibration correction, gradient error calibration correction, cycle error calibration correction, etc.

[0138] Gradient error calibration correction method: In a standard calibration device, the distance error value of each pixel can be measured at the same distance to form a frame of calibration compensation value, and finally each point is corrected and calculated with the calibration compensation value of the corresponding position. In other words, the gradient error compensation parameters are preset.

[0139] The specific implementation method in the FPGA processor is: the calibrated value = the phase value of each point + the compensation value of the corresponding point.

[0140] Cyclic error calibration correction method: In a standard calibration device, the phase difference is measured at different distances. There is a certain deviation from the ideal phase difference, but it shows a certain curve change pattern, which is summarized into a phase difference table. Finally, the actual measurement distance is determined by looking up the table. In other words, the cyclic error compensation parameters are preset.

[0141] Its specific implementation method in the FPGA processor is as follows: During initialization, the cyclic error calibration parameters pre-stored in the Flash memory are read and stored in the static RAM table inside the FPGA processor. Each calculated phase difference can determine the actual distance value by looking up this table.

[0142] Temperature calibration correction method: When the phase difference value at different temperatures passes the same distance, there is a certain deviation from the ideal phase difference, showing a certain linear relationship. After data fitting simulation, a certain linear relationship is calculated. In other words, the temperature drift coefficient is preset.

[0143] The specific implementation method in the FPGA module is: temperature calibration value = calibration value + (current temperature - reference temperature) × temperature drift coefficient. In other words, using Fig. 9 The real-time temperature data carried by the image data is used for temperature calibration.

[0144] After calibration, step 4 (Formula 3) is executed to obtain the depth distance D.

[0145] The depth distance D can be directly used as the depth value. Preferably, the depth distance D is filtered and then used as the depth value, which specifically includes confidence filtering, adaptive filtering and other filtering processes.

[0146] The implementation method of confidence filtering is: the grayscale value corresponding to the depth distance is judged. If it is greater than a certain threshold, the depth distance of the point is considered valid, that is, it can be used as a depth value; if it is less than or equal to the threshold, the depth distance of the point is considered invalid.

[0147] Adaptive filtering is implemented in the following ways: Figures 15 to 16 As shown, the FPGA processor program execution method is used as an example:

[0148] like Fig.15 , adaptive filtering is performed on each depth distance, and different orders of filter parameter settings can be supported. First, Gaussian filtering is performed on the depth distance in the column direction (Y axis) to obtain a new depth distance D_Y, and then the depth distance D_Y in the row direction (X axis) is filtered to obtain a new filter value D_X;

[0149] The specific implementation process of the column-wise (Y-axis) filtering algorithm is as follows:

[0150] Receive the first line of data of a frame and store it in the FIFO buffer (FIFO1);

[0151] Receive the second line of data of frame 1 and continue to store it in the FIFO buffer (FIFO1). At the same time, read data from FIFO1 and store it in FIFO2.

[0152] Receive the third line of data of frame 1 and continue to store it in the FIFO buffer (FIFO1). At the same time, read data from FIFO1 and store it in FIFO2. At the same time, read data from FIFO2 and store it in FIFO3.

[0153] And so on, the remaining FIFO buffers adopt similar operations.

[0154] According to the first-in-first-out principle, after the first row of data is stored in FIFO6, the seventh row of data starts to be input, and the first row of data is read out from FIFO6. We start to perform Gaussian calculation on the first data read out of each FIFO;

[0155] The filter parameter 0 (w6) is multiplied by the first number of FIFO6 to obtain sum6;

[0156] Multiply the filter parameter w+1(w7) by the first number of FIFO7 to get sum7...

[0157] Multiply the filter parameter w-1 (w5) by the first number of FIFO5 to get sum5...

[0158] SUM=sum0+sum1…+sum6+sum7…sum11;

[0159] Weight=w0+sw1…+w6+w7…w11;

[0160] Filtered depth distance = SUM / Weight;

[0161] By analogy, the filtering calculation of one row of data can be realized.

[0162] After the last line of data (line N) is input, the filter has completed filtering of N-6 lines of data, of which the last 6 lines of data of a frame are still cached in FIFO1~FIF06;

[0163] While calculating the N-6th row of data filtering, a fill mode is automatically triggered to input a row of data containing all 0s into FIFO1;

[0164] While calculating the N-5th row of data filtering, a fill mode is automatically triggered to input a row of data containing all zeros into FIFO1;

[0165] By analogy, filtering of 1 frame and 6 lines of data can be completed by doing this 6 times.

[0166] like Fig.16 As shown, the column-wise (Y-axis) filtering algorithm code state machine is specifically implemented as follows:

[0167] STATE_IDLE: Waiting state, read the first 6 lines of data of each frame and store them into FIFO;

[0168] STATE_RDY: Indicates that there are 6 lines of data in the current FIFO, waiting for the next line of data to arrive. When the next line of data arrives, the filtering result can be output;

[0169] STATE_FILTER: Perform Y-axis filtering and output the filtering result;

[0170] STATE_LAST: Indicates that the last 6 lines of data are filtered, but data is not read from DDR and FIFO is filled with data 0.

[0171] The specific implementation process of the row (X-axis) filtering algorithm is as follows:

[0172] The implementation idea is consistent with the column-wise (Y-axis) filtering algorithm process, changing from 1 FIFO cache to 1 register cache.

[0173] After the above two steps of filtering operations, the depth distance D can be adaptively filtered to obtain the depth value.

[0174] Finally, the depth value and grayscale value obtained by parallel processing are output for digital image imaging. Preferably, the FPGA processor outputs the depth value and grayscale value data through the MIPI interface and transmits them to a chip with a MIPI interface. After receiving the data, or after further digital image processing, the data can be transmitted to a terminal device 90, such as a computer or a portable smart terminal, such as an Android system terminal, through a USB.

[0175] In particular, the preferred embodiment uses DDR SDRAM memory. Preferably, the third generation DDR SDRAM memory is used. Since only one operation can be read or written at the same time, the preferred embodiment also provides a read and write priority process, such as Figures 11 to 14 , making the process reasonable and avoiding affecting the processing speed.

[0176] The DDR SDRAM memory is defined with processing priorities, from high to low: calibration correction parameter write operation, image data write operation, calibration correction parameter read operation, image data read operation.

[0177] Specifically, the calibration parameter write operation has the highest priority, which is related to the actual data processing result; the pixel data write operation has the second priority, and if it is not stored in the DDR SDRAM memory in time, it will generate additional resource overhead; the calibration data read operation has the third priority; and the pixel data read operation has the lowest priority.

[0178] The management of DDR SDRAM memory is implemented through 3-level state machine nesting, corresponding to 3 modules: ddr_ctrl, ddr_axi, ddr_raxi / ddr_waxi, such as Fig.11 As shown, ddr_ctrl is responsible for DDR address management, whether it is write address or read address; responsible for data classification management, whether it is calibration parameters or phase data; and is also the initiator of all DDR operations.

[0179] A state machine such as Fig.12 As shown, its state machine description:

[0180] STATE_IDLE: idle waiting state;

[0181] STATE_PIXELI: write phase data into DDR;

[0182] STATE_PIXELO: read phase data from DDR;

[0183] STATE_INIT: After power-on, the calibration parameters are read from Flash and stored in DDR, with the highest priority.

[0184] STATE_CALI: read calibration data from DDR;

[0185] Among them, ddr_axi initiates DDR read and write operations, and sends address commands to DDR.

[0186] Secondary state machine, such as Fig.13 As shown, its state machine description:

[0187] STATE_IDLE: idle state;

[0188] STATE_CMDW: indicates that the write address command has been sent;

[0189] STATE_CMDW: indicates that the read address command has been sent;

[0190] STATE_BURST: indicates that the reading and writing of one row of data is completed;

[0191] Among them, ddr_raxi / ddr_raxi respectively correspond to read / write operations according to DDR timing.

[0192] Three-level state machine, such as Fig.14 As shown, its state machine description:

[0193] STATE_IDLE: idle state;

[0194] STATE_WR: write data process;

[0195] STATE_RD: Read data process.

[0196] Therefore, in addition to adopting the strategy of parallel computing processing, an optimization strategy is also provided for the memory call processing to make the overall computing operation reliable.

[0197] The data calculated by the depth image imaging method can be transmitted to a computer or mobile terminal or the like by wired or wireless means after digital image processing. It has high environmental adaptability and can achieve the following effects: high resolution, up to 640*480; faster frame rate, up to 30fps, and frame processing can be adaptively changed with the input frame rate; lower latency, reaching the microsecond us level.

[0198] It should be understood by those skilled in the art that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments, and the embodiments of the present invention may be deformed or modified in any way without departing from the principles.

Claims

1. A depth image forming method, characterized in that: The following steps are involved: After initialization according to the parameters of a sensor, collecting depth image data through the sensor; Processing the depth image data in parallel to obtain grayscale values ​​and depth values; as well as Output depth value and grayscale value for digital image imaging; The step of processing the depth image data in parallel to obtain the grayscale value and the depth value comprises the steps of: Converting the information data format of the collected optical signal to obtain N phase data; Arrange N phase data, and store the N phase data in a memory for storing N phase data according to a certain data format, where N is an integer greater than or equal to 1; Calculate N phase data in parallel to obtain a depth value and a grayscale value, wherein the step of calculating the N phase data in parallel to obtain the depth value and the grayscale value comprises the steps of: calculating and obtaining a phase difference value and a grayscale value for the N phase data in parallel by processing the phase data in units of lines; wherein the phase difference value is calculated based on the phase data; and the grayscale value is calculated based on the phase data; The step of collecting depth image data further comprises the steps of: Transmitting a debugged optical signal; When the light signal hits an object and is reflected, receiving the reflected light signal; and The depth image data is obtained by sampling at a period of one-N.

2. The depth image imaging method according to claim 1, wherein before the step of collecting depth image data, the depth image imaging method further comprises the steps of: Acquiring calibration correction parameters of the sensor; and The calibration correction parameters are stored in a memory for storing calibration correction parameters.

3. The depth image imaging method according to claim 2, wherein the memory used to store the calibration correction parameters is a Flash memory. 4 . The depth image imaging method according to claim 1 , wherein the memory used to store the N phase data is a DDR SDRAM memory.

5. The depth image imaging method according to claim 2, wherein after calculating N phase data in parallel to obtain depth values ​​and grayscale values, and before outputting the depth values ​​and grayscale values ​​for digital image processing, the step of processing the depth image data in parallel to obtain grayscale values ​​and depth values ​​further comprises the steps of: Calibrate and correct the phase difference value according to the calibration correction parameter; Calculating a depth distance based on the phase difference value; and The depth distance is filtered to obtain a depth value.

6. The depth image imaging method according to claim 5, wherein the method of calibrating and correcting the phase difference is selected from at least one of the following methods: temperature calibration correction, gradient error calibration correction, and cyclic error calibration correction. 7 . The depth image imaging method according to claim 5 , wherein filtering processing performed on the depth distance is selected from at least one of the following methods: confidence filtering processing and adaptive filtering processing.

8. A depth image imaging device, characterized in that include: A parallel processor, at least one memory, a sensor and an interface component, wherein the at least one memory, the sensor and the interface component are all communicatively connected to the parallel processor, wherein calibration correction parameters of the sensor are stored in a memory for storing calibration correction parameters of the at least one memory for use by the parallel processor, wherein the sensor collects depth image data by emitting light signals and receiving reflected light signals, wherein the parallel processor processes the depth image data in parallel and performs calibration correction according to the calibration correction parameters to obtain grayscale values ​​and depth values, so that the interface component outputs grayscale values ​​and depth values ​​for digital image imaging; Wherein, the parallel processor processes the depth image data in parallel and performs calibration and correction according to the calibration and correction parameters to obtain the grayscale value and the depth value, converts the information data format of the collected light signal to obtain N phase data; sorts the N phase data, and stores the N phase data in a memory for storing N phase data of the at least one memory in a certain data format; calculates and obtains the phase difference value and the grayscale value of the N phase data in parallel by processing the phase data in units of lines; wherein N is an integer greater than or equal to 1; the phase difference value is obtained by calculation based on the phase data; the grayscale value is obtained by calculation based on the phase data; The acquisition of depth image data includes the sensor emitting a narrowband light signal after sinusoidal wave adjustment, receiving the reflected light signal after the light signal hits an object, and sampling it according to a one-N period to obtain depth image data.

9. The depth image imaging device according to claim 8, wherein the calibration correction parameters of the sensor include: Gradient error compensation parameters, cycle error compensation parameters and temperature drift coefficient. 10 . The depth image imaging device according to claim 8 , wherein the memory for storing N phase data of the at least one memory is a DDR SDRAM memory.

11. The depth image imaging device according to claim 8, wherein the parallel processor obtains phase data from the memory for storing N phase data of the at least one memory and calculates a phase difference value, calibrates and corrects the phase difference value according to the calibration correction parameters, calculates the depth distance based on the phase difference value, and filters the depth distance to obtain a depth value.

12. The depth image imaging device according to claim 11, wherein the parallel processor reads the calibration correction parameters from the memory for storing the calibration correction parameters of the at least one memory to calibrate and correct the phase difference, and the method of calibrating and correcting the phase difference value is selected from at least one of the following methods: temperature calibration correction, gradient error calibration correction, and cyclic error calibration correction. 13 . The depth image imaging device according to claim 11 , wherein the parallel processor performs filtering processing on the depth distance in at least one of the following ways: confidence filtering processing and adaptive filtering processing.

Images