Interpolation method, system and electronic device for an ifto camera
By combining the multi-phase method and the aliasing period lookup table, the problem of phase aliasing in iToF camera ranging is solved, achieving more efficient ranging performance and lower system power consumption.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SIGMASTAR TECH LTD
- Filing Date
- 2022-10-21
- Publication Date
- 2026-07-21
AI Technical Summary
Existing iToF cameras are prone to phase aliasing during ranging, which leads to a decrease in ranging performance, high computational load, and high system power consumption.
Phase data is acquired using a multi-phase method, the aliasing period is determined by a pre-established aliasing period lookup table, and the dual-frequency function is enabled or disabled based on depth data, thereby reducing computational load and power consumption.
It improves ranging performance, reduces computational load and system dynamic power consumption, and uses depth histogram analysis to determine depth fluctuations and rationally adjust the dual-frequency state to optimize system power consumption.
Smart Images
Figure CN115599163B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of ToF ranging, and more particularly to an iToF camera dealiasing method, system, and electronic device. Background Technology
[0002] Indirect time of flight (iToF) calculates the true depth of an object by measuring the phase delay between emitted light and reflected light received by an image sensor, and it has wide applications in 3D measurement. Specifically, iToF uses continuous pulse modulation and a multi-phase method to demodulate distance information based on the phase, exhibiting good robustness and strong anti-interference characteristics.
[0003] like Figure 1 As shown, the working principle of the iToF camera is as follows: the modulation module 11 controls the light-emitting module 12 to actively emit modulated light signals; the emitted light is emitted onto the surface of the target object 19, and the reflected light signal formed after reflection by the target object 19 is sampled by the photosensitive pixel array 13 of the image sensor; then the distance of the target object is calculated based on the phase shift of the emitted and reflected light. The light-emitting module 12, such as a VCSEL, an infrared emitter, or an LED, is usually driven by a modulated square wave generated by the image sensor to emit modulated pulsed light signals to the target object. Then, the image sensor receives the light signal reflected back from the target object, and the depth information of the target object is calculated by calculating the round-trip time of light.
[0004] The maximum distance that an iToF camera can measure is related to the modulation frequency of the emitted light signal. Distance measurement depends on the change in the measured phase. When the actual distance exceeds the maximum measurement range of a single frequency, the measurement range is limited. max This will cause phase aliasing. Lowering the modulation frequency can increase the maximum ranging range, but it will reduce the ranging accuracy.
[0005] Chinese patent publication (CN 113256539A) discloses a method for determining the presence of aliasing by acquiring depth and brightness maps, based on the relationship between their changes, and then performing aliasing processing on the aliased pixels. This method only requires a single frequency to identify aliased pixels, significantly reducing system power consumption. However, due to variations in the reflectivity of the object under test and the distance between the object and the sensor, there is no deterministic mapping relationship between depth and brightness, thus this method has significant limitations.
[0006] Therefore, providing a ranging method that improves ranging performance, reduces computational load, and lowers system dynamic power consumption is a problem that needs to be solved. Summary of the Invention
[0007] The technical problem to be solved by the present invention is to provide a ranging method that improves ranging performance, reduces computational load, and reduces system dynamic power consumption, and to provide an iToF camera dealiasing method, system, and electronic device.
[0008] To address the aforementioned problems, this invention provides an iToF camera dealiasing method, comprising the following steps: acquiring phase data from the raw data collected by the iToF camera using a multi-phase method; determining the aliasing period through dual-frequency measurement and according to a pre-established aliasing period lookup table; acquiring depth data based on the aliasing period and the phase data; and controlling the dual-frequency function to be enabled to redetermine the aliasing period, or controlling the dual-frequency function to be disabled, based on the depth data.
[0009] To address the aforementioned problems, this invention provides an iToF camera dealiasing system, comprising: a phase data acquisition module for acquiring phase data from raw data collected by the iToF camera using a multi-phase method; an aliasing period acquisition module for determining the aliasing period through dual-frequency measurement and based on a pre-established aliasing period lookup table; a depth data acquisition module for acquiring depth data based on the aliasing period and the phase data; and a dual-frequency switch control module for controlling the dual-frequency function to be enabled to redetermine the aliasing period or to be disabled based on the depth data.
[0010] To address the aforementioned problems, the present invention provides an electronic device, including a memory, a processor, and a computer-executable program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer-executable program, implements the steps of the iToF camera dealiasing method.
[0011] The above technical solution reduces the computational load of calculating the aliasing period in dual-frequency mode by pre-establishing an aliasing period lookup table and finding the aliasing period according to the lookup parameters, thereby reducing power consumption. Simultaneously, it uses depth histogram data analysis to determine the magnitude of depth fluctuations, and shuts down dual-frequency mode when depth fluctuations are small, further reducing system power consumption. Attached Figure Description
[0012] Appendix Figure 1 The diagram shown illustrates the working principle of an iToF camera in the prior art.
[0013] Appendix Figure 2 The diagram shows a flowchart of the specific implementation steps of the iToF camera dealiasing method of the present invention.
[0014] Appendix Figure 3AThe diagram shown is a schematic representation of an embodiment of the present invention that packages and loads the current dual-frequency switching state and modulation frequency state.
[0015] Appendix Figure 3B The image shows a depth histogram of a specific embodiment of the iToF camera dealiasing method described in this invention.
[0016] Appendix Figure 4A The diagram shown illustrates the data flow when dual-frequency operation is enabled in a specific implementation of the iToF camera dealiasing method described in this invention.
[0017] Appendix Figure 4B The diagram shown is a schematic representation of the dual-frequency on / off data flow of a specific implementation of the iToF camera dealiasing method of the present invention.
[0018] Appendix Figure 5 The diagram shown is a schematic representation of an embodiment of the iToF camera dealiasing system described in this invention.
[0019] Appendix Figure 6 The diagram shown is a schematic representation of an embodiment of the electronic device described in this invention. Detailed Implementation
[0020] The following detailed description, in conjunction with the accompanying drawings, illustrates the specific implementation methods, systems, and electronic devices for iToF camera dealiasing provided by this invention.
[0021] Appendix Figure 2 The diagram shows a flowchart of a specific implementation of the iToF camera dealiasing method of the present invention, including the following steps: Step S21, obtaining phase data based on the raw data collected by the iToF camera using a multi-phase method; Step S22, determining the aliasing period through dual-frequency measurement and according to a pre-established aliasing period lookup table; Step S23, obtaining depth data based on the aliasing period and the phase data; and Step S24, controlling the dual-frequency function to be enabled to redetermine the aliasing period, or controlling the dual-frequency function to be disabled, based on the depth data.
[0022] Referring to step S21, phase data is obtained from the raw data acquired by the iToF camera using a multi-phase method. In a specific embodiment, the step of obtaining phase data further includes: calculating phase data using a four-phase method based on raw data of different phases, and correcting the phase data; and packaging and loading the current dual-frequency function's on / off state and modulation frequency state into the end of the raw data packet.
[0023] Specifically, the four-phase method acquires the original phase data Q1 when the phase delay is 0° (i.e., 0 phase), Q2 when the phase delay is 90° (i.e., 90 phase), Q3 when the phase delay is 180° (i.e., 180 phase), and Q4 when the phase delay is 270° (i.e., 270 phase). In this embodiment, the reflected light signal is sampled at equal intervals, four times per cycle. In other specific embodiments, it may be sampled two or three times per cycle.
[0024] Specifically, the formula for calculating the phase of the depth image using the four-phase method is as follows:
[0025] phase=arctan((Q3-Q4) / (Q1-Q2))
[0026] Wherein, Q1 is the original phase data with a phase delay of 0°, Q2 is the original phase data with a phase delay of 90°, Q3 is the original phase data with a phase delay of 180°, and Q4 is the original phase data with a phase delay of 270°. In other specific embodiments, the phase of the depth image can also be calculated using a two-phase method or a three-phase method.
[0027] Due to factors such as light waveform and pixel characteristics, phase calculations are subject to errors introduced from various aspects, thus requiring phase correction. As a specific embodiment, the phase data undergoes geometric correction, temporal and spatial noise reduction, FPPN correction, Wiggling correction, and temperature error compensation to obtain calibrated phase data, thereby improving the accuracy of phase calculations.
[0028] Specifically, the current dual-frequency function on / off state and modulation frequency state are packaged and loaded into the end of the original data packet, as shown in the appendix. Figure 3A As shown. (Attached) Figure 3A In the diagram, label 31 indicates raw data, and label 32 indicates the footer, which is the location where the current dual-frequency function's on / off state and modulation frequency state are packaged and loaded.
[0029] Referring to step S22, the aliasing period is determined by dual-frequency measurement and according to a pre-established aliasing period lookup table. As a specific embodiment, the step of determining the aliasing period further includes the following steps: 1) Pre-establishing lookup parameters lut idx The aliasing period lookup table Lut corresponds one-to-one with the aliasing period n1; where Lut[lut idx ]=n1(n1∈[0,M1)); 2) In actual calculation, the first modulation frequency f1 in the dual-frequency measurement and the corresponding first measurement phase are used. and the second modulation frequency f2 and the corresponding second measurement phase Get lookup parameter lut idx ; and 3) based on the search parameter lut idx The aliasing period lookup table Lut is consulted to determine the aliasing period n1. Specifically, the first modulation frequency f1 is a high-frequency frequency (frequency higher than a preset threshold), and the second modulation frequency f2 is a low-frequency frequency (frequency lower than a preset threshold).
[0030] As a specific embodiment, the correspondence between the lookup parameters and the aliasing period is as follows:
[0031] lut idx = mod(n1*M2, M1),
[0032] Where M1 = f1 / (gcd(f1, f2)) and M2 = f2 / (gcd(f1, f2)) are given.
[0033] lut idx To find the parameters, n1 is the aliasing period and n1∈[0,M1), f1 is the first modulation frequency, and f2 is the second modulation frequency. By pre-establishing an aliasing period lookup table and finding the aliasing period according to the lookup parameters, the computational load of calculating the aliasing period in dual-frequency mode is reduced, thus lowering power consumption.
[0034] As a specific embodiment, after determining the aliasing period, the method further includes the following steps: When the dual-band function is enabled, the aliasing period is stored as a reference frame. The reference frame is stored in the original data packet. When the dual-band function is disabled, the aliasing period is determined based on the presence or absence of the reference frame; that is, when the reference frame exists, the aliasing period is determined by the aliasing period of the previous cached frame; when the reference frame is empty, the aliasing period is 0.
[0035] Referring to step S23, depth data is obtained based on the aliasing period and the phase data. As a specific embodiment, the step of obtaining depth data based on the aliasing period and the phase data further includes:
[0036] Calculate the depth using the following formula:
[0037]
[0038] Where n1 is the aliasing period corresponding to the first modulation frequency f1. The measured phase corresponding to the first modulation frequency f1, d max1 = (c / 2)*(1 / f1).
[0039] As a specific implementation, when dual-band operation is enabled, the calculation depth uses a lookup table to find the aliasing period. When dual-band operation is disabled, the calculation depth obtains the aliasing period using a reference frame. When a reference frame exists, the aliasing period is determined by the cached aliasing period of the previous frame; when the reference depth is empty, the aliasing period is 0, and the depth is calculated directly. By establishing a reference frame to obtain the aliasing period, the computational load of the aliasing period is reduced, thus lowering power consumption.
[0040] Referring to step S24, based on the depth data, the dual-frequency function is controlled to be enabled to redetermine the aliasing period, or the dual-frequency function is controlled to be disabled. As a specific embodiment, the step of controlling the dual-frequency function to be enabled to redetermine the aliasing period, or controlling the dual-frequency function to be disabled, based on the depth data, further includes: 1) dividing the maximum range of the depth image according to a predetermined stack number, and calculating the total depth value contained in each stack in each frame of the depth image to form a depth histogram, such as... Figure 3B As shown, the horizontal axis represents the bin value, and the vertical axis represents the number of depth images contained in the current bin value (the number is the coordinate value multiplied by 10). 4 1) Calculate the average value of the total depth contained in each stack in the first n frames of the depth image: avg1, avg2, ..., avg m Where m is the number of stacks divided by the maximum range of the depth; 3) Calculate the total value of each stack in the current frame: bin1, bin2, ..., bin m With the average values avg1, avg2, ..., avg m The differences are diff1, diff2, ..., diff m Sum the differences, i.e., sum(diff1+diff2+…diff). m ); and 4) if the sum is greater than a preset threshold, then the dual-frequency function is enabled; otherwise, the dual-frequency function is disabled. If the sum is greater than a preset threshold, it indicates that the pixel depth fluctuation of the current frame is large, so the dual-frequency function is enabled and the aliasing period is redefined; if the sum is less than or equal to the preset threshold, it indicates that the pixel depth fluctuation of the current frame is small, so the dual-frequency function is disabled to reduce system overhead. As a specific implementation, the preset threshold is 16 by default and can be set by the user.
[0041] Appendix Figure 4AThe diagram illustrates the data flow when dual-frequency operation is enabled in a specific implementation of the iToF camera dealiasing method described in this invention. Solid black lines with arrows indicate data flow, while dashed black lines with arrows indicate no data flow. When dual-frequency operation is enabled, phase data is acquired from the original data. The dual-frequency phase data undergoes high-frequency phase correction and low-frequency phase correction respectively, resulting in corrected high-frequency and low-frequency phases. The aliasing period is determined by searching a pre-established aliasing period lookup table based on the corrected high-frequency and low-frequency phases. Depth data is acquired through the aliasing period, and the aliasing period is simultaneously stored as a reference frame. While outputting the depth data, a depth histogram is formed based on the depth data. The histogram is used to analyze the current image fluctuations, and commands to enable or disable dual-frequency operation are output based on the magnitude of the image fluctuations.
[0042] Appendix Figure 4B The diagram illustrates the data flow during dual-frequency on / off states of a specific implementation of the iToF camera dealiasing method described in this invention. Solid black lines with arrows indicate data flow, while dashed black lines with arrows indicate no data flow. When dual-frequency is off, phase data is acquired based on the original data. High-frequency phase data undergoes high-frequency phase correction to obtain the corrected high-frequency phase. The true phase is acquired based on the reference frame stored when dual-frequency is on, and depth data is acquired based on the true phase. Simultaneously with outputting the depth data, a depth histogram is formed based on the depth data. The histogram is used to analyze current image fluctuations, and commands to turn dual-frequency on / off are output based on the magnitude of the image fluctuations.
[0043] The above technical solution reduces the computational load of calculating the aliasing period in dual-frequency mode by pre-establishing an aliasing period lookup table and finding the aliasing period according to the lookup parameters, thereby reducing power consumption. Simultaneously, it uses depth histogram data analysis to determine the magnitude of depth fluctuations, and shuts down dual-frequency mode when depth fluctuations are small, further reducing system power consumption.
[0044] Based on the same inventive concept, the present invention also provides an iToF camera dealiasing system.
[0045] Appendix Figure 5 The diagram shown is a schematic representation of a specific embodiment of the iToF camera dealiasing system. The iToF camera dealiasing system 50 in this embodiment includes: a phase data acquisition module 51, used to acquire phase data from the raw data collected by the iToF camera using a multi-phase method; an aliasing period acquisition module 52, used to determine the aliasing period through dual-frequency measurement and according to a pre-established aliasing period lookup table; a depth data acquisition module 53, used to acquire depth data based on the aliasing period and the phase data; and a dual-frequency switch control module 54, used to control the dual-frequency function to be enabled to redetermine the aliasing period or to control the dual-frequency function to be disabled based on the depth data.
[0046] In one specific implementation, during system operation, the phase data acquisition module 51 can calculate the phase based on the raw data acquired by the iToF camera and the multi-phase method. In the dual-frequency enabled state, the aliasing period acquisition module 52 determines the corresponding aliasing period based on the phase obtained from the dual-frequency measurement and a pre-established lookup table, and stores the aliasing period as a reference frame in the raw data packet. In the dual-frequency disabled state, the aliasing period acquisition module 52 determines the aliasing period based on the presence or absence of a reference frame; if a reference frame exists, the aliasing period is determined by the aliasing period of the previous cached frame; if the reference frame is empty, the aliasing period is 0, and depth is directly calculated. After determining the aliasing period, the depth data acquisition module 53 acquires depth data based on the aliasing period and the phase data. Data analysis is performed on the acquired depth data to establish a depth histogram. The fluctuation of the histogram is used to determine the magnitude of pixel depth fluctuation, and the dual-frequency switch control module 54 controls the dual-frequency function's on / off state. When the pixel depth fluctuation is large, dual-frequency is enabled to re-determine the aliasing period; when the pixel depth fluctuation is small, dual-frequency is disabled to reduce system overhead.
[0047] The above technical solution reduces the computational load of calculating the aliasing period in dual-frequency mode by pre-establishing an aliasing period lookup table and finding the aliasing period according to the lookup parameters, thereby reducing power consumption. Simultaneously, it uses depth histogram data analysis to determine the magnitude of depth fluctuations, and shuts down dual-frequency mode when depth fluctuations are small, further reducing system power consumption.
[0048] Based on the same inventive concept, the present invention also provides an electronic device.
[0049] Appendix Figure 6 The diagram shown is a schematic representation of an embodiment of the electronic device described in this invention. The electronic device 100 of this embodiment includes a memory 101, a processor 102, and a computer-executable program stored in the memory 101 and executable on the processor 102. The processor 102 implements the computer-executable program as follows: Figure 2 The steps of the iToF camera dealiasing method are detailed in the previous description and will not be repeated here.
[0050] The above technical solution reduces the computational load of calculating the aliasing period in dual-frequency mode by pre-establishing an aliasing period lookup table and finding the aliasing period according to the lookup parameters, thereby reducing power consumption. Simultaneously, it uses depth histogram data analysis to determine the magnitude of depth fluctuations, and shuts down dual-frequency mode when depth fluctuations are small, further reducing system power consumption.
[0051] It should be noted that the terms "comprising" and "having," and their variations, used in this invention document are intended to cover non-exclusive inclusion. The terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, unless explicitly indicated by the context. It should be understood that such data used interchangeably where appropriate. Furthermore, embodiments and features within embodiments of this invention can be combined with each other unless otherwise specified. In addition, descriptions of well-known components and technologies have been omitted in the above description to avoid unnecessarily obscuring the concepts of this invention. In the various embodiments described above, each embodiment focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.
[0052] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for dealiasing in an iToF camera, characterized in that, Includes the following steps: Phase data is obtained from the raw data acquired by the iToF camera using a multi-phase method; The aliasing period is determined by dual-frequency measurement and according to a pre-established aliasing period lookup table, wherein the aliasing period lookup table is obtained using formula lut. idx =mod((n1*M2, M1) to establish, where M1=f1 / (gcd(f1, f2)), M2=f2 / (gcd(f1, f2)), lut idx To find the parameters, n1 is the aliasing period and n1∈[0, M1), f1 is the first modulation frequency, and f2 is the second modulation frequency; Depth data is obtained based on the aliasing period and the phase data; as well as Based on the depth data analysis, the current frame pixel depth fluctuation magnitude is determined, and the dual-frequency function is enabled to redetermine the aliasing period, or the dual-frequency function is disabled; wherein, the step of analyzing the current frame pixel depth fluctuation magnitude based on the depth data specifically involves: performing stack partitioning and difference summation on the depth data to calculate the fluctuation value; when the fluctuation value is greater than a preset threshold, the dual-frequency function is enabled, otherwise the dual-frequency function is disabled; The step of performing stack partitioning and difference summation to calculate fluctuation values for the depth data further includes: dividing the maximum range of the depth image according to a predetermined number of stacks, and calculating the total depth value contained in each stack in each frame of the depth image to form a depth histogram; calculating the average values avg1, avg2, ..., avg of the total depth value contained in each stack in the first n frames of the depth image. m Where m is the number of stacks divided by the maximum range of the depth; calculate the total value of each stack in the current frame: bin1, bin2, ..., bin m With the average values avg1, avg2, ..., avg m The differences are diff1, diff2, ..., diff m The difference is summed to obtain the fluctuation value.
2. The method according to claim 1, characterized in that, The step of acquiring phase data further includes: Phase data is calculated using the four-phase method from raw data of different phases, and the phase data is then corrected; and Pack the current dual-frequency function on / off state and modulation frequency state into the end of the original data packet.
3. The method according to claim 1, characterized in that, The step of determining the aliasing period further includes the following steps: Pre-establish an aliasing period lookup table that corresponds one-to-one with the lookup parameters and aliasing periods; The lookup parameters are obtained by using the first modulation frequency and the corresponding first measurement phase, and the second modulation frequency and the corresponding second measurement phase in the dual-frequency measurement. as well as The aliasing period is determined by searching the aliasing period lookup table according to the search parameters.
4. The method according to claim 1, characterized in that, The method further includes: storing the aliasing period as a reference frame when the dual-frequency function is enabled.
5. The method according to claim 1, characterized in that, The step of obtaining depth data based on the aliasing period and the phase data further includes: calculating the depth according to the following formula: d=d max1 *(n1+φ1) Where n1 is the aliasing period corresponding to the first modulation frequency, φ1 is the measurement phase corresponding to the first modulation frequency, and d max1 =c / 2*1 / f1.
6. The method according to claim 1, characterized in that, When dual-band functionality is disabled, the aliasing period is determined by the aliasing period of the previous buffered frame when a reference frame is present; when the reference frame is empty, the aliasing period is 0.
7. An iToF camera dealiasing system, characterized in that, include: The phase data acquisition module is used to acquire phase data from the raw data collected by the iToF camera using a multi-phase method; The aliasing period acquisition module is used to determine the aliasing period through dual-frequency measurement and according to a pre-established aliasing period lookup table, wherein the aliasing period lookup table is obtained using the formula LUT. idx =mod(n1*M2, M1) is used to establish, where M1=f1 / (gcd(f1, f2)), M2=f2 / (gcd(f1, f2)), and LUT... idx To find the parameters, n1 is the aliasing period and n1∈[0, M1), f1 is the first modulation frequency, and f2 is the second modulation frequency; A depth data acquisition module is used to acquire depth data based on the aliasing period and the phase data; The dual-frequency switch control module is used to control the dual-frequency function to be enabled to redetermine the aliasing period or to disable the dual-frequency function based on the depth data analysis of the current frame pixel depth fluctuation. Specifically, the depth data analysis of the current frame pixel depth fluctuation involves: performing stack partitioning and difference summation on the depth data to calculate the fluctuation value; when the fluctuation value is greater than a preset threshold, the dual-frequency function is enabled; otherwise, the dual-frequency function is disabled. The dual-frequency switch control module is further configured to: divide the maximum range of the depth image according to a predetermined number of stacks, and calculate the total depth value contained in each stack in each frame of the depth image to form a depth histogram; calculate the average value avg1, avg2, ..., avg of the total depth value contained in each stack in the first n frames of the depth image. m Where m is the number of stacks divided by the maximum range of the depth; calculate the total value of each stack in the current frame: bin1, bin2, ..., bin m With the average values avg1, avg2, ..., avg m The differences are diff1, diff2, ..., diff m The difference is summed to obtain the fluctuation value.
8. An electronic device comprising a memory, a processor, and a computer-executable program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-executable program, it implements the steps of the iToF camera dealiasing method as described in any one of claims 1 to 6.