A depth information correction method, electronic device and storage medium

Abstract

This application provides a depth information correction method, electronic device, and storage medium. In response to a depth information acquisition trigger event, a target sensor is invoked to acquire depth information of a target object. The acquired depth information is corrected according to a calibration curve to obtain corrected depth information. The calibration curve is a fitting curve between the signal peak value and the depth information deviation value. By using the depth information deviation value determined based on this calibration curve to correct the depth information obtained from histogram statistics, the signal peak shift in the histogram statistics obtained from TDC statistics due to the pile-up phenomenon can be effectively corrected, thereby improving the accuracy of the depth information calculation results and the ranging accuracy.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a depth information correction method, electronic device and storage medium. Background Technology

[0002] 3D sensing is increasingly being used in various applications of electronic devices, such as 3D face unlock, camera focusing, depth image acquisition, ambient environment detection, and virtual object rendering. Currently, depth information is typically acquired based on direct time-of-flight (DTOF) measurement. The principle of DTOF ranging is based on a photodetector emitting light pulses towards the object being measured, and by measuring the time interval between the emitted and reflected light pulses, the distance to the target object is directly calculated to generate depth information.

[0003] Currently, when measuring the distance to a target object, a number of pulse beams are typically emitted in a natural environment, and the reflected beams corresponding to the emitted pulse beams are received. Then, a time digital converter (TDC) is used to record the time interval between the emitted and received pulses, and the flight time with the highest frequency is taken as the flight time for calculating the depth information of the target object. However, the single photon avalanche diode (SPAD) in the TDC has a dead time, which will cause the count peak to shift, thus affecting the accuracy of the depth information measurement. Summary of the Invention

[0004] This application provides a depth information correction method, an electronic device, and a storage medium, which can improve the accuracy of depth information measurement.

[0005] In a first aspect, embodiments of this application provide a depth information correction method, comprising: responding to a depth information acquisition trigger event, calling a target sensor to acquire depth information of a target object; and correcting the acquired depth information according to a calibration curve to obtain corrected depth information.

[0006] Here, the above depth information collection trigger event can be triggered in any of the following scenarios: the 3D face unlock function is activated when the screen is locked, the user selects the background blur shooting mode when taking a picture, the user plays AR games on an electronic device, the TOF camera (or 3D sensor) is called to collect image information when using the smart eye-tracking service, and the payment application calls the TOF camera to perform face recognition during payment or login processes.

[0007] Here, the specific operation of calling the target sensor to collect depth information can be as follows: when the target sensor is a TOF camera, the mobile phone can drive the TOF camera to perform the action of collecting depth information through the camera driver; when the target sensor is a 3D sensor, the mobile phone can drive the 3D sensor to perform the action of collecting depth information through the sensor driver.

[0008] Here, the aforementioned calibration curve can be a pre-constructed curve stored in an electronic device, representing the fitting relationship between the peak value and the depth information deviation value caused by the pile-up phenomenon. Specifically, storing the pre-constructed calibration curve in the electronic device can involve storing the coefficients corresponding to the calibration curve in the electronic device. The signal peak value is the signal peak value of the histogram statistical result corresponding to the photon reception event acquired by the target sensor.

[0009] In this way, after acquiring depth information and histogram statistics, the corresponding depth information deviation value is determined based on the signal peak value and calibration curve of the histogram statistics. Then, the acquired depth information is corrected based on the depth information deviation value. Since the calibration curve is a relationship curve obtained by fitting the relationship between the depth information deviation caused by the pile-up phenomenon and the signal peak value of the histogram curve, the depth information obtained from the histogram statistics obtained by using the depth information deviation value determined based on the calibration curve can effectively correct the signal peak shift in the histogram statistics obtained by TDC statistics caused by the pile-up phenomenon, thereby improving the accuracy of the depth information calculation results and the ranging accuracy.

[0010] In one possible implementation, the acquired depth information is corrected according to a calibration curve to obtain corrected depth information, including:

[0011] The calibration curve corresponding to the pixel is determined based on the pixel in the receiver processor of the target sensor;

[0012] The signal peak value is determined based on the histogram statistics corresponding to the photon reception events of the pixel.

[0013] Based on the calibration curve, find the depth information deviation value corresponding to the signal peak value;

[0014] The corrected depth information is calculated based on the collected depth information and the depth information deviation value.

[0015] Here, after TDC obtains the histogram statistics corresponding to the counting results, the peak value corresponding to the histogram statistics can be determined. Based on the calibration curve, the depth information deviation value corresponding to the peak value is found. Then, the depth information calculated from the histogram statistics is corrected using the depth information deviation value, and the corrected depth information can be obtained.

[0016] In one possible implementation, the calibration curve is set according to the partitioning of the pixels of the target receiving sensor.

[0017] Here, each pixel partition is assigned a calibration curve corresponding to that partition, and the pixels in that partition can use this calibration curve for depth information correction.

[0018] In one possible implementation, the step of correcting the acquired depth information according to the calibration curve to obtain the corrected depth information includes: determining the partition of the pixel based on the pixels in the receiving processor of the target sensor; and determining the calibration curve based on the partition of the pixel.

[0019] Because there are unlimited differences between different pixels, the TDC (Tension Direction of Measurement) is inconsistent. If all pixels in the sensor use the same calibration curve for depth information correction, the measurement accuracy of different pixels after depth information correction will still have a certain error. Since some pixels in the same sensor have consistent depth information deviation values ​​at the same peak value, this embodiment can divide the pixels in the sensor into partitions based on the consistency of the depth information deviation values ​​of the pixels in the sensor, and then set a calibration curve corresponding to each partition. In this way, during the depth information correction process, the partition to which the pixel belongs can be determined first, and then the calibration curve corresponding to the partition can be obtained based on the partition to which the pixel belongs. The depth information can be corrected using the calibration curve corresponding to the partition. This can improve the measurement accuracy of depth information while reducing the time and resources consumed in the calibration process.

[0020] In one possible implementation, the calibration curve is obtained by fitting a curve relating the signal peak value to the depth information deviation value based on experimental data.

[0021] Here, the above experimental data can be obtained by fixing the TOF sensor used in the experiment and making the TOF sensor parallel to the macro resolution plate, and then moving the macro resolution plate to collect histogram data and depth information at different distances. Then, the peak value and depth information deviation value (TOF_bias) are extracted based on the histogram data and depth information collected at different distances to form the above experimental data.

[0022] In one possible implementation, the calibration curve is obtained by fitting a relationship curve between the signal peak value and the depth information deviation value based on experimental data, including: fitting a relationship curve between the signal peak value and the depth information deviation value based on a cubic curve.

[0023] A cubic curve is a planar algebraic curve that intersects a straight line at three points. Experiments have shown that using a cubic curve to fit the relationship between the peak value and the depth information deviation value yields the best results.

[0024] The expression for the calibration curve obtained above based on cubic curve fitting is:

[0025] TOF_bias = a * peak 3 +b*peak 2 +c*peak+d; (1)

[0026] By substituting the discrete points into the above formula (1), the corresponding coefficients a, b, c, and d can be solved, thereby obtaining the calibration curve applicable to the TOF sensor.

[0027] In one possible implementation, the calibration curve is obtained by fitting a relationship curve between the signal peak value and the depth information deviation value based on experimental data, including: for signal peak values ​​smaller than the segment boundary, fitting a relationship curve between the signal peak value and the depth information deviation value based on a cubic curve; for signal peak values ​​greater than or equal to the segment boundary, fitting a relationship curve between the signal peak value and the depth information deviation value based on a linear curve.

[0028] Here, since the more times the curve fitting is performed, the more complex the calculation becomes and the more computational resources are required. As the peak value increases, the process of fitting the relationship curve between the peak value and the depth information deviation value using cubic curves will also require more computational resources. In order to save computing power, segmentation points are set according to the experimental results corresponding to the computational resources required for the calculation. For peak values ​​smaller than the segmentation point, cubic curves are used to fit the relationship curve between the peak value and the depth information deviation value. For peak values ​​greater than or equal to the segmentation point (i.e., high peak values), linear curves are used to fit the relationship curve between the peak value and the depth information deviation value. The two curves are then merged to obtain the calibration curve mentioned above.

[0029] The expression for the calibration curve obtained by fitting cubic and linear curves is as follows:

[0030] TOF_bias1=a*peak 3 +b*peak 2+c*peak+d;TOF_bias2=e*peak+f (2)

[0031] By inputting discrete points with peak values ​​less than the segment boundary into the cubic curve in the first part above for solving, the corresponding coefficients a, b, c, and d can be obtained. By inputting discrete points with peak values ​​greater than or equal to the segment boundary into the linear curve in the second part above for solving, the corresponding coefficients e and f can be obtained, thereby determining all the coefficients corresponding to the calibration curve.

[0032] In one possible implementation, the calibration curve is obtained by fitting the relationship curve between the signal peak value and the depth information deviation value based on the experimental data, including: determining pixel partitions based on the consistency distribution of depth information deviation values ​​of different pixels under the same signal peak value; and fitting the relationship curve corresponding to each pixel partition.

[0033] Because of the wiring differences between different pixels, the consistency of TDC cannot be guaranteed. Therefore, the calibration curves fitted for different pixels will also differ. If all pixels in the entire sensor share the same curve, the measurement accuracy of different pixels after depth information correction will still be different. After testing multiple TOF sensor (module) sensors, it was found that the depth information deviation values ​​of some pixels in the same sensor showed good consistency. Therefore, pixels with relatively consistent depth information deviation values ​​were divided into a partition. The pixels in the sensor were divided into several partitions, and the same calibration curve was used for depth information correction of pixels in the same partition. That is, for each pixel partition, a relationship curve corresponding to the pixels in that partition was fitted separately, which can effectively improve the accuracy of depth information measurement.

[0034] In a second aspect, embodiments of this application provide an electronic device, one or more processors, and a memory coupled to the one or more processors. The memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors invoke the computer instructions to cause the electronic device to perform the method as described in any of the first aspects above.

[0035] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in any of the first aspects above.

[0036] Fourthly, embodiments of this application provide a chip system applied to an electronic device, the chip system including one or more processors, the one or more processors being configured to invoke computer instructions to cause the electronic device to perform the method as described in any of the first aspects.

[0037] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform any of the methods described in the first aspect.

[0038] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of a 3D sensor in an embodiment of this application;

[0040] Figure 2 This is a schematic diagram illustrating the working principle of the SPAD in the embodiments of this application;

[0041] Figure 3 A schematic diagram illustrating the statistical results of TDC for target objects with different reflectivities provided in the embodiments of this application;

[0042] Figure 4 A schematic diagram of the hardware structure of the electronic device 100 provided in the embodiments of this application;

[0043] Figure 5 A schematic diagram of the software architecture of the electronic device 100 provided in the embodiments of this application;

[0044] Figure 6 This is an interactive diagram of a software module provided in an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of the working state of a TOF camera provided in the embodiments of this application;

[0046] Figure 8 A schematic diagram of a calibration curve provided for an embodiment of this application;

[0047] Figure 9 A schematic diagram of another calibration curve provided in an embodiment of this application;

[0048] Figure 10 This is a schematic diagram of the curves corresponding to different calibration curves between different pixels provided in the embodiments of this application;

[0049] Figure 11The distribution of depth information deviation values ​​of different pixels in the same sensor at the same peak value is provided in the embodiments of this application.

[0050] Figure 12 This is a schematic diagram of the pixel partitioning of the sensor provided in the embodiments of this application;

[0051] Figure 13 This is a schematic diagram illustrating the implementation process of a depth information correction method provided in an embodiment of this application;

[0052] Figure 14 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0053] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0054] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0055] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0056] To more clearly illustrate the technical solution of this application, the concepts involved in this application are explained as follows:

[0057] 1. Time-of-flight (TOF) imaging technology refers to an imaging technique that uses a set of infrared light (or laser pulses) invisible to the human eye to be emitted outwards. Upon encountering an object, the light reflects back to a camera, and the time difference or phase difference between emission and reflection is calculated. This data is collected to form a set of distance and depth data, thus creating a 3D model. In other words, TOF imaging technology adds depth information from the Z-axis (also the XY axis of the world coordinate system) to the traditional 2D XY-axis imaging, ultimately generating a 3D image.

[0058] 2. Direct time-of-flight (DTOF) is a method of TOF that involves emitting a light pulse and then measuring the time interval between the emitted and received light pulses to obtain the time of flight. The distance to an object can then be calculated based on the time of flight to generate depth information.

[0059] 3. Three-dimensional (3D) sensors refer to sensors capable of acquiring depth information of a target object, such as Time-of-Flight (TOF) sensors and structured light sensors. These 3D sensors can be integrated into a camera to create a TOF camera (TOF camera module) capable of acquiring depth information of the target object.

[0060] 4. A TOF camera (TOF camera module) can include a transmitter module (TX) and a receiver module (RX). The TX is used to emit infrared light or laser pulses, and the RX is used to receive reflected light and form an image. Since the TX can autonomously emit light signals for imaging, TOF images are not affected by most ambient light. Therefore, applying TOF images to unlocking services can improve the security of facial recognition.

[0061] 5. A single photon avalanche diode (SPAD) is a photodiode that operates in Geiger mode and utilizes avalanche breakdown to achieve single-photon detection.

[0062] Geiger mode refers to the operating mode in which the reverse bias voltage is greater than the avalanche breakdown voltage of a single-photon avalanche diode.

[0063] 6. A time digital converter (TDC) is a device used to identify events and provide the time of event occurrence. It can count the number of times a photon reception event occurs and generate a statistical histogram of flight time based on the time of occurrence of the photon reception event. The horizontal axis of the statistical histogram is the flight time, and the vertical axis of the statistical histogram is the number of occurrences.

[0064] 7. The signal peak value refers to the maximum value in the histogram curve corresponding to the statistical histogram of the flight time mentioned above. The time on the horizontal axis corresponding to the signal peak value is recorded as the actual flight time.

[0065] To more clearly illustrate the technical solution of this application, the working principle of the 3D sensor in electronic devices is explained below:

[0066] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a 3D sensor. Figure 1 As shown, a 3D sensor may include a transmitter module and a receiver module. The adjustable light source in the transmitter module can emit multiple light pulses within one frame. After the emitted light pulses reach the target object, they are reflected by the target object and the reflected light pulses are received by the receiver processor (sensor) in the receiver module. The TDC then performs statistics on the flight times obtained from multiple counts and performs histogram statistics (to obtain a histogram curve). The flight time with the highest frequency (the horizontal axis time corresponding to the signal peak of the histogram curve) is used as the flight time for calculating the depth information of the target object (i.e., the target flight time).

[0067] It should be noted that the aforementioned adjustable light source can be a vertical cavity surface emitting laser (VCSEL) or an edge emitting laser (EEL), etc. The aforementioned sensor is composed of a pixel array, that is, the aforementioned sensor can include n*m ​​pixels, where n is the number of rows, which is a positive integer greater than or equal to 1, and m is the number of columns, which is also a positive integer greater than or equal to 1.

[0068] In the sensor, the photon reception event of each pixel is implemented through SPAD. Each time an avalanche occurs in SPAD, it corresponds to one photon reception event. At this time, TDC can count once. TDC will count the results and obtain a histogram in the time domain. The horizontal axis (time) corresponding to the peak value of the histogram curve can be further converted into the depth information of the target object.

[0069] like Figure 2 As shown, when the SPAD's operating voltage V avalanche Higher than the avalanche breakdown voltage (V BD When (V) avalanche =V BD +V excIn this context, Vexc refers to the voltage value exceeding the avalanche breakdown voltage. When the SPAD operates in Geiger mode, the avalanche effect is triggered when a photon arrives (i.e., the sensor receives a photon), generating a large current and thus a digital signal that can be captured by the TDC. The TDC records this as the arrival of a photon. However, the SPAD cannot quickly return to its initial state after the avalanche and requires a quenching circuit to recover the current generated after the avalanche. At this time, the voltage across the SPAD will recover the avalanche breakdown voltage, thereby suppressing the avalanche current and turning off the SPAD current. Then, the SPAD is powered on again to return to its initial state. The time required from the avalanche to the return to the initial state is called the dead time.

[0070] Because SPAD needs to wait for a dead time after receiving one photon before it can receive another, in situations where the distance is close, the reflectivity is high, or the ambient light is strong, a large number of photons may arrive in a very short time. Therefore, the photons that arrive first will suppress the photons that arrive later. In other words, the photons that arrive earlier have a higher probability of triggering SPAD counting, which will lead to a photon pile-up phenomenon. This will cause the signal peak to shift forward in the histogram curve corresponding to the histogram obtained by TDC statistics.

[0071] For example, please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the statistical results of TDC for target objects with different reflectivities. For example... Figure 3 As shown, Figure 3 In the diagram, L1 represents the histogram curve corresponding to the TDC statistical results of a standard reflectivity plate with a theoretical reflectivity of 90% (i.e., the histogram curve when there is no pile-up phenomenon), L2 represents the histogram curve corresponding to the TDC statistical results of a standard reflectivity plate with a reflectivity of 5%, and L3 represents the histogram curve corresponding to the TDC statistical results of a standard reflectivity plate with an actual reflectivity of 90%. It should be noted that the curves in the above examples were obtained by measurements taken at equal distances.

[0072] Depend on Figure 3 As can be seen from L1 and L3, for a standard reflectivity plate with a reflectivity of 90%, the signal peak (peak1) of the histogram curve corresponding to the statistical results actually obtained by TDC is shifted forward by t1-t2 compared to the signal peak (peak2) of the histogram curve corresponding to the theoretically 90% reflectivity plate. Here, t1 is the abscissa time corresponding to peak2, and t2 is the abscissa time corresponding to peak1.

[0073] The pile-up phenomenon causes the histogram curve corresponding to the TDC statistics to have a forward shift in the signal peak, which will lead to the shift in the peak finding position, resulting in errors in the obtained depth information and affecting the ranging accuracy.

[0074] To address the aforementioned issues, this application provides a depth information correction method. After acquiring the histogram statistical results, the method determines the corresponding depth information deviation value based on the signal peak value and calibration curve of the histogram statistical results. Then, it corrects the acquired depth information based on this depth information deviation value. Since the calibration curve is a relationship curve obtained by fitting the relationship between the depth information deviation caused by the pile-up phenomenon and the signal peak value of the histogram curve, using the depth information deviation value determined based on this calibration curve to correct the depth information obtained from the histogram statistical results can effectively correct the forward shift of the signal peak value in the histogram statistical results obtained from the TDC statistics caused by the pile-up phenomenon. This improves the accuracy of the depth information calculation results and enhances the ranging accuracy.

[0075] The solutions provided in this application can be applied to any electronic device with a 3D sensor. These electronic devices can also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in transportation safety, wireless terminals in smart cities, and so on. The embodiments of this application do not limit the specific technologies or device forms used in the electronic devices.

[0076] To better understand the embodiments of this application, the structure of the electronic device of this application is described below:

[0077] Figure 4A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0078] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0079] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0080] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0081] In some embodiments, the processor 110 is implemented through a system on chip (SOC), that is, the processor 110 mentioned in some embodiments of this application may specifically refer to an SOC chip.

[0082] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0083] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0084] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods as described in the above embodiments.

[0085] In some embodiments, antenna 1 of the electronic device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the electronic device to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), Wireless Local Area Network (WLAN), Near Field Communication (NFC), and / or infrared (IR) technology, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0086] Electronic device 100 implements audio functions through a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, and an application processor. Examples include music playback, audio capture, and recording.

[0087] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0088] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0089] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, the electronic device detects the intensity of the touch operation based on pressure sensor 180A. The electronic device can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0090] In some embodiments, the 3D sensor is disposed below the display screen 194 (below refers to the side closer to the non-user input operation side).

[0091] Electronic device 100 can perform shooting functions through 3D sensor 10, ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0092] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0093] Camera 193 is used to capture still images or videos. An object passes through the lens, generating an optical image that is projected onto a photosensitive element. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Image Signal Processor) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for further processing. The DSP converts the digital image signal into standard RGB, YUV, or other image formats.

[0094] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0095] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0096] It is understood that the aforementioned electronic device 100 may include one or more cameras 193. When the electronic device 100 captures a target object through the camera 193, it can acquire the depth information of the target object through the 3D sensor 10. In some embodiments, some cameras 193 in the aforementioned electronic device 100 have the functions of the aforementioned 3D sensor 10, that is, the camera 193 is a depth sensing camera 193. For example, the electronic device 100 may be equipped with a TOF depth sensing camera.

[0097] The electronic device 100 having the above-described hardware structure can employ a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., in its software system. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 100.

[0098] Figure 5 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application.

[0099] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system layer, and the kernel layer.

[0100] The application layer can include a series of application packages. For example... Figure 4 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, video, instant messaging, reading, and settings.

[0101] The settings application includes a face registration function, which can be used for face unlock. The lock screen application has the function of unlocking in response to the user's unlocking operation. Specifically, the lock screen application can perform unlocking processes such as face unlock, fingerprint unlock, and password unlock. This application embodiment mainly uses face unlock as an example for explanation.

[0102] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes predefined functions, such as an activity manager, window manager, content provider, view system, resource manager, notification manager, camera service, and face recognition service.

[0103] The system layer includes system libraries, Android runtime, core libraries, and media libraries.

[0104] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0105] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0106] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0107] In some embodiments, the software architecture of the electronic device 100 may further include a Hardware Abstraction Layer (HAL). The HAL layer may include multiple modules and acts as an intermediary, separating the hardware and software. The primary purpose of this layer is to allow the software to run on hardware on which it is not intended to run. In one embodiment, the multiple modules in the HAL layer include, but are not limited to, at least one of the following modules: Bluetooth HAL, camera HAL, sensor HAL, face recognition control module, etc.

[0108] The camera HAL is the core software framework of the camera system. The camera HAL can include sensor nodes and image front end (IFE) nodes. Sensor nodes and IFE nodes are components (nodes) in the image data and control command transmission path (also called the transmission pipeline) created by the camera HAL.

[0109] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0110] The camera driver is the driver layer of the camera device, mainly responsible for interacting with the hardware.

[0111] After introducing the execution subject involved in the embodiments of this application, the possible application scenarios to which the embodiments of this application are applicable will be briefly introduced. Here, we take a mobile phone as an example for explanation.

[0112] This application's embodiments are applicable to any scenario requiring depth information, such as 3D face unlock, background blurring captured by a TOF camera, and augmented reality (AR) gaming and entertainment scenarios. Here, a TOF camera is specifically used as the 3D sensor.

[0113] The following example illustrates the 3D face unlock scenario:

[0114] like Figure 6As shown, the lock screen application in the application layer can interact with the face recognition application. The face recognition application can interact with the face recognition service in the framework layer by calling a preset application programming interface (API). The face recognition service can interact with the face recognition control module in the HAL layer. The face recognition control module can interact with the camera HAL in the HAL layer through the camera service in the framework layer, or the face recognition control module can interact directly with the camera HAL in the HAL layer. The camera HAL can include sensor nodes. Sensor nodes can interact with the camera driver in the kernel layer. The camera driver can be used to drive the TOF camera in the hardware layer to acquire image data. The image data acquired by the TOF camera can be stored in memory, and the storage location is represented by a file descriptor (FD). The memory can send the FD to the camera driver. The camera driver can pass the FD to the IFE node of the camera HAL. The IFE node can pass the FD to the camera service. The camera service can pass the FD to the face recognition control module. The face recognition control module can pass the FD to the face recognition module. The frame management module in the face recognition module can read image data (RAW Data1) from the memory according to the FD and send RAW Data1 to the TOF algorithm module. The TOF algorithm module can convert RAW Data1 into a grayscale image, determine the depth information, and perform depth information correction according to the calibration curve to obtain a corrected depth image. Then, the grayscale image and the corrected depth image are sent to the face ID algorithm module. The face ID algorithm module can be used for grayscale image matching and depth image anti-spoofing to obtain the face recognition result. The face ID algorithm module can send the face recognition result to the face recognition control module through the frame management module. The face recognition control module can feed back the processing result to the lock screen application through the face recognition service and face recognition application, so that the lock screen application can determine whether to unlock (if face recognition is successful, unlock; if face recognition fails, do not unlock, i.e., unlocking fails). Figure 6 Solid arrows can be used to represent control flow, while dashed arrows can be used to represent data flow.

[0115] For example, please refer to Figure 7 , Figure 7This is a schematic diagram illustrating the working state of a TOF camera provided in some embodiments of this application. The TOF camera 70 is disposed in the electronic device 100, located on the side of the display screen 194 opposite to its display surface. The TOF camera 70 can be fixed to the main circuit board 20 and is positioned between the main circuit board 20 and the display screen 194. The TOF camera 70 includes a light source 70a. The display screen 194 may have a light-transmitting portion (not shown in the figure), through which infrared light projected by the TOF camera 70 onto an object can pass. The light-transmitting portion can also allow infrared light reflected from the object to pass through, so that the infrared light is received by the sensor of the TOF camera 70.

[0116] The light source 70a in the aforementioned TOF camera 70 can be a vertical cavity surface emitting laser (VCSEL) or an edge emitting laser (EEL) or other similar devices.

[0117] In some embodiments, the light-transmitting portion may be a light-transmitting hole on the display screen of an electronic device. Of course, it is understood that in other embodiments, the light-transmitting portion may be divided into a first light-transmitting portion and a second light-transmitting portion spaced apart, wherein the first light-transmitting portion is used to transmit infrared light emitted by the light source 70a, and the second light-transmitting portion is used to transmit infrared light reflected by an object.

[0118] The TOF camera 70 controls the light source 70a to emit infrared light. This infrared light is required for the TOF function. For example, the light source 70a in the TOF camera 70 can be an LED or laser tube (such as a laser diode or VCSEL) to emit high-frequency modulated pulsed light. The frequency can reach approximately 1000MHz, and the wavelength is around 800nm-1000nm. Furthermore, the infrared light required for TOF also has high peak power. The infrared light reflected after reaching the target object can be received by the sensor of the TOF camera 70. The TOF camera 70 outputs a histogram curve corresponding to the received reflected light pulses, thereby obtaining the round-trip time of the infrared light reflected from different positions on the target object. This allows the determination of the distance between the phone and different positions on the target object, thus generating depth information. The phone can then correct this depth information based on the peak value of the histogram curve and the calibration curve corresponding to each pixel in the sensor. This corrects depth information errors caused by pile-up phenomena. Finally, based on the corrected depth information, the corresponding image information is determined, assisting the phone in achieving 3D face unlock.

[0119] This requires pre-constructing the aforementioned calibration curve.

[0120] Generally, the pile-up phenomenon is positively correlated with the light intensity of the received light pulse and the photon detection efficiency. The pile-up phenomenon is directly reflected in the signal peak value of the histogram curve; that is, the stronger the light intensity of the received light pulse, the more severe the pile-up phenomenon, and the higher the peak value of the histogram curve. The magnitude of the peak value is related to the depth information deviation caused by the pile-up phenomenon. Therefore, a curve showing the relationship between the peak value and the depth information deviation caused by the pile-up phenomenon (i.e., the calibration curve mentioned above) can be fitted experimentally.

[0121] The following explains the process of fitting the relationship curve between the peak value and the depth information deviation value caused by the pile-up phenomenon:

[0122] 1. Fix the TOF sensor used in the experiment and make the TOF sensor parallel to the micro resolution board (also known as the chart board).

[0123] It should be noted that a chart is a standard plate for detecting the geometric resolution of short focal length photographic lenses. In this embodiment, since the peak value in the histogram curve is related to the reflectivity and distance of the target object, considering the limitations of calibration effect and calibration environment, if a chart with low reflectivity is selected, it may not be able to cover the depth information error data under high peak values. If a chart with low reflectivity is selected, it is necessary to move a greater distance and increase the size of the chart to cover the depth information error data under low peak values. Therefore, when selecting a chart, a chart with a relatively moderate reflectivity can be selected (for example, a chart with a reflectivity of 40%-60%).

[0124] 2. Collect histogram data and depth information at different distances.

[0125] In practical applications, the chart board is moved to different positions, and then the TOF sensor is controlled to collect histogram data and depth information at different distances.

[0126] For details on the histogram data and depth information collected by the TOF sensor at different distances, please refer to the explanation of the working principle of the 3D sensor, which will not be repeated here.

[0127] 3. Extract peak values ​​and depth information bias values ​​(TOF_bias) based on the histogram data and depth information collected at different distances.

[0128] In practical applications, the aforementioned depth information deviation value (TOF_bias) can be determined based on the actual distance and the depth information collected by the TOF sensor, i.e., TOF_bias = GT - TOF_mea; where TOF_bias is the depth information deviation value, GT is the actual distance, and TOF_mea is the depth information collected by the TOF sensor.

[0129] 4. Based on cubic curve fitting, the relationship curve between the peak value and the depth information deviation value is obtained, thus obtaining the above calibration curve.

[0130] In practical applications, a cubic curve is a planar algebraic curve that intersects a straight line at three points. Experiments have shown that using a cubic curve to fit the relationship between the peak value and the depth information deviation value yields the best fitting results.

[0131] In practical applications, the peak value can be normalized first, and then the normalized peak value can be fitted with the depth information deviation value.

[0132] For example, please refer to Figure 8 , Figure 8 This is a schematic diagram of the calibration curve in this application. Figure 8 As shown, the calibration curve obtained by fitting can cover the discrete points well.

[0133] The expression for the calibration curve obtained above based on cubic curve fitting is:

[0134] TOF_bias = a * peak 3 +b*peak 2 +c*peak+d; (1)

[0135] By substituting the discrete points into the above formula (1), the corresponding coefficients a, b, c, and d can be solved, thereby obtaining the calibration curve applicable to the TOF sensor.

[0136] As the number of fitting operations increases during curve fitting, the calculation becomes more complex and requires more computational resources. As the peak value increases, the process of fitting the relationship curve between the peak value and the depth information deviation value using a cubic curve also requires more computational resources. In order to save computing power, this application provides another curve fitting method to fit the relationship curve between the peak value and the depth information deviation value.

[0137] Based on the experimental results corresponding to the computational resources required for the calculation, segmentation boundaries are set. For peak values ​​smaller than the segmentation boundaries, a cubic curve is used to fit the relationship between the peak value and the depth information deviation value. For peak values ​​greater than or equal to the segmentation boundaries (i.e., high peak values), a linear curve is used to fit the relationship between the peak value and the depth information deviation value. The two curves are then merged to obtain the calibration curve mentioned above.

[0138] For example, please refer to Figure 9 , Figure 9 A schematic diagram of another calibration curve in this application is shown, such as... Figure 9 As shown, the calibration curve includes a first part obtained by cubic curve fitting and a second part obtained by linear curve fitting.

[0139] The expression for the calibration curve obtained by fitting cubic and linear curves is as follows:

[0140] TOF_bias1=a*peak 3 +b*peak 2 +c*peak+d;TOF_bias2=e*peak+f (2)

[0141] By inputting discrete points with peak values ​​less than the segment boundary into the cubic curve in the first part above for solving, the corresponding coefficients a, b, c, and d can be obtained. By inputting discrete points with peak values ​​greater than or equal to the segment boundary into the linear curve in the second part above for solving, the corresponding coefficients e and f can be obtained, thereby determining all the coefficients corresponding to the calibration curve.

[0142] Because of wiring differences between different pixels, the consistency of TDC cannot be guaranteed, and therefore the calibration curves fitted for different pixels will also differ. For an example, please refer to... Figure 10 , Figure 10 The diagram shows the corresponding calibration curves for different pixels. Calibration curve 1 is the calibration curve for the first pixel, calibration curve 2 is the calibration curve for the second pixel, and calibration curve 3 is the calibration curve for the third pixel.

[0143] If all pixels in the entire sensor share the same curve, the measurement accuracy of different pixels will still differ after depth information correction. Please refer to [link / reference]. Figure 11 , Figure 11 This shows the distribution of depth information deviation values ​​among different pixels in the same sensor at the same peak value. Figure 11 It can be seen that the depth information deviation values ​​of some pixels within the same sensor exhibit good consistency. From... Figure 11It can be clearly seen that pixels with good consistency in the same sensor are distributed in an odd-even pattern horizontally and in a symmetrical pattern vertically.

[0144] It should be noted that the above-mentioned horizontal distribution of odd and even numbers means that the consistency of pixels in odd-numbered columns is better, and the consistency of pixels in even-numbered columns is also better. The above-mentioned vertical distribution of symmetry means that the consistency of pixels in the first row is better than that in the last row, the consistency of pixels in the second row is better than that in the second-to-last row, and so on.

[0145] After testing multiple TOF sensor (module) sensors, all of them were able to obtain the following results: Figure 11 Given the consistent distribution shown, this embodiment divides pixels with relatively consistent depth information deviation values ​​into one partition, and divides the pixels in the sensor into several partitions. Pixels in the same partition use the same calibration curve for depth information correction. That is, for each pixel partition, a relationship curve corresponding to the pixels in that partition is fitted.

[0146] Specifically, the following operations can be performed for each partition: obtain experimental data for any pixel in the partition, and then fit the corresponding relationship curve based on the experimental data (it can be the relationship curve obtained by fitting the cubic curve above, or it can be the relationship curve obtained by fitting the cubic curve and the linear curve together). That is, the relationship curve is the calibration curve corresponding to the partition.

[0147] For example, such as Figure 12 As shown, the sensor described above can include 1200 pixels (30 rows * 40 columns), according to... Figure 11The consistent distribution shown divides these 1200 pixels into 30 partitions (pixels with the same color and shape belong to the same partition). Specifically, the pixels corresponding to the first row of the first column, the first row of the third column, the first row of the fifth column, ..., the first row of the thirty-ninth column, the last row of the first column, the last row of the third column, the last row of the fifth column, ..., the last row of the thirty-ninth column all belong to the same partition and are corrected for depth information using the same calibration curve. Similarly, the pixels corresponding to the second row of the first column, the second row of the third column, the second row of the fifth column, ..., the second row of the thirty-ninth column, the second row of the first column (second to last), the second row of the third column (second to last), the second row of the fifth column (second to last), ..., the second row of the thirty-ninth column (second to last) all belong to the same partition and are corrected for depth information using the same calibration curve. The pixels corresponding to the first row of the second column, the first row of the fourth column, the first row of the sixth column, ... the first row of the fortieth column, the last row of the second column, the last row of the fourth column, the last row of the sixth column, ... the last row of the sixth column are all in the same partition and use the same calibration curve for depth information correction.

[0148] Therefore, for the aforementioned TOF sensor, only 30 calibration curves need to be fitted, and the same calibration curve corresponding to that partition is used for depth information correction of each pixel.

[0149] The calibration curve can be obtained using the cubic curve fitting method corresponding to formula (1), or it can be obtained using the cubic and linear curve fitting method corresponding to formula (2). When using the cubic curve fitting method corresponding to formula (1), four coefficients a, b, c, and d need to be determined. That is, when performing calibration curve fitting, these four coefficients of 30 calibration curves need to be determined (i.e., a total of 120 coefficients). When using the cubic and linear curve fitting method corresponding to formula (2), six coefficients a, b, c, d, e, and f need to be determined. That is, when performing calibration curve fitting, six coefficients of 30 calibration curves need to be determined (i.e., a total of 180 coefficients).

[0150] After obtaining the calibration curves for each partition of the sensor, the calibration curves are pre-stored in the electronic device. This way, when the electronic device uses the TOF sensor to acquire depth information, it can call the calibration curves stored in the electronic device to perform the corresponding depth information correction.

[0151] It should be noted that "at the time of..." in the embodiments of this application can be either at the instant when a certain situation occurs, or for a period of time after the occurrence of a certain situation. The embodiments of this application do not make specific limitations on this.

[0152] The following describes the depth information correction method provided in this application embodiment using specific implementation process examples. Figure 13 This is a schematic diagram illustrating the implementation process of a depth information correction method provided in an embodiment of this application. Figure 13 As shown, the method may include:

[0153] S1301, In response to the depth information acquisition trigger event, the target sensor is invoked to acquire the depth information of the target object.

[0154] In specific applications, the aforementioned target sensor is a sensor capable of acquiring depth information of the target object, such as the aforementioned 3D sensor, TOF sensor, and TOF camera.

[0155] The above depth information collection trigger event can be triggered in the following scenarios:

[0156] 1. The 3D face unlock function is activated when the screen is locked.

[0157] In practical applications, when a user wants to use the 3D face unlock function to unlock their phone, the aforementioned depth information acquisition trigger event can be triggered when the user's first action is detected. This first action can be an unlocking operation; that is, when the phone is in a locked state and the user's first action is detected, the depth information acquisition trigger event can be triggered, thereby inducing the phone to execute the operation of calling the TOF camera to acquire depth information. This first action includes actions such as picking up the phone, pressing the power button, operating on the screen (tap, swipe, etc.), or unplugging the charging cable.

[0158] 2. Users should select the background blur shooting mode when taking photos.

[0159] In practical applications, when a user takes a picture of a target object, they can choose a background blur shooting mode. In response to the user's click to select the background blur shooting mode (which triggers the aforementioned depth information acquisition event), the mobile phone can call the TOF camera to collect the depth information of the target object in order to identify the foreground and background, thereby achieving background blur.

[0160] 3. Users play AR games using their mobile phones.

[0161] In practical applications, when users play AR games on their mobile phones, they can choose to build a scene. In response to the user clicking the build button for scene building (which triggers the aforementioned depth information acquisition event), the mobile phone can also call the TOF camera to obtain spatial images and spatial depth information, and perform scene rendering and other operations based on this to build the corresponding virtual scene.

[0162] It is understandable that the above situations are just examples of events that trigger depth information acquisition. These events can also be triggered in other situations, such as when using smart eye-tracking services to call a TOF camera (or 3D sensor) to collect image information, or when a payment application calls a TOF camera for facial recognition during payment or login.

[0163] In this embodiment of the application, the operation of calling the target sensor to collect depth information can be as follows: when the target sensor is a TOF camera, the mobile phone can drive the TOF camera to perform the action of collecting depth information through the camera driver; when the target sensor is a 3D sensor, the mobile phone can drive the 3D sensor to perform the action of collecting depth information through the sensor driver.

[0164] Taking a TOF camera as an example, when the camera driver triggers the TOF camera to collect depth information, the aforementioned TOF camera can emit infrared light through the light source in the transmitting module. After the infrared light reaches the target object, it is reflected. The reflected infrared light can be received by the sensor in the receiving module of the TOF camera. Specifically, the SPAD in each pixel of the sensor can record photon reception events. That is, each avalanche of the SPAD corresponds to one photon reception event, and the TDC can correspond to one count. The TDC can perform histogram statistics on the flight time obtained from multiple counts to obtain a histogram curve (i.e., histogram statistics results).

[0165] S1302. Correct the collected depth information according to the calibration curve to obtain the corrected depth information.

[0166] In this embodiment of the application, the calibration curve is the pre-constructed curve stored in the mobile phone, which represents the fitting relationship between the peak value and the depth information deviation value caused by the pile-up phenomenon.

[0167] After obtaining the histogram statistics corresponding to the counting results in TDC, the peak value corresponding to the histogram statistics can be determined. Based on the calibration curve, the depth information deviation value corresponding to the peak value is found. Then, the depth information calculated from the histogram statistics is corrected using the depth information deviation value, and the corrected depth information can be obtained.

[0168] The above histogram statistics can be represented by a histogram curve.

[0169] The specific process of calculating the depth information obtained from the above histogram statistical results can be to determine the horizontal axis time corresponding to the peak value in the histogram statistical results, and then calculate the depth information based on the horizontal axis time corresponding to the peak value.

[0170] The specific calculation method is as follows: TOF_mea = cΔt / 2; where TOF_mea is the depth information collected by the 3D sensor, c is the speed of light, and Δt is the horizontal coordinate time corresponding to the peak value.

[0171] The specific calculation method for correcting the depth information obtained from the histogram statistics based on the depth information deviation value can be as follows: TOF_mea' = TOF_mea + TOF_bias; where TOF_mea' is the corrected depth information, TOF_mea is the depth information collected by the 3D sensor, and TOF_bias is the depth information deviation value determined based on the calibration curve and peak value.

[0172] In some embodiments, the calibration curve described above can be a single calibration curve, meaning that each pixel in the sensor of the TOF camera uses the same calibration curve for depth information correction.

[0173] In other embodiments, the calibration curve can also be a calibration curve set for each pixel. That is, each pixel has a corresponding calibration curve, and the depth information determined by the histogram statistics of each pixel is corrected using the calibration curve corresponding to that pixel. Because there are unlimited differences between different pixels, the TDC is inconsistent. If all pixels in the sensor use the same calibration curve for depth information correction, the measurement accuracy of different pixels after depth information correction will still have a certain error. In order to further improve the measurement accuracy of depth information, this embodiment can set a corresponding calibration curve for each pixel. In this way, when performing depth information correction, the calibration curve corresponding to the pixel can be used to determine the depth information deviation value, which can effectively improve the measurement accuracy of depth information.

[0174] In other embodiments, the calibration curve can be a calibration curve set according to the partition of a pixel. That is, a calibration curve is set for each partition of a pixel, and the pixels in that partition can use the calibration curve for depth information correction. Setting a corresponding calibration curve for each pixel would consume a lot of computing resources. Since the depth information deviation values ​​of some pixels in the same sensor are consistent at the same peak value, the embodiments of this application can partition the pixels in the sensor according to the consistency of the depth information deviation values ​​of the pixels in the sensor, and then set a calibration curve corresponding to each partition. In this way, during the depth information correction process, the partition to which the pixel belongs can be determined first, and then the calibration curve corresponding to the partition can be obtained according to the partition to which the pixel belongs, and the depth information correction can be performed using the calibration curve corresponding to the partition. This can improve the measurement accuracy of depth information while reducing the time and resources consumed in the calibration process.

[0175] In some embodiments of this application, a partition calibration curve is constructed for different pixels. The calibration curve corresponding to the partition of the pixel can be obtained according to the pixel. Then, the depth information deviation value is determined based on the peak value determined by the calibration curve corresponding to the pixel and the histogram statistical results corresponding to the pixel.

[0176] As can be seen from the above, the depth information correction method provided in this application, after obtaining the histogram statistical results, determines the corresponding depth information deviation value based on the signal peak value and calibration curve of the histogram statistical results, and then corrects the collected depth information based on the depth information deviation value. Since the calibration curve is a relationship curve obtained by fitting the relationship between the depth information deviation caused by the pile-up phenomenon and the signal peak value of the histogram curve, the depth information obtained by correcting the depth information obtained by the histogram statistical results based on the depth information deviation value determined by the calibration curve can effectively correct the situation where the signal peak value of the histogram statistical results obtained by TDC statistics shifts forward due to the pile-up phenomenon, thereby improving the accuracy of the depth information calculation results and improving the ranging accuracy.

[0177] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the method steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] This application embodiment can divide the audio channel switching method apparatus into functional modules based on the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0179] Figure 14 This is a schematic diagram of a chip structure provided in an embodiment of this application. Figure 14 As shown, chip 140 includes one or more processors 1401, communication lines 1402, communication interfaces 1403, and memory 1404.

[0180] In some implementations, memory 1404 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0181] The methods described in the embodiments of this application can be applied to, or implemented by, processor 1401. Processor 1401 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the method executed by the first or second device can be completed by integrated logic circuits in the hardware of processor 1401 or by instructions in software form. Processor 1401 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 1401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0182] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 1404, and processor 1401 reads information from memory 1404 and, in conjunction with its hardware, completes the steps of the above method.

[0183] The processor 1401, memory 1404 and communication interface 1403 can communicate with each other via communication line 1402.

[0184] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0185] This application also provides a computer program product comprising one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0186] This application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium capable of transferring a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0187] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may also include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, digital versatile discs (DVD), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers.

[0188] The above combinations should also be included within the scope of computer-readable media. The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A depth information correction method characterized by, The method comprises: in response to a collection trigger event of depth information, calling a target sensor to collect depth information of a target object; determining a partition of a pixel in a receiving processor of the target sensor according to a consistency distribution of a depth information deviation value of the pixel at a same signal peak value, fitting a calibration curve corresponding to the partition for each partition of the pixel, and correcting the collected depth information according to the calibration curve to obtain corrected depth information; wherein the calibration curve is a fitting relationship curve between a signal peak value and a depth information deviation value, the signal peak value is a signal peak value of a histogram statistical result corresponding to a photon receiving event obtained by the target sensor, and the pixels in a same partition use a same calibration curve for depth information correction.

2. The depth information correction method according to claim 1, characterized by, The method further comprises: determining a calibration curve corresponding to the pixel according to the pixel in the receiving processor of the target sensor; determining a signal peak value according to a histogram statistical result corresponding to a photon receiving event of the pixel; finding a depth information deviation value corresponding to the signal peak value based on the calibration curve; calculating corrected depth information according to the collected depth information and the depth information deviation value.

3. The depth information correction method according to claim 1 or 2, characterized by, The method further comprises: fitting a relationship curve between a signal peak value and a depth information deviation value according to experimental data to obtain the calibration curve.

4. The depth information correction method according to claim 3, characterized by, The method further comprises: fitting the relationship curve between the signal peak value and the depth information deviation value based on a cubic curve.

5. The depth information correction method according to claim 3, characterized by, The method further comprises: for a signal peak value less than a segmentation boundary point, fitting the relationship curve between the signal peak value and the depth information deviation value based on a cubic curve; for a signal peak value greater than or equal to the segmentation boundary point, fitting the relationship curve between the signal peak value and the depth information deviation value based on a linear curve.

6. An electronic device, comprising: The electronic device comprises a processor and a memory, the processor and the memory are coupled, the memory is used to store a computer program, and when the processor executes the computer program, the electronic device executes the steps of the method in any one of claims 1 to 5.

7. A computer readable storage medium characterized in that, The computer readable storage medium comprises a computer program, and when the computer program is executed on the electronic device, the electronic device executes the steps of the method in any one of claims 1 to 5.

8. A chip system, characterized by The chip system is applied to an electronic device, the chip system comprises one or more processors, the processor and the memory are coupled, the memory is used to store computer program instructions, and the one or more processors are used to call the computer instructions to make the electronic device execute the steps of the method in any one of claims 1 to 5.

Images