Depth information correction method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a depth information correction method, electronic equipment and a storage medium, and the method comprises the steps: calling a target sensor to collect the depth information of a target object in response to a collection trigger event of the depth information; correcting the collected depth information according to the calibration curve to obtain corrected depth information; wherein the calibration curve is a fitting relation curve between the signal peak value and the depth information deviation value. The depth information deviation value determined based on the calibration curve is used for correcting the depth information obtained by the histogram statistical result, so that the condition that the signal peak value of the histogram statistical result obtained by TDC statistics is forward due to the pile-up phenomenon can be effectively corrected, and the accuracy of the depth information calculation result is improved; the ranging precision is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a depth information correction method, an electronic device and a storage medium. Background Art

[0002] Three-dimensional (3D) sensing has been increasingly used in various applications of electronic devices, such as 3D face unlocking, camera focusing, depth image acquisition, surrounding environment detection, virtual object rendering, etc. At present, depth information is usually obtained based on direct time-of-flight (DTOF) measurement. The principle of DTOF ranging method is to emit light pulses to the object to be measured based on a photodetector, and directly calculate the distance of the target object by measuring the time interval between the emitted light pulse and the reflected light pulse to generate depth information.

[0003] At present, when measuring the distance of a target object, a pulse light beam is usually emitted several times in a natural environment and a reflected light beam corresponding to the emitted pulse light beam is received. Then, the time interval between the emitted pulse and the received pulse is recorded based on a time digital converter (TDC), and the flight time with the highest frequency is taken as the flight time for calculating the depth information of the target object. However, the single photon avalanche diode (SPAD) in the time digital converter has a dead time, which will cause the counting peak to shift, thereby affecting the accuracy of the depth information measurement. Summary of the invention

[0004] The embodiments of the present application provide a depth information correction method, an electronic device, and a storage medium, which can improve the accuracy of depth information measurement.

[0005] In a first aspect, an embodiment of the present application provides a depth information correction method, including: in response to a depth information collection trigger event, calling a target sensor to collect depth information of a target object; correcting the collected depth information according to a calibration curve to obtain corrected depth information.

[0006] Here, the collection trigger event of the above-mentioned depth information can be triggered in any of the following scenarios: the 3D face unlocking function in the lock screen state is activated, the user selects the background blur shooting mode when shooting, the user uses the electronic device to play AR games, the TOF camera (or 3D sensor) is called to collect image information when using the smart eye movement service, the payment application calls the TOF camera for face recognition during the payment or login process, etc.

[0007] Here, the operation of calling the target sensor to collect depth information can specifically be: when the target sensor is a TOF camera, the mobile phone can drive the TOF camera through the camera driver to perform the action of collecting depth information; when the target sensor is a 3D sensor, the mobile phone can drive the 3D sensor through the sensor driver to perform the action of collecting depth information.

[0008] Here, the calibration curve may be pre-constructed and stored in an electronic device, and is used to represent a fitting relationship curve between a peak value and a depth information deviation value caused by a pile-up phenomenon. Storing the pre-constructed calibration curve in an electronic device may specifically be storing the coefficients corresponding to the calibration curve in the electronic device. The signal peak is the signal peak of the histogram statistics corresponding to the photon reception event acquired by the target sensor.

[0009] In this way, after obtaining the depth information, after obtaining the histogram statistical results, the corresponding depth information deviation value is determined according to the signal peak value of the histogram statistical results and the calibration curve, and then the collected depth information is corrected based on the depth information deviation value. Since the calibration curve is a relationship curve obtained by fitting the relationship between the depth information deviation caused by the pile-up phenomenon and the signal peak value of the histogram curve, the depth information obtained by the histogram statistical results is corrected using the depth information deviation value determined based on the calibration curve. This can effectively correct the signal peak forward shift of the histogram statistical results obtained by TDC statistics due to the pile-up phenomenon, thereby improving the accuracy of the depth information calculation results and improving the ranging accuracy.

[0010] In a possible implementation, correcting the collected depth information according to the calibration curve to obtain the corrected depth information includes:

[0011] Determining a calibration curve corresponding to a pixel in a receiving processor of the target sensor according to the pixel;

[0012] Determine a signal peak value according to a histogram statistical result corresponding to the photon reception event of the pixel;

[0013] Finding a depth information deviation value corresponding to the signal peak value based on the calibration curve;

[0014] The corrected depth information is calculated based on the collected depth information and the depth information deviation value.

[0015] Here, after TDC obtains the histogram statistical result corresponding to the counting result, the peak value corresponding to the histogram statistical result can be determined, and the depth information deviation value corresponding to the peak value can be found based on the calibration curve. Then, the depth information calculated by the histogram statistical result is corrected using the depth information deviation value, and the corrected depth information can be obtained.

[0016] In a possible implementation manner, the calibration curve is set corresponding to the partitions of the pixels of the target receiving sensor.

[0017] Here, a calibration curve corresponding to each pixel partition is set for each pixel partition, and the pixels in the partition can use the calibration curve to perform depth information correction.

[0018] In a possible implementation, the correcting the collected depth information according to the calibration curve to obtain the corrected depth information includes: determining the partitions of the pixels according to the pixels in the receiving processor of the target sensor; and determining the calibration curve according to the partitions of the pixels.

[0019] Because there are unlimited differences between different pixels, there is inconsistency in TDC. If all pixels in the sensor use the same calibration curve to correct the depth information, there will still be certain errors in the measurement accuracy of different pixels after the depth information correction. Since the depth information deviation values ​​of some pixels in the same sensor are consistent at the same peak value, the embodiment of the present application can partition the pixels in the sensor according to the consistency of the depth information deviation values ​​of the pixels in the sensor, and then set a calibration curve corresponding to the partition for each partition. In this way, in the process of depth information correction, the partition to which the pixel belongs can be determined first, and then the calibration curve corresponding to the partition can be obtained according to the partition to which the pixel belongs, and the calibration curve corresponding to the partition can be used to correct the depth information. This can improve the measurement accuracy of the depth information while reducing the time and resources consumed in the calibration process.

[0020] In a possible implementation manner, the method further includes: fitting a relationship curve between a signal peak value and a depth information deviation value according to experimental data to obtain the calibration curve.

[0021] Here, the above experimental data can be obtained by fixing the TOF sensor used in the experiment and making the TOF sensor parallel to the macro resolution board, and then moving the macro resolution board to collect histogram data and depth information at different distances, and then extracting the peak value and depth information deviation value (TOF_bias) based on the collected histogram data and depth information at different distances to form the above experimental data.

[0022] In a possible implementation, fitting a relationship curve between a signal peak value and a depth information deviation value according to experimental data to obtain the calibration curve includes: fitting the relationship curve between the signal peak value and the depth information deviation value based on a cubic curve.

[0023] The cubic curve is a plane algebraic curve, which has three intersections with a general straight line. After testing, it is found that the cubic curve has the best fitting effect for fitting the relationship curve between the peak value and the depth information deviation value.

[0024] The expression of the calibration curve obtained based on cubic curve fitting is:

[0025] TOF_bias=a*peak 3 +b*peak 2 +c*peak+d; (1)

[0026] Substituting the discrete points into the above formula (1), the corresponding coefficients a, b, c, and d can be solved, thereby obtaining a calibration curve suitable for the TOF sensor.

[0027] In a possible implementation, the calibration curve is obtained by fitting a relationship curve between the signal peak value and the depth information deviation value according to experimental data, including: for a signal peak value less than the segmentation boundary point, fitting the relationship curve between the signal peak value and the depth information deviation value based on a cubic curve; for a signal peak value greater than or equal to the segmentation boundary point, fitting the relationship curve between the signal peak value and the depth information deviation value based on a linear curve.

[0028] Here, in the process of curve fitting, the more times of fitting, the more complicated the calculation, and the more computing resources will be needed. As the peak value increases, the process of fitting the relationship curve between the peak value and the depth information deviation value with the cubic curve will also require more computing resources. In order to save computing power, the segmentation boundary point is set according to the experimental results corresponding to the computing resources required for the calculation. For peak values ​​less than the segmentation boundary point, the cubic curve is used to fit the relationship curve between the peak value and the depth information deviation value. For peak values ​​greater than or equal to the segmentation boundary point (i.e., high peak values), the linear curve is used to fit the relationship curve between the peak value and the depth information deviation value. The above-mentioned calibration curve is obtained by merging the two curves.

[0029] The expression of the calibration curve obtained by fitting the cubic curve and the linear curve is:

[0030] TOF_bias1=a*peak 3 +b*peak 2+c*peak+d;TOF_bias2=e*peak+f (2)

[0031] By inputting the discrete points whose peak values ​​are less than the segment boundary points into the cubic curve of the first part for solution, the corresponding coefficients a, b, c, and d can be solved. By inputting the discrete points whose peak values ​​are greater than or equal to the segment boundary points into the linear curve of the second part for solution, the corresponding coefficients e and f can be solved, thereby determining all the coefficients corresponding to the calibration curve.

[0032] In one possible implementation, the calibration curve is obtained by fitting the relationship curve between the signal peak value and the depth information deviation value based on the experimental data, including: determining the pixel partition according to the consistent distribution of the depth information deviation values ​​of different pixels under the same signal peak value; for each pixel partition, fitting the relationship curve corresponding to the partition.

[0033] Due to the wiring differences between different pixels, the consistency of TDC cannot be guaranteed. Therefore, the calibration curves obtained by fitting corresponding to different pixels will also be different. If all pixels in the entire sensor share the same curve, the measurement accuracy of different pixels after depth information correction will still be different. After testing the sensors of multiple TOF sensors (modules), it is determined that the depth information deviation values ​​of some pixels in the same sensor show good consistency. Therefore, the pixels with relatively consistent distribution of depth information deviation values ​​are divided into one partition, and the pixels in the sensor are divided into several partitions. The pixels in the same partition use the same calibration curve for depth information correction, that is, for each pixel partition, the relationship curve corresponding to the pixels in the partition is fitted respectively, which can effectively improve the accuracy of depth information measurement.

[0034] In a second aspect, an embodiment of the present application provides an electronic device, one or more processors, and a memory, wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute a method as described in any one of the first aspects above.

[0035] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes instructions, and when the instructions are executed on an electronic device, the electronic device executes any method as described in the first aspect above.

[0036] In a fourth aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes any method as in the first aspect.

[0037] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes any one of the above methods in the first aspect.

[0038] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a schematic diagram of the structure of a 3D sensor in an embodiment of the present application;

[0040] Figure 2 Schematic diagram of the working principle of SPAD in the embodiment of the present application;

[0041] Figure 3 A schematic diagram of TDC statistical results corresponding to target objects with different reflectivity provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application;

[0043] Figure 5 A schematic diagram of the software architecture of the electronic device 100 provided in an embodiment of the present application;

[0044] Figure 6 A schematic diagram of the interaction of a software module provided in an embodiment of the present application;

[0045] Figure 7 A schematic diagram of the working state of a TOF camera provided in an embodiment of the present application;

[0046] Figure 8 A schematic diagram of a calibration curve provided in an embodiment of the present application;

[0047] Fig. 9 A schematic diagram of another calibration curve provided in an embodiment of the present application;

[0048] Fig.10 A schematic diagram of curves of different calibration curves corresponding to different pixels provided in an embodiment of the present application;

[0049] Fig.11The distribution of depth information deviation values ​​of different pixels in the same sensor at the same peak value provided in the embodiment of the present application;

[0050] Fig.12 is a schematic diagram of the pixel partitioning of the sensor provided in an embodiment of the present application;

[0051] Fig.13 It is a schematic diagram of an implementation flow of a depth information correction method provided in an embodiment of the present application;

[0052] Fig.14 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0054] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0055] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0056] In order to more clearly explain the technical solution of this application, the concepts involved in this application are explained as follows:

[0057] 1. Time of flight (TOF) imaging technology refers to an imaging technology that uses a group of infrared light (or laser pulses) that are invisible to the human eye to be emitted outward, reflected after encountering an object, and reflected to the camera. The time difference or phase difference from emission to reflection back to the camera is calculated, and the data is collected to form a set of distance and depth data, thereby obtaining a three-dimensional 3D model. In other words, TOF imaging technology is based on the traditional 2D XY axis (XY axis of the world coordinate system) imaging, adding depth information from the Z axis ((XY axis of the world coordinate system)) direction, and finally generating 3D image information.

[0058] 2. Direct time-of-flight (DTOF) is a TOF method, which is to emit a light pulse and then measure the time interval between the emitted light pulse and the received light pulse to get the flight time. The distance of the object can be converted according to the flight time to generate depth information.

[0059] 3. A three-dimensional (3D) sensor refers to a sensor that can obtain depth information of a target object, such as a TOF sensor, a structured light sensor, etc. The above 3D sensor can be incorporated into a camera to obtain a TOF camera (TOF camera module) that can collect depth information of a target object.

[0060] 4. The TOF camera (TOF camera module) can include a transmitter module (TX) and a receiver module (RX). TX is used to transmit infrared light or laser pulses, and RX is used to receive reflected light and form an image. Since TX can autonomously emit light signals for imaging, the TOF image is not affected by most of the light in the environment. In this way, applying the TOF image to the unlocking service can improve the security of face recognition.

[0061] 5. Single photon avalanche diode (SPAD) is a photodiode that works in Geiger mode and uses avalanche breakdown to achieve single photon detection function.

[0062] Geiger mode refers to an operating mode in which the reverse bias voltage is greater than the avalanche breakdown voltage of the single photon avalanche diode.

[0063] 6. A time digital converter (TDC) is a device used to identify events and provide the time of event occurrence. It can count the number of photon reception events and generate a statistical histogram of flight time based on the time of photon reception events. The horizontal axis of the statistical histogram is the flight time, and the vertical axis of the statistical histogram is the number of occurrences.

[0064] 7. Signal peak value refers to the maximum value in the histogram curve corresponding to the statistical histogram of the above flight time. The abscissa time corresponding to the signal peak value is recorded as the actual flight time.

[0065] In order to more clearly explain the technical solution of this application, the working principle of the 3D sensor in the electronic device is first described as follows:

[0066] See also Figure 1 , Figure 1 Figure 1 is a schematic diagram of the structure of a 3D sensor. Figure 1 As shown, the 3D sensor may include a transmitting module and a receiving module, wherein the adjustable light source in the transmitting module can emit multiple light pulses within one frame time, and after the emitted light pulse reaches the target object and is reflected by the target object, the reflected light pulse will be received by the receiving processor (sensor) in the receiving module, and then counted by the TDC, and then the flight time obtained from the multiple counting results is histogrammed (to obtain a histogram curve), and the flight time with the highest frequency (the horizontal axis time corresponding to the signal peak of the histogram curve) is used as the flight time for calculating the depth information of the target object (i.e., the target flight time).

[0067] It should be noted that the above-mentioned adjustable light source can be a device such as a vertical cavity surface emitting laser (VCSEL) or an edge emitting laser (EEL), and the above-mentioned sensor is composed of a pixel array, that is, the above-mentioned sensor can include n*m ​​pixels, where n is the number of rows, which is a positive integer greater than or equal to 1, and m is the number of columns, which is also a positive integer greater than or equal to 1.

[0068] The photon reception event of each pixel in the sensor is realized by SPAD. Each time an avalanche occurs in SPAD, it corresponds to a photon reception event. At this time, TDC can correspond to a count. TDC will count the counting results to obtain a histogram in the time domain. The horizontal coordinate (time) corresponding to the peak value of the histogram curve corresponding to the histogram can be further converted into the depth information of the target object.

[0069] like Figure 2 As shown, when the SPAD operating voltage V avalanche Higher than the avalanche breakdown voltage (V BD ) avalanche =V BD +V exc), where Vexc refers to the voltage value that exceeds the avalanche breakdown voltage. When the SPAD works in Geiger mode, when a photon arrives (that is, the sensor receives a photon), it will trigger the avalanche effect of the SPAD, generate a large current, and thus generate a digital signal that can be captured by the TDC. At this time, the TDC will record it as a photon arrival, and the SPAD cannot quickly return to its initial state after the avalanche. A quenching circuit is required to recover the current generated after the avalanche. At this time, the voltage across the SPAD will restore the avalanche breakdown voltage, thereby suppressing the avalanche current and turning off the SPAD current. Subsequently, the SPAD is powered on to return to its initial state. The time required from avalanche to returning to the initial state is called dead time.

[0070] Since SPAD needs to wait for a dead time after receiving a photon before it can receive another photon, when the distance is short, the reflectivity is high or the ambient light is strong, a large number of photons will arrive in a very short time. Therefore, the photons that arrive first will suppress the photons that arrive later, that is, the photons in front have a higher probability of triggering SPAD counting, which will cause photon pile-up, that is, the histogram curve corresponding to the histogram obtained by TDC statistics will have a signal peak shifted forward.

[0071] For example, see Figure 3 , Figure 3 Schematic diagram of TDC statistical results corresponding to target objects with different reflectivity. Figure 3 As shown, Figure 3 L1 is the histogram curve corresponding to the TDC statistical result corresponding to the standard reflectivity plate with a theoretical reflectivity of 90% (i.e., the histogram curve when there is no pile-up phenomenon), L2 is the histogram curve corresponding to the TDC statistical result corresponding to the standard reflectivity plate with a reflectivity of 5%, and L3 is the histogram curve corresponding to the TDC statistical result corresponding to the standard reflectivity plate with an actual reflectivity of 90%. It should be noted that the curves in the above examples are obtained by measuring at equal distances.

[0072] Depend on Figure 3 From L1 and L3 in the figure, we can see that for the standard reflectivity plate with a reflectivity of 90%, the signal peak (peak1) of the histogram curve corresponding to the actual statistical result of TDC is shifted by t1-t2 compared to the signal peak (peak2) of the histogram curve corresponding to the standard reflectivity plate with a theoretical reflectivity of 90%. Among them, t1 is the horizontal axis time corresponding to peak2, and t2 is the horizontal axis time corresponding to peak1.

[0073] The histogram curve corresponding to the histogram obtained by TDC statistics caused by the pile-up phenomenon has a signal peak forward shift, which will cause the peak search position to shift, resulting in errors in the obtained depth information and affecting the ranging accuracy.

[0074] Based on the above problems, an embodiment of the present application provides a depth information correction method. After obtaining the histogram statistical results, the corresponding depth information deviation value is determined according to the signal peak value of the histogram statistical results and the calibration curve, and then the collected depth information is corrected based on the depth information deviation value. Since the calibration curve is a relationship curve obtained by fitting the relationship between the depth information deviation caused by the pile-up phenomenon and the signal peak value of the histogram curve, the depth information obtained by the histogram statistical results is corrected using the depth information deviation value determined based on the calibration curve. This can effectively correct the signal peak forward shift in the histogram statistical results obtained by TDC statistics due to the pile-up phenomenon, thereby improving the accuracy of the depth information calculation results and improving the ranging accuracy.

[0075] The solution provided in the embodiment of the present application can be applied to any electronic device with a 3D sensor. The above-mentioned electronic device can also be called a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control (industrial control), a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in transportation safety, a wireless terminal in smart city, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the electronic device.

[0076] In order to better understand the embodiments of the present application, the structure of the electronic device in the embodiments of the present application is introduced below:

[0077] Figure 41 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0078] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0079] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0080] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0081] In some embodiments, the processor 110 is implemented by a system on chip (SOC), that is, the processor 110 mentioned in some embodiments of the present application may specifically refer to a SOC chip.

[0082] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0083] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuits sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0084] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0085] In some embodiments, the antenna 1 of the electronic device is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth (BT), global navigation satellite system (GNSS), wireless local area network (WLAN), near field communication (NFC), and / or infrared (IR) technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0086] The electronic device 100 implements audio functions such as music playing, audio collection, and recording through the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0087] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0088] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0089] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device detects the touch operation intensity according to the pressure sensor 180A. The electronic device can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0090] In some embodiments, the 3D sensor is disposed below the display screen 194 (the below refers to the side close to non-user input operation).

[0091] The electronic device 100 can realize the shooting function through the 3D sensor 10, ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0092] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0093] Camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0094] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0095] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0096] It is understandable that the electronic device 100 may include one or more cameras 193. When the electronic device 100 photographs the target object through the camera 193, the depth information of the target object may be obtained through the 3D sensor 10. In some embodiments, some cameras 193 in the electronic device 100 have the function of the 3D sensor 10, that is, the camera 193 is a depth sensing camera 193, for example, the electronic device 100 may be equipped with a TOF depth sensing camera.

[0097] The electronic device 100 having the above hardware structure can have a software system that adopts a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0098] Figure 5 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0099] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: application layer, application framework layer, system layer, and kernel layer.

[0100] The application layer can include a series of application packages. Figure 4 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, video, instant chat, reading, settings and other applications.

[0101] Among them, the setting application has the function of recording a face, and the recorded face can be used for face unlocking. The lock screen application has the function of unlocking in response to the user's unlocking operation. Specifically, the lock screen application can perform unlocking processes such as face unlocking, fingerprint unlocking, and password unlocking. The embodiment of this application mainly takes face unlocking as an example for explanation.

[0102] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, it may include activity manager, window manager, content provider, view system, resource manager, notification manager, camera service and face recognition service.

[0103] The system layer includes system libraries, Android runtime, core libraries, and media libraries.

[0104] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0105] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0106] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0107] In some embodiments, the software architecture of the electronic device 100 may further include a hardware abstraction layer (HAL), which may include multiple modules. The HAL layer acts as an intermediary to separate hardware and software from each other. The main purpose of this layer is to allow software to run on hardware that it is not intended to run on. In one embodiment, the multiple modules in the HAL layer include but are not limited to at least one of the following modules: Bluetooth HAL, camera HAL, sensors HAL, face recognition control module, etc.

[0108] Among them, the camera hal is the core software framework of the Camera, which can include sensor nodes and image front end (IFE) nodes. The sensor node and IFE node are components (nodes) in the image data and control instruction transmission path (also called transmission pipeline) created by the camera hal.

[0109] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0110] The camera driver is the driver layer of the Camera device and is mainly responsible for interacting with the hardware.

[0111] After introducing the execution subject involved in the embodiments of the present application, the possible application scenarios applicable to the embodiments of the present application are briefly introduced. Here, the electronic device is a mobile phone as an example for explanation.

[0112] The embodiments of the present application are applicable to any scenario where depth information needs to be obtained, such as 3D face unlocking, background blur under TOF camera shooting, augmented reality (AR) game entertainment scenario, etc. Here, the 3D sensor is specifically a TOF camera as an example.

[0113] The following is an example of a 3D face unlocking scenario:

[0114] like Figure 6As shown, the lock screen application in the application layer can interact with the face recognition application, and the face recognition application can interact with the face recognition service in the framework layer by calling a preset application programming interface (API) interface. The face recognition service can interact with the face recognition control module in the hal layer, and the face recognition control module can interact with the camera hal in the hal layer through the camera service in the framework layer, or the face recognition control module can directly interact with the camera hal in the hal layer. The camera hal may include a sensor node. The sensor node can interact with the camera driver in the kernel layer, and the camera driver can be used to drive the TOF camera in the hardware layer to collect image data. The image data collected by the TOF camera can be stored in a memory, and a file descriptor (FD) is used to indicate the storage location. The memory can send the FD to the camera driver. The camera driver can pass the FD to the IFE node of the camera hal. The IFE node can pass the FD to the camera service. The camera service can pass the FD to the face recognition control module. The face recognition control module can pass the FD to the face recognition module. The frame management module in the face recognition module can read the image data (RAW Data1) from the memory according to the FD, and send RAW Data1 to the TOF algorithm module. The TOF algorithm module can convert RAW Data1 into a grayscale image and determine the depth information, and perform depth information correction on the depth information according to the calibration curve to obtain the corrected depth image, and then send the grayscale image and the corrected depth image to the face ID algorithm module. The face ID algorithm module can be used for grayscale image matching and depth image anti-counterfeiting judgment to obtain face recognition results. The face ID algorithm module can send the face recognition results to the face recognition control module through the frame management module. The face recognition control module can feed back the processing results to the lock screen application through the face recognition service and the face recognition application, so that the lock screen application can determine whether to unlock (if the face recognition is successful, unlock; if the face recognition fails, do not unlock, that is, unlocking fails). Among them, Figure 6 The solid arrows in the figure can be used to represent control flow, and the dashed arrows can be used to represent data flow.

[0115] For example, see Figure 7 , Figure 7Schematic diagram of the working state of the TOF camera provided for some embodiments of the present application. The TOF camera 70 is arranged in the electronic device 100, and the TOF camera 70 is on the side of the display screen 194 that is away from the display surface of the display screen 194. The TOF camera 70 can be fixed on the main circuit board 20 and is between the main circuit board 20 and the display screen 194. A light source 70a is arranged in the TOF camera 70. The display screen 194 can have a light-transmitting portion (not shown in the figure), and the light-transmitting portion can allow the infrared light projected by the TOF camera 70 to pass through the object. The light-transmitting portion can also allow infrared light reflected by the object to pass through, so that the infrared light is received by the sensor of the TOF camera 70.

[0116] The light source 70a in the TOF camera 70 may be a vertical cavity surface emitting laser (VCSEL) or an edge emitting laser (EEL).

[0117] In some embodiments, the light-transmitting portion may be a light-transmitting hole on a display screen of an electronic device. Of course, it is understood that in other embodiments, the light-transmitting portion may be divided into a first light-transmitting portion and a second light-transmitting portion separated from each other, wherein the first light-transmitting portion is used to transmit infrared light emitted by the light source 70a, and the second light-transmitting portion is used to transmit infrared light reflected by an object.

[0118] The light source 70a is controlled by the TOF camera 70 to emit infrared light. The above infrared light is the infrared light required to meet the TOF function. For example, the light source 70a in the TOF camera 70 can be an LED or a laser tube (such as a laser diode or VCSEL) to emit high-frequency modulated pulse light, and the frequency can reach about 1000MHz, and the wavelength is about 800nm-1000nm. The infrared light required for TOF also has a high peak power. The infrared light reflected after reaching the target object can be received by the sensor of the TOF camera 70. The TDC will output a histogram curve corresponding to the received reflected light pulse, so as to obtain the round-trip flight time of the infrared light reflected at different positions of the target object, and obtain the distance between the mobile phone and the different positions of the target object, so as to generate depth information. The mobile phone can correct the depth information according to the peak value of the histogram curve and the calibration curve corresponding to each pixel in the sensor to correct the depth information error caused by the pile-up phenomenon, and then determine the corresponding image information according to the corrected depth information to assist the mobile phone in realizing 3D face unlocking.

[0119] This requires pre-constructing the above calibration curve.

[0120] Generally speaking, the pile-up phenomenon is positively correlated with the light intensity of the received light pulse and the photon detection efficiency. The pile-up phenomenon is directly reflected in the signal peak value (peak value) of the histogram curve, that is, the stronger the light intensity of the received light pulse, the more serious the pile-up phenomenon is, and the higher the peak value reflected in the histogram curve will be. The peak value has a certain relationship with the depth information deviation value caused by the pile-up phenomenon, so the relationship curve between the peak value and the depth information deviation value caused by the pile-up phenomenon can be fitted based on the experiment (that is, the calibration curve mentioned above).

[0121] The following is an explanation of the process of fitting the relationship curve between the peak value and the depth information deviation value caused by the pile-up phenomenon:

[0122] 1. Fix the TOF sensor used in the experiment and make it parallel to the macro resolution board (also called chart board).

[0123] It should be noted that the chart board is a standard board for detecting the geometric resolution of a short-focal-length photographic objective. In the embodiment of the present application, since the peak value in the histogram curve is related to the reflectivity and distance of the target object, considering the limitations of the calibration effect and the calibration environment, if a chart board with a lower reflectivity is selected, the depth information error data under high peak values ​​may not be covered; if a chart board with a lower reflectivity is selected, it is necessary to move a longer distance and increase the size of the chart board to cover the depth information error data under low peak values. Therefore, when selecting a chart board, a chart board with a more moderate reflectivity can be selected (for example, a chart board with a reflectivity of 40%-60%).

[0124] 2. Collect histogram data and depth information at different distances.

[0125] In a specific application, the chart board is moved to different positions, and then the TOF sensor is controlled to collect histogram data and depth information at different distances.

[0126] The content of the histogram data and depth information collected by the TOF sensor at different distances can be found in the description of the working principle of the 3D sensor, which will not be repeated here.

[0127] 3. Extract the peak value and depth information deviation value (TOF_bias) based on the collected histogram data and depth information at different distances.

[0128] In specific applications, the depth information bias value (TOF_bias) can be determined based on the actual distance and the depth information collected by the TOF sensor, that is, TOF_bias = GT-TOF_mea; wherein TOF_bias is the depth information bias value, GT is the actual distance, and TOF_mea is the depth information collected by the TOF sensor.

[0129] 4. Based on the cubic curve, the relationship curve between the peak value and the depth information deviation value is fitted to obtain the above calibration curve.

[0130] In specific applications, the cubic curve is a plane algebraic curve, which has three intersections with a general straight line. After testing, the cubic curve is used to fit the relationship curve between the peak value and the depth information deviation value, and the fitting effect is the best.

[0131] In a specific application, the peak value may be normalized first, and then fitting may be performed based on the normalized peak value and the depth information deviation value.

[0132] For example, see Figure 8 , Figure 8 Schematic diagram of the calibration curve in this application. Figure 8 As shown in Figure 3, the fitted calibration curve can cover the discrete points well.

[0133] The expression of the calibration curve obtained based on cubic curve fitting is:

[0134] TOF_bias=a*peak 3 +b*peak 2 +c*peak+d; (1)

[0135] Substituting the discrete points into the above formula (1), the corresponding coefficients a, b, c, and d can be solved, thereby obtaining a calibration curve suitable for the TOF sensor.

[0136] In the process of curve fitting, the more times of fitting, the more complicated the calculation, the more computing resources will be needed. As the peak value increases, the process of fitting the relationship curve between the peak value and the depth information deviation value using the cubic curve will also require more computing resources. In order to save computing power, another embodiment of the present application provides another curve fitting method to fit the relationship curve between the peak value and the depth information deviation value.

[0137] The segmentation boundary point is set according to the test results corresponding to the computing resources required for the calculation. For peak values ​​less than the segmentation boundary point, a cubic curve is used to fit the relationship curve between the peak value and the depth information deviation value. For peak values ​​greater than or equal to the segmentation boundary point (i.e., high peak values), a linear curve is used to fit the relationship curve between the peak value and the depth information deviation value. The above calibration curve is obtained by merging the two curves.

[0138] For example, see Fig. 9 , Fig. 9 A schematic diagram of another calibration curve in this application is shown, such as Fig. 9 As shown, the calibration curve includes a first part obtained by cubic curve fitting and a second part obtained by linear curve fitting.

[0139] The expression of the calibration curve obtained by fitting the cubic curve and the linear curve is:

[0140] TOF_bias1=a*peak 3 +b*peak 2 +c*peak+d;TOF_bias2=e*peak+f (2)

[0141] By inputting the discrete points whose peak values ​​are less than the segment boundary points into the cubic curve of the first part for solution, the corresponding coefficients a, b, c, and d can be solved. By inputting the discrete points whose peak values ​​are greater than or equal to the segment boundary points into the linear curve of the second part for solution, the corresponding coefficients e and f can be solved, thereby determining all the coefficients corresponding to the calibration curve.

[0142] Due to the wiring differences between different pixels, the consistency of TDC cannot be guaranteed, so the calibration curves obtained by fitting corresponding to different pixels will also be different. For example, please refer to Fig.10 , Fig.10 A curve schematic diagram of different calibration curves corresponding to different pixels is shown, wherein calibration curve 1 is the calibration curve corresponding to the first pixel, calibration curve 2 is the calibration curve corresponding to the second pixel, and calibration curve 3 is the calibration curve corresponding to the third pixel.

[0143] If all pixels in the entire sensor share the same curve, the measurement accuracy of different pixels will still be different after depth information correction. Fig.11 , Fig.11 The figure shows the distribution of depth information deviation values ​​of different pixels in the same sensor at the same peak value. Fig.11 It can be seen that the depth information deviation values ​​of some pixels in the same sensor show good consistency. Fig.11It can be clearly seen that the pixels with good consistency in the same sensor are distributed in odd and even numbers horizontally and symmetrically in the vertical direction.

[0144] It should be noted that the above-mentioned odd-even distribution in the horizontal direction means that the pixels in odd-numbered columns have better consistency, and the pixels in even-numbered columns also have better consistency. The above-mentioned symmetrical distribution in the vertical direction means that the pixels in the first row have better consistency with the pixels in the last row, the pixels in the second row have better consistency with the pixels in the second to last row, and so on.

[0145] After testing multiple TOF sensors (modules), we can get the following results: Fig.11 The consistency distribution is shown. Therefore, in the embodiment of the present application, pixels with relatively consistent distribution of depth information deviation values ​​are divided into a partition, and the pixels in the sensor are divided into several partitions. The pixels in the same partition use the same calibration curve for depth information correction, that is, for each pixel partition, a relationship curve corresponding to the pixels in the partition is fitted respectively.

[0146] Specifically, the following operations can be performed for each partition: obtain the experimental data of any pixel in the partition, and then fit the corresponding relationship curve based on the experimental data (it can be the relationship curve obtained by fitting the above-mentioned cubic curve, or it can be the relationship curve obtained by fitting a cubic curve combined with a linear curve), that is, the relationship curve is the calibration curve corresponding to the partition.

[0147] For example, Fig.12 As shown, the above sensor may include 1200 pixels (30 rows * 40 columns). Fig.11According to the consistency distribution shown, the 1200 pixels can be divided into 30 partitions (pixels of the same color and pattern are pixels in the same partition). Among them, the pixels corresponding to the first row of the first column, the pixels corresponding to the first row of the third column, the pixels corresponding to the first row of the fifth column...the pixels corresponding to the first row of the thirty-ninth column, the pixels corresponding to the last row of the first column, the pixels corresponding to the last row of the third column, the pixels corresponding to the last row of the fifth column...the pixels corresponding to the last row of the thirty-ninth column are in the same partition, and the same calibration curve is used for depth information correction. The pixels corresponding to the second row of the first column, the pixels corresponding to the second row of the third column, the pixels corresponding to the second row of the fifth column...the pixels corresponding to the second row of the thirty-ninth column, the pixels corresponding to the second to last row of the first column, the pixels corresponding to the second to last row of the third column, the pixels corresponding to the second to last row of the fifth column,..., the pixels corresponding to the second to last row of the thirty-ninth column are in the same partition, and the same calibration curve is used for depth information correction. The pixels corresponding to the first row of the second column, the pixels corresponding to the first row of the fourth column, the pixels corresponding to the first row of the sixth column...the pixels corresponding to the first row of the fortieth column, the pixels corresponding to the last row of the second column, the pixels corresponding to the last row of the fourth column, the pixels corresponding to the last row of the sixth column...the pixels corresponding to the last row of the sixth column are in the same partition and use the same calibration curve for depth information correction.

[0148] Therefore, for the above TOF sensor, only 30 calibration curves need to be fitted, and the pixels in each partition use the same calibration curve corresponding to the partition to perform depth information correction.

[0149] The calibration curve may use the calibration curve obtained by fitting the cubic curve corresponding to formula (1), or may use the calibration curve obtained by fitting the cubic curve and the linear curve corresponding to formula (2). When the calibration curve uses the calibration curve obtained by fitting the cubic curve corresponding to formula (1), it is necessary to determine the four coefficients a, b, c, and d, that is, when fitting the calibration curve, it is necessary to determine the four coefficients of 30 calibration curves (i.e., a total of 120 coefficients); when the calibration curve uses the calibration curve obtained by fitting the cubic curve and the linear curve corresponding to formula (2), it is necessary to determine the six coefficients a, b, c, d, e, and f, that is, when fitting the calibration curve, it is necessary to determine the six coefficients of 30 calibration curves (i.e., a total of 180 coefficients).

[0150] After obtaining the calibration curves of each partition of the sensor, the calibration curves are pre-stored in the electronic device, so that when the electronic device uses the TOF sensor to obtain depth information, the calibration curves stored in the electronic device can be called to perform corresponding depth information correction.

[0151] It should be noted that the “at…” in the embodiments of the present application may be the instant when a certain situation occurs, or may be a period of time after the occurrence of a certain situation, and the embodiments of the present application do not specifically limit this.

[0152] The depth information correction method provided in the embodiment of the present application is described below with a specific implementation process example. Fig.13 A schematic diagram of the implementation flow of a depth information correction method provided in an embodiment of the present application. Fig.13 As shown, the method may include:

[0153] S1301: In response to a depth information collection triggering event, call a target sensor to collect depth information of a target object.

[0154] In a specific application, the target sensor is a sensor capable of collecting depth information of a target object, for example, it may be the 3D sensor, the TOF sensor, the TOF camera, etc.

[0155] The following scenarios can trigger the acquisition of the above depth information:

[0156] 1. The 3D face unlocking function in the lock screen state is activated.

[0157] In a specific application, when the user wants to use the 3D face unlocking function to unlock the phone, the depth information acquisition trigger event can be triggered when the user's first operation is detected. The first operation can be an unlocking operation, that is, when the phone is in a locked state and detects the user's first operation, the depth information acquisition trigger event can be triggered, thereby triggering the phone to call the TOF camera to collect depth information. The first operation includes the user picking up the phone, or pressing the power button, or operating on the screen (clicking, sliding, etc.), or unplugging the charging cable and other operations.

[0158] 2. The user selects a shooting mode with blurred background when taking pictures.

[0159] In a specific application, when the user takes a photo of the target object, the user can select a background blur shooting mode. In response to the user clicking on the background blur shooting mode (which can trigger the above-mentioned depth information collection trigger event), the mobile phone can call the TOF camera to collect the depth information of the target object in order to identify the foreground and background, thereby achieving background blur.

[0160] 3. Users use their mobile phones to play AR games.

[0161] In specific applications, when users use mobile phones to play AR games, they can choose scene construction. In response to the user clicking the build button of the scene construction (which can trigger the above-mentioned depth information collection trigger event), the mobile phone can also call the TOF camera to obtain spatial images and spatial depth information, and perform scene rendering and other operations based on this, thereby constructing a corresponding virtual scene.

[0162] It can be understood that the above-mentioned situations are only examples of triggering events for triggering the collection of depth information. The above-mentioned depth information collection triggering events can also be triggered in other situations, such as calling the TOF camera (or 3D sensor) to collect image information when using the smart eye movement service, or calling the TOF camera for face recognition during the payment or login process of the payment application.

[0163] In an embodiment of the present application, the operation of calling the target sensor to collect depth information can specifically be: when the target sensor is a TOF camera, the mobile phone can drive the TOF camera through a camera driver to perform the action of collecting depth information; when the target sensor is a 3D sensor, the mobile phone can drive the 3D sensor through a sensor driver to perform the action of collecting depth information.

[0164] Taking the TOF camera as an example, when the camera driver triggers the TOF camera to collect depth information, the TOF camera can emit infrared light through the light source in the transmitting module. The infrared light is reflected after reaching the target object, and the reflected infrared light can be received by the sensor in the receiving module of the TOF camera. Specifically, the SPAD in each pixel in the sensor can record the photon reception event, that is, each avalanche of the SPAD corresponds to a photon reception event, and the TDC can correspond to one count. The TDC can perform histogram statistics on the flight time obtained from multiple counting results to obtain a histogram curve (that is, the histogram statistical result).

[0165] S1302: Correct the collected depth information according to the calibration curve to obtain corrected depth information.

[0166] In the embodiment of the present application, the calibration curve is the above-mentioned pre-constructed and stored in the mobile phone, which is used to represent the fitting relationship curve between the peak value and the depth information deviation value caused by the pile-up phenomenon.

[0167] After TDC obtains the histogram statistics corresponding to the counting result, it can determine the peak value corresponding to the histogram statistics, find the depth information deviation value corresponding to the peak value based on the calibration curve, and then use the depth information deviation value to correct the depth information calculated by the histogram statistics, that is, the corrected depth information can be obtained.

[0168] The above histogram statistical results can be represented by a histogram curve.

[0169] The calculation process of the depth information obtained by calculating the above histogram statistical results may specifically be to determine the abscissa time corresponding to the peak value in the histogram statistical results, and calculate the depth information according to the abscissa time corresponding to the peak value.

[0170] The specific calculation method is: TOF_mea=cΔt / 2; where TOF_mea is the depth information collected by the 3D sensor, c is the speed of light, and Δt is the horizontal coordinate time corresponding to the peak value.

[0171] The depth information calculated by the histogram statistics result is corrected according to the depth information deviation value. The specific calculation method can be: TOF_mea'=TOF_mea+TOF_bias; wherein TOF_mea' is the corrected depth information, TOF_mea is the depth information collected by the 3D sensor, and TOF_bias is the depth information deviation value determined according to the calibration curve and the peak value.

[0172] In some embodiments, the calibration curve may be a single calibration curve, that is, each pixel in the sensor of the TOF camera uses the same calibration curve to perform depth information correction.

[0173] In other embodiments, the calibration curve may also be a calibration curve set according to each pixel, that is, each pixel is set with a calibration curve corresponding to the pixel, and the depth information determined by the statistical result of the histogram corresponding to each pixel uses the calibration curve corresponding to the pixel to perform depth information correction. Because there are unlimited differences between different pixels, there is inconsistency in TDC. If all pixels in the sensor use the same calibration curve to correct the depth information, the measurement accuracy of different pixels after the depth information correction will still have certain errors. In order to further improve the measurement accuracy of the depth information, the embodiment of the present application can set a corresponding calibration curve for each pixel, so that when correcting the depth information, the calibration curve corresponding to the pixel can be used to determine the depth information deviation value, which can effectively improve the measurement accuracy of the depth information.

[0174] In other embodiments, the calibration curve may be a calibration curve set in correspondence with the pixel partition, that is, a calibration curve corresponding to the partition is set for each pixel partition, and the pixels in the partition may use the calibration curve to perform depth information correction. Setting a corresponding calibration curve for each pixel will consume more computing resources. Since the depth information deviation values ​​of some pixels in the same sensor are consistent under the same peak value, the embodiment of the present application may partition the pixels in the sensor according to the consistency of the depth information deviation values ​​of the pixels in the sensor, and then set a calibration curve corresponding to the partition for each partition. In this way, in the process of depth information correction, the partition to which the pixel belongs can be determined first, and then the calibration curve corresponding to the partition can be obtained according to the partition to which the pixel belongs, and the calibration curve corresponding to the partition can be used to perform depth information correction. This can improve the measurement accuracy of the depth information while reducing the time and resources consumed in the calibration process.

[0175] In some embodiments of the present application, partition calibration curves are constructed for different pixels. The calibration curve corresponding to the partition corresponding to the pixel can be obtained based on the pixel, and then the depth information deviation value is determined based on the peak value determined by the calibration curve corresponding to the pixel and the histogram statistics corresponding to the pixel point.

[0176] From the above, it can be seen that the depth information correction method provided in the embodiment of the present application determines the corresponding depth information deviation value according to the signal peak value of the histogram statistical result and the calibration curve after obtaining the histogram statistical result, and then corrects the collected depth information based on the depth information deviation value. Since the calibration curve is a relationship curve obtained by fitting the relationship between the depth information deviation caused by the pile-up phenomenon and the signal peak value of the histogram curve, the depth information obtained by the histogram statistical result is corrected using the depth information deviation value determined based on the calibration curve. This can effectively correct the signal peak forward shift in the histogram statistical result obtained by TDC statistics due to the pile-up phenomenon, thereby improving the accuracy of the depth information calculation result and improving the ranging accuracy.

[0177] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the method steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0178] The embodiment of the present application can divide the functional modules of the device of the audio channel switching method according to the above method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0179] Fig.14 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig.14 As shown, the chip 140 includes one or more (including two) processors 1401 , a communication line 1402 , a communication interface 1403 , and a memory 1404 .

[0180] In some implementations, the memory 1404 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0181] The method described in the above embodiment of the present application can be applied to the processor 1401, or implemented by the processor 1401. The processor 1401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the execution method of the above first device or second device can be completed by the hardware integrated logic circuit or software instructions in the processor 1401. The above processor 1401 can be a general processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1401 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0182] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1404, and the processor 1401 reads the information in the memory 1404 and completes the steps of the above method in combination with its hardware.

[0183] The processor 1401 , the memory 1404 , and the communication interface 1403 may communicate with each other via the communication line 1402 .

[0184] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0185] The present application embodiment also provides a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, the available medium may include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0186] The present application also provides a computer-readable storage medium. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0187] As a possible design, the computer readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connection line may also be appropriately referred to as a computer readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while optical discs reproduce data optically using lasers.

[0188] The above combinations should also be included in the scope of computer-readable media. The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A depth information correction method, characterized in that: include: In response to a depth information collection triggering event, calling a target sensor to collect depth information of a target object; The collected depth information is corrected according to the calibration curve to obtain the corrected depth information; wherein the calibration curve is a fitting relationship curve between the signal peak value and the depth information deviation value, and the signal peak value is the signal peak value of the histogram statistical result corresponding to the photon reception event obtained by the target sensor.

2. The depth information correction method according to claim 1, characterized in that: The step of correcting the collected depth information according to the calibration curve to obtain the corrected depth information includes: Determining a calibration curve corresponding to a pixel in a receiving processor of the target sensor according to the pixel; Determine a signal peak value according to a histogram statistical result corresponding to the photon reception event of the pixel; Finding a depth information deviation value corresponding to the signal peak value based on the calibration curve; The corrected depth information is calculated based on the collected depth information and the depth information deviation value.

3. The depth information correction method according to claim 1 or 2, characterized in that: The calibration curve is set corresponding to the partition of the pixels of the target receiving sensor.

4. The depth information correction method according to claim 3, characterized in that: The step of correcting the collected depth information according to the calibration curve to obtain the corrected depth information includes: determining a partition of the pixels according to the pixels in a receiving processor of the target sensor; The calibration curve is determined according to the partitioning of the pixels.

5. The depth information correction method according to any one of claims 1 to 4, characterized in that: Also includes: The calibration curve is obtained by fitting the relationship curve between the signal peak value and the depth information deviation value according to the experimental data.

6. The depth information correction method according to claim 5, characterized in that: The step of fitting the relationship curve between the signal peak value and the depth information deviation value according to the experimental data to obtain the calibration curve comprises: A relationship curve between the signal peak value and the depth information deviation value is fitted based on a cubic curve.

7. The depth information correction method according to claim 5, characterized in that: The step of fitting the relationship curve between the signal peak value and the depth information deviation value according to the experimental data to obtain the calibration curve comprises: For a signal peak value smaller than a segmentation boundary point, a relationship curve between the signal peak value and the depth information deviation value is fitted based on a cubic curve; For a signal peak value greater than or equal to the segment boundary point, a relationship curve between the signal peak value and the depth information deviation value is fitted based on a first-order curve.

8. The depth information correction method according to claim 5, characterized in that: The step of fitting the relationship curve between the signal peak value and the depth information deviation value according to the experimental data to obtain the calibration curve comprises: Determine pixel partitions according to the consistency distribution of depth information deviation values ​​of different pixels under the same signal peak value; For each pixel partition, a relationship curve corresponding to the partition is fitted.

9. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the processor and the memory are coupled, and the memory is used to store a computer program. When the processor executes the computer program, the electronic device executes the steps of the method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises a computer program, and when the computer program is executed on an electronic device, the electronic device executes the steps of the method according to any one of claims 1 to 8.

11. A chip system, characterized in that: The chip system is applied to an electronic device, the chip system includes one or more processors, the processor is coupled to a memory, the memory is used to store computer program instructions, and the one or more processors are used to call computer instructions so that the electronic device executes the steps of the method described in any one of claims 1 to 8.

Images