Depth compensation method of direct time of flight (dTOF) sensor and electronic equipment

Abstract

The invention is suitable for the technical field of computer application, and provides a depth compensation method for a direct time of flight (dTOF) sensor and electronic equipment, and the method comprises the steps: responding to a triggering operation for a camera application, and driving the dTOF sensor to collect original image data; acquiring the current actual driving voltage of a single photon avalanche diode (SPAD) in the dTOF sensor; obtaining the current expected driving voltage of the SPAD; and performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map. Therefore, the depth information offset caused by the adjustment error of the SPAD driving voltage is corrected through the deviation between the actual value and the theoretical value of the SPAD driving voltage, so that the accuracy of the depth information collected by the dTOF sensor is improved, and the distance measurement precision of the dTOF sensor is improved.

Description

Technical Field

[0001] The present application belongs to the field of terminal technology, and in particular, relates to a depth compensation method of a dTOF sensor, an electronic device, and a computer-readable storage medium. Background Art

[0002] When the direct time of flight (dTOF) sensor measures distance, the driving voltage of the single photon avalanche diodes (SPAD) (i.e., VSPAD voltage) needs to be dynamically adjusted as the temperature changes. The dTOF sensor adjusts the VSPAD voltage through the output current (i.e., IDAC current) of the current-type digital-to-analog converter (IDAC), and different IDAC currents can correspond to different voltage adjustment ranges.

[0003] In the related art, when the VSPAD voltage is adjusted, due to the accuracy error of the dTOF sensor's own IDAC current or the output error of the power supply, there may be a deviation between the actual value of the VSPAD voltage after adjustment and the theoretical value, thereby affecting the dark count (DCR), photon detection efficiency (PDE) and other parameters of the dTOF sensor, thereby causing the depth information collected by the dTOF sensor to be offset, affecting the ranging accuracy of the dTOF sensor. Summary of the invention

[0004] The embodiments of the present application provide a depth compensation method, electronic device and computer-readable storage medium for a dTOF sensor, which can solve the problem that when the VSPAD voltage is adjusted, there may be a deviation between the actual value of the VSPAD voltage after adjustment and the theoretical value due to the accuracy error of the dTOF sensor's own IDAC current or the output error of the power supply, thereby causing the depth information collected by the dTOF sensor to be offset, affecting the ranging accuracy of the dTOF sensor.

[0005] In a first aspect, an embodiment of the present application provides a depth compensation method for a dTOF sensor, comprising: in response to a trigger operation applied to a camera, driving the dTOF sensor to collect raw image data; obtaining the current actual driving voltage of the SPAD in the dTOF sensor; obtaining the current expected driving voltage of the SPAD; and performing depth compensation on the raw image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0006] In this way, by performing depth compensation on the original image data collected by the dTOF sensor according to the current actual driving voltage and expected driving voltage of the SPAD, the depth information offset caused by the adjustment error of the SPAD driving voltage can be corrected through the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor and further improving the ranging accuracy of the dTOF sensor.

[0007] In a possible implementation of the first aspect, after depth compensation is performed on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map, the method further includes:

[0008] During the shooting process, focus processing is performed according to the target depth map.

[0009] In this way, by using the compensated target depth map to perform focus processing during the shooting process, the focus effect of the shooting is improved, thereby improving the image shooting quality and further improving the user experience.

[0010] Optionally, in another possible implementation of the first aspect, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the depth compensation is performed on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map, including:

[0011] Generate an original depth map based on the original histogram;

[0012] The original depth map is depth compensated according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0013] In this way, by generating an original depth map based on the original histogram collected by the dTOF sensor, and then performing depth compensation on the depth information in the original depth map according to the deviation between the actual driving voltage of the SPAD and the expected driving voltage, the accuracy of the depth compensation is further improved, thereby further improving the ranging accuracy of the dTOF.

[0014] Optionally, in another possible implementation of the first aspect, the obtaining of the actual current driving voltage of the SPAD in the dTOF sensor includes:

[0015] The output voltage of the power supply corresponding to the SPAD is sampled to obtain the actual driving voltage.

[0016] Optionally, in yet another possible implementation of the first aspect, sampling the output voltage of the power supply corresponding to the SPAD to obtain the actual driving voltage includes:

[0017] When the start of frame (SOF) interrupt sent by the dTOF sensor is obtained, an analog-to-digital converter (ADC) reading thread is created;

[0018] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain a current ADC sampling value corresponding to the output voltage of the power supply;

[0019] The current ADC sampling value is released, and the actual driving voltage is determined based on the ADC sampling value.

[0020] In this way, each time the dTOF sensor acquires raw image data and frames it, an ADC reading thread is created, and the output voltage of the SPAD power supply is sampled through the ADC reading thread to obtain the current actual driving voltage of the SPAD, thereby improving the accuracy of the actual driving voltage acquisition, thereby further improving the accuracy of the depth information compensation, and further improving the ranging accuracy of the dTOF sensor.

[0021] Optionally, in yet another possible implementation of the first aspect, the obtaining of the current expected driving voltage of the SPAD includes:

[0022] Analyze and process the raw image data to determine the current voltage regulation level of the SPAD;

[0023] Obtain benchmark data corresponding to SPAD;

[0024] Determine the expected driving voltage based on the voltage regulation level and the reference data.

[0025] In this way, when the dTOF sensor adjusts the driving voltage of the SPAD through the IDAC current, the current voltage adjustment level can be determined according to the current operating temperature of the SPAD, and then the voltage adjustment is performed based on the pre-calibrated reference data according to the voltage adjustment level, and the dTOF sensor merges the collected raw data with the voltage adjustment level to generate raw image data (i.e., raw image) and outputs it each time the distance is measured. Therefore, the raw image output by the dTOF sensor can be analyzed and processed to determine the current voltage adjustment level of the SPAD, and then the current expected driving voltage of the SPAD, i.e., the theoretical value of the current driving voltage of the SPAD, is determined according to the voltage adjustment level and the reference data, thereby ensuring the accuracy of the determination of the theoretical value of the SPAD driving voltage, thereby further improving the accuracy of the depth information compensation, and further improving the ranging accuracy of the dTOF sensor.

[0026] Optionally, in another possible implementation of the first aspect, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the above-mentioned parsing and processing of the original image data to determine the current voltage adjustment level of the SPAD includes:

[0027] The metadata is parsed to determine the voltage regulation level.

[0028] In this way, every time the dTOF sensor collects a frame of raw image, it can merge the temperature, voltage adjustment level and other parameters when collecting the raw image as metadata with the collected original histogram and then output it. Therefore, the metadata in the raw image can be directly parsed to determine the voltage adjustment level corresponding to the time when the raw image was collected, that is, to determine the current voltage adjustment level of the SPAD, which not only improves the accuracy of depth information compensation and the ranging accuracy of dTOF, but also further reduces the calculation complexity of depth compensation.

[0029] Optionally, in another possible implementation of the first aspect, the reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step; accordingly, determining the expected driving voltage according to the voltage adjustment level and the reference data includes:

[0030] Determine the current voltage adjustment amplitude of the SPAD according to the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step;

[0031] The expected driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.

[0032] In this way, by pre-calibrating the reference driving voltage, reference voltage level and reference voltage adjustment step corresponding to each voltage level of SPAD, the reference data for adjusting the driving voltage of SPAD can be determined in advance. Then, the desired number of adjustment levels can be determined according to the difference between the real-time voltage adjustment level and the reference voltage level. The voltage amplitude to be adjusted, i.e., the current voltage adjustment amplitude of SPAD, can be determined according to the desired number of adjustment levels and the reference voltage adjustment step. Finally, the voltage can be adjusted based on the reference driving voltage according to the voltage adjustment amplitude to determine the current desired driving voltage of SPAD, thereby ensuring the accuracy of the theoretical value of SPAD driving voltage, thereby further improving the accuracy of the depth step, and further improving the ranging accuracy of the dTOF sensor.

[0033] Optionally, in yet another possible implementation of the first aspect, performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map includes:

[0034] Determine the current driving voltage adjustment error of the SPAD according to the difference between the actual driving voltage and the expected driving voltage;

[0035] The original image data is depth compensated according to the driving voltage adjustment error to generate a target depth map.

[0036] Optionally, in yet another possible implementation of the first aspect, the depth compensation is performed on the original image data according to the driving voltage adjustment error to generate a target depth map, including:

[0037] Determine the depth value error corresponding to the original image data according to the driving voltage adjustment error and the preset conversion ratio;

[0038] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.

[0039] In this way, by compensating for the depth value error caused by the deviation between the actual value and the theoretical value of the SPAD driving voltage, the accuracy of the depth information collected by the dTOF sensor is improved, and the ranging accuracy of the dTOF sensor is further improved.

[0040] In the second aspect, an embodiment of the present application provides a depth compensation device for a dTOF sensor, including: a first acquisition module, used to drive the dTOF sensor to acquire raw image data in response to a trigger operation applied to a camera; a first acquisition module, used to acquire the current actual driving voltage of the SPAD in the dTOF sensor; a second acquisition module, used to acquire the current expected driving voltage of the SPAD; a first compensation module, used to perform depth compensation on the raw image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0041] In a possible implementation manner of the second aspect, the apparatus further includes:

[0042] The focus module is used to perform focus processing according to the target depth map during the shooting process.

[0043] Optionally, in another possible implementation of the second aspect, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first compensation module includes:

[0044] A first generating unit, configured to generate an original depth map according to the original histogram;

[0045] The first compensation unit is used to perform depth compensation on the original depth map according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0046] Optionally, in another possible implementation of the second aspect, the first acquisition module includes:

[0047] The first sampling unit is used to sample the output voltage of the power supply corresponding to the SPAD to obtain the actual driving voltage.

[0048] Optionally, in yet another possible implementation manner of the second aspect, the first sampling unit is specifically configured to:

[0049] When the SOF interrupt sent by the dTOF sensor is obtained, an ADC reading thread is created;

[0050] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain a current ADC sampling value corresponding to the output voltage of the power supply;

[0051] The current ADC sampling value is released, and the actual driving voltage is determined based on the ADC sampling value.

[0052] Optionally, in yet another possible implementation of the second aspect, the second acquisition module includes:

[0053] A first determination unit, configured to perform analysis and processing on the original image data to determine a current voltage regulation level of the SPAD;

[0054] A first acquisition unit, used to acquire reference data corresponding to the SPAD;

[0055] The second determining unit is used to determine the expected driving voltage according to the voltage adjustment level and the reference data.

[0056] Optionally, in another possible implementation manner of the second aspect, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first determining unit is specifically configured to:

[0057] The metadata is parsed to determine the voltage regulation level.

[0058] Optionally, in another possible implementation of the second aspect, the reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step; accordingly, the second determining unit is specifically configured to:

[0059] Determine the current voltage adjustment amplitude of the SPAD according to the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step;

[0060] The expected driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.

[0061] Optionally, in yet another possible implementation of the second aspect, the first compensation module further includes:

[0062] A third determining unit, configured to determine a current driving voltage adjustment error of the SPAD according to a difference between the actual driving voltage and the expected driving voltage;

[0063] The second compensation unit is used to perform depth compensation on the original image data according to the driving voltage adjustment error to generate a target depth map.

[0064] Optionally, in yet another possible implementation manner of the second aspect, the second compensation unit is specifically configured to:

[0065] Determine the depth value error corresponding to the original image data according to the driving voltage adjustment error and the preset conversion ratio;

[0066] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.

[0067] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the depth compensation method of the dTOF sensor as described above.

[0068] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by an electronic device, implements the depth compensation method of the dTOF sensor as described above.

[0069] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the depth compensation method of the dTOF sensor as described above.

[0070] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 This is a schematic diagram of the architecture of a software system of an electronic device provided by an embodiment of the present application;

[0073] Figure 2It is a flowchart of a depth compensation method of a dTOF sensor provided in an embodiment of the present application;

[0074] Figure 3 is a schematic diagram of a main interface of an electronic device provided by an embodiment of the present application;

[0075] Figure 4 is a schematic diagram of a camera preview interface of an electronic device provided by an embodiment of the present application;

[0076] Figure 5 is a schematic diagram of a camera preview interface of another electronic device provided by an embodiment of the present application;

[0077] Figure 6 is a schematic diagram of a recording interface of an electronic device provided by an embodiment of the present application;

[0078] Figure 7 It is a structural schematic diagram of a raw image collected by a dTOF sensor provided in an embodiment of the present application;

[0079] Figure 8 It is a technical block diagram of a depth compensation method of a dTOF sensor provided by an embodiment of the present application;

[0080] Fig. 9 This is a schematic diagram of the relationship between a SPAD driving voltage and a voltage adjustment level provided in an embodiment of the present application;

[0081] Fig.10 is a timing diagram of a depth compensation method of a dTOF sensor provided in one embodiment of the present application;

[0082] Fig.11 It is a structural schematic diagram of a depth compensation device of a dTOF sensor provided in an embodiment of the present application;

[0083] Fig.12 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0084] Before introducing the method provided in the embodiment of the present application, the software system of the electronic device in the embodiment of the present application is introduced.

[0085] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software system of the electronic device.

[0086] Figure 1 is a block diagram of a software system of an electronic device provided in an embodiment of the present application. Figure 1The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer (framework), the hardware abstraction layer (HAL), and the kernel layer (kernel). In addition, Figure 1 The relationship between the hardware layer and the software system is also shown. As an example of the present application, the hardware layer includes a dTOF sensor and an ADC. The dTOF sensor is used to collect image data. The ADC is used to sample the VSPAD voltage to determine the actual value of the VSPAD voltage, that is, the actual driving voltage of the SPAD.

[0087] The application layer can include a series of application packages. Figure 1 As shown, the application package may include a camera application. In addition, the application package may also include instant messaging, calling, gallery, map, navigation, WLAN, Bluetooth, music, video, short message and other applications, which are not limited in the present application embodiment.

[0088] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. As an example of this application, Figure 1 As shown, the application framework layer may include a camera service (CameraService). The camera service is used to monitor whether the camera application is triggered to collect images, and when the camera application is triggered and collects images, the hardware abstraction layer and the kernel layer notify the dTOF sensor of the hardware layer to collect images. dTOF sensor node, image front-end lightweight node (IFE lite node), depth node (depth node) and auto focus node (AF node) and dTOF algorithm.

[0089] The kernel layer is the layer between hardware and software. As an example, Figure 1 As shown, the kernel layer may include a camera driver. In addition, the kernel layer may also include at least a display driver, an audio driver, a sensor driver, etc., which is not limited in the embodiments of the present application.

[0090] In the embodiment of the present application, as an example, the interaction process between the software system and the hardware layer of the embodiment of the present application and the interaction process between the layers of the software system are as follows: when the camera application is triggered (such as obtaining the user's trigger operation on the camera application icon, the trigger operation on the shooting control, etc.), the camera application of the application layer sends an instruction to the camera service of the application framework layer, and then the camera service sends an instruction to the dTOF sensor node of the hardware abstraction layer, and then the dTOF sensor node sends an instruction to the camera driver of the kernel layer, so that the camera driver starts streaming (streamOn) to the dTOF sensor of the hardware layer to drive the dTOF sensor to collect raw image data; thereafter, when the dTOF sensor collects a frame of raw image data and outputs the frame, it sends an SOF interrupt and raw image data to the camera driver; the camera driver sends the SOF interrupt to the dTOF sensor node, and when the dTOF sensor node receives the SOF interrupt, it creates an ADC thread through ReadRequest to The VSPAD voltage is sampled through the ADC in the camera driver hardware layer to obtain the current ADC sampling value of the SPAD, and the current ADC sampling value of the SPAD collected is obtained through the camera driver in the ADC thread; then, the dTOF sensor node publishes the ADC sampling value and raw image data received from the camera driver to the depth node through the image front-end lightweight node; then, the depth node sends the received ADC sampling value and raw image data to the dTOF algorithm; finally, the dTOF algorithm determines the current actual driving voltage of the SPAD based on the ADC sampling value, and parses the raw image data to determine the current expected driving voltage of the SPAD, and generates an original depth map based on the raw image data, and then performs depth compensation on the original depth map based on the current actual driving voltage and expected driving voltage of the SPAD to generate a target depth map; then, the target depth map is sent to the depth node, and the depth node sends the target depth map to the autofocus node to use the target depth map for autofocus.

[0091] The depth compensation method, device, electronic device, storage medium and computer program of the dTOF sensor provided by the present application are described in detail below with reference to the accompanying drawings.

[0092] In the above Figure 1 Based on the electronic device shown, the depth compensation method of the dTOF sensor provided by the embodiment of the present application is introduced below. Figure 2 , Figure 2 is a flow chart of a depth compensation method of a dTOF sensor provided in an embodiment of the present application. As an example but not a limitation, the method can be Figure 1 The electronic device shown in the figure is executed, and the electronic device is implemented by the interaction of the above modules. The method may include part or all of the following contents:

[0093] Step 201 : in response to a trigger operation on a camera application, driving a dTOF sensor to collect raw image data.

[0094] During the use of electronic devices, in some application scenarios, users need the electronic device to be able to automatically detect the distance between the target object and the target object in its detection sensing area, and control certain components of the electronic device or perform certain functions according to the distance. For example, in a photo-taking scene, the electronic device can perform autofocus processing according to the distance between the target object and the electronic device, so it is necessary to measure the distance through the ranging sensor in the electronic device, and perform autofocus processing according to the ranging result. As an example, the ranging sensor in the electronic device can be a dTOF sensor. The depth compensation method of the dTOF sensor of the embodiment of the present application can be applied to the camera focus scene, and can also be applied to any ranging scene through the dTOF sensor, and the embodiment of the present application does not limit this. In addition, the dTOF sensor of the embodiment of the present application can be either a ranging sensor or an image sensor, so that both ranging and image acquisition can be performed.

[0095] It should be noted that the depth compensation method of the dTOF sensor provided in the embodiment of the present application can also be applied to electronic devices such as mobile phones, sports cameras (GoPro), digital cameras, tablet computers, desktops, laptops, handheld computers, notebook computers, vehicle-mounted equipment, ultra-mobile personal computers (ultra-mobile personal computers, UMPC), netbooks, cellular phones, personal digital assistants (personal digital assistants, PDA), augmented reality (augmented reality, AR) \ virtual reality (virtual reality, VR) devices, wearable devices (such as wearable smart bracelets, etc.), smart home devices (such as smart refrigerators, smart TVs, etc.), and the embodiments of the present application are not limited to this.

[0096] Among them, the first operation can be any operation input by the user in the electronic device that can trigger the autofocus function.

[0097] For example, autofocus is usually performed during the preview process, so the trigger operation can be any operation that can trigger the display of the camera preview interface. For example, the trigger operation can be to trigger the camera application icon in the main interface of the electronic device, trigger the shutter control in the camera operation interface, trigger the end recording control during the camera recording process, trigger the pause recording control during the camera recording process, and so on. For another example, assuming that autofocus can also be performed during the camera recording process, the trigger operation can also be to trigger the recording control in the camera operation interface; or, during the preview or recording process, if the target object in the field of view changes, it will trigger refocusing, and the trigger operation can also be the change of the target object in the field of view.

[0098] It should be noted that the above examples are only exemplary and cannot be regarded as limitations of the present application. In actual use, the type of trigger operation may be related to the functions of the electronic device and the camera application. Therefore, according to actual needs and specific application scenarios, any operation that can trigger autofocus can be set as the trigger operation of the embodiment of the present application, and the embodiment of the present application does not limit this.

[0099] The raw image data may refer to the raw image captured by the dTOF sensor.

[0100] As a possible implementation, Figure 3 As shown, the main interface 300 of the electronic device displays a page with application icons, which may include multiple application icons (for example, weather application icon, calendar application icon, album application icon, note application icon, email application icon, application store application icon, setting application icon, etc.). A page indicator may also be displayed below the above multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple application icons (for example, camera application icon 301, browser application icon, information application icon, dial application icon) below the page indicator. These application icons remain displayed when the page is switched. It can be understood that the camera application icon 301 is an icon of a camera application (i.e., a camera application). The camera application icon 301 can be used to trigger the launch of the camera application.

[0101] The electronic device may detect a trigger operation on the camera application icon 301, and in response to the trigger operation, the electronic device may display a Figure 4 The camera preview interface 400 is shown. The camera preview interface 400 is a shooting interface of the default shooting mode of the camera application, and the user can preview the image and complete the shooting on this interface; and the automatic focusing process can also be performed during the display of the camera preview interface 400.

[0102] like Figure 4As shown, the camera preview interface 400 may include a preview window 401, a camera mode option 402, an album shortcut control 403, a shutter control 404, and a camera flip control.

[0103] The preview window 401 can be used to display a preview image. The preview image displayed in the preview window 401 is an original image captured by the camera of the electronic device based on the viewing range.

[0104] One or more shooting mode options may be displayed in the camera mode option 402. The one or more shooting mode options may include: a night scene mode option, a smart portrait mode option, a photo mode option, a video mode option 4021, and more options. It is understandable that the camera mode option 402 may also include more or fewer shooting mode options.

[0105] The album quick control 403 can be used to open the album application. After the user triggers the electronic device to open the album application through the album quick control 403, the captured images and videos can be viewed. In addition, the album quick control 403 can also display thumbnails of the captured images or videos.

[0106] As a possible implementation, the shutter control 404 can be used to trigger the camera to collect images and complete the photo operation. The electronic device can automatically return to and display the camera preview interface 400 after obtaining the user's trigger operation on the shutter control 404 and completing the photo.

[0107] As a possible implementation, triggering the video mode option 4021 can enter the video mode and display the following: Figure 5 The camera preview interface 500 (video preview interface) shown in FIG. 4 is substantially the same as the controls included in the camera preview interface 400, except that the camera preview interface 500 may further include a video start control 501. The video start control 501 is used to trigger the electronic device to start video recording. After obtaining the user's trigger operation on the video start control 501, the electronic device may display a video start control 501. Figure 6 The recording interface 600 shown in FIG. The recording interface 600 includes an end recording control 601 and a pause recording control 602. The end recording control 601 can be used to end recording, and the pause recording control 602 can be used to pause recording. After obtaining the user's trigger operation on the end recording control 601, the electronic device can end the recording and display the following information: Figure 5 The camera preview interface 500 shown in the figure can also stop recording and display the camera preview interface after obtaining the user's trigger operation on the pause recording control 602.

[0108] It can be understood that the trigger operations mentioned in the embodiments of the present application may include but are not limited to touch (for example, click, etc.), voice control, gesture and other operations, and the present application does not impose any restrictions on this.

[0109] In an embodiment of the present application, after obtaining the trigger operation of the user on the camera application, it can be determined that there is currently a ranging or focusing requirement, so that the dTOF sensor can be driven to emit light pulses and receive echo data corresponding to the emitted light pulses, and then generate original image data, i.e., a raw image, based on the echo data.

[0110] As a possible implementation, the raw image data may include a raw histogram and metadata corresponding to the raw histogram. The raw histogram may refer to a histogram generated based on the echo data received by the SPAD in the dTOF sensor, which can be used to calculate the distance between the electronic device and the object being measured; the metadata may include various parameter information of the dTOF sensor when collecting the raw image data of the frame, such as the operating temperature during collection, the voltage adjustment level corresponding to the SPAD, etc.

[0111] As an example, Figure 7 As shown in the figure, it is a schematic diagram of the structure of a raw image collected by a dTOF sensor provided in an embodiment of the present application, wherein the size of the raw image is 31×2560, the first line is metadata, and the 2nd to 31st lines are original histograms. The metadata may include parameter information such as the operating temperature when the raw image of the frame is collected, the voltage adjustment level (Bvd_current(IDAC)) corresponding to the SPAD, the error flag (errorFlag), and binmode.

[0112] It should be noted that the above examples are only exemplary and cannot be regarded as limitations of the present application. In actual use, the data structure of the raw image collected by the dTOF sensor and the parameter information contained in the metadata can be determined according to actual needs and specific application scenarios, and the embodiments of the present application do not limit this.

[0113] Step 202: Obtain the actual current driving voltage of the SPAD in the dTOF sensor.

[0114] Among them, SPAD refers to a photoelectric detection avalanche diode with single-photon detection capability and has the characteristics of high sensitivity. When the operating voltage is higher than the avalanche breakdown voltage of SPAD, SPAD works in Geiger mode, that is, when a photon arrives, it triggers the avalanche effect of SPAD: due to the high reverse bias electric field inside SPAD, a small amount of electrons generated by photon conversion causes SPAD to produce an avalanche state. At this time, the photoelectric conversion gain is theoretically infinite, generating a digital signal that can be captured by the time to digital converter (TDC), which is recorded as a photon arriving, thereby realizing the detection of the echo corresponding to the emitted light pulse.

[0115] It should be noted that when the dTOF sensor measures distance, the driving voltage of the SPAD needs to be dynamically adjusted as the temperature changes. Figure 8 As shown, it is a technical block diagram of a depth compensation method of a dTOF sensor provided in an embodiment of the present application, wherein the power supply of the SPAD can be a direct current (DC) to direct current power supply (DCDC power supply), and the dTOF sensor adjusts the driving voltage of the SPAD through the IDAC current, that is, adjusts the output voltage VSPAD of the DCDC power supply. For example, the voltage adjustment step corresponding to each IDAC is 128mV. If the SPAD needs a higher voltage due to the increase in temperature, for example, 6 gears need to be added, that is, the VSPAD voltage needs to be increased by 768mV in total. Due to the accuracy error of the IDAC current of the dTOF sensor itself or the output error of the DCDC power supply, the final VSPAD voltage is only increased by 668mV; therefore, the difference of 100mV will affect the parameters such as DCR and PDE of the dTOF sensor, and then affect the depth information collected by the dTOF sensor. For example, the specific impact of the adjustment error of the VSPAD voltage on the depth information can be 0.08mm / mV, that is, the adjustment error of 100mV will cause the depth information to be offset by about 8mm. Therefore, the embodiment of the present application can perform targeted compensation for the depth information offset caused by the adjustment error according to the difference between the actual value and the theoretical value of the SPAD driving voltage, so as to improve the accuracy of the depth information collected by the dTOF sensor and the ranging accuracy of the dTOF sensor.

[0116] The actual driving voltage may refer to the driving voltage corresponding to the SPAD when the dTOF sensor collects raw image data.

[0117] As a possible implementation method, after the dTOF sensor collects and outputs the original image data, that is, after each frame output by the dTOF sensor, the output voltage of the power supply corresponding to the SPAD can be collected, and the collected output voltage of the power supply can be determined as the current actual driving voltage of the SPAD.

[0118] Furthermore, since the output voltage VSPAD of the power supply corresponding to the SPAD is a continuous analog signal, the actual driving voltage can be obtained by sampling, so that the actual instantaneous value of the VSPAD voltage is used as the current actual driving voltage of the SPAD, so as to improve the accuracy of the actual value of the SPAD voltage. That is, in a possible implementation of the embodiment of the present application, the above step 202 may include:

[0119] The output voltage of the power supply corresponding to the SPAD is sampled to obtain the actual driving voltage.

[0120] In an embodiment of the present application, after the dTOF sensor collects the original image data and outputs the frame, the output voltage of the power supply corresponding to the SPAD can be sampled immediately, and the sampled value can be determined as the actual driving voltage.

[0121] Furthermore, the actual current driving voltage of the SPAD can be obtained by ADC sampling, and after the dTOF sensor frames out, an independent thread can be created to collect and obtain ADC sampling values, so as to determine the actual current driving voltage of the SPAD according to the ADC sampling values, so as to further improve the accuracy of the actual value of the SPAD voltage, and further improve the accuracy of the depth compensation of the dTOF sensor. That is, in a possible implementation of the embodiment of the present application, the above step 202 may include:

[0122] When the SOF interrupt sent by the dTOF sensor is obtained, an ADC reading thread is created;

[0123] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain a current ADC sampling value corresponding to the output voltage of the power supply;

[0124] The current ADC sampling value is released, and the actual driving voltage is determined based on the ADC sampling value.

[0125] As a possible implementation method, in the native architecture of the software system of the electronic device, there is no thread or channel for collecting the output voltage of the power supply corresponding to the SPAD. Therefore, Figure 1As shown, after the dTOF sensor collects the raw image data and frames it, it can send a SOF interrupt to the camera driver, and the camera driver sends the SOF interrupt to the dTOF sensor node to notify the dTOF sensor to create an independent ADC thread through ReadRequest, and drive the ADC to perform ADC sampling on the output voltage VSPAD of the power supply through the camera driver in the ADC thread (such as Figure 8 As shown), the current ADC sampling value corresponding to the output voltage of the power supply is collected and returned to the dTOF sensor node through the ADC thread; then, the dTOF sensor node publishes the acquired current ADC sampling value to the depth node; finally, the depth node sends the ADC sampling value to the dTOF algorithm, so that the dTOF algorithm determines the current actual driving voltage of the SPAD based on the current ADC sampling value. For example, the current ADC sampling value can be determined as the current actual driving voltage of the SPAD. Figure 8 As shown, after the dTOF algorithm obtains the current sampling value from the ADC through the ADC thread, the actual driving voltage Vspad_real is determined according to the current sampling value.

[0126] Through the above analysis, the embodiment of the present application creates an ADC reading thread each time the dTOF sensor acquires raw image data and frames it, and performs ADC sampling on the output voltage of the SPAD power supply through the ADC reading thread to obtain the current actual driving voltage of the SPAD, thereby improving the accuracy of the actual driving voltage acquisition, and further improving the accuracy of the depth information compensation, and further improving the ranging accuracy of the dTOF sensor.

[0127] Step 203, obtaining the current expected driving voltage of the SPAD.

[0128] The expected driving voltage may refer to a theoretical value of the SPAD driving voltage determined according to the current operating temperature.

[0129] As a possible implementation method, the dTOF sensor can adjust the driving voltage of the SPAD according to the real-time operating temperature, and after determining the current expected driving voltage of the SPAD according to the current operating temperature, the expected driving voltage can be stored in the currently collected raw image data. Therefore, the raw image data collected by the dTOF sensor can be analyzed and processed to determine the current expected driving voltage of the SPAD.

[0130] Furthermore, since the dTOF sensor can adjust the driving voltage of the SPAD through the IDAC current gear (different IDAC current gears correspond to different voltage adjustment levels), that is, the dTOF sensor can determine the IDAC current gear corresponding to the operating temperature according to the current operating temperature, and then the power supply can adjust the output voltage according to the IDAC current gear received from the dTOF sensor, that is, adjust the driving voltage of the SPAD; therefore, the raw data collected by the dTOF sensor can include the IDAC current gear. That is, in a possible implementation of the embodiment of the present application, the above step 203 may include:

[0131] Analyze and process the raw image data to determine the current voltage regulation level of the SPAD;

[0132] Obtain benchmark data corresponding to SPAD;

[0133] Determine the expected driving voltage based on the voltage regulation level and the reference data.

[0134] The voltage adjustment level may refer to the IDAC current level currently used to adjust the driving voltage of the SPAD.

[0135] The reference data corresponding to the SPAD may refer to a reference value for adjusting the driving voltage of the SPAD determined by calibrating the electronic equipment on the production line. In other words, when adjusting the driving voltage of the SPAD, the driving voltage of the SPAD may be adjusted based on the reference data corresponding to the SPAD.

[0136] As a possible implementation method, when the dTOF sensor adjusts the driving voltage of the SPAD through the IDAC current gear, the current IDAC current gear can be stored as the current voltage adjustment level in the currently collected raw image data, so that the raw image data collected by the dTOF sensor can be analyzed and processed to determine the current voltage adjustment level of the SPAD; after determining the current voltage adjustment level of the SPAD, the expected driving voltage corresponding to the current voltage adjustment level can be calculated based on the pre-calibrated benchmark data.

[0137] Therefore, when the dTOF sensor adjusts the driving voltage of the SPAD through the IDAC current, the current voltage adjustment level can be determined according to the current operating temperature of the SPAD, and then the voltage adjustment is performed based on the pre-calibrated reference data according to the voltage adjustment level, and the dTOF sensor combines the collected raw data with the voltage adjustment level to generate and output the raw image data (i.e., the raw image) each time the distance is measured. Therefore, the raw image output by the dTOF sensor can be analyzed to determine the current voltage adjustment level of the SPAD, and then the current expected driving voltage of the SPAD, i.e., the theoretical value of the current driving voltage of the SPAD, is determined according to the voltage adjustment level and the reference data, thereby ensuring the accuracy of the determination of the theoretical value of the SPAD driving voltage, thereby further improving the accuracy of the depth information compensation, and further improving the ranging accuracy of the dTOF sensor.

[0138] Furthermore, when the original image data includes the original histogram and metadata corresponding to the original histogram, parameter information such as the operating temperature and voltage adjustment level when the original image data is collected can be stored in the metadata. That is, in a possible implementation of the embodiment of the present application, the above-mentioned original image data can include the original histogram and metadata corresponding to the original histogram; accordingly, the above-mentioned analysis and processing of the original image data to determine the current voltage adjustment level of the SPAD can include:

[0139] The metadata is parsed to determine the voltage regulation level.

[0140] As a possible implementation method, every time the dTOF sensor collects a frame of original image data (i.e., raw image), the operating temperature, voltage regulation level and other parameters when the raw image is collected can be merged with the collected original histogram as metadata and then output. Therefore, the metadata in the raw image can be directly parsed to determine the voltage regulation level corresponding to the time when the raw image was collected, that is, to determine the current voltage regulation level of the SPAD, thereby not only improving the accuracy of depth information compensation and the ranging accuracy of dTOF, but also further reducing the calculation complexity of depth compensation.

[0141] As an example, Figure 1 As shown in the figure, after the dTOF sensor in the hardware layer collects the raw image, the raw image can be transmitted to the dTOF algorithm through the camera driver in the kernel layer, the sensor node in the hardware abstraction layer, the image front-end lightweight node, and the depth node. The dTOF algorithm can parse the metadata in the raw image based on the acquired raw image, and parse the metadata to determine the current voltage regulation level of the SPAD. Figure 8As shown, in hardware, the dTOF algorithm can exchange data with the dTOF sensor through the mobile industry processor interface (MIPI) to obtain the raw image collected by the dTOF sensor, and then parse the current voltage regulation level Bvd_current of the SPAD.

[0142] Furthermore, when calibrating the SPAD, the initial voltage value, initial voltage adjustment level, voltage adjustment step length corresponding to each voltage adjustment level, and other data used when adjusting the driving voltage of the SPAD may be calibrated. That is, in a possible implementation of the embodiment of the present application, the above-mentioned reference data may include a reference driving voltage, a reference voltage level, and a reference voltage adjustment step length; accordingly, the above-mentioned determination of the expected driving voltage based on the voltage adjustment level and the reference data may include:

[0143] Determine the current voltage adjustment amplitude of the SPAD according to the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step;

[0144] The expected driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.

[0145] Among them, the reference driving voltage may refer to the initial voltage value used when adjusting the driving voltage of the SPAD; the reference voltage level may refer to the initial voltage level used when adjusting the driving voltage of the SPAD; and the driving voltage value corresponding to the reference voltage level is the reference driving voltage, that is, if the current voltage adjustment level of the SPAD is determined to be the reference voltage level, then the current expected driving voltage of the SPAD can be determined to be the reference driving voltage.

[0146] The reference voltage adjustment step may refer to the voltage value of the SPAD driving voltage when the SPAD driving voltage is adjusted by one voltage adjustment level. That is, if the SPAD driving voltage is increased by one voltage adjustment level, the SPAD driving voltage is increased by one reference voltage adjustment step; if the SPAD driving voltage is reduced by one voltage adjustment level, the SPAD driving voltage is reduced by one reference voltage adjustment step.

[0147] For example, if Fig. 9As shown, it is a schematic diagram of the relationship between a SPAD driving voltage and a voltage adjustment level provided in an embodiment of the present application. Wherein, the horizontal axis represents the voltage adjustment level Bvd (i.e., IDAC gear position) of the SPAD, and the vertical axis represents the driving voltage Vspad of the SPAD, in volts (V), the reference driving voltage is Vspad1, the reference voltage level is Bvd1, and the reference voltage adjustment step is 130mV. If the driving voltage of the SPAD increases by one voltage adjustment level, the driving voltage of the SPAD increases by 130mV; if the driving voltage of the SPAD decreases by one voltage adjustment level, the driving voltage of the SPAD decreases by 130mV. Moreover, it can be seen from the figure that when the operating temperature increases, the driving voltage and voltage adjustment level of the SPAD both increase; when the operating temperature decreases, the driving voltage and voltage adjustment level of the SPAD both decrease.

[0148] As a possible implementation method, the difference between the current voltage adjustment level of the SPAD and the reference voltage level can be determined as the current desired adjustment level number, and the product of the current desired adjustment level number and the reference voltage adjustment step can be determined as the current voltage adjustment amplitude of the SPAD, and then the sum of the reference driving voltage and the current voltage adjustment amplitude of the SPAD can be determined as the desired driving voltage.

[0149] like Figure 8 As shown, the expected driving voltage can be determined by the following formula:

[0150] Vspad_calc=Vspad1+(Bvd_current-Bvd1)×Bvd_LSB

[0151] Among them, Vspad_calc is the expected driving voltage, Vspad1 is the reference driving voltage, Bvd1 is the reference voltage level, Bvd_current is the current voltage adjustment level of SPAD, and Bvd_LSB is the reference voltage adjustment step.

[0152] Through the above analysis, the embodiment of the present application pre-calibrates the reference driving voltage, reference voltage level and reference voltage adjustment step corresponding to each voltage level corresponding to the SPAD, and determines the reference data for adjusting the driving voltage of the SPAD in advance. Thereafter, the desired number of adjustment levels can be determined according to the difference between the real-time voltage adjustment level and the reference voltage level, and the voltage amplitude to be adjusted, that is, the current voltage adjustment amplitude of the SPAD, can be determined according to the desired number of adjustment levels and the reference voltage adjustment step. Finally, the voltage can be adjusted based on the reference driving voltage according to the voltage adjustment amplitude to determine the current desired driving voltage of the SPAD, thereby ensuring the accuracy of the determination of the theoretical value of the SPAD driving voltage, thereby further improving the accuracy of the depth step, and further improving the ranging accuracy of the dTOF sensor.

[0153] Step 204 , performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0154] As a possible implementation method, since the offset of the depth information collected by the dTOF sensor is caused by the error between the actual value and the theoretical value of the SPAD's driving voltage, the strategy for depth compensation of the original image data can be determined according to the difference between the current actual driving voltage of the SPAD and the expected driving voltage, so as to eliminate the depth offset caused by the error between the actual value and the theoretical value of the SPAD's driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor and further improving the ranging accuracy of the dTOF.

[0155] Furthermore, the offset of the depth information caused by the adjustment error of the driving voltage can be quantified according to the difference between the actual value and the theoretical value of the SPAD driving voltage, so as to further improve the accuracy of the depth information compensation. That is, in a possible implementation of the embodiment of the present application, the above step 204 can include:

[0156] Determine the current driving voltage adjustment error of the SPAD according to the difference between the actual driving voltage and the expected driving voltage;

[0157] The original image data is depth compensated according to the driving voltage adjustment error to generate a target depth map.

[0158] As a possible implementation method, the difference between the current actual driving voltage of the SPAD and the expected driving voltage can be determined as the current driving voltage adjustment error of the SPAD. Then, the depth value error corresponding to the driving voltage adjustment error can be determined according to the conversion relationship between the driving voltage adjustment error and the depth value, and then the original image data can be depth compensated according to the depth value error to generate a target depth map.

[0159] Furthermore, the conversion relationship between the voltage regulation error of the SPAD and the depth offset can be pre-calibrated to further improve the accuracy of the depth compensation. That is, in a possible implementation of the embodiment of the present application, the depth compensation of the original image data based on the driving voltage regulation error to generate the target depth map may include:

[0160] Determine the depth value error corresponding to the original image data according to the driving voltage adjustment error and the preset conversion ratio;

[0161] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.

[0162] The preset conversion ratio may refer to the ratio between the pre-calibrated depth value error and the driving voltage adjustment error. For example, if the preset conversion ratio is 0.08 mm / mV, it means that every 1 mV voltage adjustment error will result in a 0.08 mm depth value error.

[0163] As a possible implementation method, the conversion relationship between the voltage adjustment error and the depth offset of the SPAD can be calibrated in advance to determine the conversion ratio between the driving voltage adjustment error and the depth value error as a preset conversion ratio; thus, after determining the current driving voltage adjustment error of the SPAD, the product between the current driving voltage adjustment error of the SPAD and the preset conversion ratio can be determined as the depth value error corresponding to the original image data, and then each depth value corresponding to the original image data can be compensated according to the depth value error to generate a target depth map. For example, after determining the depth value error, for a depth value corresponding to the original image data, the sum of the depth value and the depth value error can be determined as the target depth value corresponding to the depth value; and so on, each target depth value corresponding to the original image data can be determined to generate a target depth map.

[0164] As an example, Figure 8 As shown, the original image data can be depth compensated by the following formula:

[0165] Tof_cali=Tof_Raw+(Vspad_real-Vspad_calc)×ratio_Tof_VSPAD

[0166] Among them, Tof_cali is the target depth map, Tof_Raw is the original depth map corresponding to the original image data, Vspad_real is the current actual driving voltage of SPAD, Vspad_calc is the current expected driving voltage of SPAD, and ratio_Tof_VSPAD is the preset conversion ratio, for example, ratio_Tof_VSPAD=0.08mm / mV.

[0167] For example, if atio_Tof_VSPAD=0.08mm / mV, Vspad_real-Vspad_calc=100mV, that is, the current driving voltage adjustment error of SPAD is 100mV, then it can be determined that the depth value error corresponding to the original image data is 8mm, so 8mm can be compensated for each depth value corresponding to the original image data to generate a target depth map.

[0168] Through the above analysis, the embodiment of the present application improves the accuracy of the depth information collected by the dTOF sensor by compensating for the depth value error caused by the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby further improving the ranging accuracy of the dTOF sensor.

[0169] Furthermore, when the original image data collected by the dTOF sensor includes an original histogram and metadata corresponding to the original histogram, an original depth map can be first generated according to the original histogram, and then depth compensation is performed on the original depth map to further improve the accuracy of depth compensation. That is, in a possible implementation of the embodiment of the present application, the above step 204 may include:

[0170] Generate an original depth map based on the original histogram;

[0171] The original depth map is depth compensated according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0172] As a possible implementation, Figure 1 As shown, the depth node in the hardware abstraction layer can parse the raw image data obtained from the dTOF sensor to determine the original histogram in the raw image data, and then send the original histogram to the dTOF algorithm. The dTOF algorithm can generate an original depth map based on the original histogram, and then perform depth compensation on the original depth map based on the current actual driving voltage and expected driving voltage of the SPAD to generate a target depth map.

[0173] It should be noted that the specific method of performing depth compensation on the original depth map according to the current actual driving voltage and the expected driving voltage of the SPAD may be the same as that in the aforementioned embodiment and will not be described in detail here.

[0174] Through the above analysis, the embodiment of the present application generates an original depth map based on the original histogram collected by the dTOF sensor, and then performs depth compensation on the depth information in the original depth map according to the deviation between the actual driving voltage of the SPAD and the expected driving voltage, thereby further improving the accuracy of the depth compensation and further improving the ranging accuracy of the dTOF.

[0175] Furthermore, when the depth compensation method of the dTOF sensor of the embodiment of the present application is applied to a photo shooting scene, the compensated depth map can be used for autofocus to further improve the autofocus effect. That is, in a possible implementation of the embodiment of the present application, after the above step 204, the following may also be included:

[0176] During the shooting process, focus processing is performed according to the target depth map.

[0177] As a possible implementation method, after compensating the depth information collected by the dTOF sensor, the accurate distance between the object being measured and the electronic device can be determined according to the target depth map. Figure 1 As shown, the dTOF algorithm can send the target depth map generated after compensation to the autofocus node, so that the autofocus node can determine the exact distance between the electronic device and the object under measurement according to the target depth map, and perform autofocus processing according to the accurate distance measurement result, thereby further improving the autofocus effect during shooting, improving the image capture quality, and further improving the user experience.

[0178] The depth compensation method of the dTOF sensor provided in the embodiment of the present application performs depth compensation on the original image data collected by the dTOF sensor according to the current actual driving voltage and expected driving voltage of the SPAD, so as to correct the depth information offset caused by the adjustment error of the SPAD driving voltage through the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor, and further improving the ranging accuracy of the dTOF sensor.

[0179] Fig.10 A timing diagram of a depth compensation method for a dTOF sensor provided in an embodiment of the present application is shown.

[0180] like Fig.10 As shown, the timing diagram of the depth compensation method of the dTOF sensor involves the sensor node in the hardware abstraction layer in the software system, the camera driver in the kernel layer, the dTOF sensor in the hardware layer, the ADC driver in the kernel layer and the depth node in the hardware abstraction layer. Specifically, it includes the following steps:

[0181] S1001: The sensor node of the hardware abstraction layer drives the camera of the core layer to start streaming (streamOn);

[0182] S1002: The camera driver of the kernel layer starts streaming (streamOn) to the dTOF sensor of the hardware layer;

[0183] S1003: The dTOF sensor at the hardware layer sends a SOF interrupt to the camera driver at the kernel layer after the frame is output;

[0184] S1004: The camera driver of the kernel layer notifies the sensor node of the hardware abstraction layer of the SOF interrupt time;

[0185] S1005: The sensor node of the hardware abstraction layer creates an ADC reading process ReadRequest();

[0186] S1006: The sensor node of the hardware abstraction layer reads the ADC sampling value from the camera driver through v4l2_ioctl through the ADC reading process;

[0187] S1007: The camera driver of the kernel layer reads the ADC sampling value from the ADC driver through the ADC reading process;

[0188] S1008: The camera driver of the kernel layer returns the ADC sampling value to the sensor node of the hardware abstraction layer;

[0189] S1009: The sensor node of the hardware abstraction layer publishes the ADC sampling value;

[0190] S1010: The depth node of the hardware abstraction layer obtains the ADC sampling value released by the sensor node through getAdcValue().

[0191] It should be noted that the specific implementation process and principle of each step in the timing diagram can refer to the detailed description of the aforementioned embodiment, which will not be repeated here.

[0192] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0193] Corresponding to the depth compensation method of the dTOF sensor described in the above embodiment, Fig.11 A structural block diagram of a depth compensation device of a dTOF sensor provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0194] Reference Fig.11 , the device 1100, comprises:

[0195] A first acquisition module 1101 is used to drive the dTOF sensor to acquire raw image data in response to a trigger operation on the camera application;

[0196] A first acquisition module 1102 is used to acquire a current actual driving voltage of the SPAD in the dTOF sensor;

[0197] The second acquisition module 1103 is used to acquire the current expected driving voltage of the SPAD;

[0198] The first compensation module 1104 is used to perform depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0199] In actual use, the depth compensation device of the dTOF sensor provided in the embodiment of the present application can be configured in any electronic device to execute the aforementioned depth compensation method of the dTOF sensor.

[0200] The depth compensation device of the dTOF sensor provided in the embodiment of the present application performs depth compensation on the original image data collected by the dTOF sensor according to the current actual driving voltage and expected driving voltage of the SPAD, so as to correct the depth information offset caused by the adjustment error of the SPAD driving voltage through the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor, and further improving the ranging accuracy of the dTOF sensor.

[0201] In a possible implementation of the present application, the apparatus 1100 further includes:

[0202] The focus module is used to perform focus processing according to the target depth map during the shooting process.

[0203] Furthermore, in another possible implementation of the present application, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first compensation module 1104 includes:

[0204] A first generating unit, configured to generate an original depth map according to the original histogram;

[0205] The first compensation unit is used to perform depth compensation on the original depth map according to the actual driving voltage and the expected driving voltage to generate a target depth map.

[0206] Furthermore, in another possible implementation of the present application, the first acquisition module 1102 includes:

[0207] The first sampling unit is used to sample the output voltage of the power supply corresponding to the SPAD to obtain the actual driving voltage.

[0208] Furthermore, in another possible implementation of the present application, the first sampling unit is specifically used to:

[0209] When the SOF interrupt sent by the dTOF sensor is obtained, an ADC reading thread is created;

[0210] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain a current ADC sampling value corresponding to the output voltage of the power supply;

[0211] The current ADC sampling value is released, and the actual driving voltage is determined based on the ADC sampling value.

[0212] Furthermore, in another possible implementation of the present application, the second acquisition module 1103 includes:

[0213] A first determination unit, configured to perform analysis and processing on the original image data to determine a current voltage regulation level of the SPAD;

[0214] A first acquisition unit, used to acquire reference data corresponding to the SPAD;

[0215] The second determining unit is used to determine the expected driving voltage according to the voltage adjustment level and the reference data.

[0216] Further, in another possible implementation of the present application, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first determination unit is specifically configured to:

[0217] The metadata is parsed to determine the voltage regulation level.

[0218] Furthermore, in another possible implementation of the present application, the reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step; accordingly, the second determining unit is specifically configured to:

[0219] Determine the current voltage adjustment amplitude of the SPAD according to the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step;

[0220] The expected driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.

[0221] Furthermore, in another possible implementation of the present application, the first compensation module 1104 further includes:

[0222] A third determining unit, configured to determine a current driving voltage adjustment error of the SPAD according to a difference between the actual driving voltage and the expected driving voltage;

[0223] The second compensation unit is used to perform depth compensation on the original image data according to the driving voltage adjustment error to generate a target depth map.

[0224] Furthermore, in another possible implementation of the present application, the second compensation unit is specifically configured to:

[0225] Determine the depth value error corresponding to the original image data according to the driving voltage adjustment error and the preset conversion ratio;

[0226] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.

[0227] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0228] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0229] In order to implement the above embodiments, the present application also proposes an electronic device.

[0230] Fig.12 A schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0231] See also Fig.12 The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0232] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0233] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0234] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0235] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0236] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0237] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0238] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0239] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is an integer greater than 1.

[0240] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0241] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0242] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0243] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0244] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0245] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, in a shooting scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing. In a possible implementation of the embodiment of the present application, the distance sensor 180F can be a dTOF sensor.

[0246] It should be noted that the implementation process and technical principles of the electronic device of this embodiment refer to the aforementioned explanation of the depth compensation method of the dTOF sensor in the embodiment of the present application, and will not be repeated here.

[0247] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0248] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0249] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0250] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0251] In the above-described embodiments, specific details such as specific system structures, technologies, etc. are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0252] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0253] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0254] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0255] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0256] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0257] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0258] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0259] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A depth compensation method for a direct time-of-flight (dTOF) sensor, characterized in that: include: In response to a trigger operation applied to the camera, driving the dTOF sensor to collect raw image data; Obtaining a current actual driving voltage of a single photon avalanche diode (SPAD) in the dTOF sensor; Obtaining a current desired driving voltage of the SPAD; Depth compensation is performed on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.

2. The method according to claim 1, characterized in that After performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map, the method further includes: During the shooting process, focusing processing is performed according to the target depth map.

3. The method according to claim 1, characterized in that The original image data includes an original histogram and metadata corresponding to the original histogram, and the depth compensation is performed on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map, including: Generate an original depth map according to the original histogram; The original depth map is depth compensated according to the actual driving voltage and the expected driving voltage to generate the target depth map.

4. The method according to claim 1, characterized in that The obtaining the actual current driving voltage of the SPAD in the dTOF sensor includes: The output voltage of the power supply corresponding to the SPAD is sampled to obtain the actual driving voltage.

5. The method according to claim 4, characterized in that The sampling of the output voltage of the power supply corresponding to the SPAD to obtain the actual driving voltage includes: When the frame header SOF interrupt sent by the dTOF sensor is obtained, an analog-to-digital conversion ADC reading thread is created; Performing ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain a current ADC sampling value corresponding to the output voltage of the power supply; The current ADC sampling value is released, and the actual driving voltage is determined according to the ADC sampling value.

6. The method according to claim 1, characterized in that The obtaining of the current expected driving voltage of the SPAD includes: Analyze and process the raw image data to determine the current voltage regulation level of the SPAD; Obtaining benchmark data corresponding to the SPAD; The expected driving voltage is determined according to the voltage regulation level and the reference data.

7. The method according to claim 6, characterized in that The original image data includes an original histogram and metadata corresponding to the original histogram, and the parsing and processing of the original image data to determine the current voltage regulation level of the SPAD includes: The metadata is parsed to determine the voltage regulation level.

8. The method according to claim 6, characterized in that The reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step. The determining the expected driving voltage according to the voltage adjustment level and the reference data includes: Determining a current voltage adjustment amplitude of the SPAD according to a difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step; The expected driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.

9. The method according to any one of claims 1 to 8, characterized in that: The performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map includes: Determining a current driving voltage adjustment error of the SPAD according to a difference between the actual driving voltage and the expected driving voltage; The original image data is depth compensated according to the driving voltage adjustment error to generate the target depth map.

10. The method according to claim 9, characterized in that The step of performing depth compensation on the original image data according to the driving voltage adjustment error to generate the target depth map includes: Determining a depth value error corresponding to the original image data according to the driving voltage adjustment error and a preset conversion ratio; According to the depth value error, depth compensation is performed on the original image data to generate the target depth map.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by an electronic device, the method according to any one of claims 1 to 10 is implemented.

Images