Depth compensation method of direct time of flight (dTOF) sensor and electronic equipment
Patent Information
- Application Number
- CN202480050189.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-31
- Filing Date
- 2024-07-19
- Publication Date
- 2026-03-17
Smart Images
Figure CN121693679A_ABST
Abstract
Description
Depth compensation method and electronic device for direct time-of-flight (dTOF) sensor
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 31, 2023, with application number 202311440945.0 and application name “Depth compensation method and electronic equipment for direct time-of-flight dTOF sensor”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application belongs to the field of terminal technology, and in particular relates to a depth compensation method for a dTOF sensor, an electronic device, and a computer-readable storage medium. Background Art
[0003] When measuring distance using a direct time of flight (dTOF) sensor, the drive voltage (VSPAD) of the single photon avalanche diodes (SPADs) needs to be dynamically adjusted as the temperature changes. dTOF sensors adjust VSPAD voltage using the output current (IDAC current) of a current-mode digital-to-analog converter (IDAC). Different IDAC currents correspond to different voltage adjustment ranges.
[0004] In related technologies, when adjusting the VSPAD voltage, due to the accuracy error of the dTOF sensor's own IDAC current or the output error of the power supply, there may be a deviation between the actual value of the VSPAD voltage after adjustment and the theoretical value, thereby affecting parameters such as the dTOF sensor's dark-count-rate (DCR) and photon detection efficiency (PDE), thereby causing the depth information collected by the dTOF sensor to be offset, affecting the ranging accuracy of the dTOF sensor.
[0005] Summary of the Invention
[0006] Embodiments of the present application provide a depth compensation method, electronic device, and computer-readable storage medium for a dTOF sensor, which can solve the problem that when adjusting the VSPAD voltage, due to the accuracy error of the dTOF sensor's own IDAC current or the output error of the power supply, there may be a deviation between the actual value of the VSPAD voltage after adjustment and the theoretical value, thereby causing the depth information collected by the dTOF sensor to be offset, affecting the ranging accuracy of the dTOF sensor.
[0007] In a first aspect, an embodiment of the present application provides a depth compensation method for a dTOF sensor, comprising: driving the dTOF sensor to collect raw image data in response to a trigger operation applied to a camera; obtaining the current actual driving voltage of the SPAD in the dTOF sensor; obtaining the current expected driving voltage of the SPAD; and performing depth compensation on the raw image data based on the actual driving voltage and the expected driving voltage to generate a target depth map.
[0008] In this way, by performing depth compensation on the original image data collected by the dTOF sensor based on the current actual driving voltage and expected driving voltage of the SPAD, the depth information offset caused by the adjustment error of the SPAD driving voltage can be corrected through the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor and further improving the ranging accuracy of the dTOF sensor.
[0009] In a possible implementation of the first aspect, after performing depth compensation on the original image data based on the actual driving voltage and the expected driving voltage to generate the target depth map, the method further includes:
[0010] During the shooting process, focus processing is performed based on the target depth map.
[0011] In this way, by using the compensated target depth map to perform focus processing during the shooting process, the focus effect of the shooting is improved, thereby improving the image shooting quality and further improving the user experience.
[0012] Optionally, in another possible implementation of the first aspect, the raw image data includes a raw histogram and metadata corresponding to the raw histogram; accordingly, performing depth compensation on the raw image data based on the actual driving voltage and the expected driving voltage to generate a target depth map includes:
[0013] Generate the original depth map according to the original histogram;
[0014] The original depth map is depth compensated according to the actual driving voltage and the expected driving voltage to generate a target depth map.
[0015] In this way, by generating an original depth map based on the original histogram collected by the dTOF sensor, and then performing depth compensation on the depth information in the original depth map based on the deviation between the actual driving voltage of the SPAD and the expected driving voltage, the accuracy of the depth compensation is further improved, thereby further improving the ranging accuracy of the dTOF.
[0016] Optionally, in another possible implementation of the first aspect, obtaining the actual current driving voltage of the SPAD in the dTOF sensor includes:
[0017] The output voltage of the power supply corresponding to the SPAD is sampled to obtain the actual driving voltage.
[0018] Optionally, in yet another possible implementation of the first aspect, sampling the output voltage of the power supply corresponding to the SPAD to obtain the actual driving voltage includes:
[0019] When the start of frame (SOF) interrupt sent by the dTOF sensor is obtained, an analog-to-digital converter (ADC) reading thread is created;
[0020] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain the current ADC sampling value corresponding to the output voltage of the power supply;
[0021] Publish the current ADC sampling value and determine the actual driving voltage based on the ADC sampling value.
[0022] In this way, by creating an ADC reading thread each time the dTOF sensor acquires raw image data and frames it out, and using the ADC reading thread to perform ADC sampling on the output voltage of the SPAD power supply to obtain the current actual driving voltage of the SPAD, the accuracy of the actual driving voltage acquisition is improved, thereby further improving the accuracy of the depth information compensation and further improving the ranging accuracy of the dTOF sensor.
[0023] Optionally, in another possible implementation of the first aspect, obtaining the current expected driving voltage of the SPAD includes:
[0024] Analyze the raw image data to determine the current voltage regulation level of the SPAD;
[0025] Obtain benchmark data corresponding to SPAD;
[0026] Determine the desired driving voltage based on the voltage regulation level and benchmark data.
[0027] In this way, when the dTOF sensor adjusts the driving voltage of the SPAD through the IDAC current, it can determine the current voltage adjustment level according to the current operating temperature of the SPAD, and then perform voltage adjustment based on the pre-calibrated reference data according to the voltage adjustment level, and the dTOF sensor merges the collected raw data with the voltage adjustment level to generate raw image data (i.e., raw image) and output it each time it measures distance. Therefore, the raw image output by the dTOF sensor can be parsed to determine the current voltage adjustment level of the SPAD, and then, based on the voltage adjustment level and the reference data, the current expected driving voltage of the SPAD, i.e., the theoretical value of the current driving voltage of the SPAD, is determined, thereby ensuring the accuracy of the theoretical value of the SPAD driving voltage, thereby further improving the accuracy of the depth information compensation, and further improving the ranging accuracy of the dTOF sensor.
[0028] Optionally, in another possible implementation of the first aspect, the raw image data includes a raw histogram and metadata corresponding to the raw histogram; accordingly, the parsing and processing of the raw image data to determine the current voltage adjustment level of the SPAD includes:
[0029] The metadata is parsed to determine the voltage regulation level.
[0030] In this way, every time the dTOF sensor collects a frame of raw image, it can merge the temperature, voltage adjustment level and other parameters when collecting the raw image as metadata with the collected original histogram and then output it. Therefore, the metadata in the raw image can be directly parsed to determine the voltage adjustment level corresponding to the time when the raw image was collected, that is, to determine the current voltage adjustment level of the SPAD, which not only improves the accuracy of depth information compensation and the ranging accuracy of dTOF, but also further reduces the computational complexity of depth compensation.
[0031] Optionally, in another possible implementation of the first aspect, the reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step; accordingly, determining the expected driving voltage based on the voltage adjustment level and the reference data includes:
[0032] Determine the current voltage adjustment amplitude of the SPAD based on the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step size;
[0033] The desired driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.
[0034] In this way, by pre-calibrating the reference driving voltage, reference voltage level and reference voltage adjustment step corresponding to the SPAD, the reference data for adjusting the driving voltage of the SPAD can be determined in advance. Then, the desired number of adjustment levels can be determined based on the difference between the real-time voltage adjustment level and the reference voltage level. The voltage amplitude that needs to be adjusted, that is, the current voltage adjustment amplitude of the SPAD, can be determined based on the desired number of adjustment levels and the reference voltage adjustment step. Finally, the voltage adjustment can be performed on the basis of the reference driving voltage according to the voltage adjustment amplitude to determine the current desired driving voltage of the SPAD, thereby ensuring the accuracy of the theoretical value of the SPAD driving voltage, thereby further improving the accuracy of the depth step, and further improving the ranging accuracy of the dTOF sensor.
[0035] Optionally, in yet another possible implementation of the first aspect, performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map includes:
[0036] Determine the current driving voltage adjustment error of the SPAD based on the difference between the actual driving voltage and the expected driving voltage;
[0037] The error is adjusted according to the driving voltage and depth compensation is performed on the original image data to generate a target depth map.
[0038] Optionally, in yet another possible implementation of the first aspect, performing depth compensation on the original image data according to the driving voltage adjustment error to generate a target depth map includes:
[0039] Determine the depth value error corresponding to the original image data based on the driving voltage adjustment error and the preset conversion ratio;
[0040] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.
[0041] In this way, by compensating for the depth value error caused by the deviation between the actual value and the theoretical value of the SPAD driving voltage, the accuracy of the depth information collected by the dTOF sensor is improved, and the ranging accuracy of the dTOF sensor is further improved.
[0042] In second aspect, an embodiment of the present application provides a depth compensation device for a dTOF sensor, including: a first acquisition module, used to drive the dTOF sensor to collect raw image data in response to a trigger operation applied to a camera; a first acquisition module, used to obtain the current actual driving voltage of the SPAD in the dTOF sensor; a second acquisition module, used to obtain the current expected driving voltage of the SPAD; a first compensation module, used to perform depth compensation on the raw image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.
[0043] In a possible implementation of the second aspect, the apparatus further includes:
[0044] The focus module is used to perform focus processing according to the target depth map during the shooting process.
[0045] Optionally, in another possible implementation of the second aspect, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first compensation module includes:
[0046] A first generating unit, configured to generate an original depth map according to the original histogram;
[0047] The first compensation unit is configured to perform depth compensation on the original depth map according to the actual driving voltage and the expected driving voltage to generate a target depth map.
[0048] Optionally, in another possible implementation of the second aspect, the first acquisition module includes:
[0049] The first sampling unit is used to sample the output voltage of the power supply corresponding to the SPAD to obtain an actual driving voltage.
[0050] Optionally, in yet another possible implementation of the second aspect, the first sampling unit is specifically configured to:
[0051] When receiving the SOF interrupt sent by the dTOF sensor, create an ADC reading thread;
[0052] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain the current ADC sampling value corresponding to the output voltage of the power supply;
[0053] Publish the current ADC sampling value and determine the actual driving voltage based on the ADC sampling value.
[0054] Optionally, in yet another possible implementation of the second aspect, the second acquisition module includes:
[0055] A first determining unit is configured to analyze and process the original image data to determine a current voltage adjustment level of the SPAD;
[0056] A first acquisition unit, configured to acquire reference data corresponding to the SPAD;
[0057] The second determining unit is configured to determine the expected driving voltage according to the voltage adjustment level and the reference data.
[0058] Optionally, in another possible implementation of the second aspect, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first determining unit is specifically configured to:
[0059] The metadata is parsed to determine the voltage regulation level.
[0060] Optionally, in another possible implementation of the second aspect, the reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step; accordingly, the second determining unit is specifically configured to:
[0061] Determine the current voltage adjustment amplitude of the SPAD based on the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step size;
[0062] The desired driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.
[0063] Optionally, in yet another possible implementation of the second aspect, the first compensation module further includes:
[0064] a third determining unit, configured to determine a current driving voltage adjustment error of the SPAD according to a difference between the actual driving voltage and the expected driving voltage;
[0065] The second compensation unit is configured to perform depth compensation on the original image data according to the driving voltage adjustment error to generate a target depth map.
[0066] Optionally, in yet another possible implementation of the second aspect, the second compensation unit is specifically configured to:
[0067] Determine the depth value error corresponding to the original image data based on the driving voltage adjustment error and the preset conversion ratio;
[0068] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.
[0069] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the depth compensation method for the dTOF sensor as described above.
[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by an electronic device, the depth compensation method of the dTOF sensor as described above is implemented.
[0071] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the depth compensation method of the dTOF sensor as described above.
[0072] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0074] FIG1 is a schematic diagram of the architecture of a software system of an electronic device provided in one embodiment of the present application;
[0075] FIG2 is a schematic flow chart of a depth compensation method for a dTOF sensor provided in an embodiment of the present application;
[0076] FIG3 is a schematic diagram of a main interface of an electronic device provided in one embodiment of the present application;
[0077] FIG4 is a schematic diagram of a camera preview interface of an electronic device provided by an embodiment of the present application;
[0078] FIG5 is a schematic diagram of a camera preview interface of another electronic device provided by an embodiment of the present application;
[0079] FIG6 is a schematic diagram of a recording interface of an electronic device provided in an embodiment of the present application;
[0080] FIG7 is a schematic structural diagram of a raw image collected by a dTOF sensor according to an embodiment of the present application;
[0081] FIG8 is a technical block diagram of a depth compensation method for a dTOF sensor provided in one embodiment of the present application;
[0082] FIG9 is a schematic diagram showing the relationship between a SPAD driving voltage and a voltage adjustment level provided in an embodiment of the present application;
[0083] FIG10 is a timing diagram of a depth compensation method for a dTOF sensor provided in one embodiment of the present application;
[0084] FIG11 is a schematic structural diagram of a depth compensation device for a dTOF sensor according to an embodiment of the present application;
[0085] FIG12 is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0086] Before introducing the method provided in the embodiment of the present application, the software system of the electronic device in the embodiment of the present application is introduced.
[0087] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to exemplify the software system of the electronic device.
[0088] Figure 1 is a block diagram of a software system of an electronic device provided in an embodiment of the present application. Referring to Figure 1, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, namely, the application layer, the application framework layer (framework), the hardware abstraction layer (HAL), and the kernel layer (kernel) from top to bottom. In addition, Figure 1 also shows the relationship between the hardware layer and the software system. As an example of the present application, the hardware layer includes a dTOF sensor and an ADC. The dTOF sensor is used to collect image data. The ADC is used to sample the VSPAD voltage to determine the actual value of the VSPAD voltage, that is, the actual driving voltage of the SPAD.
[0089] The application layer can include a series of application packages. As shown in Figure 1, an application package can include a camera application. In addition, the application package can also include instant messaging, calling, gallery, map, navigation, WLAN, Bluetooth, music, video, short message and other applications, which are not limited in this embodiment of the application.
[0090] The application framework layer provides an application programming interface (API) and a programming framework for the applications of the application layer. The application framework layer includes some predefined functions. As an example of the present application, as shown in Figure 1, the application framework layer may include a camera service (CameraService). Among them, the camera service is used to monitor whether the camera application is triggered to perform image acquisition, and when the camera application is triggered and performs image acquisition, the dTOF sensor of the hardware layer is notified through the hardware abstraction layer and the kernel layer to perform image acquisition. dTOF sensor node, image front-end lightweight node (IFE lite node), depth node (depth node) and autofocus node (AF node) and dTOF algorithm.
[0091] The kernel layer is the layer between hardware and software. As an example, as shown in Figure 1, the kernel layer may include a camera driver. In addition, the kernel layer may also include at least a display driver, an audio driver, a sensor driver, etc., which are not limited in the embodiments of the present application.
[0092] In the embodiment of the present application, as an example, the interaction process between the software system and the hardware layer of the embodiment of the present application and the interaction process between the various layers of the software system are as follows: when the camera application is triggered (such as obtaining the user's trigger operation on the camera application icon, the trigger operation on the shooting control, etc.), the camera application of the application layer sends an instruction to the camera service of the application framework layer, and then the camera service sends an instruction to the dTOF sensor node of the hardware abstraction layer, and then the dTOF sensor node sends an instruction to the camera driver of the kernel layer, so that the camera driver starts streaming (streamOn) to the dTOF sensor of the hardware layer to drive the dTOF sensor to collect raw image data; thereafter, when the dTOF sensor collects a frame of raw image data and outputs the frame, it sends an SOF interrupt and raw image data to the camera driver; the camera driver sends the SOF interrupt to the dTOF sensor node, and when the dTOF sensor node receives the SOF interrupt, it creates an ADC thread through ReadRequest to The ADC in the camera driver hardware layer samples the VSPAD voltage to obtain the current ADC sampling value of the SPAD, and the current ADC sampling value of the SPAD collected is obtained through the camera driver in the ADC thread; then, the dTOF sensor node publishes the ADC sampling value and raw image data received from the camera driver to the depth node through the image front-end lightweight node; then, the depth node sends the received ADC sampling value and raw image data to the dTOF algorithm; finally, the dTOF algorithm determines the current actual driving voltage of the SPAD based on the ADC sampling value, and parses the raw image data to determine the current expected driving voltage of the SPAD, and generates an original depth map based on the raw image data, and then performs depth compensation on the original depth map based on the current actual driving voltage and expected driving voltage of the SPAD to generate a target depth map; then, the target depth map is sent to the depth node, and the depth node sends the target depth map to the autofocus node to use the target depth map for autofocus.
[0093] The depth compensation method, device, electronic device, storage medium and computer program of the dTOF sensor provided in the present application are described in detail below with reference to the accompanying drawings.
[0094] Based on the electronic device shown in FIG1 above, the depth compensation method of the dTOF sensor provided by the embodiment of the present application is described below. Please refer to FIG2, which is a flow chart of the depth compensation method of the dTOF sensor provided by an embodiment of the present application. As an example and not a limitation, the method can be executed by the electronic device shown in FIG1, and the electronic device is implemented by the interaction of the various modules described above. The method may include some or all of the following contents:
[0095] Step 201 : In response to a trigger operation applied to a camera, drive the dTOF sensor to collect raw image data.
[0096] During the use of electronic devices, in some application scenarios, users need the electronic device to be able to automatically detect the distance between the target object and itself within its detection sensing area, and control certain components of the electronic device or perform certain functions based on the distance. For example, in a photo-taking scenario, the electronic device can perform autofocus processing based on the distance between the target object and the electronic device, so it is necessary to measure the distance through the ranging sensor in the electronic device, and perform autofocus processing based on the ranging result. As an example, the ranging sensor in the electronic device may be a dTOF sensor. The depth compensation method of the dTOF sensor in the embodiment of the present application can be applied to the camera focus scene, and can also be applied to any ranging scene by the dTOF sensor, and the embodiment of the present application does not limit this. In addition, the dTOF sensor in the embodiment of the present application can be either a ranging sensor or an image sensor, so that both ranging and image acquisition can be performed.
[0097] It should be noted that the depth compensation method of the dTOF sensor provided in the embodiment of the present application can also be applied to electronic devices such as mobile phones, sports cameras (GoPro), digital cameras, tablet computers, desktop computers, laptops, handheld computers, notebook computers, vehicle-mounted equipment, ultra-mobile personal computers (UMPC), netbooks, cellular phones, personal digital assistants (PDA), augmented reality (AR) and virtual reality (VR) devices, wearable devices (such as wearable smart bracelets, etc.), smart home devices (such as smart refrigerators, smart TVs, etc.), and the embodiment of the present application is not limited to this.
[0098] The first operation may be any operation input by a user in the electronic device that can trigger the autofocus function.
[0099] For example, autofocus is usually performed during the preview process, so the trigger operation can be any operation that can trigger the display of the camera preview interface. For example, the trigger operation can be triggering the camera application icon in the main interface of the electronic device, triggering the shutter control in the camera operation interface, triggering the end recording control during the camera recording process, triggering the pause recording control during the camera recording process, etc. For another example, assuming that autofocus can also be performed during the camera recording process, the trigger operation can also be triggering the recording control in the camera operation interface; or, during the preview or recording process, if the target object in the field of view changes, it will trigger refocusing, and the trigger operation can also be a change in the target object in the field of view.
[0100] It should be noted that the above examples are merely illustrative and should not be construed as limiting the present application. In actual use, the type of trigger operation may be related to the functions of the electronic device and the camera application. Therefore, any operation that can trigger autofocus may be set as a trigger operation in the embodiments of the present application based on actual needs and specific application scenarios, and the embodiments of the present application do not limit this.
[0101] The raw image data may refer to the raw image captured by the dTOF sensor.
[0102] As a possible implementation, as shown in Figure 3, the main interface 300 of the electronic device displays a page with application icons, which may include multiple application icons (for example, a weather application icon, a calendar application icon, an album application icon, a note application icon, an email application icon, an application store application icon, a settings application icon, etc.). A page indicator may also be displayed below the above-mentioned multiple application icons to indicate the positional relationship between the currently displayed page and other pages. There are multiple application icons (for example, a camera application icon 301, a browser application icon, a message application icon, and a dial application icon) below the page indicator. These application icons remain displayed when the page is switched. It can be understood that the camera application icon 301 is an icon for a camera application (i.e., a camera application). The camera application icon 301 can be used to trigger the launch of the camera application.
[0103] The electronic device can detect a trigger operation on the camera application icon 301. In response to the trigger operation, the electronic device can display a camera preview interface 400 as shown in FIG4 . The camera preview interface 400 is the default shooting mode shooting interface of the camera application. The user can preview the image and complete the shooting on this interface; and the autofocus process can also be performed while the camera preview interface 400 is displayed.
[0104] As shown in FIG. 4 , the camera preview interface 400 may include a preview window 401 , a camera mode option 402 , an album shortcut control 403 , a shutter control 404 , and a camera flip control.
[0105] The preview window 401 can be used to display a preview image. The preview image displayed in the preview window 401 is an original image captured by the camera of the electronic device based on the viewing range.
[0106] One or more shooting mode options may be displayed in the camera mode options 402. These one or more shooting mode options may include: a night scene mode option, a smart portrait mode option, a photo mode option, a video mode option 4021, and more options. It is understood that the camera mode options 402 may also include more or fewer shooting mode options.
[0107] The album shortcut control 403 can be used to open the album application. After the user triggers the electronic device to open the album application through the album shortcut control 403, the user can view the images and videos taken. In addition, the album shortcut control 403 can also display thumbnails of the images or videos taken.
[0108] As a possible implementation, the shutter control 404 can be used to trigger the camera to capture an image and complete the photo-taking operation. The electronic device can automatically return to and display the camera preview interface 400 after obtaining the user's triggering operation on the shutter control 404 and completing the photo-taking.
[0109] As one possible implementation, triggering the recording mode option 4021 can enter recording mode and display the camera preview interface 500 (recording preview interface) shown in FIG5 . The controls included in the camera preview interface 500 are substantially the same as those included in the camera preview interface 400 , except that the camera preview interface 500 also includes a recording start control 501 . The recording start control 501 is used to trigger the electronic device to start recording. Upon receiving a user trigger operation on the recording start control 501 , the electronic device can display the recording interface 600 shown in FIG6 . The recording interface 600 includes an end recording control 601 and a pause recording control 602 . The end recording control 601 can be used to end recording, while the pause recording control 602 can be used to pause recording. Upon receiving a user trigger operation on the end recording control 601 , the electronic device can end recording and display the camera preview interface 500 shown in FIG5 . Alternatively, upon receiving a user trigger operation on the pause recording control 602 , the electronic device can pause recording and display the camera preview interface.
[0110] It will be understood that the triggering operations mentioned in the embodiments of the present application may include but are not limited to touch (for example, click, etc.), voice control, gestures and other operations, and the present application does not impose any restrictions on this.
[0111] In an embodiment of the present application, after obtaining the user's trigger operation on the camera application, it can be determined that there is currently a ranging or focusing requirement, so that the dTOF sensor can be driven to emit light pulses and receive echo data corresponding to the emitted light pulses, and then generate original image data, i.e., a raw image, based on the echo data.
[0112] As a possible implementation, the raw image data may include a raw histogram and metadata corresponding to the raw histogram. The raw histogram may refer to a histogram generated based on the echo data received by the SPAD in the dTOF sensor, which can be used to calculate the distance between the electronic device and the object being measured; the metadata may include various parameter information of the dTOF sensor when collecting the raw image data of the frame, such as the operating temperature during collection and the voltage adjustment level corresponding to the SPAD.
[0113] As an example, Figure 7 shows a schematic diagram of the structure of a raw image captured by a dTOF sensor provided in an embodiment of the present application. The raw image is 31×2560 pixels in size, with the first row containing metadata and rows 2–31 containing the original histogram. The metadata may include parameter information such as the operating temperature when the raw image was captured, the voltage regulation level (Bvd_current(IDAC)) corresponding to the SPAD, an error flag (errorFlag), and binmode.
[0114] It should be noted that the above examples are merely illustrative and should not be construed as limiting the present application. In actual use, the data structure of the raw image captured by the dTOF sensor and the parameter information contained in the metadata can be determined based on actual needs and specific application scenarios, and the present application embodiments do not limit this.
[0115] Step 202: Obtain the actual current driving voltage of the SPAD in the dTOF sensor.
[0116] SPAD refers to a photodetector avalanche diode with high sensitivity and single-photon detection capabilities. When the operating voltage is higher than the SPAD's avalanche breakdown voltage, the SPAD operates in Geiger mode. This means that when a photon arrives, it triggers the SPAD's avalanche effect. Due to the high reverse-bias electric field inside the SPAD, the small amount of electrons generated by photon conversion causes the SPAD to enter an avalanche state. At this point, the photoelectric conversion gain is theoretically infinite, generating a digital signal that can be captured by a time-to-digital converter (TDC). This is then recorded as the arrival of a photon, enabling the detection of the echo corresponding to the emitted light pulse.
[0117] It should be noted that when the dTOF sensor measures distance, the SPAD drive voltage needs to be dynamically adjusted as the temperature changes. As shown in Figure 8, a technical block diagram of a depth compensation method for a dTOF sensor provided in an embodiment of the present application is shown, in which the SPAD power supply can be a direct current (DC) to direct current power supply (DCDC power supply). The dTOF sensor adjusts the SPAD drive voltage through the IDAC current, that is, adjusts the output voltage VSPAD of the DCDC power supply. For example, the voltage adjustment step size corresponding to each IDAC is 128mV. If the SPAD requires a higher voltage due to temperature increase, for example, 6 gears need to be added, that is, the VSPAD voltage needs to be increased by 768mV in total. Due to the accuracy error of the dTOF sensor's own IDAC current or the output error of the DCDC power supply, the final VSPAD voltage is only increased by 668mV. Therefore, the difference of 100mV will affect the DCR, PDE and other parameters of the dTOF sensor, thereby affecting the depth information collected by the dTOF sensor. For example, the specific impact of the VSPAD voltage adjustment error on the depth information can be 0.08mm / mV, that is, a 100mV adjustment error will cause the depth information to be offset by about 8mm. Therefore, the embodiment of the present application can perform targeted compensation for the depth information offset caused by the adjustment error based on the difference between the actual value and the theoretical value of the SPAD driving voltage, so as to improve the accuracy of the depth information collected by the dTOF sensor and the ranging accuracy of the dTOF sensor.
[0118] The actual driving voltage may refer to the driving voltage corresponding to the SPAD when the dTOF sensor collects raw image data.
[0119] As a possible implementation method, after the dTOF sensor collects and outputs the original image data, that is, after each frame output by the dTOF sensor, the output voltage of the power supply corresponding to the SPAD can be collected, and the collected output voltage of the power supply can be determined as the current actual driving voltage of the SPAD.
[0120] Furthermore, since the output voltage VSPAD of the power supply corresponding to the SPAD is a continuous analog signal, the actual driving voltage can be obtained by sampling, so that the actual instantaneous value of the VSPAD voltage is used as the current actual driving voltage of the SPAD, thereby improving the accuracy of the actual value of the SPAD voltage. That is, in one possible implementation of the embodiment of the present application, the above step 202 may include:
[0121] The output voltage of the power supply corresponding to the SPAD is sampled to obtain the actual driving voltage.
[0122] In an embodiment of the present application, after the dTOF sensor collects the original image data and outputs the frame, the output voltage of the power supply corresponding to the SPAD can be sampled immediately, and the sampled value can be determined as the actual driving voltage.
[0123] Furthermore, the current actual driving voltage of the SPAD can be obtained by ADC sampling, and after the dTOF sensor outputs a frame, an independent thread can be created to collect and obtain the ADC sampling value to determine the current actual driving voltage of the SPAD based on the ADC sampling value, so as to further improve the accuracy of the actual value of the SPAD voltage, thereby further improving the accuracy of the depth compensation of the dTOF sensor. That is, in a possible implementation of the embodiment of the present application, the above step 202 may include:
[0124] When receiving the SOF interrupt sent by the dTOF sensor, create an ADC reading thread;
[0125] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain the current ADC sampling value corresponding to the output voltage of the power supply;
[0126] Publish the current ADC sampling value and determine the actual driving voltage based on the ADC sampling value.
[0127] As a possible implementation method, in the native architecture of the electronic device's software system, there is no thread or channel for collecting the output voltage of the power supply corresponding to the SPAD. Therefore, as shown in Figure 1, after the dTOF sensor collects the raw image data and frames it, it can send an SOF interrupt to the camera driver, and the camera driver sends the SOF interrupt to the dTOF sensor node to notify the dTOF sensor to create an independent ADC thread through ReadRequest, and in the ADC thread, the camera driver drives the ADC to perform ADC sampling on the output voltage VSPAD of the power supply (as shown in Figure 8). Then, the current ADC sampling value corresponding to the output voltage of the power supply is returned to the dTOF sensor node through the ADC thread; thereafter, the dTOF sensor node publishes the obtained current ADC sampling value to the depth node; finally, the depth node sends the ADC sampling value to the dTOF algorithm, so that the dTOF algorithm determines the current actual driving voltage of the SPAD based on the current ADC sampling value. For example, the current ADC sampling value can be determined as the current actual driving voltage of the SPAD. As shown in Figure 8, after the dTOF algorithm obtains the current sampling value from the ADC through the ADC thread, it determines the actual driving voltage Vspad_real based on the current sampling value.
[0128] Through the above analysis, the embodiment of the present application creates an ADC reading thread each time the dTOF sensor acquires the original image data and frames it out, and performs ADC sampling on the output voltage of the SPAD power supply through the ADC reading thread to obtain the current actual driving voltage of the SPAD, thereby improving the accuracy of the actual driving voltage acquisition, and further improving the accuracy of the depth information compensation, and further improving the ranging accuracy of the dTOF sensor.
[0129] Step 203: Obtain the current expected driving voltage of the SPAD.
[0130] The expected driving voltage may refer to a theoretical value of the SPAD driving voltage determined according to the current operating temperature.
[0131] As a possible implementation method, the dTOF sensor can adjust the driving voltage of the SPAD according to the real-time operating temperature, and after determining the current expected driving voltage of the SPAD based on the current operating temperature, the expected driving voltage can be stored in the currently collected original image data. Therefore, the original image data collected by the dTOF sensor can be analyzed and processed to determine the current expected driving voltage of the SPAD.
[0132] Furthermore, since the dTOF sensor can adjust the driving voltage of the SPAD through the IDAC current gear (different IDAC current gears correspond to different voltage adjustment levels), that is, the dTOF sensor can determine the IDAC current gear corresponding to the operating temperature based on the current operating temperature, and then the power supply can adjust the output voltage according to the IDAC current gear received from the dTOF sensor, that is, adjust the driving voltage of the SPAD; therefore, the raw data collected by the dTOF sensor can include the IDAC current gear. That is, in a possible implementation method of the embodiment of the present application, the above step 203 may include:
[0133] Analyze the raw image data to determine the current voltage regulation level of the SPAD;
[0134] Obtain benchmark data corresponding to SPAD;
[0135] Determine the desired driving voltage based on the voltage regulation level and benchmark data.
[0136] The voltage adjustment level may refer to the IDAC current level currently used to adjust the driving voltage of the SPAD.
[0137] The reference data corresponding to the SPAD may refer to a reference value for adjusting the SPAD's drive voltage, determined by calibrating the electronic device on the production line. In other words, when adjusting the SPAD's drive voltage, the SPAD's drive voltage may be adjusted based on the reference data corresponding to the SPAD.
[0138] As a possible implementation method, when the dTOF sensor adjusts the driving voltage of the SPAD through the IDAC current gear, the current IDAC current gear can be stored as the current voltage adjustment level in the currently collected original image data, so that the original image data collected by the dTOF sensor can be analyzed and processed to determine the current voltage adjustment level of the SPAD; after determining the current voltage adjustment level of the SPAD, the expected driving voltage corresponding to the current voltage adjustment level can be calculated based on the pre-calibrated benchmark data.
[0139] Therefore, when the dTOF sensor adjusts the driving voltage of the SPAD through the IDAC current, it can determine the current voltage adjustment level according to the current operating temperature of the SPAD, and then perform voltage adjustment based on the pre-calibrated reference data according to the voltage adjustment level, and the dTOF sensor merges the collected raw data with the voltage adjustment level to generate raw image data (i.e., raw image) and output it each time it measures distance. Therefore, the raw image output by the dTOF sensor can be parsed to determine the current voltage adjustment level of the SPAD, and then, based on the voltage adjustment level and the reference data, the current expected driving voltage of the SPAD, i.e., the theoretical value of the current driving voltage of the SPAD, is determined, thereby ensuring the accuracy of the theoretical value of the SPAD driving voltage, thereby further improving the accuracy of the depth information compensation, and further improving the ranging accuracy of the dTOF sensor.
[0140] Furthermore, when the raw image data includes a raw histogram and metadata corresponding to the raw histogram, parameter information such as the operating temperature and voltage adjustment level when the raw image data was collected can be stored in the metadata. That is, in one possible implementation of the embodiment of the present application, the raw image data can include a raw histogram and metadata corresponding to the raw histogram; accordingly, the above-mentioned parsing and processing of the raw image data to determine the current voltage adjustment level of the SPAD can include:
[0141] The metadata is parsed to determine the voltage regulation level.
[0142] As a possible implementation method, every time the dTOF sensor collects a frame of original image data (i.e., raw image), the operating temperature, voltage adjustment level and other parameters when the raw image is collected can be merged with the collected original histogram as metadata and then output. Therefore, the metadata in the raw image can be directly parsed to determine the voltage adjustment level corresponding to the time when the raw image was collected, that is, to determine the current voltage adjustment level of the SPAD, thereby not only improving the accuracy of depth information compensation and the ranging accuracy of dTOF, but also further reducing the computational complexity of depth compensation.
[0143] As an example, as shown in Figure 1, after the dTOF sensor in the hardware layer collects the raw image, the raw image can be transmitted to the dTOF algorithm through the camera driver in the kernel layer, the sensor node in the hardware abstraction layer, the image front-end lightweight node, and the depth node. The dTOF algorithm can parse the metadata in the raw image based on the obtained raw image and parse the metadata to determine the current voltage regulation level of the SPAD. As shown in Figure 8, in hardware, the dTOF algorithm can exchange data with the dTOF sensor through the mobile industry processor interface (MIPI) to obtain the raw image collected by the dTOF sensor and then parse the current voltage regulation level Bvd_current of the SPAD.
[0144] Furthermore, when calibrating the SPAD, the initial voltage value, initial voltage adjustment level, voltage adjustment step size corresponding to each voltage adjustment level, and other data used when adjusting the SPAD's driving voltage can be calibrated. That is, in one possible implementation of the embodiment of the present application, the above-mentioned reference data may include a reference driving voltage, a reference voltage level, and a reference voltage adjustment step size; accordingly, the above-mentioned determination of the expected driving voltage based on the voltage adjustment level and the reference data may include:
[0145] Determine the current voltage adjustment amplitude of the SPAD based on the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step size;
[0146] The desired driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.
[0147] Among them, the reference driving voltage may refer to the initial voltage value used when adjusting the driving voltage of the SPAD; the reference voltage level may refer to the initial voltage level used when adjusting the driving voltage of the SPAD; and the driving voltage value corresponding to the reference voltage level is the reference driving voltage, that is, if it is determined that the current voltage adjustment level of the SPAD is the reference voltage level, then it can be determined that the current expected driving voltage of the SPAD is the reference driving voltage.
[0148] The reference voltage adjustment step size may refer to the voltage value by which the SPAD's driving voltage changes when the SPAD's driving voltage is adjusted by one voltage adjustment level. That is, if the SPAD's driving voltage increases by one voltage adjustment level, the SPAD's driving voltage increases by one reference voltage adjustment step size; and if the SPAD's driving voltage decreases by one voltage adjustment level, the SPAD's driving voltage decreases by one reference voltage adjustment step size.
[0149] For example, as shown in Figure 9, a schematic diagram of the relationship between the SPAD driving voltage and the voltage adjustment level provided in an embodiment of the present application is provided. The horizontal axis represents the SPAD voltage adjustment level Bvd (i.e., IDAC gear position), and the vertical axis represents the SPAD driving voltage Vspad, in volts (V). The reference driving voltage is Vspad1, the reference voltage level is Bvd1, and the reference voltage adjustment step is 130mV. If the SPAD driving voltage increases by one voltage adjustment level, the SPAD driving voltage increases by 130mV; if the SPAD driving voltage decreases by one voltage adjustment level, the SPAD driving voltage decreases by 130mV. In addition, it can be seen from the figure that when the operating temperature increases, the SPAD driving voltage and voltage adjustment level both increase; when the operating temperature decreases, the SPAD driving voltage and voltage adjustment level both decrease.
[0150] As a possible implementation method, the difference between the current voltage adjustment level of the SPAD and the reference voltage level can be determined as the current desired adjustment level number, and the product of the current desired adjustment level number and the reference voltage adjustment step can be determined as the current voltage adjustment amplitude of the SPAD, and then the sum of the reference driving voltage and the current voltage adjustment amplitude of the SPAD can be determined as the desired driving voltage.
[0151] As shown in FIG8 , the desired driving voltage can be determined by the following formula: Vspad_calc=Vspad1+(Bvd_current−Bvd1)×Bvd_LSB
[0152] Among them, Vspad_calc is the expected driving voltage, Vspad1 is the reference driving voltage, Bvd1 is the reference voltage level, Bvd_current is the current voltage adjustment level of SPAD, and Bvd_LSB is the reference voltage adjustment step.
[0153] Through the above analysis, the embodiment of the present application pre-calibrates the reference driving voltage, reference voltage level and reference voltage adjustment step corresponding to the SPAD, and determines the reference data for adjusting the driving voltage of the SPAD in advance. Then, the desired number of adjustment levels can be determined according to the difference between the real-time voltage adjustment level and the reference voltage level, and the voltage amplitude to be adjusted can be determined according to the desired number of adjustment levels and the reference voltage adjustment step, that is, the current voltage adjustment amplitude of the SPAD. Finally, the voltage adjustment can be performed on the basis of the reference driving voltage according to the voltage adjustment amplitude to determine the current desired driving voltage of the SPAD, thereby ensuring the accuracy of the determination of the theoretical value of the SPAD driving voltage, thereby further improving the accuracy of the depth step, and further improving the ranging accuracy of the dTOF sensor.
[0154] Step 204 : Perform depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.
[0155] As a possible implementation method, since the offset of the depth information collected by the dTOF sensor is caused by the error between the actual value and the theoretical value of the SPAD's driving voltage, the strategy for depth compensation of the original image data can be determined based on the difference between the current actual driving voltage of the SPAD and the expected driving voltage to eliminate the depth offset caused by the error between the actual value and the theoretical value of the SPAD's driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor and further improving the ranging accuracy of the dTOF.
[0156] Furthermore, the offset of the depth information caused by the adjustment error of the driving voltage can be quantified based on the difference between the actual value and the theoretical value of the SPAD driving voltage, so as to further improve the accuracy of the depth information compensation. That is, in a possible implementation of the embodiment of the present application, the above step 204 can include:
[0157] Determine the current driving voltage adjustment error of the SPAD based on the difference between the actual driving voltage and the expected driving voltage;
[0158] The error is adjusted according to the driving voltage and depth compensation is performed on the original image data to generate a target depth map.
[0159] As a possible implementation method, the difference between the current actual driving voltage of the SPAD and the expected driving voltage can be determined as the current driving voltage adjustment error of the SPAD. Then, the depth value error corresponding to the driving voltage adjustment error can be determined based on the conversion relationship between the driving voltage adjustment error and the depth value, and then the original image data can be depth compensated based on the depth value error to generate a target depth map.
[0160] Furthermore, the conversion relationship between the SPAD voltage regulation error and the depth offset can be pre-calibrated to further improve the accuracy of depth compensation. That is, in one possible implementation of the embodiment of the present application, the depth compensation of the original image data based on the driving voltage regulation error to generate the target depth map can include:
[0161] Determine the depth value error corresponding to the original image data based on the driving voltage adjustment error and the preset conversion ratio;
[0162] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.
[0163] The preset conversion ratio may refer to a ratio between a pre-calibrated depth error and a drive voltage adjustment error. For example, a preset conversion ratio of 0.08 mm / mV may indicate that a voltage adjustment error of 1 mV will result in a depth error of 0.08 mm.
[0164] As a possible implementation method, the conversion relationship between the voltage adjustment error and the depth offset of the SPAD can be calibrated in advance to determine the conversion ratio between the driving voltage adjustment error and the depth value error as a preset conversion ratio; thus, after determining the current driving voltage adjustment error of the SPAD, the product between the current driving voltage adjustment error of the SPAD and the preset conversion ratio can be determined as the depth value error corresponding to the original image data, and then each depth value corresponding to the original image data can be compensated according to the depth value error to generate a target depth map. For example, after determining the depth value error, for a depth value corresponding to the original image data, the sum of the depth value and the depth value error can be determined as the target depth value corresponding to the depth value; and so on, each target depth value corresponding to the original image data can be determined to generate a target depth map.
[0165] As an example, as shown in FIG8 , depth compensation can be performed on the original image data using the following formula: Tof_cali=Tof_Raw+(Vspad_real−Vspad_calc)×ratio_Tof_VSPAD
[0166] Among them, Tof_cali is the target depth map, Tof_Raw is the original depth map corresponding to the original image data, Vspad_real is the current actual driving voltage of SPAD, Vspad_calc is the current expected driving voltage of SPAD, and ratio_Tof_VSPAD is the preset conversion ratio, for example, ratio_Tof_VSPAD = 0.08mm / mV.
[0167] For example, if atio_Tof_VSPAD = 0.08mm / mV, Vspad_real - Vspad_calc = 100mV, that is, the current driving voltage adjustment error of the SPAD is 100mV, then it can be determined that the depth value error corresponding to the original image data is 8mm, so that 8mm can be compensated for each depth value corresponding to the original image data to generate a target depth map.
[0168] Through the above analysis, the embodiment of the present application improves the accuracy of the depth information collected by the dTOF sensor by compensating for the depth value error caused by the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby further improving the ranging accuracy of the dTOF sensor.
[0169] Furthermore, when the raw image data collected by the dTOF sensor includes a raw histogram and metadata corresponding to the raw histogram, an original depth map can be first generated based on the raw histogram, and then depth compensation can be performed on the original depth map to further improve the accuracy of depth compensation. That is, in a possible implementation of the embodiment of the present application, the above step 204 may include:
[0170] Generate the original depth map according to the original histogram;
[0171] The original depth map is depth compensated according to the actual driving voltage and the expected driving voltage to generate a target depth map.
[0172] As a possible implementation method, as shown in Figure 1, the depth node in the hardware abstraction layer can parse and process the raw image data obtained from the dTOF sensor to determine the original histogram in the raw image data, and then send the original histogram to the dTOF algorithm. The dTOF algorithm can generate an original depth map based on the original histogram, and then perform depth compensation on the original depth map based on the current actual driving voltage and the expected driving voltage of the SPAD to generate a target depth map.
[0173] It should be noted that the specific method of performing depth compensation on the original depth map based on the current actual driving voltage and the expected driving voltage of the SPAD can be the same as that in the aforementioned embodiment and will not be repeated here.
[0174] Through the above analysis, the embodiment of the present application generates an original depth map based on the original histogram collected by the dTOF sensor, and then performs depth compensation on the depth information in the original depth map according to the deviation between the actual driving voltage and the expected driving voltage of the SPAD, thereby further improving the accuracy of the depth compensation and further improving the ranging accuracy of the dTOF.
[0175] Furthermore, when the depth compensation method of the dTOF sensor of the embodiment of the present application is applied to a photo shooting scene, the compensated depth map can be used for autofocus to further improve the autofocus effect. That is, in a possible implementation of the embodiment of the present application, after the above step 204, the following steps may also be included:
[0176] During the shooting process, focus processing is performed based on the target depth map.
[0177] As a possible implementation method, after compensating the depth information collected by the dTOF sensor, the exact distance between the object being measured and the electronic device can be determined based on the target depth map. Therefore, as shown in Figure 1, the dTOF algorithm can send the target depth map generated after compensation to the autofocus node, so that the autofocus node can determine the exact distance between the electronic device and the object being measured based on the target depth map, and perform autofocus processing based on the accurate distance measurement results, thereby further improving the autofocus effect during shooting, improving the image shooting quality, and further improving the user experience.
[0178] The depth compensation method of the dTOF sensor provided in the embodiment of the present application performs depth compensation on the original image data collected by the dTOF sensor according to the current actual driving voltage and the expected driving voltage of the SPAD, so as to correct the depth information offset caused by the adjustment error of the SPAD driving voltage through the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor and further improving the ranging accuracy of the dTOF sensor.
[0179] FIG10 shows a timing diagram of a depth compensation method for a dTOF sensor provided in an embodiment of the present application.
[0180] As shown in Figure 10, the timing diagram of the depth compensation method for the dTOF sensor involves the sensor node in the hardware abstraction layer of the software system, the camera driver in the kernel layer, the dTOF sensor in the hardware layer, the ADC driver in the kernel layer, and the depth node in the hardware abstraction layer. Specifically, it includes the following steps:
[0181] S1001: The sensor node of the hardware abstraction layer drives the camera of the core layer to start streaming (streamOn);
[0182] S1002: The camera driver of the kernel layer starts streaming (streamOn) to the dTOF sensor of the hardware layer;
[0183] S1003: The dTOF sensor at the hardware layer sends an SOF interrupt to the camera driver at the kernel layer after generating a frame.
[0184] S1004: The camera driver of the kernel layer notifies the sensor node of the hardware abstraction layer of the SOF interrupt time;
[0185] S1005: The sensor node of the hardware abstraction layer creates an ADC reading process ReadRequest();
[0186] S1006: The sensor node of the hardware abstraction layer reads the ADC sampling value from the camera driver through the ADC reading process via v4l2_ioctl;
[0187] S1007: The camera driver of the kernel layer reads the ADC sampling value from the ADC driver through the ADC reading process;
[0188] S1008: The camera driver of the kernel layer returns the ADC sampling value to the sensor node of the hardware abstraction layer;
[0189] S1009: The sensor node of the hardware abstraction layer publishes the ADC sampling value;
[0190] S1010: The depth node of the hardware abstraction layer obtains the ADC sampling value released by the sensor node through getAdcValue().
[0191] It should be noted that the specific implementation process and principle of each step in the timing diagram can be referred to the detailed description of the above embodiment and will not be repeated here.
[0192] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0193] Corresponding to the depth compensation method of the dTOF sensor described in the above embodiment, Figure 11 shows a structural block diagram of the depth compensation device of the dTOF sensor provided in the embodiment of the present application. For the sake of convenience, only the parts related to the embodiment of the present application are shown.
[0194] 11 , the apparatus 1100 includes:
[0195] A first acquisition module 1101 is configured to drive the dTOF sensor to acquire raw image data in response to a trigger operation applied to the camera;
[0196] A first acquisition module 1102 is configured to acquire a current actual driving voltage of the SPAD in the dTOF sensor;
[0197] The second acquisition module 1103 is used to obtain the current expected driving voltage of the SPAD;
[0198] The first compensation module 1104 is configured to perform depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.
[0199] In actual use, the depth compensation device for a dTOF sensor provided in the embodiment of the present application can be configured in any electronic device to execute the aforementioned depth compensation method for a dTOF sensor.
[0200] The depth compensation device of the dTOF sensor provided in the embodiment of the present application performs depth compensation on the original image data collected by the dTOF sensor according to the current actual driving voltage and the expected driving voltage of the SPAD, so as to correct the depth information offset caused by the adjustment error of the SPAD driving voltage through the deviation between the actual value and the theoretical value of the SPAD driving voltage, thereby improving the accuracy of the depth information collected by the dTOF sensor and further improving the ranging accuracy of the dTOF sensor.
[0201] In a possible implementation of the present application, the apparatus 1100 further includes:
[0202] The focus module is used to perform focus processing according to the target depth map during the shooting process.
[0203] Furthermore, in another possible implementation of the present application, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first compensation module 1104 includes:
[0204] A first generating unit, configured to generate an original depth map according to the original histogram;
[0205] The first compensation unit is configured to perform depth compensation on the original depth map according to the actual driving voltage and the expected driving voltage to generate a target depth map.
[0206] Furthermore, in another possible implementation of the present application, the first obtaining module 1102 includes:
[0207] The first sampling unit is used to sample the output voltage of the power supply corresponding to the SPAD to obtain an actual driving voltage.
[0208] Furthermore, in another possible implementation of the present application, the first sampling unit is specifically configured to:
[0209] When receiving the SOF interrupt sent by the dTOF sensor, create an ADC reading thread;
[0210] Perform ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain the current ADC sampling value corresponding to the output voltage of the power supply;
[0211] Publish the current ADC sampling value and determine the actual driving voltage based on the ADC sampling value.
[0212] Furthermore, in another possible implementation of the present application, the second obtaining module 1103 includes:
[0213] A first determining unit is configured to analyze and process the original image data to determine a current voltage adjustment level of the SPAD;
[0214] A first acquisition unit, configured to acquire reference data corresponding to the SPAD;
[0215] The second determining unit is configured to determine the expected driving voltage according to the voltage adjustment level and the reference data.
[0216] Furthermore, in another possible implementation of the present application, the original image data includes an original histogram and metadata corresponding to the original histogram; accordingly, the first determining unit is specifically configured to:
[0217] The metadata is parsed to determine the voltage regulation level.
[0218] Furthermore, in another possible implementation of the present application, the reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step; accordingly, the second determining unit is specifically configured to:
[0219] Determine the current voltage adjustment amplitude of the SPAD based on the difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step size;
[0220] The desired driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.
[0221] Furthermore, in another possible implementation of the present application, the first compensation module 1104 further includes:
[0222] a third determining unit, configured to determine a current driving voltage adjustment error of the SPAD according to a difference between the actual driving voltage and the expected driving voltage;
[0223] The second compensation unit is configured to perform depth compensation on the original image data according to the driving voltage adjustment error to generate a target depth map.
[0224] Furthermore, in another possible implementation of the present application, the second compensation unit is specifically configured to:
[0225] Determine the depth value error corresponding to the original image data based on the driving voltage adjustment error and the preset conversion ratio;
[0226] According to the depth value error, depth compensation is performed on the original image data to generate a target depth map.
[0227] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.
[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0229] In order to implement the above embodiments, the present application also proposes an electronic device.
[0230] FIG12 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
[0231] 12 , the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0232] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0233] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.
[0234] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0235] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.
[0236] In some embodiments, the processor 110 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0237] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.
[0238] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.
[0239] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is an integer greater than one.
[0240] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
[0241] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.
[0242] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0243] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.
[0244] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created by the electronic device 100 during use (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0245] Distance sensor 180F is used to measure distance. Electronic device 100 can measure distance using infrared or laser. In some embodiments, in a shooting scenario, electronic device 100 can use distance sensor 180F to measure distance to achieve fast focusing. In one possible implementation of the embodiment of the present application, distance sensor 180F can be a dTOF sensor.
[0246] It should be noted that the implementation process and technical principles of the electronic device of this embodiment can be found in the aforementioned explanation of the depth compensation method of the dTOF sensor of the embodiment of the present application, and will not be repeated here.
[0247] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.
[0248] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0249] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0250] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.
[0251] In the above embodiments, specific details such as specific system structures and technologies are provided for illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present application with unnecessary details.
[0252] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.
[0253] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0254] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.
[0255] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0256] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.
[0257] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0258] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0259] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. The above-mentioned embodiments are only used to illustrate the technical scheme of the present application, rather than to limit it; although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical schemes recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not deviate the essence of the corresponding technical schemes from the spirit and scope of the technical schemes of the various embodiments of the present application, and should all be included in the scope of protection of the present application.
Claims
1. A depth compensation method for a direct time-of-flight (dTOF) sensor, characterized in that: include: In response to a trigger operation applied to the camera, driving the dTOF sensor to collect raw image data; Obtaining a current actual driving voltage of a single photon avalanche diode (SPAD) in the dTOF sensor; Obtaining a current desired driving voltage of the SPAD; Depth compensation is performed on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map.
2. The method according to claim 1, characterized in that After performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map, the method further includes: During the shooting process, focusing processing is performed according to the target depth map.
3. The method according to claim 1, characterized in that The original image data includes an original histogram and metadata corresponding to the original histogram, and the depth compensation is performed on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map, including: Generate an original depth map according to the original histogram; The original depth map is depth compensated according to the actual driving voltage and the expected driving voltage to generate the target depth map.
4. The method according to claim 1, characterized in that The obtaining the actual current driving voltage of the SPAD in the dTOF sensor includes: The output voltage of the power supply corresponding to the SPAD is sampled to obtain the actual driving voltage.
5. The method according to claim 4, characterized in that The sampling of the output voltage of the power supply corresponding to the SPAD to obtain the actual driving voltage includes: When the frame header SOF interrupt sent by the dTOF sensor is obtained, an analog-to-digital conversion ADC reading thread is created; Performing ADC sampling on the output voltage of the power supply through the ADC reading thread to obtain a current ADC sampling value corresponding to the output voltage of the power supply; The current ADC sampling value is released, and the actual driving voltage is determined according to the ADC sampling value.
6. The method according to claim 1, characterized in that The obtaining of the current expected driving voltage of the SPAD includes: Analyze and process the raw image data to determine the current voltage regulation level of the SPAD; Obtaining benchmark data corresponding to the SPAD; The expected driving voltage is determined according to the voltage regulation level and the reference data.
7. The method according to claim 6, characterized in that The original image data includes an original histogram and metadata corresponding to the original histogram, and the parsing and processing of the original image data to determine the current voltage regulation level of the SPAD includes: The metadata is parsed to determine the voltage regulation level.
8. The method according to claim 6, characterized in that The reference data includes a reference driving voltage, a reference voltage level, and a reference voltage adjustment step. The determining the expected driving voltage according to the voltage adjustment level and the reference data includes: Determining a current voltage adjustment amplitude of the SPAD according to a difference between the voltage adjustment level and the reference voltage level and the reference voltage adjustment step; The expected driving voltage is determined according to the reference driving voltage and the voltage adjustment amplitude.
9. The method according to any one of claims 1 to 8, characterized in that: The performing depth compensation on the original image data according to the actual driving voltage and the expected driving voltage to generate a target depth map includes: Determining a current driving voltage adjustment error of the SPAD according to a difference between the actual driving voltage and the expected driving voltage; The original image data is depth compensated according to the driving voltage adjustment error to generate the target depth map.
10. The method according to claim 9, characterized in that The step of performing depth compensation on the original image data according to the driving voltage adjustment error to generate the target depth map includes: Determining a depth value error corresponding to the original image data according to the driving voltage adjustment error and a preset conversion ratio; According to the depth value error, depth compensation is performed on the original image data to generate the target depth map.
11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 10.
12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by an electronic device, the method according to any one of claims 1 to 10 is implemented.