Methods, systems, and electronic devices for reducing power consumption of an iToF camera

Abstract

The application provides a method, system and electronic device for reducing power consumption of an iToF camera. The method for reducing power consumption of an iToF camera comprises the following steps: obtaining a corresponding depth image according to an original phase image collected by the iToF camera; obtaining a phase of the depth image based on a multi-phase method; obtaining a depth of the depth image according to the phase; obtaining gray scale image information of one phase output by the depth image after multi-phase; performing time domain motion detection according to the depth; performing space domain motion detection according to the gray scale image information; obtaining a motion vector according to a result of the time domain motion detection and a result of the space domain motion detection; comparing the motion vector with a preset threshold value, and adjusting an output timing of the depth image to switch the iToF camera to a low power consumption mode when it is determined that the motion vector is less than the preset threshold value. The above technical solution judges the motion state of a target in an image through time domain motion and space domain motion, dynamically adjusts the frame rate of the system in real time, reduces the number of light emitting times, and reduces the power consumption of the system.

Description

Technical Field

[0001] This invention relates to the field of ToF ranging, and more particularly to a method, system, and electronic device for reducing the power consumption of an iToF camera. Background Technology

[0002] Indirect time of flight (iToF) calculates the true depth of an object by measuring the phase delay between emitted light and reflected light received by an image sensor, and it has wide applications in 3D measurement. Specifically, iToF uses continuous pulse modulation and a multi-phase method to demodulate distance information based on the phase, exhibiting good robustness and strong anti-interference characteristics.

[0003] like Figure 1 As shown, the working principle of the iToF camera is as follows: the modulation module 11 controls the light-emitting module 12 to actively emit modulated light signals; the emitted light is emitted onto the surface of the target object 19, and the reflected light signal formed after reflection by the target object 19 is sampled by the photosensitive pixel array 13 of the image sensor; then, the distance of the target object is calculated based on the phase shift of the emitted light and the reflected light. The light-emitting module 12, such as a VCSEL, an infrared emitter, or an LED, is usually driven by a modulated square wave generated by the image sensor to emit modulated light pulses to the target object. Then, the image sensor receives the light emitted back by the object, and the depth information of the measured object is calculated by calculating the round-trip time of light.

[0004] Because iToF cameras require active light pulse emission for ranging, they consume significant power, greatly limiting their application in embedded systems. Depending on the ranging range, the system's dynamic power consumption typically ranges from 20W to 300mW. As the ranging range increases, greater power is needed to maintain good ranging performance, leading to increased dynamic power consumption. This poses a significant challenge to the embedded system's battery and thermal balance.

[0005] US Patent Publication (US20210174474A1) discloses a method that uses a phase image as input to a machine learning component, which performs aliasing based on phase shift information and then calculates depth information. This approach reduces the phase data required to generate depth images through deep learning, thus significantly reducing the power consumption of time-of-flight depth imaging and depth calculation systems. While machine learning effectively reduces the need for a single phase image, it does not significantly improve the dynamic power consumption per unit time. Furthermore, the additional hardware acceleration unit required for machine learning adds a substantial cost to the overall system.

[0006] Japanese patent publication (JP2017223648A) discloses a kind of fusion through RGB image and depth image, according to RGB image determines image position, then according to image position whether change decides whether to carry out the capture of depth image. Through the fusion of 2D and 3D image, can be well according to the change of position information to capture depth information, in some static scene greatly reduces the dynamic power consumption of system, but due to the need of additional RGB image, the introduction of RGB camera, for the cost of system has been greatly improved, the synchronization between two cameras also brings certain challenge.

[0007] Therefore, under the premise of not improving the production cost, finding a kind of method that can reduce the power consumption of iToF camera is the problem to be solved at present. SUMMARY

[0008] The technical problem to be solved by the present application is to find a method for reducing the power consumption of iToF camera, and to provide a method, system and electronic device for reducing the power consumption of iToF camera.

[0009] In order to solve the above problems, the present application provides a method for reducing the power consumption of iToF camera, comprising the following steps: obtaining the corresponding depth image according to the original phase image collected by the iToF camera; obtaining the phase of the depth image based on the multi-phase method; obtaining the depth of the depth image according to the phase; obtaining the gray scale image information of one phase output by the depth image after multi-phase; performing time domain motion detection according to the depth; performing spatial domain motion detection according to the gray scale image information; obtaining motion vector according to the time domain motion detection result and the spatial domain motion detection result; comparing the motion vector with the preset threshold value, and adjusting the output timing of the depth image when it is determined that the motion vector is less than the preset threshold value, so as to switch the iToF camera to low power consumption mode.

[0010] To address the aforementioned issues, this invention provides a system for reducing the power consumption of an iToF camera, comprising: a depth image acquisition module for acquiring a corresponding depth image based on an original phase image captured by the iToF camera; a phase acquisition module for acquiring the phase of the depth image based on a multi-phase method; a depth acquisition module for acquiring the depth of the depth image based on the phase; a grayscale acquisition module for acquiring the grayscale of the grayscale image based on the phase; a temporal motion detection module for performing temporal motion detection and acquiring a temporal motion detection result; a spatial motion detection module for performing spatial motion detection and acquiring a spatial motion detection result; a motion vector acquisition module for acquiring a motion vector based on the temporal motion detection result and the spatial motion detection result; and a processing module for comparing the motion vector with a preset threshold, and adjusting the output timing of the depth image when the motion vector is determined to be less than the preset threshold, thereby switching the iToF camera to a low-power mode.

[0011] To address the aforementioned problems, the present invention provides an electronic device, including a memory, a processor, and a computer-executable program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer-executable program, implements the steps of the above-described method for reducing the power consumption of an iToF camera.

[0012] The above technical solution combines depth image and grayscale image information, judges the motion state of the target in the image through temporal and spatial motion, dynamically adjusts the system frame rate in real time, reduces the number of light emission times, and reduces system power consumption. Attached Figure Description

[0013] Appendix Figure 1 The diagram shown illustrates the working principle of an iToF camera in the prior art.

[0014] Appendix Figure 2 The diagram shown is a flowchart illustrating the steps of a specific implementation of the method for reducing power consumption in an iToF camera according to the present invention.

[0015] Appendix Figure 3 The diagram shown illustrates the sampling process of the original phase image acquired by the iToF camera, which is a specific embodiment of the method for reducing the power consumption of the iToF camera according to the present invention.

[0016] Appendix Figure 4 The figure shown is a temporal depth histogram of a specific implementation of the method for reducing the power consumption of an iToF camera according to the present invention.

[0017] Appendix Figure 5A The figure shown is a timing diagram of the output of a depth image in normal mode.

[0018] Appendix Figure 5BThe output timing diagram of the low-power mode of the embodiment of the method for reducing power consumption of an iToF camera is shown.

[0019] The output timing diagram of the low-power mode of the embodiment of the method for reducing power consumption of an iToF camera is shown. Figure 6 The schematic diagram of an embodiment of the system for reducing power consumption of an iToF camera is shown.

[0020] The output timing diagram of the low-power mode of the embodiment of the method for reducing power consumption of an iToF camera is shown. Figure 7 The schematic diagram of an embodiment of the electronic device is shown. DETAILED DESCRIPTION

[0021] The method, system and electronic device for reducing power consumption of an iToF camera provided by the present application are described in detail below with reference to the accompanying drawings.

[0022] The output timing diagram of the low-power mode of the embodiment of the method for reducing power consumption of an iToF camera is shown. Figure 2 The flowchart of the steps of the embodiment of the method for reducing power consumption of an iToF camera is shown, which includes: step S21, obtaining a corresponding depth image according to an original phase image collected by an iToF camera; step S22, obtaining a phase of the depth image based on a multi-phase method; step S23, obtaining a depth of the depth image according to the phase; step S24, obtaining gray-scale image information of a phase output by the depth image after multi-phase; step S25, performing time-domain motion detection according to the depth; step S26, performing spatial-domain motion detection according to the gray-scale image information; step S27, obtaining a motion vector according to the time-domain motion detection result and the spatial-domain motion detection result; and step S28, comparing the motion vector with a preset threshold value, and adjusting an output timing of the depth image to switch the iToF camera to a low-power mode when it is determined that the motion vector is less than the preset threshold value.

[0023] Referring to step S21, a corresponding depth image is obtained according to an original phase image collected by an iToF camera. As a specific embodiment, the sampling process of the original phase image is as shown in FIG. 2. Figure 3 As shown, the reflected light signal is sampled at equal intervals, and four times per cycle. Phase data Q1 is obtained when the phase delay is 0° (i.e. 0 phase), phase data Q2 is obtained when the phase delay is 90° (i.e. 90 phase), phase data Q3 is obtained when the phase delay is 180° (i.e. 180 phase), and phase data Q4 is obtained when the phase delay is 270° (i.e. 270 phase). In other specific embodiments, it can also be sampled twice or three times per cycle.

[0024] Continuing to refer to step S22, a phase of the depth image is obtained based on a multi-phase method. As a specific embodiment, the step of obtaining the phase of the depth image based on the multi-phase method further includes: calculating the phase of the depth image by using a four-phase method, and the calculation formula is:

[0025] phase = arctan((Q3 - Q4) / (Q1 - Q2))

[0026] wherein Q1 is the phase data when the phase delay is 0°, Q2 is the phase data when the phase delay is 90°, Q3 is the phase data when the phase delay is 180°, and Q4 is the phase data when the phase delay is 270°. In other embodiments, the phase of the depth image can also be calculated by using two-phase method or three-phase method.

[0027] With reference back to step S23, the depth of the depth image is obtained according to the phase. As an embodiment, the step of obtaining the depth of the depth image according to the phase further comprises calculating the depth by using the following formula:

[0028] d = c*phase / (2*f*phase max )+d max *n

[0029] wherein c is the speed of light, phase is the phase obtained based on the multi-phase method, phase max is a phase period (i.e. 2π), f is the modulation frequency of the emitted light, d max = (c / 2)*(1 / f), n is the frame number and n ∈ [0, 1,...]. The modulation frequency f and the frame number n are set by the user, and different modulation frequencies f and frame numbers n can be set according to the requirements of different scenes.

[0030] With reference back to step S24, the gray scale image information of one phase output after the multi-phase of the depth image is obtained. One frame of depth image is calculated by four phases, and the gray scale image information is also calculated by four phases. Generally, considering the influence of ambient light, a gray scale image of one phase is output after four phases as a statistical ambient light. The gray scale image information is used for time domain motion detection.

[0031] With reference back to step S25, time domain motion detection. As an embodiment, the step of time domain motion detection further comprises the following steps: dividing the maximum range of the depth according to a predetermined stack number; counting the total value of the depth contained in each stack in each frame of the depth image, forming a time domain depth histogram, and obtaining a time domain motion detection result.

[0032] Considering the influence of noise, the step of counting the total value of the depth contained in each stack in each frame of the depth image, forming a time domain depth histogram, and obtaining a time domain motion detection result further comprises: calculating the average value avg1, avg2,..., avg mWhere m is the number of stacks divided by the maximum range of the depth; calculate the total value of each stack in the current frame: bin1, bin2, ..., bin m With the average values ​​avg1, avg2, ..., avg m The differences are: diff1 = bin1 - avg1, diff2 = bin2 - avg2, ..., diff m =bin m -avg m The temporal motion detection result sum is obtained using the following formula (i.e., summing all the differences). time As a criterion for judging the motion of a target in the time domain:

[0033] sum time =sum(diff1,diff2,…,diff m ).

[0034] The number of bin values, m, can be any value, but is preferably 32. The larger the bin value, the larger the required RAM cache space, which can be set according to the user's needs.

[0035] Reference Appendix Figure 4 The temporal depth histogram shown has the horizontal axis representing the bin value and the vertical axis representing the number of depth images contained in the current bin value (the number is the coordinate value multiplied by 10). 4 As a specific embodiment, the bin value is 32. Temporal denoising is performed on the depth image using a first-order IIR filter to obtain the temporally denoised depth images of the current frame and the previous n frames. The average values ​​avg1, avg2, ..., avg of each stack in the previous n frames are calculated. 32 And calculate the total values ​​bin1, bin2, ..., bin32 of each stack in the current frame and the average values ​​avg1, avg2, ..., avg 32 The differences bin1, bin2, ..., bin 32 The time-domain motion detection result is obtained by summing all the differences. time =sum(diff1,diff2,…,diff 32 The depth images of the previous n frames compared with the current frame depth image are subjected to IIR filtering and averaging to reduce the impact of noise on image fluctuations.

[0036] Continuing with step S26, spatial motion detection. In one specific implementation, the spatial motion detection step further includes: calculating the sum of squared differences between the current grayscale image and the previous frame grayscale image using the following formula, based on the grayscale image information, to obtain the spatial motion detection result sum. spatialAs the basis for judging the movement of the target in the spatial domain:

[0037] sum spatial =∑((pre (i，j) -cur (i，j) )^2)

[0038] wherein pre (i，j) is the point coordinate of the previous frame of the gray scale image, cur (i，j) is the point coordinate of the current gray scale image, wherein i and j are the size of the sliding window. As a specific embodiment, the size of the sliding window is 5*5.

[0039] With reference back to step S27, the motion vector is obtained according to the time domain motion detection result and the spatial domain motion detection result. The step of calculating the motion vector according to the time domain motion detection result and the spatial domain motion detection result further comprises: calculating the motion vector motion vector using the following formula:

[0040] motion vector = sum time *gain1+sum spatial *gain2

[0041] wherein sum time is the time domain motion detection result, sum spatial is the spatial domain motion detection result, gain1 is the weight of the time domain motion detection result, and gain2 is the weight of the spatial domain motion detection result. Wherein gain1 and gain2 are set by the user and can be adjusted according to the actual scene.

[0042] With reference back to step S28, the motion vector is compared with a preset threshold, and when it is determined that the motion vector is less than the preset threshold, the output timing of the depth image is adjusted to switch the iToF camera to a low power consumption mode.

[0043] The step of adjusting the output timing of the depth image to switch the iToF camera to the low power consumption mode further comprises: outputting a preset frame of gray scale image and a frame of multi-phase plus gray scale image. As a specific embodiment, when the iToF camera is switched to the low power consumption mode, a preset frame of gray scale image is outputted first, and then a frame of four-phase depth image plus one-phase gray scale image (illustrated as 4 frames) is outputted, as shown in the accompanying Figure 5B While in the normal mode, the output timing of the depth image is four-phase depth image plus one-phase gray scale image (illustrated as 2 frames) per frame, as shown in the accompanying Figure 5AAs shown. In other specific embodiments, after outputting a preset frame grayscale image, a frame of two-phase depth image plus a phase grayscale image, or a frame of three-phase depth image plus a phase grayscale image, can also be output. In low-power mode, only the four-phase depth image and one-phase grayscale image of the current frame are output, reducing the output of depth images and lowering power consumption.

[0044] The above technical solution performs motion statistics by using temporal depth map histogram and spatial difference to dynamically switch the depth image frame rate, reduce the number of emission times, and achieve the goal of power reduction; it also reduces the output of four-phase depth images to achieve the goal of power reduction; and it dynamically adjusts the output timing of the depth map based on motion vector and threshold judgment.

[0045] Based on the same inventive concept, the present invention also provides a system for reducing the power consumption of an iToF camera.

[0046] Appendix Figure 6 The diagram shown is a schematic representation of a specific embodiment of the iToF camera power reduction system of the present invention. This embodiment of the iToF camera power reduction system includes: a depth image acquisition module 61, used to acquire a corresponding depth image based on the original phase image captured by the iToF camera; a phase acquisition module 62, used to acquire the phase of the depth image based on a multi-phase method; a depth acquisition module 63, used to acquire the depth of the depth image based on the phase; a grayscale acquisition module 64, used to acquire grayscale image information of one phase output after multi-phase processing of the depth image; a temporal motion detection module 65, used to perform temporal motion detection and acquire temporal motion detection results; a spatial motion detection module 66, used to perform spatial motion detection and acquire spatial motion detection results; a motion vector acquisition module 67, used to acquire motion vectors based on the temporal motion detection results and the spatial motion detection results; and a processing module 68, used to compare the motion vectors with a preset threshold, and when the motion vectors are determined to be less than the preset threshold, adjust the output timing of the depth image to switch the iToF camera to a low-power mode. The working methods of each module can be found in the following references. Figure 2 The descriptions of the corresponding steps in the method for reducing the power consumption of the iToF camera shown are not repeated here.

[0047] In some embodiments, the spatial motion detection module 66 obtains the sum of squared differences between the current grayscale image and the previous grayscale image based on the grayscale image information, so as to obtain the spatial motion detection result as the basis for judging the motion of the target in the spatial domain.

[0048] In some embodiments, when the iToF camera switches to the low-power mode, a preset frame of grayscale image is output first, and then one frame of four-phase depth image and one phase of grayscale image are output. In the low-power mode, only the four-phase depth image and the one phase of grayscale image of the current frame are output, the output of the depth image is reduced, and the power consumption is reduced.

[0049] The technical solution described above performs motion statistics through time-domain depth histogram and spatial difference, performs dynamic switching of the frame rate of the depth image, reduces the number of light emissions, reduces the output of the four-phase depth image, and dynamically adjusts the output timing of the depth image according to the motion vector and the threshold value, so as to achieve the goal of reducing power consumption.

[0050] Based on the same inventive concept, the application further provides an electronic device.

[0051] Attached Figure 7 FIG. 1 shows a schematic diagram of an embodiment of the electronic device described in the application. The electronic device 100 described in the embodiment includes a memory 101, a processor 102, and a computer executable program stored on the memory 101 and executable on the processor 102. When the processor 102 executes the computer executable program, the processor 102 implements the method for reducing the power consumption of the iToF camera as described above. Figure 2 The steps of the method for reducing the power consumption of the iToF camera are described in detail above, and will not be described here again.

[0052] The technical solution described above performs motion statistics through time-domain depth histogram and spatial difference, performs dynamic switching of the frame rate of the depth image, reduces the number of light emissions, reduces the output of the four-phase depth image, and dynamically adjusts the output timing of the depth image according to the motion vector and the threshold value, so as to achieve the goal of reducing power consumption.

[0053] It should be noted that the references in the description to "one embodiment", "an embodiment", "an example embodiment", "some embodiments", and the like indicate that the described embodiment can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted within the knowledge of those skilled in the relevant art that such feature, structure, or characteristic can be implemented in connection with other embodiments whether or not explicitly described.

[0054] Generally, terms can be understood at least partially from their usage in context. For example, the term "one or more," as used herein, depends at least partially on the context and can be used to describe any feature, structure, or characteristic in a singular sense, or in a plural sense, to describe a combination of features, structures, or characteristics. Similarly, terms such as "a," "a," or "the" can also be understood, at least partially on the context, to express either a singular or plural usage. Furthermore, the term "based on" can be understood not necessarily to express an exclusive set of factors, but rather, alternatively, also at least partially on the context, to allow for the presence of other factors that are not necessarily explicitly described. It should also be noted in this specification that "connection / coupling" refers not only to a direct coupling of one component to another, but also to an indirect coupling of one component to another via an intermediate component.

[0055] It should be noted that the terms "comprising" and "having," and their variations, used in this invention document are intended to cover non-exclusive inclusion. The terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, unless explicitly indicated by the context. It should be understood that such data used interchangeably where appropriate. Furthermore, embodiments and features within embodiments of this invention can be combined with each other unless otherwise specified. In addition, descriptions of well-known components and technologies have been omitted in the above description to avoid unnecessarily obscuring the concepts of this invention. In the various embodiments described above, each embodiment focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0056] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for reducing the power consumption of an iToF camera, characterized in that, Includes the following steps: Based on the raw phase image captured by the iToF camera, obtain the corresponding depth image; The phase of the depth image is obtained based on the multi-phase method; The depth of the depth image is obtained based on the phase. Obtain the grayscale image information of one phase output after multi-phase processing of the depth image; Perform temporal motion detection based on the depth; Spatial motion detection is performed based on the grayscale image information; Motion vectors are obtained based on temporal and spatial motion detection results; The motion vector is compared with a preset threshold, and when the motion vector is determined to be less than the preset threshold, the output timing of the depth image is adjusted to switch the iToF camera to a low-power mode.

2. The method according to claim 1, characterized in that, The step of obtaining the phase of the depth image based on the multi-phase method further includes: calculating the phase of the depth image using the four-phase method, and the calculation formula is as follows: Wherein, Q1 is the phase data when the phase delay is 0°, Q2 is the phase data when the phase delay is 90°, Q3 is the phase data when the phase delay is 180°, and Q4 is the phase data when the phase delay is 270°.

3. The method according to claim 1, characterized in that, The step of obtaining the depth of the depth image based on the phase further includes: calculating the depth using the following formula: Where c is the speed of light, phase is the phase obtained based on the multi-phase method, phase_max is one phase period, f is the modulation frequency of the emitted light, d_max = (c / 2)*(1 / f), and n is the preset number of frames and n∈[0,1……].

4. The method according to claim 1, characterized in that, The time-domain motion detection steps further include the following steps: The maximum range of the depth is divided according to the predetermined stack number; The total depth values ​​contained in each stack of each frame of the depth image are counted to form a temporal depth histogram, and the temporal motion detection results are obtained.

5. The method according to claim 4, characterized in that, The step of forming a temporal depth histogram by calculating the total depth values ​​contained in each stack of each frame of the depth image and obtaining the temporal motion detection results further includes: Calculate the average value of the total depth contained in each stack in the first n frames of the depth image, avg_1, avg_2, ..., avg_m, where m is the number of stacks at the maximum range of the depth. Calculate the differences between the total values ​​bin_1, bin_2, ..., bin_m of each stack in the current frame and the average values ​​avg_1, avg_2, ..., avg_m, and the differences dif_1, dif_2, ..., dif_m. The temporal motion detection result sum is obtained using the following formula. time As a criterion for judging the motion of a target in the time domain: sum time =sum(dif_1,dif_2,...,dif_m).

6. The method according to claim 1, characterized in that, The aforementioned airspace motion detection steps further include: Based on the grayscale image information, the sum of squared differences between the current grayscale image and the previous grayscale image is calculated using the following formula to obtain the spatial motion detection result sum. spatial As a basis for judging the motion of a target in the airspace: sum spatiai =∑((pre (i,i) -cur (i,i) )^2) Among them, pre (i，j) The coordinates of the points in the previous grayscale image, cur (i，j) These are the coordinates of a point in the current grayscale image.

7. The method according to claim 1, characterized in that, The step of calculating the motion vector based on the temporal motion detection results and the spatial motion detection results further includes: The motion vector is calculated using the following formula: motion vector =sum time *gain1+sum spatial *gain2 Where, sum time For time-domain motion detection results, sum spatial The result represents the spatial motion detection result, gain1 represents the weight of the temporal motion detection result, and gain2 represents the weight of the spatial motion detection result.

8. The method according to claim 1, characterized in that, The step of adjusting the output timing of the depth image to switch the iToF camera to a low-power mode further includes: Output a preset frame grayscale image and a single frame multi-phase grayscale image.

9. A system for reducing power consumption in an iToF camera, characterized in that, include: The depth image acquisition module is used to acquire the corresponding depth image based on the raw phase image captured by the iToF camera; A phase acquisition module is used to acquire the phase of the depth image based on a multi-phase method; A depth acquisition module is used to acquire the depth of the depth image based on the phase. The grayscale acquisition module is used to acquire the grayscale image information of one phase output by the depth image after multi-phase processing; The temporal motion detection module is used to perform temporal motion detection and obtain temporal motion detection results; The airspace motion detection module is used to perform airspace motion detection and obtain airspace motion detection results; The motion vector acquisition module is used to acquire motion vectors based on temporal domain motion detection results and spatial domain motion detection results. The processing module is used to compare the motion vector with a preset threshold, and when it is determined that the motion vector is less than the preset threshold, adjust the output timing of the depth image to switch the iToF camera to a low-power mode.

10. An electronic device comprising a memory, a processor, and a computer-executable program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer-executable program, it implements the steps of the method for reducing the power consumption of an iToF camera as described in any one of claims 1 to 8.

Images