A self-adjusting method for exposure time and a self-adjusting depth camera for exposure time

Abstract

The present application provides a method for self-adjusting exposure time and a depth camera with self-adjusting exposure time. The method for self-adjusting exposure time includes: projecting a pulse emission light beam to a target in a scene; collecting a pulse reflection light beam reflected by the target in a current frame period and generating a corresponding rawphase diagram, wherein the depth camera includes a collector composed of at least one collection unit, each collection unit includes at least two taps, and the raw value of each raw pixel in the rawphase diagram represents the amount of charge generated by each tap collecting the pulse reflection light beam and / or ambient light; calculating the overexposure ratio according to the number of maximum raw values ​​in the rawphase diagram; adjusting the exposure time of the tap in the next frame period according to the overexposure ratio. The present application reduces or even eliminates the unreliability of the traditional AE algorithm that simply uses the average amplitude of the current exposure time to judge the brightness of the scene, and effectively avoids the problem that the traditional AE algorithm has a slow convergence speed and is very easy to fail to converge when performing multiple iterations.

Description

[Technical field]

[0001] The present application relates to the technical field of depth measurement, and in particular to a method for self-adjusting exposure time and a depth camera with self-adjusting exposure time. [Background technology]

[0002] Depth camera refers to an electronic product that can obtain the depth information of the target in the scene and generate the corresponding depth image accordingly. Its depth detection function can be realized by various 3D (3-Dimension) sensing technologies, such as structured light, monocular / binocular, dTOF (direct Time-of-Flight, direct measurement of light flight time) and iTOF (indirect Time-of-Flight, indirect measurement of light flight time), etc. In the application of iTOF, the AE (Automatic Exposure) algorithm can be used to realize real-time and automatic adjustment of exposure time (based on the changes in brightness of the scene), so that the depth camera can obtain better depth detection effect regardless of the scene, thus making up for the defects of fixed exposure time, such as serious loss of depth in low-reflection scenes and easy overexposure in high-reflection scenes.

[0003] In the related art, the logic of the AE algorithm is: pre-set an empirical optimal amplitude A for the exposure time op and a fixed adjustment rate u, calculate the average amplitude A of the current exposure time nav With the optimal amplitude A op The deviation between n , if the deviation D n The absolute value of is less than the preset threshold D τ , then keep the current exposure time unchanged, otherwise follow the iterative formula T n+1 =T n -u*D n To adjust the exposure time, T n Indicates the current exposure time, T n+1 Indicates the next exposure time. The AE algorithm's adjustment logic for exposure time is relatively simple, but there are also many problems:

[0004] When there is a small and overexposed close-range target in the scene (close-range means the distance between the target and the depth camera is relatively close), since the area of ​​the close-range target is not large, the average amplitude A of the current exposure time is nav It may be small. According to the logic of AE algorithm, the exposure time will be increased. However, in fact, since the close target is overexposed, the exposure time should not be increased but reduced. This shows that the average amplitude A of the current exposure time is not enough. navIt is unreliable to judge the brightness of the scene, which reduces the reliability of automatic exposure time adjustment through the AE algorithm;

[0005] For the iterative formula T in the AE algorithm n+1 =T n -u*D n , the range of its final convergent integration time (i.e., exposure time) is very wide, as long as the average amplitude A of the current exposure time nav With the optimal amplitude A op The deviation between n Does not exceed the preset threshold D τ This will result in inconsistent exposure times when the AE algorithm returns to the same position from different directions in the scene multiple times. This shows that the AE algorithm has no clear definition of the optimal exposure time, and the final convergence is only a suitable exposure time, not the optimal one. Taking the shooting of a white wall as an example, assuming that the depth camera moves from a distance of 300mm to a distance of 500mm from the wall, and the exposure time is t1 at a distance of 300mm from the wall and it is overexposed, the exposure time will be reduced according to the logic of the AE algorithm, for example, to t2; assuming that the depth camera moves from a distance of 500mm to a distance of 300mm from the wall, the exposure time is t2 and it is overexposed. Move from a position 700mm away from the wall to a position 500mm away from the wall. When the distance from the wall is 700mm, the exposure time is t3 and the brightness is dark. According to the logic of the AE algorithm, the exposure time will be increased, for example, to t4. For the above two cases, although the depth camera is finally at a distance of 500mm from the wall, the final converged exposure times t2 and t4 of the two are likely to be different. This is an inevitable defect of the AE algorithm, which is called the inconsistency of the convergence result, that is, when returning to the same position from different directions in the scene multiple times, the final converged exposure time of the AE algorithm is different.

[0006] The convergence speed of the AE algorithm is affected by the preset threshold D τ The impact of the preset threshold D τ The larger the value, the more likely it is that the above-mentioned convergence results will be inconsistent, that is, when returning to the same position from different directions in the scene multiple times, the exposure time at which the AE algorithm finally converges is different and varies greatly. The preset threshold D τ The smaller it is, the longer the AE algorithm needs to converge, and it is very easy to fail to converge (jump back and forth between two exposure times).

[0007] Therefore, it is necessary to improve the process of the above AE algorithm. [Summary of the invention]

[0008] The present application provides a method for self-adjusting exposure time and a depth camera with self-adjusting exposure time, aiming to solve the problems of poor reliability, slow convergence speed and easy convergence failure in automatic exposure time adjustment of AE algorithm in related technologies.

[0009] In order to solve the above-mentioned technical problems existing in the related technology, the first aspect of an embodiment of the present application provides a self-adjusting method for exposure time, which includes: projecting a pulse emission light beam to a target in a scene; collecting a pulse reflection light beam reflected by the target in a current frame period, and generating a corresponding rawphase diagram based on this, wherein the raw value of each raw pixel in the rawphase diagram respectively represents the amount of charge generated by each tap collecting the pulse reflection light beam and / or ambient light; calculating the overexposure ratio according to the number of maximum raw values ​​in the rawphase diagram; and adjusting the exposure time of the tap in the next frame period according to the overexposure ratio.

[0010] In some embodiments, the overexposure ratio is calculated based on the number of maximum raw values ​​in the rawphase map, including: determining the maximum raw value of the raw pixel based on the number of bits of the rawphase map; counting the number of target raw pixels in the rawphase map, wherein the raw value of the target raw pixel is equal to the maximum raw value; dividing the number of target raw pixels in the rawphase map by the total number of raw pixels in the rawphase map to obtain the overexposure ratio.

[0011] In some embodiments, adjusting the exposure time of the tap in the next frame period according to the overexposure ratio includes: comparing the overexposure ratio with a preset overexposure ratio threshold; calculating the optimal exposure time according to the comparison result; and adjusting the exposure time of the tap in the next frame period with reference to the optimal exposure time and the preset exposure time range. Wherein, adjusting the exposure time of the tap in the next frame period with reference to the optimal exposure time and the preset exposure time range includes: if the optimal exposure time is within the preset exposure time range, setting the exposure time of the tap in the next frame period as the optimal exposure time; if the optimal exposure time exceeds the exposure time range, setting the exposure time of the tap in the next frame period as the maximum exposure time or the minimum exposure time defined by the exposure time range. Wherein, calculating the optimal exposure time according to the comparison result includes: if the overexposure ratio is greater than 0 and less than or equal to the overexposure ratio threshold, using the current exposure time as the optimal exposure time; if the overexposure ratio is equal to 0 or greater than the overexposure ratio threshold, generating a corresponding IR map according to the rawphase map; and calculating the optimal exposure time in combination with the rawphase map and the IR map.

[0012] In some embodiments, when the overexposure ratio is greater than the overexposure ratio threshold, the optimal exposure time is calculated in combination with the rawphase map and the IR map, including: dividing the rawphase map into an overexposure area and a non-overexposure area, wherein the raw value of each raw pixel in the overexposure area is the maximum raw value, and the raw value of each raw pixel in the non-overexposure area is less than the maximum raw value; corresponding to the overexposure area, searching for a first maximum IR pixel with the largest IR value among the corresponding IR pixels in the IR map, and corresponding to the first maximum IR pixel, searching for a first minimum raw pixel with the smallest raw value among the corresponding raw pixels in the rawphase map; corresponding to the non-overexposure area, searching for a second maximum IR pixel with the largest IR value among the corresponding IR pixels in the IR map, and corresponding to the second maximum IR pixel, searching for a second minimum raw pixel with the smallest raw value among the corresponding raw pixels in the rawphase map; and calculating the optimal exposure time according to the raw values ​​of the first minimum raw pixel and the second minimum raw pixel. Calculating the optimal exposure time according to the raw values ​​of the first minimum raw pixel and the second minimum raw pixel includes: calculating the optimal exposure time according to the first calculation formula t opt / t in =(B min -Pd) / (A min -Pd) calculates the optimal exposure time, where t opt represents the optimal exposure time, t in represents the current exposure time, Pd represents the dark field value of the collector in the depth camera, A min Indicates the raw value of the first minimum raw pixel, B min Indicates the raw value of the second smallest raw pixel.

[0013] In some embodiments, when the overexposure ratio is equal to 0, the optimal exposure time is calculated in combination with the rawphase map and the IR map, including: searching for the third largest IR pixel with the largest IR value among the IR pixels in the IR map; corresponding to the third largest IR pixel, searching for the largest raw pixel with the largest raw value among the corresponding raw pixels in the rawphase map; and calculating the optimal exposure time according to the raw value of the largest raw pixel. Calculating the optimal exposure time according to the raw value of the largest raw pixel includes: calculating according to the second calculation formula t opt / t in =(M-Pd) / (C max -Pd) calculates the optimal exposure time, where to p t represents the optimal exposure time, t in Indicates the current exposure time, M indicates the maximum raw value, Pd indicates the dark field value of the collector in the depth camera, C max Indicates the raw value of the maximum raw pixel.

[0014] According to a second aspect of an embodiment of the present application, there is provided a depth camera with self-adjusting exposure time, comprising: a transmitter for projecting a pulsed emission light beam to a target in a scene; a collector, comprising at least one collection unit, and each collection unit comprises at least two taps for collecting a pulsed reflection light beam reflected by the target in a current frame period, and generating a corresponding rawphase diagram, wherein the raw value of each raw pixel in the rawphase diagram respectively represents the amount of charge generated by each tap collecting the pulsed reflection light beam and / or ambient light; a control and processor for calculating an overexposure ratio according to the number of maximum raw values ​​in the rawphase diagram, and adjusting the exposure time of the tap in the next frame period according to the overexposure ratio.

[0015] From the above description, it can be seen that compared with the related art, the beneficial effects of the present application are:

[0016] A pulsed emission beam is projected to the target in the scene by the transmitter, and the pulsed reflection beam reflected by the target in the current frame period is collected by the collector, and the corresponding rawphase diagram is generated accordingly. Then, the overexposure ratio can be calculated according to the generated rawphase diagram, and the brightness of the scene is indicated by the overexposure ratio, so that the exposure time of the tap in the next frame period can be adjusted according to the overexposure ratio. For example, when the overexposure ratio exceeds the preset threshold, it means that the current overexposure is serious, and the exposure time needs to be reduced to "just not overexposed" to achieve the optimal exposure time; when the overexposure ratio is equal to zero, it means that there is no overexposure at present, and the exposure time needs to be increased to "just overexposed" to achieve the optimal exposure time; when the overexposure ratio is greater than zero, but does not exceed the preset threshold, it means that the current exposure time is appropriate and does not need to be changed. It can be seen that the present application uses the overexposure ratio of the rawphase graph to judge the brightness of the scene, thereby reducing or even eliminating the unreliability of the traditional AE algorithm that simply uses the average amplitude of the current exposure time to judge the brightness of the scene. In addition, the present application directly adjusts the exposure time of the tap in the next frame period according to the overexposure ratio, which is different from the multiple iterations of the traditional AE algorithm, thereby effectively avoiding the problem of slow convergence speed and easy convergence failure of the traditional AE algorithm during iteration.

Brief Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the related technologies or the embodiments of the present application, the drawings required for use in the description of the related technologies or the embodiments of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, not all embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1A schematic diagram of a depth camera provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of the acquisition of an image sensor in a continuous frame period provided by an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a flow chart of a method for self-adjusting exposure time provided in an embodiment of the present application;

[0021] Figure 4 A diagram showing the correspondence between pixels in a rawphase map and an IR map provided in an embodiment of the present application. [Specific implementation method]

[0022] In order to make the purpose, technical solutions and advantages of the present application more obvious and easy to understand, the present application will be clearly and completely described below in conjunction with the embodiments of the present application and the corresponding drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. It should be understood that the various embodiments of the present application described below are only used to explain the present application and are not used to limit the present application, that is, based on the various embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0023] Figure 1 The schematic diagram of the depth camera provided in the embodiment of the present application, the depth camera 10 includes a control and processor 13 and a transmitter 11 and a collector 12 connected to the control and processor 13. Specifically, the transmitter 11 is used to continuously transmit a pulsed emission light beam 30 whose amplitude is modulated in time sequence to a target 20 in the scene, and at least part of the pulsed emission light beam 30 is reflected by the target 20 to form a pulsed reflection light beam 40, and at least part of the pulsed reflection light beam 40 is received by the collector 12 and converted into a corresponding electrical signal, and then the control and processor 13 will synchronize the trigger signals of the transmitter 11 and the collector 12, and receive and process the electrical signal generated by the collector 12 to calculate the flight time of the pulsed reflection light beam 40 relative to the pulsed emission light beam 30, and then the depth information of the target 20 can be calculated according to the calculated flight time.

[0024] In the embodiment of the present application, the transmitter 11 includes a light source 111, an emitting optical element 112 and a driver 113. Specifically, the light source 111 can project an amplitude-modulated pulse emission light beam 30 to the emitting optical element 112 at a certain timing under the control of the driver 113 (which can be further controlled by the processor 13). For example, the light source 111 projects a pulse modulated light beam, a square wave modulated light beam, a sine wave modulated light beam, etc. to the emitting optical element 112 at a certain frequency under the control of the driver 113; after receiving the pulse emission light beam 30 projected by the light source 111, the emitting optical element 112 will project the received pulse emission light beam 30 to the target 20 in the scene, and at the same time, the pulse emission light beam 30 can be collimated, expanded, diffracted, and other modulation operations can be performed. Among them, the light source 111 can be a single light source such as a light emitting diode (LED), an edge emitting laser (EEL), a vertical cavity surface emitting laser (VCSEL), etc., or it can be a VCSEL array light source chip formed by generating multiple VCSELs on a single semiconductor substrate; the emitting optical element 112 can be a combination of one or more lenses, microlens arrays, diffractive optical elements (DOEs), diffusers, etc.

[0025] In the embodiment of the present application, the collector 12 includes an image sensor 121, a filter unit 122 and a lens unit 123. Specifically, the lens unit 123 is used to receive at least part of the pulse reflection light beam 40 reflected by the target 20 in the scene, and image it on at least part of the image sensor 121; the filter unit 122 is set to a narrow band filter that matches the wavelength of the light source 111, which is used to suppress the background light noise of the remaining bands; the image sensor 121 can be an array image sensor composed of a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), an avalanche photodiode (APD), a single photon avalanche photodiode (SPAD), etc., and the size of the array represents the resolution of the depth camera 10, such as 320×240, etc. In addition, the collector 12 can also include other common structures in the art, such as a readout circuit (not shown in the figure) electrically connected to the image sensor 121, which generally includes a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC) and other devices. One or more combinations of devices.

[0026] Furthermore, the image sensor 121 includes at least one acquisition unit. Unlike conventional image sensors that are only used for taking pictures, each acquisition unit here includes at least two taps, and each tap is used to store and read or discharge the charge signal generated by the acquisition of the pulse reflection light beam 40 (or ambient light) under the control of the corresponding electrode. In actual applications, the image sensor 121 can switch the taps in a certain order in a single frame period (or a single exposure time) to acquire the corresponding pulse reflection light beam 40, and convert the acquired pulse reflection light beam 40 into a corresponding electrical signal, so that the image sensor 121 can output a rawphase map in each frame period, and the pixel value (referred to as raw value) of each pixel (referred to as raw pixel) in the rawphase map is the amount of charge accumulated by each tap due to the acquisition of the pulse reflection light beam 40 (or ambient light), and the rawphase map also includes the phase difference information of the pulse reflection light beam 40 relative to the pulse emission light beam 30. As an example, when the image sensor 121 includes 100*100 acquisition units and each acquisition unit includes 3 taps, the rawphase diagram includes 300*100 raw pixels.

[0027] Further, Figure 2 The image sensor provided in the embodiment of the present application is a schematic diagram of acquisition within a continuous frame period, wherein the entire frame period T is divided into two time periods Ta and Tb, wherein Ta represents the time period in which each tap in the acquisition unit acquires the pulse reflection light beam 40 and stores the corresponding charges, and Tb represents the time period in which the charge signal generated by each tap in the acquisition unit due to the acquisition of the pulse reflection light beam 40 is read out. As an example, each acquisition unit in the image sensor 121 includes three taps, represented by s1, s2 and s3, respectively, wherein s1 and s2 are used to collect the pulse reflection light beam 40, and s3 is used to collect ambient light; in general modulation and demodulation, the exposure time of each tap in a continuous frame period T is fixed, that is, assuming that the pulse reflection light beam 40 is collected by taps s1 and taps s2 (tap s1 and tap s2 will also collect ambient light) and accumulate charges Q1 and Q2 respectively, tap s3 is used to collect ambient light and accumulate charge Q3, at this time, the image sensor 121 outputs a rawphase map in each frame period T, and then the control and processor 13 can calculate the depth information of the target 20 according to the rawphase map output by the image sensor 121, and the depth detection range D of the depth camera 10 is also limited to a single pulse width time Th, that is:

[0028]

[0029] In the embodiment of the present application, the control and processor 13 can provide the modulation signal (transmission signal) required when the light source 111 projects the pulse emission light beam 30, and the light source 111 will project the pulse emission light beam 30 to the target 20 in the scene under the control of the modulation signal. For example, the modulation signal provided by the control and processor 13 is a square wave signal or a pulse signal, and the light source 111 will perform timing amplitude modulation under the modulation of the modulation signal to generate a square wave modulated light beam or a pulse modulated light beam and emit it outward. In addition, the control and processor 13 can also provide a demodulation signal (acquisition signal) of each tap in each acquisition unit of the image sensor 121, and each tap will collect the pulse reflection light beam 40 reflected back by the target 20 in the scene under the control of the demodulation signal, and convert the collected pulse reflection light beam 40 into a corresponding electrical signal, and after the acquisition is completed, a rawphase map can be output. Further, after the control and processor 13 receives the rawphase map from the image sensor 121, it can process the received rawphase map to calculate the depth information of the target 20 (equivalent to generating a depth image of the target 20). Among them, the control and processor 13 can be an independent dedicated circuit, such as a dedicated SOC chip, FPGA chip, ASIC chip, etc. composed of a central processing unit (CPU), memory, bus, etc., or it can be a general processing circuit. For example, when the depth camera 10 is integrated into a smart terminal such as a mobile phone, TV, computer, etc., the processing circuit in the smart terminal can serve as at least a part of the control and processor 13.

[0030] In order to improve the depth detection effect of the depth camera 10 in multiple scenes, the AE algorithm can be used to realize the real-time and automatic adjustment of the exposure time of each tap in each acquisition unit of the image sensor 121 (based on the brightness change of the scene), so that the depth camera 10 can obtain a better depth detection effect regardless of the scene it is in, thereby making up for the defects of fixed exposure time, such as serious depth loss in low-reflection scenes and easy overexposure in high-reflection scenes. However, the traditional AE algorithm has many shortcomings, such as: simply using the average amplitude of the current exposure time to judge the brightness of the scene, which reduces the reliability of automatic adjustment of the exposure time by the AE algorithm; inconsistency of convergence results, that is, when returning to the same position from different directions in the scene multiple times, the exposure time that the AE algorithm finally converges is different; multiple iterations are required, the convergence time is long and it is very easy to fail to converge (jumping back and forth between two exposure times). In order to solve these problems, an embodiment of the present application provides a method for self-adjusting exposure time, which can be applied to the above-mentioned depth camera 10, so that the control and processor 13 of the depth camera 10 can calculate the overexposure ratio through the rawphase diagram output by the image sensor 121, and directly adjust the exposure time of the tap in the next frame period according to the calculated overexposure ratio. This method no longer judges the brightness of the scene by the average amplitude of the current exposure time, does not require multiple iterations, and clearly defines the optimal exposure time.

[0031] Figure 3 A flow chart of a method for self-adjusting exposure time provided in an embodiment of the present application is provided. The method for self-adjusting exposure time is based on that the raw value of a raw pixel in a rawphase diagram is proportional to the exposure time, and includes the following steps 301 to 304.

[0032] Step 301: Project a pulsed emission light beam toward a target in a scene.

[0033] In an embodiment of the present application, when self-adjusting the exposure time, it is necessary to project a pulse emission beam 30 to a target 20 in the scene through the transmitter 11. At least part of the pulse emission beam 30 will be reflected by the target 20 to form a pulse reflection beam 40, and at least part of the pulse reflection beam 40 will be received by the collector 12.

[0034] Step 302: Collect the pulse reflected light beam reflected by the target in the current frame period and generate a corresponding rawphase diagram.

[0035] In the embodiment of the present application, after the pulse reflection light beam 40 is formed by reflection by the target 20, it is necessary to collect the pulse reflection light beam 40 reflected by the target 20 in the current frame period through the collector 12 (specifically, the image sensor 121 in the collector 12) and generate a corresponding rawphase diagram; wherein the image sensor 121 includes at least one collection unit, each collection unit includes at least two taps, in which case the raw value of each raw pixel in the rawphase diagram respectively represents the amount of charge generated by each tap collecting the pulse reflection light beam 40 and / or ambient light.

[0036] Step 303: Calculate the overexposure ratio according to the number of maximum raw values ​​in the rawphase graph.

[0037] In an embodiment of the present application, after the image sensor 121 generates the rawphase diagram of the current frame period, the control and processor 13 needs to receive the rawphase diagram from the image sensor 121, and calculate the overexposure ratio according to the number of maximum raw values ​​in the received rawphase diagram. The overexposure ratio will be used to determine the brightness of the scene and as a basis for adjusting the tap exposure time in the next frame period.

[0038] In some embodiments, the method specifically includes: determining the maximum raw value of a raw pixel according to the number of bits of a rawphase map, wherein the maximum raw value is defined as the raw value of a target raw pixel; counting the number of target raw pixels in the rawphase map; and dividing the number of target raw pixels in the rawphase map by the total number of raw pixels in the rawphase map, thereby obtaining the overexposure ratio. For example, if the rawphase map is 10 bits, then the maximum raw value of a raw pixel is 1023. In this case, the number of target raw pixels in the rawphase map whose raw value is equal to 1023 is directly counted, and the number of target raw pixels in the rawphase map whose raw value is equal to 1023 is divided by the total number of raw pixels in the rawphase map. The final result is the overexposure ratio of the rawphase map.

[0039] Step 304: adjust the exposure time of the tap in the next frame period according to the overexposure ratio.

[0040] In the embodiment of the present application, after the overexposure ratio is calculated according to the rawphase diagram from the image sensor 121, the control and processor 13 needs to adjust the exposure time of the tap in the next frame period according to the calculated overexposure ratio, such as increasing the exposure time, reducing the exposure time, or keeping the current exposure time unchanged, etc., so as to finally optimize the exposure time of the tap in the next frame period.

[0041] In some implementations, step 304 specifically includes the following steps:

[0042] Step 3041, comparing the overexposure ratio with a preset overexposure ratio threshold, and calculating the optimal exposure time according to the comparison result;

[0043] In these embodiments, an overexposure ratio threshold value can be pre-set, and the calculated overexposure ratio can be compared with the overexposure ratio threshold value. The result of the comparison can indicate the brightness of the scene, so that an optimal exposure time can be calculated according to the comparison result, and then this optimal exposure time can be selected as the exposure time of the tap in the next frame period, so that the exposure time of the tap in the next frame period reaches the optimal. It can be understood that these embodiments have a clear definition of the optimal exposure time, that is, the maximum exposure time without overexposure. For each scene, these embodiments will return the optimal exposure time in the scene. When the scene is determined, the optimal exposure time is a fixed value, that is, the maximum exposure time without overexposure in the scene, that is, each scene has its own optimal exposure time; for example, when the depth camera 10 shoots a flat white wall 300mm away from itself, the exposure time is less than 500μs without overexposure, but the exposure time is higher than 500μs and overexposure occurs, then 500μs is the optimal exposure time in this scene.

[0044] As one implementation method, the following situations are specifically included: if the overexposure ratio is greater than 0 and less than or equal to the overexposure ratio threshold, the current exposure time is used as the optimal exposure time. In this implementation method, if the comparison result is that the overexposure ratio is greater than 0 and less than or equal to the overexposure ratio threshold, it means that the current exposure time is appropriate, that is, there is no need to adjust the exposure time, and the current exposure time can be directly used as the optimal exposure time.

[0045] If the overexposure ratio is equal to 0 or greater than the overexposure ratio threshold, a corresponding IR map is generated according to the rawphase map, and the optimal exposure time is calculated in combination with the rawphase map and the IR map, wherein the pixel value (called IR value) of each pixel (called IR pixel) in the IR map represents the amount of charge generated by each acquisition unit collecting the pulse reflected light beam and / or ambient light. If the comparison result is that the overexposure ratio is greater than the overexposure ratio threshold, it means that the current overexposure is serious and the exposure time needs to be reduced; if the comparison result is that the overexposure ratio is equal to 0, it means that although there is no overexposure, the brightness of the current scene is dim and the exposure time needs to be increased. It can be understood that the exposure time needs to be adjusted only when the overexposure ratio is equal to 0 or greater than the overexposure ratio threshold. In this case, the corresponding IR map can be generated according to the rawphase map first, and then the rawphase map and the IR map can be combined to calculate the optimal exposure time.

[0046] It should be noted that there is a corresponding relationship between each raw pixel in the rawphase diagram and each IR pixel in the corresponding IR diagram. From the foregoing description, it can be found that each acquisition unit in the image sensor 121 includes at least two taps, and the raw value of each raw pixel in the rawphase diagram respectively represents the amount of charge generated by each tap when collecting the pulse reflection light beam 40 and / or the ambient light, and the IR value of each IR pixel in the IR diagram respectively represents the amount of charge generated by each acquisition unit when collecting the pulse reflection light beam and / or the ambient light (that is, it respectively represents the sum of the amount of charge generated by all taps of each acquisition unit when collecting the pulse reflection light beam and / or the ambient light). Then, taking the target acquisition unit including two target taps as an example, in this case, the raw values ​​of the two target raw pixels in the rawphase diagram are summed to obtain the IR value of the target IR pixel in the IR diagram. This is the corresponding relationship between each raw pixel in the rawphase diagram and each IR pixel in the corresponding IR diagram. The raw values ​​of the multiple raw pixels corresponding to each acquisition unit in the rawphase map are summed up respectively to obtain the IR value of each IR pixel in the corresponding IR map; and the IR map corresponding to the rawphase map is generated according to all the IR values. Figure 4 The rawphase diagram shown is a corresponding relationship diagram between pixels in the IR diagram. There are 8 IR pixels in the IR diagram, which respectively represent that 8 acquisition units A, B, C, D, E, F, G and H respectively collect the charge generated by the corresponding pulse reflection light beam 40 and / or the ambient light. Each of the 8 acquisition units includes two taps (for example, acquisition unit A includes taps a1 and a2, acquisition unit B includes taps b1 and b2, acquisition unit C includes taps c1 and c2, and so on). In this case, there are 1 The 6 raw pixels respectively represent the amount of charge generated by the 16 taps of the 8 acquisition units respectively collecting the corresponding pulse reflected light beam 40 and / or ambient light. Taking the acquisition unit A and its taps a1 and a2 as an example, the raw values ​​of the two raw pixels in the rawphase diagram representing the amount of charge generated by the taps a1 and a2 respectively collecting the corresponding pulse reflected light beam 40 and / or ambient light can be summed to obtain the IR value of the IR pixel in the IR diagram representing the amount of charge generated by the acquisition unit A collecting the corresponding pulse reflected light beam 40 and / or ambient light.

[0047] In a specific implementation of the present embodiment, when the overexposure ratio is greater than the overexposure ratio threshold, the rawphase map is divided into an overexposed area and a non-overexposed area, wherein the raw values ​​of each raw pixel in the overexposed area are all the maximum raw values ​​described above, and the raw values ​​of each raw pixel in the non-overexposed area are all less than the maximum raw value described above. Corresponding to the overexposed area, a first maximum IR pixel with the largest IR value is searched among the corresponding IR pixels in the IR map, and corresponding to the first maximum IR pixel, a first minimum raw pixel with the smallest raw value is searched among the corresponding raw pixels in the rawphase map; corresponding to the non-overexposed area, a second maximum IR pixel with the largest IR value is searched among the corresponding IR pixels in the IR map, and corresponding to the second maximum IR pixel, a second minimum raw pixel with the smallest raw value is searched among the corresponding raw pixels in the rawphase map; and the optimal exposure time is calculated according to the raw values ​​of the first minimum raw pixel and the second minimum raw pixel.

[0048] Since the overexposure ratio is greater than the overexposure ratio threshold, it means that the current overexposure is serious, so the exposure time needs to be reduced. In this case, the adjustment target of the exposure time is "just not overexposed", that is, the raw value of the first minimum raw pixel can be reduced to the raw value of the second minimum raw pixel. At the same time, since the raw value of the raw pixel in the rawphase diagram is proportional to the exposure time, it can be calculated according to the first calculation formula t opt / t in =(B min -Pd) / (A min -Pd) calculates the optimal exposure time; where to pt represents the optimal exposure time, t in represents the current exposure time, Pd represents the dark field value of the collector 12 (specifically, the image sensor 121), A min Indicates the raw value of the first minimum raw pixel, B min Indicates the raw value of the second smallest raw pixel.

[0049] It can be understood that, when calculating the optimal exposure time, the present specific implementation uses the raw value of the raw pixel with the smallest raw value (i.e., the first smallest raw pixel) among several raw pixels corresponding to the first maximum IR pixel in the rawphase diagram, and the raw value of the raw pixel with the smallest raw value (i.e., the second smallest raw pixel) among several raw pixels corresponding to the second maximum IR pixel in the rawphase diagram, instead of using the IR values ​​of the first maximum IR pixel and the second maximum IR pixel. Taking the first maximum IR pixel as an example, this is because the acquisition unit corresponding to the first maximum IR pixel is overexposed, which means that at least one of the charges generated by the corresponding pulse reflection light beam 40 and / or ambient light collected by multiple taps in the acquisition unit corresponding to the first maximum IR pixel is inaccurate. Therefore, the first smallest raw pixel with the smallest raw value among several raw pixels corresponding to the first maximum IR pixel in the rawphase diagram is used. The first smallest raw pixel belongs to the background noise and usually will not overflow due to overexposure. Similarly, the second largest IR pixel is also used. In addition, the dark field value Pd of the image sensor 121 is not affected by the integration time (i.e., exposure time), which is usually a fixed gain caused by digital-to-analog conversion. Therefore, the raw values ​​of the first minimum raw pixel and the second minimum raw pixel need to be subtracted from the dark field value Pd of the image sensor 121, and the remaining part is positively correlated with the integration time.

[0050] When the overexposure ratio is equal to 0, the third largest IR pixel with the largest IR value is found among the IR pixels in the IR map; corresponding to the third largest IR pixel, the maximum raw pixel with the largest raw value is found among the corresponding raw pixels in the rawphase map; and the optimal exposure time is calculated according to the raw value of the maximum raw pixel.

[0051] In this specific implementation, since the overexposure ratio is equal to 0, it means that although there is no overexposure, the brightness of the current scene is dim, so the exposure time needs to be increased. In this case, the adjustment target of the exposure time is "just overexposure", that is, the raw value of the brightest raw pixel in the rawphase image can reach the maximum raw value mentioned above. At the same time, since the raw value of the raw pixel in the rawphase image is proportional to the exposure time, it can be calculated according to the second calculation formula t opt / t in =(M-Pd) / (C max -Pd) calculates the optimal exposure time; where M represents the maximum raw value mentioned above (when the rawphase map is 10 bits, M is equal to 1023), C max Indicates the raw value of the maximum raw pixel.

[0052] Step 3042: Using the optimal exposure time and the preset exposure time range as reference, adjust the exposure time of the tap in the next frame period.

[0053] Furthermore, these embodiments also pre-set an exposure time range, and this exposure time range can be determined according to the depth detection range D of the depth camera 10 mentioned above, so that no matter how the exposure time is adjusted, the final exposure time will be within this exposure time range. This is also the reason why when adjusting the exposure time of the tap in the next frame period, it is necessary to refer not only to the calculated optimal exposure time, but also to the exposure time range.

[0054] As one implementation method, it specifically includes: if the optimal exposure time is within a preset exposure time range, the exposure time of the tap in the next frame period is set to the optimal exposure time; if the optimal exposure time exceeds the exposure time range, the exposure time of the tap in the next frame period is set to the maximum exposure time or the minimum exposure time defined by the exposure time range. In this embodiment, when the calculated optimal exposure time is within the preset exposure time range, it means that the calculated optimal exposure time meets the limitation requirements of the exposure time range (that is, the exposure time will always be within the preset exposure time range), so the exposure time of the tap in the next frame period can be set to the optimal exposure time; when the calculated optimal exposure time exceeds the preset exposure time range, that is, exceeds the maximum exposure time or the minimum exposure time defined by the exposure time range, it means that the calculated optimal exposure time does not meet the limitation requirements of the exposure time range, so when the calculated optimal exposure time exceeds the maximum exposure time defined by the exposure time range, the maximum exposure time defined by the exposure time range is directly selected as the exposure time of the tap in the next frame period; similarly, when the calculated optimal exposure time exceeds the minimum exposure time defined by the exposure time range, the minimum exposure time defined by the exposure time range is directly selected as the exposure time of the tap in the next frame period. For example, when the depth camera 10 is used to perform depth detection on a target 20 that is 8 meters away indoors, since the target 20 is very far away from the depth camera 10, the rawphase diagram output by the image sensor 121 may be very dark even at the maximum exposure time defined by the exposure time range. However, since the exposure time range has defined the maximum exposure time, that is, it cannot be increased any further, even if the calculated optimal exposure time is greater than the defined maximum exposure time, only the maximum exposure time defined by the exposure time range can be selected as the exposure time of the tap in the next frame period; similarly, the same is true when the calculated optimal exposure time is less than the minimum exposure time defined by the exposure time range.

[0055] As can be seen from the above, the embodiment of the present application projects a pulsed emission light beam 30 to the target 20 in the scene through the transmitter 11, and collects the pulsed reflection light beam 40 reflected by the target 20 in the current frame period through the collector 12, and generates a corresponding rawphase diagram accordingly. Then, the overexposure ratio can be calculated according to the generated rawphase diagram, and the brightness of the scene is indicated by the overexposure ratio, so that the exposure time of the tap in the next frame period can be adjusted according to the overexposure ratio. For example, when the overexposure ratio exceeds the preset threshold, it means that the current overexposure is serious, and the exposure time needs to be reduced to "just not overexposed" to achieve the optimal exposure time; when the overexposure ratio is equal to zero, it means that there is no overexposure at present, and the exposure time needs to be increased to "just overexposed" to achieve the optimal exposure time; when the overexposure ratio is greater than zero but does not exceed the preset threshold, it means that the current exposure time is appropriate and does not need to be changed. It can be understood that the embodiment of the present application utilizes the overexposure ratio of the rawphase diagram to determine the brightness of the scene, thereby reducing or even eliminating the unreliability of the traditional AE algorithm that simply uses the average amplitude of the current exposure time to determine the brightness of the scene. The embodiment of the present application directly adjusts the exposure time of the tap in the next frame period according to the overexposure ratio, which is different from the multiple iterations of the traditional AE algorithm, thereby effectively avoiding the problem of slow convergence speed and easy convergence failure of the traditional AE algorithm during iteration.

[0056] The above implementation is only a preferred implementation of the embodiment of the present application, and it is not the only limitation of the described content; in this regard, those skilled in the art can flexibly set it according to the actual application scenario based on the embodiment of the present application.

[0057] In summary, the self-adjusting method for exposure time provided in the embodiment of the present application uses the overexposure ratio of the rawphase diagram to judge the brightness of the scene, thereby reducing or even eliminating the unreliability of the traditional AE algorithm that simply uses the average amplitude of the current exposure time to judge the brightness of the scene; and directly adjusts the exposure time of the tap in the next frame period according to the overexposure ratio, which is different from the multiple iterations of the traditional AE algorithm, thereby effectively avoiding the problem that the traditional AE algorithm has slow convergence speed and is very easy to fail to converge during iteration; importantly, there is a clear definition of the optimal exposure time, which is the maximum exposure time without overexposure. For each scene, the optimal exposure time for the scene will be returned. When the scene is determined, the optimal exposure time is a fixed value (that is, the maximum exposure time without overexposure in the scene), that is, each scene has its own optimal exposure time, thereby effectively avoiding the inconsistency of the convergence results in the traditional AE algorithm (that is, when returning to the same position from different directions in the scene for multiple times, the exposure time at which the AE algorithm finally converges is different).

[0058] It should be noted that each embodiment in the present application is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For product-type embodiments, since they are similar to method-type embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method-type embodiments.

[0059] It should also be noted that, in the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0060] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in the present application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown in the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the present application.

Claims

1. A method for self-adjusting exposure time, characterized in that: include: Projects a pulsed emission beam at targets in the scene; Collecting the pulse reflected light beam reflected by the target in the current frame period and generating a corresponding rawphase diagram; wherein the raw value of each raw pixel in the rawphase diagram respectively represents the amount of charge generated by each tap collecting the pulse reflected light beam and / or ambient light; Calculating the overexposure ratio according to the number of maximum raw values ​​in the rawphase graph; Comparing the overexposure ratio with a preset overexposure ratio threshold; If the overexposure ratio is greater than 0 and less than or equal to the overexposure ratio threshold, the current exposure time is used as the optimal exposure time; If the overexposure ratio is greater than the overexposure ratio threshold, generating a corresponding IR map according to the rawphase map, and calculating an optimal exposure time in combination with the rawphase map and the IR map; When the overexposure ratio is equal to 0, searching for a third maximum IR pixel having a maximum IR value among the IR pixels of the IR map; Corresponding to the third maximum IR pixel, searching for a maximum raw pixel having a maximum raw value among the corresponding raw pixels in the rawphase diagram, and calculating an optimal exposure time according to the raw value of the maximum raw pixel; The exposure time of the tap in the next frame period is adjusted with reference to the optimal exposure time and the preset exposure time range.

2. The method for self-adjusting exposure time according to claim 1, characterized in that: The calculating the overexposure ratio according to the number of maximum raw values ​​in the rawphase graph includes: Determining a maximum raw value of the raw pixel according to the number of bits of the rawphase map; Counting the number of target raw pixels in the rawphase graph; wherein the raw value of the target raw pixel is equal to the maximum raw value; The overexposure ratio is obtained by dividing the number of the target raw pixels in the rawphase image by the total number of the raw pixels in the rawphase image.

3. The method for self-adjusting exposure time according to claim 1, characterized in that: The step of adjusting the exposure time of the tap in the next frame period by taking the optimal exposure time and the preset exposure time range as reference includes: If the optimal exposure time is within the preset exposure time range, setting the exposure time of the tap in the next frame period to the optimal exposure time; If the optimal exposure time exceeds the exposure time range, the exposure time of the tap in the next frame period is set to the maximum exposure time or the minimum exposure time defined by the exposure time range.

4. The method for self-adjusting exposure time according to claim 1, characterized in that: When the overexposure ratio is greater than the overexposure ratio threshold, the calculating the optimal exposure time in combination with the rawphase map and the IR map includes: Dividing the rawphase map into an overexposed area and a non-overexposed area; wherein the raw value of each raw pixel in the overexposed area is the maximum raw value, and the raw value of each raw pixel in the non-overexposed area is less than the maximum raw value; Corresponding to the overexposed area, searching for a first maximum IR pixel with a maximum IR value among the corresponding IR pixels in the IR map; and corresponding to the first maximum IR pixel, searching for a first minimum raw pixel with a minimum raw value among the corresponding raw pixels in the rawphase map; Corresponding to the non-overexposed area, searching for a second maximum IR pixel with a maximum IR value among the corresponding IR pixels in the IR map; and corresponding to the second maximum IR pixel, searching for a second minimum raw pixel with a minimum raw value among the corresponding raw pixels in the rawphase map; An optimal exposure time is calculated according to the raw values ​​of the first minimum raw pixel and the second minimum raw pixel.

5. The method for self-adjusting exposure time according to claim 4, characterized in that: The calculating the optimal exposure time according to the raw values ​​of the first minimum raw pixel and the second minimum raw pixel includes: According to the first calculation formula t opt / t in =(B min -Pd) / (A min -Pd) calculates the optimal exposure time; where t opt represents the optimal exposure time, t in represents the current exposure time, Pd represents the dark field value of the collector in the depth camera, A min represents the raw value of the first minimum raw pixel, B min Indicates the raw value of the second smallest raw pixel.

6. The method for self-adjusting exposure time according to claim 1, characterized in that: The calculating the optimal exposure time according to the raw value of the maximum raw pixel includes: According to the second calculation formula t opt / t in =(M-Pd) / (C max -Pd) calculates the optimal exposure time; where t opt represents the optimal exposure time, t in represents the current exposure time, M represents the maximum raw value, Pd represents the dark field value of the collector in the depth camera, C max Indicates the raw value of the maximum raw pixel.

7. A self-adjusting exposure time depth camera, characterized in that: include: A transmitter, used to project a pulsed emission beam toward a target in the scene; A collector, comprising at least one collection unit, and each of the collection units comprises at least two taps, for collecting a pulse reflected light beam reflected by the target in a current frame period, and generating a corresponding rawphase diagram; wherein the raw value of each raw pixel in the rawphase diagram respectively represents the amount of charge generated by each tap collecting the pulse reflected light beam and / or ambient light; A control and processor, used to calculate the overexposure ratio according to the number of maximum raw values ​​in the rawphase map; compare the overexposure ratio with a preset overexposure ratio threshold; if the overexposure ratio is greater than 0 and less than or equal to the overexposure ratio threshold, use the current exposure time as the optimal exposure time; if the overexposure ratio is greater than the overexposure ratio threshold, generate a corresponding IR map according to the rawphase map, and calculate the optimal exposure time in combination with the rawphase map and the IR map; when the overexposure ratio is equal to 0, find the third maximum IR pixel with the largest IR value among the IR pixels in the IR map; corresponding to the third maximum IR pixel, find the maximum raw pixel with the largest raw value among the corresponding raw pixels in the rawphase map, and calculate the optimal exposure time according to the raw value of the maximum raw pixel; with the optimal exposure time and the preset exposure time range as reference, adjust the exposure time of the tap in the next frame period.

Images