A depth camera and method to eliminate motion artifacts

Abstract

The application relates to a depth camera and method for removing motion artifacts, the depth camera comprising: an emitter for emitting a pulsed light beam to a target in a spatial region within a plurality of frame periods; a collector for collecting a reflected pulsed light beam reflected by the target within each frame period and generating a raw phase image; wherein the collector comprises an image sensor composed of a plurality of pixels, each pixel comprising a plurality of taps, each tap being used for collecting the reflected pulsed light beam or background light to generate a charge amount, a pixel value in the raw phase image being the charge amount generated by the tap; a control and processing circuit receiving a plurality of raw phase images and processing to obtain an IR image corresponding to each raw phase image; determining a motion pixel according to a pixel value in the IR image, and correcting a pixel value in a raw phase image corresponding to the motion pixel to obtain a corrected raw phase image; and calculating a target depth image according to the corrected raw phase image. Through the implementation of the application, the accuracy of removing motion artifacts is improved.

Description

Technical Field

[0001] This invention belongs to the field of depth camera technology, and particularly relates to a depth camera and method for eliminating motion artifacts. Background Technology

[0002] The Time-of-Flight (TOF) principle allows for distance measurement of targets to obtain depth maps containing the target's depth values. TOF-based depth cameras are widely used in consumer electronics, unmanned aerial vehicles, and AR / VR. A typical TOF-based depth camera includes a transmitter, a collector, and control and processing circuitry. The transmitter continuously emits light signals into the target scene. The light signals reflected by the target are collected by the collector, which outputs a charge. The control and processing circuitry receives and processes this charge to calculate the time of flight of the pulse to and from the target point, further calculating the distance from the target point to the measurement system. In TOF technology, the technique of directly measuring the time of flight of light is called dToF (direct TOF). The technique of periodically modulating the emitted light signal, measuring the phase delay of the reflected light signal relative to the emitted light signal, and then calculating the time of flight from the phase delay is called indirect TOF (indirect TOF). Furthermore, based on the modulation and demodulation type, it can be divided into continuous wave (CW) modulation and demodulation and pulse modulation (PM) modulation and demodulation.

[0003] Currently, Time-of-Flight (TOF) technology is mainly applied in depth cameras built using tapped sensors. Multiple taps are controlled sequentially within a certain integration time to accumulate charge and acquire multiple raw phase images. These raw phase images are then used for depth calculation. If the target or module moves during the integration time, the scene recorded in each raw phase image will differ, resulting in motion artifacts in the depth map. Related technologies generally employ several assumptions to eliminate these artifacts: assuming the target is moving at a constant speed, assuming the target moves only once within the integration time, or constructing a highly idealized mathematical model to optimize the solution. However, these assumptions are too idealistic and cannot realistically eliminate motion artifacts, leading to poor results. Summary of the Invention

[0004] This invention provides a depth camera and method for eliminating motion artifacts, thereby solving the technical problem of poor motion artifact removal performance.

[0005] On one hand, the present invention provides a depth camera for eliminating motion artifacts, comprising: a transmitter for emitting pulsed light beams toward a target in a spatial region within multiple frame periods; a collector for acquiring reflected pulsed light beams reflected back from the target and generating a rawphase map within each frame period; wherein the collector includes an image sensor composed of multiple pixels, each pixel including multiple taps, each tap being used to acquire the reflected pulsed light beams or background light to generate a charge, and the pixel values ​​in the rawphase map being the charge generated by the taps; a control and processing circuit for receiving multiple rawphase maps and processing them to obtain an IR map corresponding to each rawphase map; determining moving pixels based on the pixel values ​​in the IR map, and correcting the pixel values ​​in the rawphase map corresponding to the moving pixels to obtain a corrected rawphase map; and calculating a target depth map based on the corrected rawphase map.

[0006] Secondly, the present invention provides a method for eliminating motion artifacts, the method comprising: emitting a pulsed light beam toward a target in a spatial region within multiple frame periods; acquiring reflected pulsed light beams from the target within each frame period and generating a rawphase image; pixel values ​​in the rawphase image representing the charge generated by tapping the reflected pulsed light beams or background light; receiving multiple rawphase images and processing them to obtain an IR image corresponding to each rawphase image; determining moving pixels based on the pixel values ​​in the IR images and correcting the pixel values ​​in the rawphase images corresponding to the moving pixels to obtain a corrected rawphase image; and calculating a target depth map based on the corrected rawphase image.

[0007] Thirdly, the present invention provides a depth camera for eliminating motion artifacts, comprising: a transmitter for emitting pulsed light beams having a first frequency or a second frequency toward a target in a spatial region at consecutive frame periods; a collector for acquiring reflected pulsed light beams of the first frequency reflected back from the target and generating a first rawphase image, and acquiring reflected pulsed light beams of the second frequency reflected back from the target and generating a second rawphase image; wherein the collector includes an image sensor composed of multiple pixels, each pixel including multiple taps, each tap being used to acquire the reflected pulsed light beams or background light to generate a charge, and the pixel value in the rawphase image being the charge generated by the tap; and a control and processing circuit for receiving the first rawphase image and processing it to obtain a first depth image and a corresponding first IR image, receiving the second rawphase image and processing it to obtain a second depth image and a corresponding second IR image, fusing the first depth image and the second depth image to obtain a target depth image, and determining moving pixels by comparing the pixel values ​​in the first IR image and the second IR image, and correcting the depth value in the target depth image corresponding to the moving pixel.

[0008] Fourthly, the present invention provides a method for eliminating motion artifacts, the method further comprising: emitting a pulse beam having a first frequency or a second frequency toward a target in a spatial region during a continuous frame period; acquiring the reflected pulse beam of the first frequency reflected back from the target and generating a first rawphase image, and acquiring the reflected pulse beam of the second frequency reflected back from the target and generating a second rawphase image; the pixel values ​​in the rawphase images are the amount of charge generated by tapping the reflected pulse beam or background light; receiving the first rawphase image and processing it to obtain a corresponding first depth image and a corresponding first IR image, and receiving the second rawphase image and processing it to obtain a second depth image and a corresponding second IR image; fusing the first depth image and the second depth image to obtain a target depth image; determining moving pixels by comparing the pixel values ​​in the first IR image and the second IR image, and correcting the depth values ​​in the target depth image corresponding to the moving pixels.

[0009] Fifthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps in the above-described method.

[0010] As can be seen from the above embodiments of the present invention, by comparing the pixel values ​​of the IR images of all acquired rawphase images through the control and processing circuit, moving pixels in the rawphase images are determined, and then the pixel values ​​of the moving pixels are corrected to eliminate artifacts in the rawphase images. Then, the depth map of the target area is calculated based on the corrected rawphase images. By determining the moving pixels through the differences in pixel values ​​between IR images, the accuracy of determining moving pixels is improved, thereby ensuring the accuracy of motion artifact removal. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a depth camera according to an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the depth camera optical signal transmission and acquisition method according to an embodiment of the present invention;

[0014] Figure 3 This application embodiment captures multiple raw phase images of a dynamic hand.

[0015] Figure 4 For this application Figure 3 The motion artifacts are shown in the subtraction diagram of the two corresponding rawphase images;

[0016] Figure 5 For this application Figure 3 The corresponding depth diagram showing the presence of motion artifacts;

[0017] Figure 6 This is a flowchart of a method for eliminating motion artifacts according to an embodiment of this application;

[0018] Figure 7 This is a flowchart illustrating a method for eliminating motion artifacts according to another embodiment of this application. Detailed Implementation

[0019] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] Explanation of relevant terms in this invention:

[0021] Motion artifacts: Within a certain integration time, the iTOF (indirect TOF) module needs to acquire multiple raw phase images to calculate the charge of the scene. If the target or the module moves during this integration (exposure) time, the scene recorded in each raw phase image will be different. This is ultimately reflected in the depth map as "an incorrect depth ring around the edge of the object moves with the object".

[0022] IR value: The intensity of the light signal after it has passed through the target object; the light signal is preferably infrared light.

[0023] iTOF module: Includes iTOF image sensor, light source and microcontroller. The working principle of iTOF module is to emit modulated single-frequency light signal into the scene through light source, and then the iTOF image sensor receives the single-frequency light signal reflected back by the target object in the scene. The phase difference between the emitted signal and the received signal is calculated based on the cumulative charge during the exposure (integration) time, thereby obtaining the depth (distance value) of the target object.

[0024] Figure 1 This is a schematic diagram of a depth camera according to an embodiment of the present invention. The depth camera 10 includes a transmitter 11, a collector 12, and a control and processing circuit 13 connected to the transmitter and the collector. The transmitter 11 continuously emits a time-modulated emission beam 30 towards a target object 20. At least a portion of the emission beam is reflected by the target point to form a reflected beam 40. At least a portion of the reflected beam 40 is received by the collector 12 and generates an electrical signal. The control and processing circuit 13 synchronizes the trigger signals of the transmitter 11 and the collector 12, receives the electrical signal, processes it, calculates the flight time of the reflected beam 40 relative to the emission beam 30, and further calculates the depth information of the target based on the flight time.

[0025] The transmitter 11 includes a light source 111, an emitting optical element 112, and a driver 113. The light source 111 can be a single light source such as a light-emitting diode (LED), an edge-emitting laser (EEL), or a vertical-cavity surface-emitting laser (VCSEL), or it can be a VCSEL array light source chip formed by generating multiple VCSEL light sources on a single semiconductor substrate. The light source 111 can emit a beam of light outward with a certain timing amplitude under the control of the driver 113 (which can be further controlled by the control and processing circuit 13). For example, in one embodiment, the light source 111 emits a pulse-modulated beam, a square-wave modulated beam, or a sinusoidal-wave modulated beam at a certain frequency under control. The emitting optical element 112 receives the beam emitted by the light source 111 and emits it outward, and can simultaneously collimate, expand, diffract, etc., the beam before emitting the beam 30 outward. The emitting optical element 112 can be one or more lenses, microlens arrays, diffractive optical elements (DOE), diffusers, etc.

[0026] The acquisition unit 12 includes an iTOF image sensor 121, a filtering unit 122, and a lens unit 123. The lens unit 123 receives and images at least a portion of the speckled pattern beam reflected back from the target object onto at least a portion of the TOF image sensor 121. The filtering unit 122 is configured as a narrowband filter matched to the wavelength of the light source to suppress background light noise in other bands. The iTOF image sensor 121 can be an image sensor composed of a charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), avalanche diode (AD), single-photon avalanche diode (SPAD), etc., and the array size represents the resolution of the depth camera, such as 320x240. Generally, a readout circuit (not shown in the figure) composed of one or more of the following devices are also connected to the image sensor 121: a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC), etc.

[0027] Generally, the iTOF image sensor 121 includes at least one pixel. Compared to traditional image sensors used only for taking pictures, each pixel here includes two or more taps (used to store, read, or discharge charge signals generated by incident photons under the control of corresponding electrodes), such as three taps. Within a single frame period (or a single exposure time), the taps are switched sequentially in a certain order to acquire the corresponding light signals and convert them into electrical signals. This allows the iTOF image sensor 121 to output a raw phase image in each frame period. The pixel values ​​in the raw phase image represent the amount of charge accumulated by each tap. This raw phase image contains information about the phase difference between the reflected beam and the emitted beam. Assuming the iTOF image sensor 121 includes 100*100 pixels, and each pixel includes three taps, the pixel value in the raw phase image is 300*100.

[0028] The control and processing circuit 13 can be an independent dedicated circuit, such as a dedicated SOC chip, FPGA chip, ASIC chip, etc., which includes a CPU, memory, bus, etc. It can also include general processing circuits. For example, when the depth camera is integrated into a smart terminal such as a mobile phone, TV, or computer, the processing circuit in the terminal can be at least a part of the control and processing circuit 13.

[0029] In some embodiments, the control and processing circuit 13 provides the modulation signal (emission signal) required for the light source 111 to emit laser light, and the light source emits a beam of light toward the target object under the control of the modulation signal. For example, in one embodiment, the modulation signal is a square wave signal or a pulse signal, and the amplitude of the light source is time-modulated under the modulation signal to generate a square wave signal or a pulse signal for outward emission. It also provides the demodulation signal (acquisition signal) of each tap in each pixel of the iTOF image sensor 121. Under the control of the demodulation signal, the taps acquire the reflected beam of light containing the target object and convert it into an electrical signal. After acquisition, the iTOF image sensor outputs a rawphase image. The control and processing circuit 13 receives the rawphase image and processes it to calculate the target depth map.

[0030] In some embodiments, each pixel in the iTOF image sensor includes three taps. In conventional modulation and demodulation methods, the exposure time of each tap is fixed and constant within consecutive frame periods. That is, it is assumed that the reflected light signal is collected by the first and second taps (which also collect ambient light signals), accumulating charge Q1 and Q2; the third tap is used to collect the ambient light signal's accumulated charge Q3. In this case, the iTOF image sensor outputs a raw phase image in each frame period. The control and processing circuit can calculate the target's depth information based on the raw phase image. Thus, the depth camera's measurement range is limited to a single pulse width time Th. Specifically, c is a coefficient.

[0031]

[0032] Example 1

[0033] Figure 2 This is a schematic diagram of a depth camera optical signal transmission and acquisition method according to an embodiment of the present invention. In some embodiments of this application, in order to improve detection accuracy and detection range, the control and processing circuit 13 can choose to receive multiple raw phase images within a continuous frame period to calculate the depth map of the target area, which can improve detection accuracy compared to a single raw phase image. Alternatively, the control and processing circuit 13 adjusts the demodulation signal (acquisition signal) provided to each tap in each pixel of the iTOF image sensor 121, and selects a rotating sampling mode to acquire multiple raw phase images to calculate the depth map of the target area, that is, the exposure time of each tap is different in each frame period, which can effectively expand the detection range. Figure 2 As shown, the entire frame period T is divided into two time periods, Ta and Tb, where Ta represents the time period during which each tap of the pixel performs charge acquisition and storage, and Tb represents the time period during which the charge signal is read out. Specifically, the acquisition and storage time period is divided into three exposure moments. In the first frame period, the first, second, and third taps sequentially activate the charge accumulation signal; in the second frame period, the second, third, and first taps sequentially activate the charge accumulation signal; and in the third frame period, the third, first, and second taps sequentially activate the charge accumulation signal. This control method is called the rotating sampling mode. Three raw phase images are output for each of the three frame periods. These three raw phase images are used to calculate the depth map of the target area, thus extending the measurement range of the depth camera to three times the pulse width time Th. However, in the above detection method, if the target or the depth camera moves, the target corresponding to multiple raw phase images will not be at the same location. The phase information contained in each raw phase image will change, resulting in motion artifacts in the final calculated depth map. Figure 3The image shown is a display of multiple raw phase images of a dynamically captured hand. In this embodiment, six raw phase images were captured consecutively. Figure 4 The image shows the motion artifacts resulting from subtracting the first and sixth rawphase images. When multiple rawphase images are used for depth calculation, noticeable motion artifacts will appear in the calculated depth map. Figure 5 The image shown illustrates the depth characteristics of motion artifacts. Therefore, it's necessary to assess the operational status; if the target is moving, the raw phase image needs to be corrected.

[0034] The charge values ​​of three raw phase images are acquired using a rotary sampling mode. The charge value sampled by tap a at the first exposure time in the first raw phase image is denoted as Q. a1 Tap b is sampled at the second exposure time and recorded as Q. b2 Tap c is sampled at the third exposure time and recorded as Q. c3 In the second raw phase image, the charge sampled by tap b at the first exposure time is denoted as Q. b1 The charge sampled by tap c at the second exposure time is denoted as Q. c2 The charge sampled by tap a at the third exposure time is denoted as Q. a3 In the third raw phase image, the charge sampled by tap c at the first exposure time is denoted as Q. c1 The charge sampled by tap a at the second exposure time is denoted as Q. a2 The charge sampled by tap b at the third exposure time is denoted as Q. b3 .

[0035] If there is no motion, the depth map is directly calculated from the three raw phase images. The calculation method is as follows: calculate the charge amount sampled at the same exposure time in multiple frame periods, where the charge amount sampled at the first exposure time is Q1 = Q a1 +Q b1 +Q c1 The charge sampled at the second exposure time is Q2 = Q a2 +Q b2 +Q c2 The charge sampled at the third exposure time is Q3 = Q a3 +Q b3 +Q c3 Based on the charge quantities Q1, Q2, and Q3, a judgment is made to determine whether to obtain the tap containing the reflected light signal that excites electrons or the tap containing only the background signal. Assuming that after the judgment, the two total charge quantities containing the reflected light signal are received sequentially (in chronological order), they are denoted as Q1, Q2, and Q3 respectively. A With Q B The total charge containing only the background light signal is denoted as Q. OThe control and processing circuit then calculates the target's depth using the following formula:

[0036]

[0037] Where m is 0, 1, or 2 respectively. If the reflected light signal is collected by the first tap for the first time within the frame period, then m = 0, Q A =Q1, Q B =Q2, Q0 = Q3; if the reflected light signal is collected by the second tap for the first time within the frame period, then m = 1, Q A =Q3, Q B =Q1, Q0 = Q2; if the reflected light signal is collected by the third tap for the first time within the frame period, then m = 2, Q A =Q3, Q B =Q1 (current pulse period Ta), Q0 = Q2. For example... Figure 2 The figure shows the case where m = 1.

[0038] In some embodiments, the control and processing circuit receives multiple rawphase images, processes them to obtain an IR image corresponding to each rawphase image, determines moving pixels based on the pixel values ​​in the IR image, and corrects the pixel values ​​in the corresponding rawphase image to obtain a corrected rawphase image. The target depth map is then calculated based on the corrected rawphase image. The following mainly uses the rotary sampling mode as an example to illustrate the specific processing method:

[0039] The control and processing circuit receives multiple rawphase images and processes them to obtain the corresponding IR image for each rawphase image. This means acquiring the IR image captured by the iTOF image sensor in each frame period. The pixel value (denoted as IR value) in the IR image corresponds to the amount of charge accumulated by each pixel within the frame period. Each pixel accumulates charge through multiple configured taps, and the amount of charge accumulated by each pixel within the frame period is equal to the sum of the charges accumulated by the multiple taps. Therefore, the corresponding IR value can be calculated based on the pixel value in the rawphase image, thus obtaining the corresponding IR image. Further, moving pixels are determined based on the pixel values ​​in each IR image. This involves comparing the IR values ​​in the three IR images. If the object is not moving, the IR values ​​of the same pixel in the three IR images should be the same or similar. If the difference is large, it indicates that motion has occurred. In one embodiment, an IR difference threshold is set. If the difference between any two IR values ​​of a pixel in the three IR images is greater than or equal to the IR difference threshold, it indicates that there is motion in the scene corresponding to that pixel, and the pixel is identified as a moving pixel. If the difference between any two IR values ​​of a pixel in the three IR images is less than the IR difference threshold, it indicates that the pixel is not a moving pixel. IR values ​​represent the intensity of the light signal collected by a pixel. If the target object moves, the intensity of the light signal collected by the pixel will change. For example, in the first frame period, the pixel collects the light signal reflected by the target, and in the second frame period, due to the movement of the target object, the pixel collects the ambient light signal. Therefore, determining moving pixels by the difference between IR values ​​is relatively accurate, thus ensuring the accuracy of motion artifact removal.

[0040] The control and processing circuit selects a reference rawphase image from multiple rawphase images. The IR image corresponding to the reference rawphase image is the reference IR image. The other rawphase images are the non-reference rawphase images and their corresponding non-reference IR images. The moving pixels in the non-reference rawphase image are determined by comparing the IR values ​​in the non-reference IR images with the reference IR values. Furthermore, the raw values ​​of the moving pixels, i.e., the pixel values, are corrected according to a preset motion pixel correction rule to obtain the corrected rawphase image.

[0041] In one embodiment, any one of multiple rawphase images is selected as a reference image, such as the second rawphase image. The second IR image then becomes the reference IR image. Moving pixels are identified by determining whether other IR values ​​are similar to the reference IR values. For example, a pixel A in the second rawphase image might have an IR value corresponding to the IR value of a hand, while in the third rawphase image, the IR value of pixel A might correspond to the IR value of a background wall. Because the scene differs, the pixel's IR value is different, making pixel A a moving pixel. The raw value of the moving pixel A in the third rawphase image needs to be corrected to eliminate motion artifacts. Specifically, the IR value of each pixel in the other two IR images is compared one by one with the reference IR value in the corresponding reference IR image to select all moving pixels. Then, each pixel is corrected according to a preset motion pixel correction rule.

[0042] Once a moving pixel is identified, the control and processing circuit selects the rawphase image corresponding to the moving pixel from the previous depth detection cycle, where the depth detection cycle includes multiple frame cycles. It calculates the difference between the pixel value in the IR image corresponding to the rawphase image from the previous depth detection cycle and the pixel value in the reference IR image. If the difference is less than a difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the rawphase image from the previous depth detection cycle. If the difference is not less than the difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the reference rawphase image.

[0043] In this application, a round-robin sampling mode is adopted, where three frame periods correspond to one depth detection period, outputting one depth map. The previous depth detection period also includes three frame periods, with one rawphase map output in each frame period. Therefore, the historical frame corresponding to the first rawphase map in the current depth detection period is the first rawphase map in the previous depth detection period. Similarly, the historical frame corresponding to the second rawphase map in the current depth detection period is the second rawphase map in the previous depth detection period. Furthermore, the IR map corresponding to the rawphase and the reference rawphase follow the same correspondence rules.

[0044] Specifically, the preset motion pixel correction rules include the following cases:

[0045] 1. If only some of the non-reference rawphase images contain moving pixels, and the IR value of the historical frame's IR image corresponding to the rawphase image containing the moving pixel is less than the IR value in the reference IR image, then the raw value in the historical frame's rawphase image will replace the raw value of the moving pixel.

[0046] Specifically, during the calibration process, the moving pixels in each rawphase image are inconsistent; some pixels are moving pixels in one rawphase image but not in others. Moving pixels in each rawphase image are determined by comparing the baseline IR value with other IR values. For example, if a pixel B is selected, and the difference between its IR value in the third IR image and the baseline IR value is less than the IR difference threshold, while the difference between its IR value in the first IR image and the baseline IR value is not less than the IR difference threshold, then pixel B in the first rawphase image is a moving pixel, and its corresponding raw value needs calibration. The IR value of pixel B in the IR image corresponding to the first rawphase image in the previous depth probing cycle is compared with the baseline IR value in the current depth probing cycle. If it is less than the IR difference threshold, the raw value of pixel B in the first rawphase image in the previous depth probing cycle replaces the raw value of pixel B in the first rawphase image in the current probing cycle.

[0047] 2. If each of the multiple non-reference rawphase images contains a moving pixel, and the IR value of the historical frame's IR image corresponding to each rawphase image is less than the IR value in the reference IR image, then the raw value in the historical frame's rawphase image replaces the raw value of the moving pixel.

[0048] Specifically, for example, if we select a pixel C, and the difference between the IR value in the third IR image and the reference IR value is not less than the IR difference threshold, and the IR value in the first IR image is also not less than the reference IR value, then pixel C in both the first and third rawphase images is a moving pixel, and its corresponding raw value needs to be corrected. Since we selected the second rawphase image as the reference, we need to correct the other two rawphase images. Similarly, we select the raw values ​​from historical frames for correction. We compare the IR value of pixel C in the corresponding IR images of the first and third rawphase images from the previous depth probing period with the reference IR value in the current depth probing period. If it is less than the IR difference threshold, then the raw value of pixel C in the first and third rawphase images from the previous depth probing period replaces the raw value of pixel C in the first and third rawphase images from the current probing period.

[0049] 3. If rules 1 and 2 are not satisfied, the raw values ​​of the non-baseline rawphase diagram are corrected using the raw values ​​in the baseline rawphase diagram.

[0050] According to rules 1 and 2, once the moving pixels are determined, their raw values ​​need to be corrected using the raw values ​​of historical frames. The difference between the IR value of the historical frame and the current baseline IR value must be less than the IR difference threshold; otherwise, correction is not possible. In this case, the raw values ​​in the baseline rawphase image are directly used to correct the raw values ​​in the non-baseline rawphase image. The current depth detection cycle can be considered to no longer use a rotating sampling mode, but a non-rotating sampling mode. Understandably, this correction mode combines three rawphase images into one. For faster calculation, the baseline rawphase can also be directly selected for depth calculation.

[0051] After obtaining the corrected rawphase image, the control and processing circuit performs depth calculation based on the corrected rawphase image, and the specific calculation process is the same as above.

[0052] Example 2

[0053] In some embodiments, to improve detection accuracy and range, depth cameras can also employ multi-frequency fusion for depth detection, such as a dual-frequency fusion algorithm. The following explanation uses two frequencies as an example. In one embodiment, the transmitter alternately emits two frequencies of light signals towards the target area. The two frequencies correspond to two pulse periods Ta and two pulse widths Th. The first frequency light signal is emitted in the first frame period, and the second frequency light signal is emitted in the second frame period. The first frequency is lower than the second frequency. During the first frame period, the collector acquires the reflected light signal and generates an electrical signal. The control and processing circuit processes the electrical signal to calculate a first depth map. Similarly, during the second frame period, a second depth map is calculated, and the first and second depth maps are fused to calculate the target depth map.

[0054] In one embodiment, each pixel in the iTOF image sensor includes three taps, taking taps a, b, and c as an example. The exposure time of each tap is fixed and unchanging within consecutive frame periods, assuming that the reflected light signal is collected by the first and second taps (which also collect ambient light signals), accumulating charge Q1 and Q2; the third tap is used to collect the ambient light signal and accumulate charge Q3. At this time, the iTOF image sensor outputs a first rawphase image (corresponding to a first frequency) in the first frame period, and the control and processing circuit can calculate a first depth map of the target scene based on the first rawphase image; in the second frame period, it outputs a second rawphase image (corresponding to a second frequency), and the control and processing circuit can calculate a second depth map of the target scene based on the second rawphase image; further, the first and second depth maps are fused to obtain the target depth map, and the specific calculation process is as follows:

[0055] The first and second flight times were calculated based on the amount of charge accumulated at the taps:

[0056]

[0057] The corresponding number of the first and second winding cycles are calculated based on the first and second flight times: t1 + n1 × Ta1 = t2 + n2 × Ta2;

[0058] The first depth value is calculated based on the first number of winding cycles and the first flight time: D1 = c(t1 + n1 × Ta1) / 2;

[0059] The second depth value is calculated based on the second number of winding cycles and the second flight time: D2=c(t2+n2×Ta2) / 2;

[0060] The fusion depth is then: D = wD1 + (1 - w)D2;

[0061] Where n1 and n2 are integers, representing the number of winding cycles; D1 and D2 correspond to the depth values ​​measured at the first and second frequencies, respectively; w is the weight, which is set according to the specific system scheme.

[0062] If there is motion, the reflected light signals collected by the same pixel in the first frame period and the second frame period will come from different target scenes, and the corresponding depth values ​​will also deviate. At this time, the fused depth map will produce motion artifacts, and the moving pixels need to be corrected.

[0063] The control and processing circuit is also used to receive the first raw phase image, process it to obtain the corresponding first IR image, receive the second raw phase image, process it to obtain the corresponding second IR image, compare the IR pixel values ​​in the first IR image and the second IR image to determine the moving pixel, and correct the depth value in the target depth image corresponding to the moving pixel to obtain an accurate target depth image.

[0064] Specifically, the process begins by identifying moving pixels. This involves selecting one frequency's rawphase image as the baseline rawphase and determining its corresponding baseline IR image. The other frequency's rawphase image is used as a non-baseline rawphase image and a non-baseline IR image. Moving pixels are then identified based on the IR values ​​(pixel values) in the IR images. The principle behind this identification is detailed in the description of Embodiment 1: comparing the IR values ​​of the same pixel in the non-baseline IR image with those in the baseline IR image. If the object is not moving, the pixel values ​​of the same pixel should be the same or similar in both IR images. A significant difference indicates motion. In one embodiment, for example, a first rawphase image is selected as the baseline rawphase image. A difference threshold is set, and the IR difference between the second and first IR images is calculated. If the difference between the two IR values ​​corresponding to a pixel is greater than or equal to the difference threshold, it indicates that there is motion in the scene corresponding to that pixel, and the pixel is identified as a moving pixel. If the difference between the two IR values ​​of a pixel is less than the IR difference threshold, the pixel is not considered a moving pixel.

[0065] The moving pixels in the target depth map are obtained through the IR map. It is necessary to correct the erroneous depth values ​​of these moving pixels to the correct depth values. Taking pixel E as an example, assuming that pixel E is determined to be a moving pixel through the first and second IR maps, it is necessary to calculate the accurate number of winding cycles n2 corresponding to pixel E. This is used to calculate the accurate second depth value corresponding to pixel E and the final corrected depth map in the target depth map.

[0066] It is understandable that in practical applications, the motion pixels in the second depth map can be corrected first by determining the motion pixels before performing depth fusion calculation. The description of the above embodiments does not limit the specific execution order.

[0067] Specifically, in the reference IR image, the neighborhood range of pixel E is taken, and non-moving pixels with similar pixel values ​​to pixel E are found within the neighborhood range. Since the distance values ​​of non-moving pixels are correct, the number of winding cycles of pixel E is calculated using the distance value D3 of non-moving pixels, that is: D3=t2+n2×Ta2. The number of winding cycles n2 corresponding to the second flight time is solved, and the corrected depth value of pixel E is further calculated according to the calculation process of the fusion depth value described above.

[0068] To reduce errors, multiple non-moving pixels with IR values ​​close to that of pixel E are usually searched. However, the distance values ​​of these pixels are not necessarily reliable, mainly for two reasons: 1. There may be misjudgments, where moving pixels are mistakenly identified as non-moving pixels, and the distance value in this case is incorrect; 2. The selected non-moving pixels may happen to be located at the edge of the foreground and background of the image, and the corresponding distance values ​​are also unreliable. Those unreliable distance values ​​are then discarded.

[0069] In some embodiments, the non-moving pixels used to calculate the winding period of pixel E are determined through the following two steps: First, pixels located at the edges are removed from the target depth map corresponding to the neighborhood of pixel E; Second, the mean depth, maximum depth, and minimum depth of all non-moving pixels in the neighborhood are calculated, and the maximum or minimum depth is used as the first depth value corresponding to the non-moving pixel to calculate the number of winding periods of the moving pixel based on the mean depth. Specifically, if the mean depth is closest to the maximum depth, the maximum depth is used as the reference to calculate the winding period of pixel E; otherwise, if the mean depth is closest to the minimum depth, the minimum depth is used as the reference to calculate the winding period of pixel E, and the corrected depth value of pixel E is further calculated. In one embodiment, the choice between the maximum depth and the mean depth, and the absolute value of the difference between the minimum depth and the mean depth, can be used to determine whether to use the maximum depth or the minimum depth as the reference.

[0070] In some embodiments, if the depth values ​​of some pixels are still not corrected after the final correction, median filtering is used for optimization.

[0071] In some embodiments, the control and processing circuit can further regulate the taps to acquire optical signals using a rotating sampling mode. The control and processing circuit controls the transmitter to emit a pulse beam with a first frequency during a first depth detection cycle, and regulates multiple taps in the pixel to acquire reflected pulse beams or background light according to the rotating sampling mode during the first depth detection cycle to generate charge, causing the collector to output multiple first raw phase images. Additionally, the control and processing circuit controls the transmitter to emit a pulse beam with a second frequency during a second depth detection cycle, and regulates multiple taps in the pixel to acquire reflected pulse beams or background light according to the rotating sampling mode during the second depth detection cycle to generate charge, causing the collector to output multiple second raw phase images. The first and second depth detection cycles are generated alternately. For the single-frequency regulation and sampling process, please refer to Embodiment 1, which will not be elaborated further here. The specific example will still be a three-tap setup.

[0072] Specifically, the iTOF image sensor acquires three first raw phase images at the first frequency and three second raw phase images at the second frequency. If motion is present, the raw phase images at a single frequency need to be corrected first, and then the depth map fused from the two frequencies needs to be corrected. Therefore, the multiple raw phase images at a single frequency are first corrected, and the depth map is calculated based on the corrected raw phase images. For the correction of multiple raw phase images at a single frequency, see Example 1. When performing motion correction on the depth maps at two frequencies, the reference raw phase image from the single-frequency correction can be selected as the reference for correcting the depth values ​​of moving pixels.

[0073] Example 3

[0074] In an exemplary embodiment, see [reference] Figure 6 This is a flowchart illustrating a method for eliminating motion artifacts in one embodiment of this application. The method includes:

[0075] Step S600: Emit pulse beams to targets in the space region within multiple frame periods.

[0076] In this embodiment, pulse beams of the same frequency are emitted within multiple frame periods.

[0077] Step S601: Acquire the reflected pulse beams from the target in each frame period and generate a rawphase map; the pixel values ​​in the rawphase map are the charge generated by the tap-acquired reflected pulse beams or background light.

[0078] In this embodiment, the amount of charge generated by the reflected pulse beam or background light is acquired through a tap rotation mode. Taking three taps a, b, and c as an example, in the first frame period, the first tap, the second tap, and the third tap sequentially activate the charge accumulation signal; in the second frame period, the second tap, the third tap, and the first tap sequentially activate the charge accumulation signal; in the third frame period, the third tap, the first tap, and the second tap sequentially activate the charge accumulation signal; thus, three rawphase images are obtained.

[0079] Furthermore, each cycle includes three exposure times, and the charge amount of three raw phase images is acquired according to the rotational sampling mode. The charge amount sampled by tap a at the first exposure time in the first raw phase image is denoted as Q. a1 Tap b is sampled at the second exposure time and recorded as Q. b2 Tap c is sampled at the third exposure time and recorded as Q. c3 In the second raw phase image, the charge sampled by tap b at the first exposure time is denoted as Q. b1 The charge sampled by tap c at the second exposure time is denoted as Q. c2 The charge sampled by tap a at the third exposure time is denoted as Q. a3 In the third raw phase image, the charge sampled by tap c at the first exposure time is denoted as Q. c1 The charge sampled by tap a at the second exposure time is denoted as Q. a2 The charge sampled by tap b at the third exposure time is denoted as Q. b3 .

[0080] Step S602: Receive multiple rawphase images and process them to obtain the IR image corresponding to each rawphase image.

[0081] In this embodiment, the IR map collected in each frame period is obtained. The pixel value of the IR map (denoted as IR value) corresponds to the amount of charge accumulated by each pixel in the frame period. The pixel accumulates charge through multiple configured taps. The amount of charge accumulated by each pixel in the frame period is equal to the sum of the charge accumulated by the multiple taps. Therefore, the corresponding IR value can be calculated based on the pixel value (raw value) in the raw phase map, and the corresponding IR map can be obtained.

[0082] Step S603: Determine the moving pixels based on the pixel values ​​in the IR image, and correct the pixel values ​​in the rawphase image corresponding to the moving pixels to obtain the corrected rawphase image.

[0083] In this embodiment, moving pixels are determined based on the pixel values ​​in each IR image. This involves comparing the IR values ​​in the three IR images. If the object is not moving, the IR values ​​of the same pixel should be the same or similar in the three IR images. A large difference indicates movement. In one embodiment, an IR difference threshold is set. If the difference between any two IR values ​​of a pixel in the three IR images is greater than or equal to the IR difference threshold, it indicates that there is motion in the scene corresponding to that pixel, and the pixel is identified as a moving pixel. If the difference between any two IR values ​​of a pixel in the three IR images is less than the IR difference threshold, the pixel is not considered a moving pixel. The IR value represents the intensity of the light signal collected by the pixel. If the target object moves, the intensity of the light signal collected by the pixel will change. For example, in the first frame period, the pixel collects the light signal reflected by the target; in the second frame period, due to the movement of the target object, the pixel collects the ambient light signal. Therefore, determining moving pixels by the difference between IR values ​​is relatively accurate, thus ensuring the accuracy of motion artifact removal.

[0084] In this embodiment, any frame is selected from multiple rawphase images to determine the reference rawphase image and its corresponding reference IR image. The other rawphase images are the non-reference rawphase images and non-reference IR images. The difference between the pixel values ​​in the reference IR image and the pixel values ​​in the non-reference IR images is calculated, and pixels with a difference not less than a difference threshold are identified as moving pixels in the non-reference rawphase images. Furthermore, the raw values ​​(i.e., pixel values) of the moving pixels are corrected according to a preset motion artifact correction rule to obtain the corrected rawphase image. The motion artifact correction rule has been described in the above embodiments and will not be repeated here.

[0085] Step S604: Calculate the target depth map based on the corrected rawphase map.

[0086] In this embodiment, the specific calculation process has been described in the above embodiments and will not be repeated here.

[0087] In an exemplary embodiment, step S603 includes:

[0088] The rawphase image corresponding to the moving pixel is selected from the rawphase image of the previous depth detection period, which includes multiple frame periods. The difference between the pixel value in the IR image corresponding to the rawphase image of the previous depth detection period and the pixel value in the reference IR image is calculated. If the difference is less than the difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the rawphase image of the previous depth detection period. If the difference is not less than the difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the reference rawphase image.

[0089] For a detailed explanation of the above steps, please refer to the description in Example 2, which will not be repeated here.

[0090] Example 4

[0091] In an exemplary embodiment, see [reference] Figure 7 This is a flowchart illustrating a method for eliminating motion artifacts according to another embodiment of this application; the method specifically includes:

[0092] Step S700: Emit a pulse beam with a first frequency or a second frequency to a target in the space region during consecutive frame periods.

[0093] In this embodiment, pulse beams of a first frequency and a second frequency are alternately emitted within consecutive frame periods. For example, a pulse beam of the first frequency is emitted in the previous frame period, and a pulse beam of the second frequency is emitted in the current frame period.

[0094] Step S701: Acquire a reflected pulse beam of the first frequency reflected back from the target and generate a first rawphase image; acquire a reflected pulse beam of the second frequency reflected back from the target and generate a second rawphase image; the pixel values ​​in the rawphase image are the amount of charge generated by the tap acquiring the reflected pulse beam or background light.

[0095] Step S702: Receive the first rawphase image and process it to obtain the corresponding first depth image and the corresponding first IR image; receive the second rawphase image and process it to obtain the second depth image and the corresponding second IR image.

[0096] In one embodiment, each pixel in the iTOF image sensor includes three taps, taking taps a, b, and c as an example. The exposure time of each tap is fixed and constant within consecutive frame periods, i.e., it is assumed that the reflected light signal is collected by the first and second taps (the first and second taps also collect the ambient light signal) and accumulates charge Q1 and Q2; the third tap is used to collect the ambient light signal and accumulate charge Q3. At this time, the iTOF image sensor outputs a first rawphase image (corresponding to the first frequency) in the first frame period, and the control and processing circuit can calculate the first depth map of the target scene based on the first rawphase image; in the second frame period, it outputs a second rawphase image (corresponding to the second frequency).

[0097] Step S703: Merge the first depth map and the second depth map to obtain the target depth map.

[0098] A first depth map of the target scene can be calculated based on the first rawphase map; a second rawphase map is output during the second frame period, and the control and processing circuit can calculate the second depth map of the target scene based on the second rawphase map; further, the first depth map and the second depth map are fused to obtain the target depth map. The specific calculation process can be found in Embodiment 2.

[0099] Step S704: Determine the moving pixels by comparing the pixel values ​​in the first IR image and the second IR image, and correct the depth value in the target depth image corresponding to the moving pixels.

[0100] One of two frequencies is selected as the reference rawphase image, and the corresponding reference IR image is determined. The other frequency is used as a non-reference rawphase image and a non-reference IR image. Moving pixels are determined based on the IR values ​​(pixel values) in the IR images. The principle of determining moving pixels based on IR values ​​can be found in the description of Embodiment 1. That is, the IR value of the same pixel in the non-reference IR image is compared with the IR value of the pixel in the reference IR image. If the object is not moving, the pixel value of the same pixel in the two IR images should be the same or similar. If the difference is large, it indicates that motion has occurred. In one embodiment, for example, a first rawphase image is selected as the reference rawphase image, a difference threshold is set, and the IR difference between the second IR image and the first IR image is calculated. If the difference between the two IR values ​​corresponding to a pixel is greater than or equal to the difference threshold, it indicates that there is motion in the scene corresponding to that pixel, and the pixel is determined as a moving pixel; if the difference between the two IR values ​​of a pixel is less than the IR difference threshold, it indicates that the pixel is not a moving pixel.

[0101] The moving pixels in the target depth map are obtained through the IR map. It is necessary to correct the erroneous depth values ​​of these moving pixels to the correct values. Taking pixel E as an example, assuming pixel E is determined to be a moving pixel through the first and second IR maps, the accurate number of winding cycles n2 corresponding to pixel E needs to be calculated. This is used to calculate the accurate second depth value of pixel E and the final corrected depth map in the target depth map. Specifically, the neighborhood of pixel E is taken in the reference IR map. Within this neighborhood, non-moving pixels with similar pixel values ​​to pixel E are found. Since the distance values ​​of the non-moving pixels are correct, the number of winding cycles for pixel E is calculated using the distance value D3: D3 = t2 + n2 × Ta2. This solves for the number of winding cycles n2 corresponding to the second flight time. Furthermore, the corrected depth value of pixel E is calculated based on the previously described process for calculating the fused depth value.

[0102] In an exemplary embodiment, step S704 includes:

[0103] Select the first rawphase image and the corresponding first IR image as the reference rawphase image and reference IR image; determine the neighborhood range of the moving pixel in the reference IR image, and the non-moving pixels within the neighborhood range; calculate the number of winding cycles of the moving pixel based on the first depth value corresponding to the non-moving pixel; calculate the corrected depth value of the moving pixel based on the number of winding cycles.

[0104] In an exemplary embodiment, step S704 further includes;

[0105] Determine the neighborhood range of the moving pixel in the reference IR image; remove pixels located at the edge of the neighborhood range; calculate the mean depth, maximum depth, and minimum depth of all non-moving pixels in the neighborhood range; based on the mean depth, determine whether the maximum or minimum depth is used as the first depth value corresponding to the non-moving pixel to calculate the number of winding cycles of the moving pixel.

[0106] For a detailed explanation of the above steps, please refer to the description in Example 2, which will not be repeated here.

[0107] Example 5

[0108] This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program. When the program is executed by a processor, it implements the corresponding function. The computer-readable storage medium of this embodiment is used for computer programs, which, when executed by a processor, implement the motion artifact elimination method described in the above embodiments.

[0109] It is understood that the content described in the above embodiments is similar to the content described in the optimization method in Embodiment 1. For details, please refer to the content of the method described in Embodiment 1, which will not be repeated here.

[0110] The above is a description of the method and apparatus for detecting task execution programs provided by the present invention. For those skilled in the art, based on the ideas of the embodiments of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A depth camera for eliminating motion artifacts, characterized in that, include: A transmitter used to emit pulsed beams of light to a target in a space region over multiple frame periods; A collector is configured to acquire a reflected pulse beam reflected back from a target and generate a rawphase map in each frame period; wherein the collector includes an image sensor consisting of multiple pixels, each pixel including multiple taps, each tap being used to acquire the reflected pulse beam or background light to generate a charge, and the pixel values ​​in the rawphase map being the charge generated by the taps. The control and processing circuit receives at least three rawphase images, adds the charge collected by the taps corresponding to each pixel in each rawphase image to obtain a corresponding IR image; selects any frame from the multiple rawphase images to determine a reference rawphase image and a corresponding reference IR image; calculates the difference between the pixel value in the reference IR image and the pixel value in each non-reference IR image, and determines the pixels with a difference not less than a difference threshold as moving pixels in the corresponding non-reference rawphase image; corrects the pixel value of the moving pixels in each non-reference rawphase image according to a preset motion pixel correction rule to obtain multiple corrected rawphase images, wherein the corrected rawphase images are rawphase images after motion artifact elimination; and calculates a target depth map based on the multiple corrected rawphase images.

2. The depth camera according to claim 1, characterized in that, Each frame period includes multiple exposure moments, and each tap acquires the reflected pulse beam or background light at the corresponding exposure moment; The control and processing circuit controls multiple taps in the pixel to collect the reflected pulse beam or background light according to the rotating sampling mode in multiple frame periods to generate charge, so that the collector outputs multiple raw phase images.

3. The depth camera according to claim 2, characterized in that, The plurality of taps includes a first tap, a second tap, and a third tap; the plurality of frame periods includes a first frame period, a second frame period, and a third frame period; the collector is further configured to: During the first frame period, the first tap, the second tap, and the third tap collect the reflected pulse beam or background light to generate charge, thereby obtaining the first rawphase image. During the second frame period, the second tap, the third tap, and the first tap collect the reflected pulse beam or background light to generate charge, thereby obtaining the second rawphase image. During the third frame period, the third tap, the first tap, and the second tap collect the charge generated by the reflected pulse beam or background light to obtain the third rawphase image.

4. The depth camera according to claim 1, characterized in that, The control and processing circuit is also used for: Select the rawphase image of the moving pixel in the previous depth detection cycle, wherein the depth detection cycle includes the plurality of frame cycles; The difference is calculated based on the pixel values ​​in the IR image corresponding to the raw phase image in the previous depth detection cycle and the pixel values ​​in the reference IR image; If the difference is less than the difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the raw phase image of the previous depth detection cycle; If the difference is not less than the difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the reference rawphase image.

5. A method for eliminating motion artifacts, characterized in that, The method includes: Firing pulsed beams at targets in the space region over multiple frame periods; Within each frame period, a reflected pulse beam from the target is acquired and a rawphase map is generated; the pixel values ​​in the rawphase map are the amount of charge generated by the tap acquired from the reflected pulse beam or background light. Receive at least three rawphase images, and sum the charge amounts collected by the taps corresponding to each pixel in each rawphase image to obtain the corresponding IR image; From the multiple rawphase images, any frame is selected to determine the reference rawphase image and the corresponding reference IR image; the difference between the pixel value in the reference IR image and the pixel value in each non-reference IR image is calculated, and the pixels with the difference not less than the difference threshold are determined as the moving pixels in the corresponding non-reference rawphase image; the pixel values ​​of the moving pixels in each non-reference rawphase image are corrected according to the preset motion pixel correction rule to obtain multiple corrected rawphase images, and the corrected rawphase images are the rawphase images after eliminating motion artifacts. The target depth map is calculated based on multiple corrected rawphase images.

6. The method for eliminating motion artifacts according to claim 5, characterized in that, The pixel values ​​of the moving pixels in each of the non-reference raw phase images are corrected according to a preset motion pixel correction rule, including: Select the rawphase image of the moving pixel in the previous depth detection cycle, wherein the depth detection cycle includes the plurality of frame cycles; The difference is calculated based on the pixel values ​​in the IR image corresponding to the raw phase image in the previous depth detection cycle and the pixel values ​​in the reference IR image; If the difference is less than the difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the raw phase image of the previous depth detection cycle; If the difference is not less than the difference threshold, the pixel value of the moving pixel is corrected using the pixel value in the reference rawphase image.

7. A depth camera for eliminating motion artifacts, characterized in that, include: A transmitter for alternately emitting pulse beams with a first frequency and a second frequency to a target in a space region in consecutive frame periods; A collector is configured to acquire a reflected pulse beam of the first frequency reflected back from a target and generate a first rawphase image, and to acquire a reflected pulse beam of the second frequency reflected back from a target and generate a second rawphase image; wherein the collector includes an image sensor composed of multiple pixels, each pixel including multiple taps, each tap being used to acquire the reflected pulse beam or background light to generate a charge, and the pixel value in the rawphase image being the charge generated by the tap. The control and processing circuit receives the first rawphase image and processes it to obtain a first depth image and a corresponding first IR image; receives the second rawphase image and processes it to obtain a second depth image and a corresponding second IR image; fuses the first depth image and the second depth image to obtain a target depth image; selects the first rawphase image and the corresponding first IR image as a reference rawphase image and a reference IR image, and selects the second rawphase image and the corresponding second IR image as a non-reference rawphase image and a non-reference IR image; calculates the IR difference between the second IR image and the first IR image; identifies pixels whose IR difference is greater than or equal to a preset difference threshold as moving pixels; determines the neighborhood range of the moving pixels in the reference IR image and the non-moving pixels within the neighborhood range; calculates the number of winding cycles of the moving pixels based on the depth values ​​corresponding to the non-moving pixels; calculates the corrected depth value of the moving pixels based on the number of winding cycles; and corrects the depth value in the target depth image corresponding to the moving pixels to obtain a depth image after eliminating motion artifacts.

8. The depth camera according to claim 7, characterized in that, Each frame period includes multiple exposure moments, and each tap acquires the reflected pulse beam or background light at the corresponding exposure moment; The control and processing circuit controls the transmitter to emit a pulse beam with the first frequency during the first depth detection cycle, and adjusts multiple taps in the pixel to collect the reflected pulse beam or background light according to the rotating sampling mode during the first depth detection cycle to generate charge so that the collector outputs multiple first raw phase images. Furthermore, the control and processing circuit controls the transmitter to emit a pulse beam with the second frequency during the second depth detection cycle, and adjusts multiple taps in the pixel to collect the reflected pulse beam or background light according to the rotating sampling mode during the second depth detection cycle to generate charge so that the collector outputs multiple second raw phase images. The first depth detection period and the second depth detection period include a plurality of the frame periods.

9. A method for eliminating motion artifacts, characterized in that, The method further includes: Pulsed beams with a first frequency and a second frequency are alternately emitted toward a target in the space region in consecutive frame periods; The system acquires a reflected pulse beam of the first frequency reflected back from the target and generates a first rawphase image, and acquires a reflected pulse beam of the second frequency reflected back from the target and generates a second rawphase image; the pixel values ​​in the rawphase image are the amount of charge generated by the tap acquiring the reflected pulse beam or background light. The first raw phase image is received and processed to obtain a corresponding first depth image and a corresponding first IR image; the second raw phase image is received and processed to obtain a second depth image and a corresponding second IR image. The target depth map is obtained by fusing the first depth map and the second depth map; The first rawphase image and its corresponding first IR image are selected as the reference rawphase image and reference IR image, and the second rawphase image and its corresponding second IR image are selected as the non-reference rawphase image and non-reference IR image. The IR difference between the second IR image and the first IR image is calculated. Pixels whose IR difference is greater than or equal to a preset difference threshold are identified as moving pixels. The neighborhood range of the moving pixels in the reference IR image and the non-moving pixels within the neighborhood range are determined. The number of winding cycles of the moving pixels is calculated based on the depth value corresponding to the non-moving pixels. The corrected depth value of the moving pixels is calculated based on the number of winding cycles, and the depth value in the target depth image corresponding to the moving pixels is corrected to obtain a depth image after motion artifact elimination.

10. The method for eliminating motion artifacts according to claim 9, characterized in that, Determining the neighborhood range of the moving pixel in the reference IR map, and the non-moving pixels within the neighborhood range, includes: Determine the neighborhood range of the moving pixel in the reference IR map; Remove pixels located at the edges of the neighborhood range; Calculate the mean depth, maximum depth, and minimum depth of all non-moving pixels within the neighborhood; Based on the mean depth, the maximum or minimum depth value is used as the first depth value corresponding to the non-moving pixel to calculate the number of winding cycles of the moving pixel.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer-readable storage medium stores a computer program that can be executed by at least one processor to cause the at least one processor to perform the steps of the method for eliminating motion artifacts as described in any one of claims 5 or 6, or to perform the steps of the method for eliminating motion artifacts as described in any one of claims 9 or 10.

Images