Moving image noise reduction device and moving image noise reduction method

The video noise reduction device and method address blurring and low generation rate issues by separating image data into frequency components, estimating motion vectors with an event-based sensor, and performing motion compensation to enhance noise reduction accuracy and speed.

JP2024164863APending Publication Date: 2024-11-28JVC KENWOOD CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023080527
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Conventional video noise reduction methods using frame averaging lead to blurring of moving subjects, and existing techniques for high-speed event-based vision sensors do not effectively increase the generation rate or address memory processing speed for accurate noise reduction.

Method used

A video noise reduction device and method that separates image data into low and high frequency components, uses an event-based sensor to estimate motion vectors at a higher frame rate, and performs motion compensation to reduce noise by adding these components at a predetermined ratio.

Benefits of technology

Achieves highly accurate video noise reduction with improved motion estimation and reduced blurring, enabling high-speed processing and effective noise reduction in moving images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024164863000001_ABST
    Figure 2024164863000001_ABST
Patent Text Reader

Abstract

To provide a moving image noise reduction technique with high accuracy.SOLUTION: A filter part separates image data from an image sensor for performing imaging at a predetermined frame rate into a low frequency component and a first high frequency component. A motion vector estimation part 34 reads out data from an event sensor for asynchronously outputting information of a pixel whose luminance is changed at a frame rate higher than the predetermined frame rate, to estimate a motion vector. A motion compensation part 32 performs motion compensation on the basis of the motion vector. The filter part adds the first high frequency component and a second high frequency component extracted from an image obtained by the motion compensation at a predetermined ratio, to generate a third high frequency component and adds the low frequency component and the third high frequency component, to reduce the noise of the image data.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an apparatus and method for reducing moving image noise. [Background technology]

[0002] In order to reduce noise in captured video, conventional technology reduces random noise by, for example, simply adding up several frames of video between frames, dividing by the number of frames, and averaging. The averaging is performed using a moving average that updates the frame interval for calculating the average for each frame. In this case, there is an issue that blurring occurs for moving subjects, and the image obtained by averaging has noise and blur. On the other hand, when measuring the distance to a stationary object precisely using a TOF (Time Of Flight) sensor, there is no problem with blurring, so this noise reduction method is effective, and the more frames are averaged, the more effective it becomes, so it is often used for noise reduction in distance measuring cameras.

[0003] A high-speed, low-latency event-based vision sensor (EVS) (also called "event-based sensor") has been developed that asynchronously detects luminance changes in each pixel of an image and outputs the coordinates of pixels with luminance changes along with time information. Patent Document 1 discloses a processing technique for signals generated by an asynchronous event-based sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special table number 2021-508975 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to improve the accuracy of noise reduction in videos, it is necessary to capture the motion of a subject at a high frame rate for a blurred video, estimate a motion vector with high accuracy, and perform noise reduction using a motion-compensated image. Patent Document 1 does not disclose a technology for acquiring image motion information using an event-based sensor, and although it estimates a generation rate from asynchronous data output from an event-based sensor, it does not disclose a method for increasing the speed of memory read and write, which is necessary to make the generation rate compatible with high frames.

[0006] The present invention has been made in view of the above circumstances, and has an object to provide a highly accurate moving image noise reduction technique. [Means for solving the problem]

[0007] In order to solve the above problems, a video noise reduction device according to an embodiment of the present invention includes a filter unit that separates image data from an image sensor that captures images at a predetermined frame rate into a low frequency component and a first high frequency component, a motion vector estimation unit that reads data from an event sensor that asynchronously outputs information on pixels whose luminance has changed at a frame rate higher than the predetermined frame rate to estimate a motion vector, and a motion compensation unit that performs motion compensation based on the motion vector. The filter unit generates a third high frequency component by adding the first high frequency component and a second high frequency component extracted from an image obtained by the motion compensation at a predetermined ratio, and reduces noise in the image data by adding the low frequency component and the third high frequency component.

[0008] Another aspect of the present invention is a method for reducing noise in a moving image. This method includes a filtering step of separating image data from an image sensor capturing an image at a predetermined frame rate into a low frequency component and a first high frequency component, a motion vector estimation step of estimating a motion vector by reading data from an event sensor that asynchronously outputs information on pixels whose luminance has changed at a frame rate higher than the predetermined frame rate, and a motion compensation step of performing motion compensation based on the motion vector. The filtering step generates a third high frequency component by adding the first high frequency component and a second high frequency component extracted from an image obtained by the motion compensation at a predetermined ratio, and reduces noise in the image data by adding the low frequency component and the third high frequency component.

[0009] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0010] According to the present invention, a highly accurate moving image noise reduction technique can be provided. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a configuration diagram of an event-based sensor. [Diagram 2] 2(a) to 2(c) are diagrams illustrating an event signal output by an event-based sensor. [Diagram 3] 1 is a diagram showing a basic configuration of a moving image noise reduction device according to a first embodiment. [Figure 4] FIG. 13 is a diagram illustrating a block range used for motion vector estimation. [Diagram 5] FIG. 2 is a block diagram of a motion vector estimation unit. [Figure 6] 5 is a timing chart illustrating an operation of a motion vector estimating unit. [Figure 7]10 is a timing chart illustrating the operation of the motion vector estimating section in the case of a faster shutter speed. [Figure 8] FIG. 2 is a diagram illustrating how image pixels are read out horizontally and vertically. [Figure 9] FIG. 2 is a diagram illustrating the movement of a subject within a frame. [Figure 10] 10 is a diagram for explaining code data of an event-based sensor indicating a change in brightness due to the movement of the subject in FIG. 9. FIG. [Figure 11A] FIG. 2 is a diagram showing vertical coefficients and surrounding coefficients of each pixel of a block of an image. [Figure 11B] FIG. 13 is a diagram showing weighting coefficients for the amount of vertical motion of each pixel in an image block. [Figure 12A] 11B is a diagram showing a result of multiplying the sign data on the bright side (plus side) of the EVS output in FIG. 10 by the weighting coefficient of the vertical motion amount in FIG. 11B. [Figure 12B] 11B is a diagram showing a result of multiplying the sign data on the dark side (negative side) of the EVS output in FIG. 10 by the weighting coefficient of the vertical motion amount in FIG. 11B. [Figure 13A] FIG. 2 illustrates the horizontal and peripheral coefficients of each pixel in a block of an image. [Figure 13B] FIG. 13 is a diagram showing weighting coefficients for the amount of lateral motion of each pixel in an image block. [Figure 14A] 13B is a diagram showing a result of multiplying the sign data on the bright side (plus side) of the EVS output in FIG. 10 by the weighting coefficient of the lateral movement amount in FIG. 13B. [Figure 14B] 13B is a diagram showing a result of multiplying the sign data on the dark side (negative side) of the EVS output in FIG. 10 by the weighting coefficient of the lateral movement amount in FIG. 13B. [Figure 15] FIG. 13 is a diagram illustrating the movement of a subject in the previous frame. [Figure 16] 16 is a diagram for explaining code data of an event-based sensor indicating a change in brightness due to the movement of the subject in FIG. 15. FIG. [Figure 17A]17 is a diagram showing a result of multiplying the positive sign data of the brightness change in FIG. 16 by the weighting coefficient of the vertical movement amount in FIG. 11B. [Figure 17B] 17 is a diagram showing a result of multiplying the negative sign data of the brightness change in FIG. 16 by the weighting coefficient of the vertical movement amount in FIG. 11B. [Figure 18A] 17 is a diagram showing a result of multiplying the positive sign data of the brightness change in FIG. 16 by the weighting coefficient of the horizontal movement amount in FIG. 13B. [Figure 18B] 17 is a diagram showing a result of multiplying the negative sign data of the brightness change in FIG. 16 by the weighting coefficient of the horizontal movement amount in FIG. 13B. [Figure 19] FIG. 13 is a diagram showing a method for moving a block that is a motion search target using a motion vector. [Figure 20A] FIG. 13 is a diagram illustrating a one-pixel unit search in an 8×8 pixel area in the first search. [Figure 20B] This shows a method for performing a search by interpolating in 1 / 4 pixel units in the second search. [Figure 21] FIG. 1 is a diagram illustrating conventional blocking matching. [Figure 22] FIG. 1 is a diagram for explaining the arrangement of an EVS and an RGB sensor using a prism. [Figure 23] 23(a) and 23(b) are diagrams for explaining the arrangement of an EVS, an RGB sensor, and a TOF sensor using a prism. [Figure 24] FIG. 13 is a diagram illustrating a basic configuration of a moving image noise reduction device according to a fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Embodiment 1) In the first embodiment, a configuration will be described in which the motion of a subject is captured at a high frame rate by using output data from an EVS, a motion vector is estimated with high accuracy, and noise is reduced using a motion-compensated image.

[0013] In this embodiment, the R, G, and B data of an image is acquired using an RGB sensor, while at the same time luminance change data of the image is acquired using an event-based sensor (EVS). Using the luminance change data, a motion vector is calculated within one frame of the RGB sensor, motion compensation is performed, and the motion-compensated image is used to perform noise reduction by inter-frame moving average.

[0014] 1 is a configuration diagram of an EVS 10. The EVS 10 includes a light receiving unit 12 and a luminance change detection unit 16. The light receiving unit 12 includes a current-voltage converter 13 and a photodiode 14. The luminance change detection unit 16 includes an asynchronous difference detector 17 and a comparator 18.

[0015] In the light receiving section 12, the photodiode 14 converts incident light for each pixel into a current and supplies the current to the current-voltage converter 13. The current-voltage converter 13 converts the current detected in each pixel into a voltage and supplies the voltage to the luminance change detection section 16.

[0016] In the luminance change detection unit 16, an asynchronous difference detector 17 asynchronously detects the difference between the detected voltage of each pixel and a reference voltage, and provides the difference to a comparator 18. The comparator 18 outputs a positive event signal if the voltage difference exceeds a positive threshold, outputs a negative event signal if the voltage difference exceeds a negative threshold, and outputs zero otherwise.

[0017] In this way, the EVS10 converts the light detected for each pixel into a voltage, compares the converted voltage with a threshold level (hereinafter simply called "threshold level") with an arbitrary width and voltage level, and sets the positive event signal to 1 if the light is brighter than the positive threshold level, sets the negative event signal to 1 if the light is darker than the negative threshold level, and outputs 0 for both otherwise. When either the positive or negative threshold level is exceeded, the voltage level of the reference threshold level is reset to the voltage level at that time. When both are 0, the data of that pixel is not output. Since nothing is output when it is 0, it is sufficient to output data representing one polarity only for pixels in the image where the brightness has changed by a certain threshold level width, that is, one bit of data is output. It also outputs data indicating which address position of the x and y coordinates of the screen the output is at.

[0018] 2(a) to 2(c) are diagrams for explaining the event signal output by the EVS 10. It is assumed that the subject moves along the arrow in the figure. In FIG. 2(a), the luminance of the pixel at the position of the reference symbol 15a exceeds the threshold, and an event signal (x1, y1, t1, ±1) including the pixel coordinates, time, and light / dark polarity is output. In FIG. 2(b), the luminance of the pixel at the position of the reference symbol 15b exceeds the threshold, and an event signal (x2, y2, t2, ±2) including the pixel coordinates, time, and light / dark polarity is output. In FIG. 2(c), the luminance of the pixel at the position of the reference symbol 15c exceeds the threshold, and an event signal (x3, y3, t3, ±3) including the pixel coordinates, time, and light / dark polarity is output.

[0019] In this way, the EVS10 does not output multiple bits for each pixel like the 12-bit A / D data of a normal sensor, but outputs only pixel data at the timing when a certain threshold level is exceeded due to a change in luminance, so that an event signal can be output asynchronously at a very high speed on the order of microseconds. This embodiment uses the output data of the EVS10, which can detect luminance changes at such a high speed.

[0020] FIG. 3 is a diagram showing a basic configuration of the moving image noise reducing device 200 according to the first embodiment.

[0021] A RAW signal acquired from an RGB sensor such as a camera undergoes camera processing to become R, G, B or Y, Cb, Cr moving image data, which is input to the moving image noise reduction device 200. Alternatively, each of the R, Gr, Gb, and B signals of a BAYER pattern may be input to the moving image noise reduction device 200. Furthermore, a distance measurement image output from information obtained from a TOF sensor may be input to the moving image noise reduction device 200. For ease of explanation, an example in which a Y signal is input will be described here. The input Y signal is input to a low pass filter (LPF (Low Pass Filter)) unit 20.

[0022] The LPF unit 20 includes a horizontal low-pass filter, for example, 0.25Z^(-2)+0.5Z^(-1)+0.25Z^0, where Z is the scanning time of one pixel. The LPF unit 20 further includes a vertical low-pass filter, for example, 0.25L^(-2)+0.5L^(-1)+0.25L^0, where L is the scanning time of one line.

[0023] At the same time, the Y signal is delayed in a dly (delay) circuit 22, and the LPF output and the delay are combined in a subtraction unit 23, and the LPF signal is subtracted from the Y signal to obtain a high-frequency signal. This high-frequency signal is divided by a coefficient X in a division unit 24. This X is used to adjust the gain of the inter-frame moving average, and this X is used to adjust the amount of noise reduction. In an addition unit 25, this 1 / X multiplied high-frequency signal NHPF is added to a high-frequency signal FDR that is motion-compensated and delayed by one frame, which will be described later. In an addition unit 27, the result of this addition is added to the output result of the LPF unit 20, to obtain a YDNR signal with reduced noise that has been subjected to inter-frame moving average. The YDNR signal is simultaneously stored in a frame memory 30.

[0024] The motion vector estimation unit 34 calculates and predicts the motion vector of a moving subject from the data of the EVS 10, and provides the calculated motion vector to the motion compensation unit 32. The motion compensation unit 32 performs motion compensation by shifting each block by the amount of the motion vector in accordance with the motion vector, and generates a predicted image.

[0025] The motion vector estimation unit 34 determines a search start point and performs a motion search, which will be described later. Fig. 4 is a diagram for explaining the block range used for motion vector estimation. Here, as shown in Fig. 4, a motion vector estimation block range 42 is set to be wider than a motion compensation target block range 40. In the example of Fig. 4, the motion compensation target block range 40 is set to 8x8, and the motion vector estimation block range 42 is set to 24x24.

[0026] 5 is a block diagram of the motion vector estimation unit 34. The motion vector estimation unit 34 reads out motion vectors at time intervals synchronized with the cameras.

[0027] The y-coordinate EVS_y, x-coordinate EVS_x, and brightness code EVS_pm of the coordinate address of a pixel where a brightness change has exceeded the threshold level are input from the EVS to the buffering block asynchronously. The buffering block is a memory that replaces the asynchronously input EVS signal with an internal clock and absorbs processing delays within the block, and uses a FIFO memory, for example.

[0028] From the buffering outputs EVS_yb and EVS_xb, write addresses EVS_WADRS for frame memory A+, frame memory B+, frame memory A-, and frame memory B- are created to temporarily store the light and dark codes in these memories. Frame memory A+ and frame memory B+ are memories for storing pixels whose EVS has changed to the lighter side during a given period, which will be described later. Frame memory A- and frame memory B- are memories for storing pixels whose EVS has changed to the darker side during a given period, which will be described later.

[0029] If the RGB sensor operates at a frame rate of, for example, 1 / 60 seconds, the EVS outputs data asynchronously, on the order of microseconds at the maximum, so the frame memory is read out so that it can be updated at any period and synchronized with the RGB sensor data. For this reason, the EVS data is stored in a memory area corresponding to the address in these frame memories. The frame memory can be read out quickly, for example, at a period 1 / 24 times that of the RGB sensor, and the output data can be updated, so that it can be extracted as data with a period 1 / 24 times that of the RGB data. This 1 / 24 period is a period that is convenient for the circuit configuration of the 24 x 24 pixel block division in Figure 4, but it may be changed to any period depending on the pixel block division or the frame memory configuration. For ease of understanding, the explanation is given here using a period 1 / 24 times that of RGB.

[0030] Since the period is 1 / 24 of the RGB sensor, extremely high-speed processing is required. However, this is difficult to achieve with a normal FPGA hardware configuration due to the processing speed constraints. In this embodiment, the following innovation is used to achieve this.

[0031] FIG. 6 is a timing chart for explaining the operation of the motion vector estimation unit 34. The SSG in FIG. 5 generates a x24frm pulse that is 1 / 24 times the period of the RGB sensor. The RGB sensor VSYNC in FIG. 6 becomes a HIGH signal once every 1 / 60 second period. In contrast, the x24frm signal becomes a pulse that repeats HIGH and LOW at a 1 / 1440 second period, which is 1 / 24, as shown in FIG. 6. Frame memory A+ and frame memory B+, which store pixels that have changed toward the brighter side, write EVS data on one side in accordance with the HIGH and LOW of this x24frm, while the other reads and writes 0. Similarly, frame memory A- and frame memory B-, which store pixels that have changed toward the darker side, read and write 0 on one side, while the other writes EVS data in accordance with the HIGH and LOW of this x24frm.

[0032] As shown in Figure 6, when x24frm is LOW, frame memory A+ and frame memory A- write to addresses corresponding to the sign of the sensor brightness, and when it is HIGH, the written data is read out in order starting from address 0 at the top of the screen. Normally, to read one pixel at a time from memory, it is necessary to read it at a clock speed 24 times faster than the RGB signal, but by noting that EVS brightness data is 1 bit, we have devised a way to handle 24 lines as 24-bit data, which makes it possible to read 24 lines at once. This makes it possible to read out at the same speed as the clock speed used for normal RGB. The method for converting this data to 24 bits is shown below.

[0033] For example, if the pixel size of the EVS is 1280 (x-axis) × 720 (y-axis), EVS_yb is expressed in 10 bits as EVS_yb[9:0], and EVS_xb is expressed in 11 bits as EVS_xb[10:0]. The write address EVS_WADRS at that time can be calculated by the following formula, assuming that the motion vector estimation block range is 24 × 24. EVS_WADRS=INT(EVS_yb / 24)×2^11+EVS_xb (1)

[0034] EVS_pm is input as 1 if the pixel becomes brighter, and as 0 if it becomes darker. EVS_pm read out by the clock inside the block through the buffer is set as EVS_pmb. BITSHIFT is calculated from EVS_yb as shown in equation (2), and the value obtained by bit-shifting EVS_pmb by the amount of BITSHIFT using the shift register is output as shown in equation (3). BITSHIFT=EVS_yb-INT(EVS_yb / 24)×24 (2) ShiftRegister=2^BITSHIFT×EVS_pmb (3)

[0035] For example, when EVS_yb is 1000, BITSHIFT = 16 and ShiftRegister = 2^16. This value is output in sync with EVS_WADRS to the frame memory A+ and frame memory B+ blocks for bright direction detection. The inverted signal of EVS_pmb is 1 when it gets dark and 0 when it gets bright. It is input to the shift register and, according to EVS_WADRS, the output signals of equations (2) and (3) are output to the frame memory A- and frame memory B- blocks. In this way, 24 lines can be written to one address in the frame memory. At the same time, the shift register generates a YADR signal which indicates which bit out of 24 BITs the line address of EVS_yb where the event occurred is. It outputs the YADR signal with the following formula. YADR=2^BITSHIFT (4)

[0036] As mentioned above, data is written to frame memory A and frame memory B according to the x24frm pulse, but since multiple pieces of data for adjacent 24 lines may be input during the x24frm write period, the address of EVS_WADRS that was once saved is read out during the write phase, and the 24-bit data read out is ORed (logical sum) with the signal newly generated by the shift register. If the bit value of the YADR set to 1 among the 24 bits read out at this time is 1, and the bit newly generated by the shift register and indicated by YADR is 0, that bit is set to 0 without ORing it. In order to read out the 24-bit data that was once saved in this way, EVS_WADRS passes through a register that delays it by one clock (CLK) and is input to the frame memory address. This allows the latest light and dark code data to be written during the x24frm write pulse period.

[0037] The above writing is performed for frame memory A+ and frame memory A- when x24frm is LOW, and for frame memory B+ and frame memory B- when x24frm is HIGH, and the following reading and writing of 0 are performed for frame memory A+ and frame memory A- when x24frm is HIGH, and for frame memory B+ and frame memory B- when x24frm is LOW. The read address EVS_RADRS is generated by the R address generator in Figure 5, and if the pixel size is, for example, 1280 (x) x 720 (y), the addresses for all pixels of 1280 x 720 are generated in order from address 0, as given by equation (5) so as to comply with the rule of equation (1), and the data in the frame memory is read out during the read period indicated by x24frm. EVS_RADRS=INT(yadrs / 24)×2^11+xadrs (5) xadrs=0 to 1279, yadrs=0 to 29

[0038] As soon as one address is read, 24 bits of 0 are written to that address. Writing 0 is performed by generating addresses for all 1280 x 720 pixels in order starting from address 0, just like EVS_RADRS. By switching the read data from frame memory A+ and frame memory B+, and the read data from frame memory A- and frame memory B- with the x24frm pulse, plus and minus sign data for 24 frames of all 60 frames is read out.

[0039] This read data is code data for 24 lines per data, but because motion vector calculations are performed in units of 8 pixels x 8 lines, a delay adjusted 48-line memory is used, with 24 pixels read out horizontally, shifted by 8 pixels, and 24 lines of data read out vertically, shifted by 8 lines, as shown in Figure 8. Reading is done at a clock speed more than three times faster than writing. At the same time, plus sign data and minus sign data for 24 times 60 frames is read out, which is delayed by the frame memory in frame units of 1 / (24 x 60) seconds, and similarly, 24 pixels are read out by shifting by 8 pixels, and 24 lines of data by shifting by 8 lines, as shown in Figure 8. By doing this, vector calculations for each block can be performed including overlapping parts with adjacent blocks.

[0040] Next, we will explain how to calculate motion vectors. Figure 9 shows a circular subject within a 24 x 24 pixel area moving to the upper right in 1 / (24 x 60) second frame units. The numbers in the figure are the code values ​​of the Y signal. Figure 10 shows the code data of the EVS signal indicating the change in brightness due to the movement of the second and third circles in Figure 9.

[0041] To detect the direction of movement, the vertical weighting coefficient for the Y axis and the horizontal weighting coefficient for the X axis are multiplied with the positive and negative signs, respectively, and the sum of 24 x 24 pixels is calculated for the current frame in 24 x 60 frame units, as explained below, a similar sum is calculated for the previous frame, and the difference between the two is taken to find the amount of movement in the Y and X directions.

[0042] FIG. 11A shows a vertical coefficient that increases by 1 for each line for a 24×24 pixel block. Around the block in FIG. 11A, numbers that get smaller toward the four corners of the block are written, and the number obtained by dividing the number by 8 is the peripheral coefficient. That is, in the case of 2, the peripheral coefficient is 2 / 8, and in the case of 8, the peripheral coefficient is 8 / 8=1. The weighting coefficient of the vertical motion amount is a value obtained by multiplying the vertical coefficient of each pixel in FIG. 11A by the peripheral coefficient, as shown in FIG. 11B. For example, in the case of a pixel with X=1 and Y=3, the vertical coefficient is 3, the peripheral coefficient in the X direction is 2 / 8, and the peripheral coefficient in the Y direction is 6 / 8, so the weighting coefficient of the vertical motion amount is 3×(2 / 8)×(6 / 8)≒0.6. The reason for multiplying the peripheral coefficient is to reduce the contribution of the data to the motion estimation in the case of a motion that crosses the block boundary.

[0043] Figure 13A shows the horizontal coefficient, which increases by 1 for every horizontal pixel for a 24x24 pixel block. Around the block in Figure 13A are numbers that get smaller as you get closer to the four corners of the block, and the peripheral coefficient is calculated by dividing this number by 8. The weighting coefficient for the horizontal motion amount is calculated by multiplying the horizontal coefficient of each pixel in Figure 13A by the peripheral coefficient, as shown in Figure 13B.

[0044] Figure 12A shows the result of multiplying the bright side (positive side) sign data of the EVS output in Figure 10 by the weighting coefficient of the vertical motion amount in Figure 11B. This calculation is performed in the weighting section on the vertical positive side in Figure 5, and the result is called VP. There are 20 pixels that are positive in the EVS output in Figure 10, and the total weighting value in Figure 12A for these positive pixels is 185, and the average is 185 / 20 ≒ 9.25.

[0045] Figure 12B shows the result of multiplying the dark side (negative side) sign data of the EVS output in Figure 10 by the weighting coefficient of the vertical motion amount in Figure 11B. This calculation is performed in the weighting section on the vertical negative side in Figure 5, and the result is VM. There are 19 pixels that are negative in the EVS output in Figure 10, and the sum of the weighting values ​​in Figure 12B for these negative pixels is -235, and the average is -235 / 19 ≒ -12.37.

[0046] Figure 14A shows the result of multiplying the bright side (positive side) sign data of the EVS output in Figure 10 by the horizontal motion amount weighting coefficient in Figure 13B. This calculation is performed in the horizontal positive side weighting section in Figure 5, and the result is called HP. There are 20 pixels that are positive in the EVS output in Figure 10, and the total weighting value in Figure 14A for these positive pixels is 359, and the average is 359 / 20 ≈ 17.95.

[0047] Figure 14B shows the result of multiplying the dark side (negative side) code data of the EVS output in Figure 10 by the horizontal motion amount weighting coefficient in Figure 13B. This calculation is performed in the horizontal negative side weighting section in Figure 5, and the result is called HM. There are 19 pixels that are negative in the EVS output in Figure 10, and the sum of the weighting values ​​in Figure 14B for these negative pixels is -290, and the average is -290 / 19 ≒ -15.26.

[0048] In this way, the weighted values ​​on the positive side of 24x24 pixel blocks in the vertical motion direction are summed for all 24x24 pixels, and then divided by the number of positive side 24x24 pixel blocks to find the average value. In the example of Figures 12A and 12B, the average on the vertical positive side is 9.25, and the average on the vertical negative side is -12. This average value calculation is performed by the vertical P1BLOCK averaging unit and vertical M1BLOCK averaging unit in Figure 5, and the VPSUM signal and VMSUM signal are output. VPSUM=9.25, VMSUM=-12.37.

[0049] The weighted values ​​on the positive side of 24x24 pixel blocks in the horizontal motion direction are summed for all 24x24 pixels, and then divided by the number of positive sides of the 24x24 pixel blocks to find the average value. In the example of Figures 14A and 14B, the average of the horizontal positive side is 18, and the average of the vertical negative side is -15. This calculation is performed for the horizontal P1BLOCK average block and the horizontal M1BLOCK average block in Figure 5, and the HPSUM signal and HMSUM signal are output. HPSUM=17.95, HMSUM=-15.26.

[0050] The above calculation of the average brightness change of the 24 x 24 pixel block is also performed on the previous frame. This is done in the same way for the image of the previous frame in units of 1 / 24 times the period of 60 frames in the delay adjustment frame memory of Figure 5.

[0051] Figure 15 shows the movement of the subject and the state of the Y signal in the previous frame. Figure 16 shows the code data of the EVS signal due to the brightness change caused by the movement of the first and second circles in Figure 15. There are 18 pixels of positive code data and 18 pixels of negative code data.

[0052] Figure 17A shows the result of multiplying the positive sign data of brightness change in Figure 16 by the weighting coefficient of the vertical motion amount in Figure 11B. The weighting value of the positive pixels is 217 in total, and 217 / 18 ≈ 12.06 on average.

[0053] Figure 17B shows the result of multiplying the negative sign data of brightness change in Figure 16 by the weighting coefficient of the vertical motion amount in Figure 11B. The weighting values ​​of the negative pixels sum to -288, and the average is -288 / 18 ≈ -16.0.

[0054] In Figure 5, the average value on the vertical positive side is the VPSUM1f signal, and the average value on the vertical negative side is the VMSUM1f signal. VPSUM1f=12.06, VMSUM1f=-16.0.

[0055] Figure 18A shows the result of multiplying the positive brightness change data in Figure 16 by the weighting coefficient of the horizontal motion amount in Figure 13B. The weighting value of the positive pixels is 278 in total, and the average is 278 / 18 ≈ 15.44.

[0056] Figure 18B shows the result of multiplying the negative sign data of brightness change in Figure 16 by the weighting coefficient of the horizontal movement amount in Figure 13B. The weighting values ​​of the negative pixels sum to -240, and average -240 / 18 ≈ -13.33.

[0057] In Figure 5, the average value on the positive horizontal side is the HPSUM1f signal, and the average value on the negative horizontal side is the HMSUM1f signal. HPSUM1f=15.44, HMSUM1f=-13.33.

[0058] The vertical motion vector is calculated by taking the difference between the weighted average of the vertical motion amount in the current frame and the weighted average of the vertical motion amount in the previous frame. This difference is the average vertical movement amount of the subject within the 24 x 24 block. This difference is calculated in both positive and negative values, and the average is taken as the vertical motion vector amount VVECT1Fr for one frame of the 24 x 60 frame cycle.

[0059] The negative average is treated as a negative value, and the vertical motion vector amount VVECT1Fr is calculated using the following formula. VVECT1Fr=((VPSUM-VPSUM1f)+(-1)×(VMSUM-VMSUM1f)) / 2 (6)

[0060] 17A and 17B, VVECT = ((9.25 - 12.06) - (-12.37 + 16.0)) / 2 ≈ -3.2. When this value is negative, it indicates a movement toward the top of the screen, and when this value is positive, it indicates a movement toward the bottom of the screen.

[0061] The horizontal motion vector is calculated by taking the difference between the weighted average of the horizontal motion amount in the current frame and the weighted average of the horizontal motion amount in the previous frame. This difference is the average horizontal movement amount of the subject within the 24 x 24 block. This difference is calculated in both positive and negative values, and the average is taken as the horizontal motion vector amount HVECT1Fr for one frame of the 24 x 60 frame cycle.

[0062] The negative average is treated as a negative value, and the horizontal motion vector amount HVECT1Fr is calculated using the following formula. HVECT1Fr=((HPSUM-HPSUM1f)+(-1)×(HMSUM-HMSUM1f)) / 2 (7)

[0063] 18A and 18B, HVECT = ((17.95 - 15.44) - (-15.26 + 13.33)) / 2 = 2.22. When this value is negative, it indicates movement to the left of the screen, and when it is positive, it indicates movement to the right of the screen.

[0064] The vector amount of VVECT1Fr and HVECT1Fr is the amount of movement in a very short time of 24 x 60 = 1440 fps, so the movement of a normal subject is within a few pixels of 24 x 24 pixels for a 1280 x 720 pixel sensor. This vector calculation is performed for each 8 x 8 block in order, and the block number value output by the block number counter is used as an address and saved in the block-specific motion vector 24Fr addition memory. The calculation is performed so that the vertical and horizontal vectors are added in units of 1440 fps to the block-specific motion vector 24Fr addition memory, and after adding up for 24 frames, the sums VVECT24Fr and HVECT24Fr are output and saved in the next block-specific motion vector memory. After output, the added value in the block-specific motion vector 24Fr addition memory is reset to 0. The vector resulting from this addition for 24 frames becomes a vector in units of 60 FPS, and becomes the motion vector for the R, G, and B images.

[0065] Here, the addition of the block-specific motion vector 24Fr addition memory is reset to 0 when the 60FrmVrst pulse is HIGH, but this timing can be changed according to the shutter accumulation time of the RGB sensor to obtain a more accurate motion vector.

[0066] Figure 6 shows the reset timing 60FrmVrst when the RGB sensor has a shutter speed of 1 / 60 seconds. The top of Figure 6 shows the movement of the subject within 8 x 8 pixels. In this example, the RGB sensor operates at 60 FPS, and its synchronization signal VSYNC is at the timing shown in Figure 6. Here, the RGB sensor is a global shutter sensor that captures images at the same time for all pixels. When the shutter speed is 1 / 60 seconds, the subject is exposed from the VSYNC timing to the next VSYNC timing, so the subject captured by the RGB sensor is blurred to the upper right for 1 / 60 seconds of movement, as shown in the state before VSYNC in the RGB 1 / 60 seconds accumulation column in Figure 6. Furthermore, if the subject moves down at the next VSYNC, it is blurred to the lower right. On the other hand, EVS detects luminance changes asynchronously at high speed, so there is no blur. When calculating a motion vector from this blur-free EVS output data, it is good that the start and end points for calculating the vector are the timing of the center of the subject's blur on the RGB sensor.

[0067] Starting from the part of the subject displayed as a black circle in Fig. 6, calculating a vector between the part displayed as the next black circle will determine the vector at the center timing of the subject's blur on the RGB sensor, which will enable comparison with the center of the blur in the next frame when removing noise between frames. For this reason, the reset timing of 60FrmVrst is set to the midpoint between VSYNCs plus a delay of one frame, as shown in Fig. 6. This one-frame delay takes into account the delay caused by the internal processing memory. The reset timing of this 60FrmVrst is generated from VSYNC with the delay amount given by the following formula. If the frame rate of the RGB sensor is RGBFPS, the accumulation cycle of the frame memory for the event sensor is EVSFPS, and the shutter speed is RGBSH, the reset timing of 60FrmVrst, 60FrmVrstTiming, can be calculated using the following formula. 60FrmVrstTiming=INT((RGBSH / 2) / (1 / EVSFPS))×(1 / EVSFPS)+1 / EVSFPS (8)

[0068] For example, if the frame rate of the RGB sensor is RGBFP = 60 (FPS), the shutter speed RGBSH = 1 / 60 (sec), and the accumulation rate of the frame memory for the event sensor is EVSFPS = 24 × RGBFP, then from equation (8), it becomes 0.009027778 seconds. This is equivalent to 13 frames of the change period of x24frm.

[0069] Figure 7 shows an example when the shutter speed RGBSH is 1 / 1500 seconds. The delay amount is calculated using equation (8) according to the shutter speed. This is one frame of the pulse change period of x24frm.

[0070] At the timing of 60FrmVrst, which changes with this shutter speed, vectors for 24 frames in 1440 FPS units are added in a block-specific motion vector 24Fr addition memory, and this added vector value is written to a block-specific motion vector memory. The data in this memory is then read out block by block at the timing of VSYNC to obtain the motion parameters of VVECT and HVECT.

[0071] In this way, the motion parameters are acquired in the motion vector estimation unit 34, and the motion parameters are input to the motion compensation unit 32 in Fig. 3. The motion compensation unit 32 moves the range of the 8 x 8 pixel block division by the negative value of the motion parameters for the Y signal delayed by one frame at the frame rate of the RGB sensor, and obtains the newly input Y signal difference.

[0072] Fig. 19 is a diagram showing a method of moving a block to be subjected to motion search using a motion vector. In the case of Fig. 19, the motion vector is 10.2 in the horizontal direction HVECT and -10.5 in the vertical direction VVECT. Data of an 8x8 pixel area obtained by subtracting the motion vector from the Y signal of the previous frame is taken out from the frame memory of one frame in Fig. 3. If the motion vector is small, it is obtained by interpolating with a phase accuracy of 1 / 4 pixel. Between this data and the 8x8 pixel area of ​​the newly input Y signal, the 8x8 area obtained by the motion vector of the one frame behind is shifted in the order shown in Fig. 20A to find the correlation and search for the data.

[0073] Fig. 20A shows a one-pixel search in an 8x8 pixel area in the first search, and Fig. 20B shows a method of searching by interpolating pixel data in 1 / 4 pixel units in the second search. When the correlation value falls below a certain threshold, a phase with a high degree of correlation is found by further searching by interpolating data in 1 / 4 pixel units. If all searches are completed and the correlation is still below the threshold, no movement is performed and the frame difference of the same block is taken.

[0074] This search can be performed with higher accuracy when removing noise between frames, but it also leads to a delay in processing, so it does not have to be performed if it is not necessary.

[0075] This search has a narrower search range than the wide search used in conventional block matching as shown in Figure 21. This is because when calculating the motion vector, the subject is tracked at a high speed of 24 x 60 FPS, and the vector is calculated, so the accuracy is high at the time of calculating the motion vector.

[0076] A search is performed in this manner, and the 8×8 area at the point in time when a match is found with the previous frame is subjected to high pass filtering (HPF (High Pass Filter)) in the HPF unit 28 in FIG.

[0077] The HPF unit 28 includes a horizontal high-pass filter, such as -0.25Z^(-2)+0.5Z^(-1)-0.25Z^0, where Z is the scanning time of one pixel. Furthermore, the HPF unit 28 includes a vertical high-pass filter, such as -0.25L^(-2)+0.5L^(-1)-0.25L^0, where L is the scanning time of one line.

[0078] In the case of 3TAP as described above, the HPF unit 28 uses data of a 10×10 area that is actually one pixel wider above, below, left and right than the 8×8 area so that the high-pass filtering of the 8×8 area can be performed accurately.

[0079] The high frequency components extracted by the HPF unit 28 are multiplied by (1-1 / X) in the multiplication unit 26 in Fig. 3 to obtain the FDR. This FDR is calculated for each frame and added to the NHPF multiplied by 1 / X in the adder 25 to obtain the cyclically moving averaged high frequency components. This moving averaged high frequency component is added to the LPF in the adder 27 in Fig. 3. In this way, a frame-by-frame moving average value with motion compensation is obtained, and noise reduction is performed for random noise that has no correlation between frames.

[0080] (Embodiment 2) Since the EVS is a special sensor, the sensor lineup is limited, and the number of pixels of the EVS and the number of pixels of the RGB sensor may not match. In the second embodiment, when there is a difference in the number of effective pixels between the EVS and the RGB sensor, the values ​​of the horizontal motion vector and the vertical motion vector described in the first embodiment are multiplied by the ratio of the number of effective pixels between the event sensor and the RGB sensor.

[0081] For example, if the EVS is 1280 x 720 pixels and the RGB sensor is 1920 x 1080 pixels, and both are effective pixels with the same angle of view, the horizontal ratio HPIX_Ratio and the vertical ratio VPIX_Ratio can be calculated using the following equations. HPIX_Ratio=1920 / 1280=1.5 VPIX_Ratio=1080 / 720=1.5

[0082] The motion vector estimation unit 34 multiplies the horizontal motion vector HVECT and the vertical motion vector VVECT by the horizontal ratio HPIX_Ratio and the vertical ratio VPIX_Ratio, respectively, to obtain and output a corrected horizontal motion vector HVECT' and a corrected vertical motion vector VVECT'. HVECT'=HVECT×HPIX_Ratio ···(9) VVECT'=VVECT×VPIX_Ratio ···(10)

[0083] (Embodiment 3) In the third embodiment, a prism is used to arrange the EVS and the RGB sensor in the first and second embodiments so that they have the same optical axis. The RGB sensor is an example of an image sensor. A sensor that acquires black and white image data may be used instead of the RGB sensor.

[0084] 22 is a diagram for explaining the arrangement of an EVS and an RGB sensor using a prism. The image light incident from the lens is split by the prism, and the EVS and RGB sensor images are arranged so as to have the same optical axis. The noise reduction process of the first or second embodiment is performed using the output data of the RGB sensor and the output data of the EVS obtained by this configuration. The noise reduction process may be performed on each of the R, G, and B signals of the RGB sensor, or on each of the Y, Cb, and Cr signals obtained by converting R, G, and B into Y, Cb, and Cr, or on each channel of the Bayer pattern signals of the R, Gr, Gb, and B signals.

[0085] (Embodiment 4) In the fourth embodiment, a prism is used to arrange the EVS, the RGB sensor, and the TOF (Time Of Flight) sensor on the same optical axis as in the first and second embodiments. The TOF sensor is an example of a distance sensor.

[0086] 23(a) and 23(b) are diagrams explaining the arrangement of an EVS using a prism, an RGB sensor, and a TOF sensor. Visible ambient light and near-infrared light emitted from a VCSEL (Vertical Cavity Surface Emitting Laser) of a laser diode array, for example, infrared light with a wavelength of 940 nm in this case, reflected by an object, are incident on the prism through a lens. The prism separates the visible light and near-infrared light with a visible light reflecting dichroic film, for example, as shown in FIG. 23, and incidents the near-infrared light on the TOF sensor. The visible light is further split by the prism and incident on the RGB sensor and the EVS.

[0087] The prism structure may be either the arrangement shown in FIG. 23(a) or the arrangement shown in FIG. 23(b). The RGB sensor, the EVS, and the TOF sensor are arranged on the same optical axis. The TOF sensor has an internal processor and outputs distance data D. The noise reduction process of the first or second embodiment is performed using the output data of the RGB sensor, the output data of the EVS, and the distance data of the TOF sensor. The noise reduction process may be performed on each of the R, G, and B signals of the RGB sensor, each of the Y, Cb, and Cr signals obtained by converting R, G, and B to Y, Cb, and Cr, or each of the channels of the Bayer pattern signals of the R, Gr, Gb, and B signals. At the same time, the noise reduction of the first or second embodiment is also performed on the distance data D of the TOF sensor. At this time, the motion vector estimation unit 34 of FIG. 3 may be common to both the RGB image and the distance measurement image.

[0088] (Embodiment 5) In embodiment 5, in the arrangement of embodiment 4, noise reduction is performed on the distance data output from the TOF sensor by frame-to-frame moving averaging using the above-mentioned motion compensation, but in this case, the noise reduction is performed over the entire frequency band without separating the distance data into low and high frequency components.

[0089] Fig. 24 is a configuration diagram of a video noise reduction device 210 of embodiment 5. The configuration for extracting high-frequency components and the configuration for adding high-frequency components to low-frequency components are omitted from the configuration of the video noise reduction device 200 of Fig. 3. The rest is as explained in embodiment 1.

[0090] When distance image data from the TOF sensor is input to video noise reduction device 210, it is divided by coefficient X in divider 24 without being separated into low frequency components and high frequency components, to obtain NHPF for the entire frequency band. Distance image data for the entire frequency band that is delayed by one frame and motion compensated and output from motion compensation unit 32 is multiplied by (1-1 / X) in multiplier 26 to obtain FDR. In adder 25, the NHPF for the entire frequency band multiplied by 1 / X and the FDR for the entire frequency band multiplied by (1-1 / X) are added together to obtain a YDNR signal for the entire frequency band with reduced noise that has been moved between frames.

[0091] For distance data from a distance measurement sensor, noise reduction in low frequency areas may be more effective when using the distance data to create a point cloud for 3D modeling. For this reason, noise reduction is performed across the entire frequency band without separating the frequency bands with an LPF and an HPF. On the other hand, for video signals from an RGB sensor, frequency bands are separated as in the first embodiment, and noise reduction is performed on high frequency components. In this case, the motion vector estimation unit 34 in FIG. 3 may be common to both RGB video and distance measurement video.

[0092] The various processes of the video noise reduction devices 200, 210 described above can be realized not only as a device using hardware such as a CPU and memory, but also as firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by recording it on a recording medium readable by a computer, etc., or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as a data broadcast of terrestrial or satellite digital broadcasting.

[0093] According to the video noise reduction devices 200 and 210, by calculating a high frame rate motion vector using EVS and then performing block matching, it is possible to improve the tracking of a moving subject, and there is an advantage that it is not necessary to expand block matching over a wide range. In addition, EVS has no concept of a shutter, and since EVS has a wide dynamic range from dark to bright areas, it has good tracking even for dark subjects.

[0094] In addition, because motion can be estimated using two types of 1-bit signals, bright and dark, for pixels where a change in brightness has occurred, calculation speed is improved and data can be updated at high speed. This improves tracking of moving objects and reduces the search range between frames. As a result, noise reduction accuracy for moving objects is improved and image lag between frames is reduced. This allows high-quality images and distance data to be obtained, while also reducing the circuit scale and processing time of the entire device.

[0095] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0096] 10 EVS, 12 light receiving unit, 13 current-voltage converter, 14 photodiode, 16 brightness change detection unit, 17 asynchronous difference detector, 18 comparator, 20 LPF unit, 22 dly circuit, 23 subtraction unit, 24 division unit, 25 addition unit, 26 multiplication unit, 27 addition unit, 28 HPF unit, 30 frame memory, 32 motion compensation unit, 34 motion vector estimation unit, 36 SSG, 200, 210 video noise reduction device.

Claims

1. a filter unit that separates image data from an image sensor that captures images at a predetermined frame rate into a low frequency component and a first high frequency component; a motion vector estimation unit that estimates a motion vector by reading data from an event sensor that asynchronously outputs information on pixels whose luminance has changed at a frame rate higher than the predetermined frame rate; a motion compensation unit that performs motion compensation based on the motion vector, the filter unit generates a third high-frequency component by adding the first high-frequency component and a second high-frequency component extracted from the image obtained by the motion compensation in a predetermined ratio, and reduces noise in the image data by adding the low-frequency component and the third high-frequency component.

2. 2. The moving image noise reduction device according to claim 1, wherein a block range of image data for estimating a motion vector is larger than a block range of image data for performing motion compensation.

3. 3. The moving image noise reduction device according to claim 1, wherein, when estimating the motion vector, a calculation start timing of the motion vector is changed according to a shutter speed of the image sensor.

4. The video noise reduction device described in claim 1, characterized in that when performing noise reduction on distance measurement image data from a distance sensor that detects the distance to a subject, the filter unit reduces noise in the distance measurement image data by adding a distance measurement image of the entire frequency band from the distance sensor and a distance measurement image of the entire frequency band obtained by the motion compensation in a predetermined ratio without separating the distance measurement image data into low frequency components and high frequency components.

5. A filtering step of separating image data from an image sensor capturing images at a predetermined frame rate into a low frequency component and a first high frequency component; a motion vector estimation step of estimating a motion vector by reading data from an event sensor that asynchronously outputs information on pixels whose luminance has changed at a frame rate higher than the predetermined frame rate; a motion compensation step of performing motion compensation based on the motion vector, The method for reducing noise in a moving image, characterized in that the filtering step generates a third high-frequency component by adding the first high-frequency component and a second high-frequency component extracted from the image obtained by the motion compensation in a predetermined ratio, and reduces noise in the image data by adding the low-frequency component and the third high-frequency component.

Citation Information

Patent Citations

  • Method for outputting a signal from an event-based sensor and event-based sensor using such a method

    JP2021508975A