An image processing method and electronic device for mitigating LED flicker based on motion compensation

Abstract

This invention discloses an image processing method and electronic device for mitigating LED flicker based on motion compensation, applicable to flicker suppression caused by pulsed LED light sources in vehicle cameras, intelligent transportation systems, and consumer electronic devices. When the alternating on / off states of LEDs cause block matching failures in traditional methods, this invention accurately predicts motion vectors through weighted fusion of spatial / temporal proximity vectors; it also solves the problem of cross-frame coordinate mapping misalignment by generating a reverse motion vector map, ensuring the accuracy of forward / backward pixel tracing; finally, through cross-frame weighted fusion, combining mean and maximum value weights, flicker is eliminated while avoiding the loss of LED brightness information. This invention solves the problem that existing technologies require special CMOS sensor hardware to handle LED flicker.

Description

Technical Field

[0001] This invention relates to an image processing method and apparatus, and more particularly to an image processing method and electronic device for LED flicker mitigation (LFM). Background Technology

[0002] In autonomous driving systems, cameras often capture image artifacts from LED light sources such as traffic lights and taillights due to LED flicker. This causes the system to fail to accurately identify the status of electronic road signs and traffic lights, or to have difficulty distinguishing between turn signals and taillight signals. These identification errors directly affect the vehicle's decision-making and control, thereby interfering with key functions such as automatic braking, high-speed cruise, and lane departure detection, and posing serious safety hazards.

[0003] The mechanism behind LED flickering is primarily due to the fact that LED light sources typically employ pulse width modulation (PWM) to achieve energy saving and extended lifespan. Their operating frequency is generally above 90Hz, meaning a pulse period of ≤11ms and a duty cycle typically not exceeding 50%. This implies that the LED rapidly completes a switching between on (power-on) and off (power-off) state within each cycle. The human eye cannot perceive this high-frequency flickering due to the persistence of vision. However, the camera's exposure time may fall precisely within the LED's "off" cycle, especially when the camera uses short exposures (e.g., 3ms) to handle high dynamic range scenes. In this case, the image sensor will capture the LED as off. If the target is an LED array, artifacts of partial brightness and darkness may appear in the same frame, with some LEDs on and others off. These brightness and darkness anomalies caused by the asynchrony between exposure and the LED driving cycle manifest as LED flickering in the image.

[0004] Regarding LED flicker mitigation, most manufacturers currently extend the exposure time of the CMOS image sensor to be longer than the LED's drive cycle. This ensures that the complete on / off cycle of the LED is captured within the exposure time, avoiding the situation where only the off portion of the LED is captured. However, if long exposures are used to capture details in dark areas, bright areas will be overexposed; using short exposures to avoid overexposure may miss the LED's emission pulses. To address this, manufacturers use specialized sensors, as follows:

[0005] OmniVision's Split Pixel technology employs a pixel-splitting structure for image sensors. Smaller photodiodes extend exposure time to ensure LED pulse capture, while larger photodiodes enable shorter exposures, ensuring higher sensitivity and color compensation in low-light conditions. Sensors like the OX01A10 and OXO2A10, based on OmniVision's 4.2-micron OmniBSI pixel architecture, utilize pixel splitting technology to provide superior high dynamic range and low-light performance.

[0006] SmartSens' QCell technology tightly arranges four pixels of the same color on an image sensor, enabling a four-in-one output. Utilizing this special monochromatic four-pixel structure, it achieves a multi-exposure-like effect in a single exposure. The four monochromatic pixels function as a single superpixel, increasing the photosensitive area and charge capacity, thus allowing for extended exposure times to cover the full flicker cycle of the LED.

[0007] Sony's SubPixel technology employs a pixel-size separation structure in its image sensor while adding Dual Conversion Gain (DCG) technology. The charge accumulated in each pixel is converted into two voltage modes: High Conversion Gain (HCG) and Low Conversion Gain (LCG), thus achieving different sensitivities. Therefore, the SubPixel sensor can acquire four images with different sensitivities simultaneously, providing a rich data foundation for subsequent HDR synthesis and LFM processing.

[0008] Onsemi's Super-Exposure Pixels technology equips each pixel on the image sensor with an overflow memory that can store extra charge beyond the normal pixel threshold. This means that even in very bright conditions, the sensor can continue to capture image data without losing information due to overexposure. A 120 dB dynamic range is achieved in a single exposure, avoiding missed LED pulses due to short exposures.

[0009] Intel's D400 series cameras can automatically detect the flicker frequency and period of LED light sources and automatically set the exposure time to an integer multiple of the light source period based on this information. However, simply using long exposure can lead to overexposure of the image. To address this, the image sensor supports a dual-conversion gain mode, which can switch between two gain modes within a single frame. The high-gain path captures details in dark areas, while the low-gain path preserves details in bright areas.

[0010] The TIER IV camera ensures sufficient light is captured during the LED's illuminated period to obtain a complete LED image by extending the exposure time beyond the LED's emission cycle. To prevent overexposure, a special image sensor design significantly improves its saturation signal level, preventing pixel saturation even when the exposure time exceeds the LED's emission cycle.

[0011] In summary, almost all manufacturers currently use special image sensors to achieve LED flicker mitigation. These special image sensors include discrete pixels, quad-pixel binning, and dual-conversion gain pixels.

[0012] However, special image sensors are expensive and consume a lot of power, and there is currently no way to achieve LED flicker mitigation using ordinary image sensors. Summary of the Invention

[0013] Purpose of the invention: To address the aforementioned existing technologies, this invention proposes an image processing method for mitigating LED flicker based on motion compensation. This method is applicable to suppressing flicker caused by pulsed LED light sources in vehicle cameras, intelligent transportation systems, and consumer electronic devices, and solves the problem that existing technologies require special CMOS sensor hardware to handle LED flicker.

[0014] Technical solution: An image processing method for mitigating LED flicker based on motion compensation, comprising: Step 1: Calculating motion vectors for each pixel in each frame of the video and generating inverse motion vector vectors; Step 2: Identifying flickering LED pixels through flicker detection and then achieving flicker mitigation through cross-frame weighted fusion.

[0015] Furthermore, in step 1, generating the motion vector diagram of the first frame of the video includes: dividing the frame image into several pixel blocks of size a*a, each pixel block being called a "matching block"; starting from the upper left of the image, traversing each matching block in the image, and performing the following steps:

[0016] Step 1.1.1: Calculate the motion vector of the first matching block:

[0017] For the first matching block, the SAD algorithm is used to search for the most similar reference matching block in the second frame image, and then the motion vector of the current matching block is obtained, and the motion vector is assigned to all pixels of the current matching block.

[0018] Step 1.1.2: Use a recursive algorithm to obtain the motion vector for each subsequent matching block:

[0019] Take the motion vectors of the four calculated neighboring matching blocks around the current matching block. The positions of these four neighboring blocks are the upper left, upper, upper right, and left of the current matching block, respectively. If there is no neighboring matching block at a certain position, the motion vector of the neighboring matching block is defaulted to 0. By mapping "current matching block coordinates + neighboring matching block motion vectors", several initial reference positions in the second frame are obtained. Within a set range centered on each initial reference position, the SAD algorithm is used to search for the most similar reference matching block, obtain the motion vector of the current matching block, and assign it to all pixels in the block.

[0020] Furthermore, in step 1, generating the motion vector diagram for each subsequent frame of the first frame includes: dividing the current frame image into several pixel blocks of size b*b, each block being called a "prediction block"; starting from the upper left of the image, traversing each prediction block in the image, and performing the following steps:

[0021] Step 1.2.1: Generate 15 candidate motion vectors for the current prediction block, specifically as follows:

[0022] One zero motion vector;

[0023] Four spatial neighbor motion vectors are taken from the motion vector diagram from the current frame to the next frame. The motion vectors of the four calculated spatial neighbor prediction blocks around the current prediction block are located at the top, left, top left, and top right positions.

[0024] Nine temporally neighboring motion vectors are taken from the reverse motion vector map from the previous frame to the current frame. The reverse motion vectors are the nine temporally neighboring pixel blocks around the center of the current prediction block in the previous frame. The nine positions are center, top, bottom, left, right, top left, bottom left, top right, and bottom right.

[0025] One block matching motion vector is the motion vector of the current predicted block obtained by searching for similar blocks in the reference frame using the SAD algorithm;

[0026] Step 1.2.2: Calculate the weights of the 15 candidate motion vectors;

[0027] Motion vector index k=0, weight of zero motion vector ,in The mean of the motion vectors of the surrounding predicted blocks. The base values ​​for static scene weights. The time correlation strength parameter; the motion vector index k = 1~4, and the weight of spatially neighboring motion vectors. ,in This represents the mean absolute pixel difference between the current predicted block and its neighboring predicted blocks. For positional weights, This is the spatial weight decay coefficient;

[0028] Motion vector index k = 5~13, weight of time-near motion vectors ,in A basic priority factor set based on spatial location. For the motion vector of the neighboring predicted block, The current predicted block motion vector, This is the time-weighted decay coefficient. It is the mean absolute difference of pixels between the current predicted block and the corresponding block at the same position in the previous frame;

[0029] Motion vector index k=14, weight of block matching motion vector ,in This is the sum of the absolute pixel differences between the current predicted block and the reference matching block;

[0030] Step 1.2.3: Fuse the 15 candidate motion vectors by weighted averaging to obtain the final motion vector of the current prediction block. ,in Let k be the candidate motion vector; Assign to all pixels in the current prediction block.

[0031] Furthermore, in step 1, generating the reverse motion vector image for each frame includes: traversing each pixel of the current frame and performing the following two operations:

[0032] 1.3.1: Invert the direction of the motion vector of the current pixel to obtain the reverse motion vector;

[0033] 1.3.2: Calculate the mapped coordinates of the current pixel in the next frame using "current pixel coordinates + motion vector", and assign the reverse motion vector to the pixel with the mapped coordinates in the next frame.

[0034] Furthermore, step 2 includes:

[0035] 2.1: Flicker Detection:

[0036] For the current pixel, the pixel value is denoted as C. Based on the motion vector diagram, the mapped coordinates of the current pixel in the next frame are calculated, and the pixel value at these mapped coordinates is taken as the backward mapped pixel value B. Based on the reverse motion vector diagram, the mapped coordinates of the current pixel in the previous frame are obtained, and the pixel value at these mapped coordinates is taken as the forward mapped pixel value F. Based on the current pixel value C, the backward mapped pixel value B, and the forward mapped pixel value F, two difference indicators are calculated: forward difference and backward difference. The forward difference is the pixel difference between the current pixel value C and the forward mapped pixel value F, and the backward difference is the pixel difference between the current pixel value C and the backward mapped pixel value B. When both difference indicators are simultaneously greater than a preset flicker threshold, the current pixel is determined to be a flickering pixel.

[0037] 2.2: Flicker mitigation:

[0038] If the current pixel is determined to be a flickering pixel, then the pixel value is weighted and fused across frames according to the following formula:

[0039] in, The final pixel value after fusion. Weighted by mean, Weights are based on the maximum value.

[0040] If a pixel is not identified as flickering, its current pixel value is retained. After each pixel in the current frame is fused, the image after LED flicker mitigation is obtained.

[0041] Furthermore, in step 1, the value of b is less than the value of a.

[0042] Furthermore, the handling of abnormal cases when generating the reverse motion vector is as follows: for pixels without a reverse motion vector, a zero motion vector (0,0) is assigned by default; for pixels with multiple reverse motion vectors, the motion vector is determined by the last assignment.

[0043] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the motion-compensated LED flicker mitigation image processing method.

[0044] Beneficial effects: When the alternation of LED on / off states causes block matching failure in traditional methods, the method of this invention accurately predicts motion vectors through weighted fusion of spatial / temporal proximity vectors; and solves the problem of cross-frame coordinate mapping misalignment by generating a reverse motion vector map, ensuring the accuracy of forward / backward pixel tracing; finally, through cross-frame weighted fusion, combining the weights of the mean and maximum values, flicker is eliminated while avoiding the loss of LED brightness information.

[0045] 1. The method of the present invention does not require special sensors from manufacturers such as OmniVision and Sony, nor does it require high-power sensor auxiliary circuits (such as dual-conversion gain switching and charge overflow storage). It can directly use ordinary CMOS image sensors to achieve the same LFM effect.

[0046] 2. The method of this invention, through a dynamic weighting mechanism of candidate vectors, can still accurately predict motion vectors even when LED flicker causes block matching failure (e.g., the LED is on in the current frame and off in the previous frame). By designing a reverse motion vector diagram, it solves the coordinate mapping misalignment problem caused by directly inverting the vector, ensuring accurate tracking of pixel positions across frames. Ultimately, it achieves LFM effect without the need for a special CMOS sensor, significantly reducing cost and power consumption. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a schematic diagram illustrating the reason for using the reverse motion vector in the method of this invention. Detailed Implementation

[0049] The invention will now be further explained with reference to the accompanying drawings.

[0050] This embodiment presents a motion-compensated LED flicker mitigation (LFM) image processing method. It processes the input video frame sequence, and the overall process is as follows: Figure 1 .

[0051] Step 1: Motion compensation.

[0052] The goal of motion compensation is to calculate a motion vector for each pixel in each frame of a video to describe the displacement of the pixel between adjacent frames, and at the same time generate a reverse motion vector map to solve the problem of misalignment in subsequent cross-frame coordinate mapping.

[0053] This step is implemented in three sub-steps, and because the first frame has no historical reference information, the first frame and subsequent frames are treated differently.

[0054] 1.1: Generate the motion vector diagram of the first frame of the video.

[0055] The frame image is divided into several pixel blocks of size a*a, each pixel block is called a "matching block". To ensure that neighboring blocks have been calculated during subsequent recursion, each matching block in the image is traversed starting from the top left of the image. In this embodiment, the value of a is 16.

[0056] 1.1.1: Calculate the motion vector of the first matching block.

[0057] For the first matching block, within an r*r pixel block range centered at the same position in the second frame, search for the most similar reference matching block. The similarity criterion for matching blocks adopts the Sum of Absolute Differences (SAD) algorithm, and its calculation formula is as follows:

[0058]

[0059] Where: i is the i-th pixel in the matching block; N is the total number of pixels in the matching block, N=a*a; , , These are the weights for the red, green, and blue channels, used to adjust the contribution of each channel to color difference; , , This represents the RGB channel value of the i-th pixel in the current matching block; , , The RGB channel values ​​of the pixels at the corresponding positions of the reference matching blocks are used. In this embodiment, the value of r is 3, meaning that the search is performed within a 3*3 area of ​​surrounding matching blocks.

[0060] The reference matching block with the smallest SAD value is the reference matching block that is most similar to the current matching block. The motion vector of the current matching block is obtained by subtracting the coordinates of the most similar reference matching block from the coordinates of the current matching block. This motion vector is then assigned to all pixels in the current matching block, meaning that pixels within the block share the same motion vector.

[0061] 1.1.2: Use a recursive algorithm to obtain the motion vector of each subsequent matching block.

[0062] For each subsequent matching block in the first frame image, the search range is narrowed recursively by taking advantage of the similar motion characteristics of adjacent matching blocks, thereby improving computational efficiency.

[0063] Specifically, the motion vectors of the four calculated neighboring matching blocks around the current matching block are taken. The positions of these four neighboring blocks are the upper left, upper, upper right, and left of the current matching block, respectively. If a neighboring matching block does not exist at a certain position, such as when the current matching is an edge block, the motion vector of that neighboring matching block is defaulted to 0.

[0064] By mapping the coordinates of the current matching block to the motion vectors of neighboring matching blocks, several initial reference positions in the second frame are obtained. Within an r*r pixel block range centered on each initial reference position, the SAD algorithm is used to search for the most similar reference matching block, thereby obtaining the motion vector of the current matching block and assigning it to all pixels within the block.

[0065] Example: The motion vector of the first matching block in the first frame is calculated to be (3,0). When calculating the motion vector of the second matching block at coordinates (2,0) in the first frame, referencing the motion vector (3,0) of the first matching block to its left, the coordinates of the initial reference position in the second frame are mapped to (2+3, 0+0) = (5,0). Within a 3x3 area around the initial reference position (5,0), the SAD algorithm is used to search for the most similar reference matching block to the current matching block. Since the upper left, upper, and upper right matching blocks of the matching block at coordinates (2,0) do not exist, their corresponding motion vector is assumed to be (0,0), and the corresponding initial reference position coordinates in the second frame are (2,0). Within a 3x3 area around the initial reference position (2,0), the SAD algorithm is used to search for the most similar reference matching block to the current matching block. The reference matching block with the smallest SAD value among the two SAD calculations is the most similar reference matching block to the current matching block.

[0066] 1.2: Generate motion vector graphics for each subsequent frame of the video.

[0067] Starting from the second frame, the motion vector of the previous frame is used as a reference. Therefore, a multi-candidate vector dynamic weighting strategy is adopted to solve the problem of block matching failure caused by the inability of LED blocks in bright and dark frames to be similar through SAD.

[0068] The current frame image is divided into several pixel blocks of size b*b, each block being called a "prediction block". Starting from the top left of the image, each prediction block in the image is traversed.

[0069] 1.2.1: Generate 15 candidate motion vectors for the current prediction block, covering four scenarios: "static, spatial proximity, temporal proximity, and block matching", to ensure that there is still a reliable reference when the LEDs are on and off.

[0070] The 15 candidate motion vectors are as follows:

[0071] ① One zero motion vector, that is, assuming the predicted block is stationary, the motion vector is (0,0).

[0072] ② Four spatial neighbor motion vectors are taken from the motion vector diagram from the current frame to the next frame, representing the motion vectors of the four spatially neighboring predicted blocks surrounding the current predicted block that have already been calculated. Since the traversal of the predicted blocks starts from the top left, among the eight predicted blocks surrounding the current predicted block, only the top, left, top-left, and top-right predicted blocks have been calculated.

[0073] ③ Nine temporally neighboring motion vectors are taken from the reverse motion vector map from the previous frame to the current frame. These are the reverse motion vectors of the nine temporally neighboring pixel blocks surrounding the current prediction block at its mapped coordinates in the previous frame: center, top, bottom, left, right, top left, bottom left, top right, and bottom right. This covers the temporal motion continuity. Here, the size of the temporally neighboring pixel block is also b*b.

[0074] ④ One block matching motion vector is the motion vector of the current predicted block obtained by searching for similar blocks in the reference frame using the SAD algorithm. This is used to ensure the vector accuracy of the static environment around the LED. The reference frame is the frame following the current frame.

[0075] In this embodiment, the value of b is 4, which is smaller than the value of a in the first frame processing, in order to improve vector accuracy.

[0076] 1.2.2: Calculate the weights of the 15 candidate motion vectors:

[0077] A dynamic weight is assigned to each candidate motion vector. The higher the weight, the greater the contribution of the candidate motion vector to the final motion vector.

[0078] ① Zero motion vector weight If the motion vector index k=0, then:

[0079]

[0080] in, The mean value of the motion vectors of the surrounding predicted blocks reflects the intensity of scene motion; the larger the value, the more intense the motion. This is the base value for static scene weights, typically ranging from 0.1 to 0.3. This is a time correlation strength parameter, typically set to 1.0, used to quantify the strength of motion continuity in a video frame sequence. When the value is small, a slight increase in motion intensity will significantly reduce the zero vector weight, making it suitable for scenarios with intense motion.

[0081] ② Spatial Proximity Motion Vector Weights If the motion vector index k = 1~4, then:

[0082]

[0083] in, This is the mean absolute pixel difference between the current prediction block and its neighboring prediction blocks. It is calculated first using the SAD algorithm described in step 1.1.1. After calculating the value, divide it by the total number of pixels N within the matching block to obtain the mean value. A larger value indicates that the two blocks are less similar; For positional weights, since horizontal / vertical movement is more continuous, top / left is usually greater than top-left / top-right. " / " indicates "or". The values ​​for the top / left direction are generally 0.4 to 0.6, and the values ​​for the top-left / top-right direction are generally 0.2 to 0.4, which are adjusted according to the continuity of the motion. This is the spatial weight decay coefficient, typically ranging from 1.0 to 2.0, used to control the decay rate of the influence of spatial proximity on the weights. The larger the exponent, the slower the decay of the term. Even if two blocks are not very similar, they can still get relatively high weights. That is, the similarity requirement is relaxed, allowing more distant or less similar neighboring points to contribute weight to the center point.

[0084] ③ Time-nearest motion vector weights If the motion vector index k = 5~13, then:

[0085]

[0086] in, Priority factors are set based on spatial location, typically center > top, bottom, left, right > corners. The center value is generally 0.6~0.8, the top, bottom, left and right values ​​are generally 0.4~0.6, and the corner values ​​are generally 0.2~0.4; For the motion vector of the neighboring predicted block, The current predicted block motion vector, The squared Euclidean distance between the motion vectors of neighboring predicted blocks and the motion vector of the current predicted block is used to measure the similarity of their motions. The more similar the motion vectors are, the greater the weight. This is the time weight decay coefficient, typically ranging from 0.5 to 2.0, used to control the decay rate of the influence of time proximity on the weights. The larger the exponent, the slower the decay, meaning that the difference in motion vectors has a weaker impact on weight decay, and even a slightly larger difference can still result in a relatively high weight. This is the mean absolute pixel difference between the current predicted block and the corresponding block at the same position in the previous frame. It is calculated first using the SAD algorithm from step 1.1.1. After calculating the value, divide it by the total number of pixels N in the matching block to obtain the mean value, which is used to measure the degree of change of the image content over time. The more drastic the change between the previous frame and the current frame, the lower the reference value of the neighboring prediction blocks.

[0087] ④ Block matching motion vector weights If the motion vector index k=14, then:

[0088]

[0089] in, The sum of the absolute pixel differences between the current predicted block and the reference matching block is calculated using the SAD formula. The larger the value, the less similar the two blocks are, and the lower the accuracy of the vector.

[0090] 1.2.3: Calculate the motion vector of the current prediction block :

[0091] The motion vector of the current prediction block is obtained by fusing 15 candidate motion vectors through weighted averaging:

[0092]

[0093] in, This is the k-th candidate motion vector; The weight is the weight of the k-th candidate motion vector.

[0094] Will As the final motion vector, it is assigned to all pixels in the current prediction block.

[0095] The reason for using a weighted average method to predict the final motion vector, rather than calculating it through block matching using the SAD algorithm, is that in images exhibiting LED flickering, adjacent frames of LEDs alternate between on and off states. In this situation, the block matching method based on the SAD algorithm cannot classify on and off LEDs as similar blocks, thus failing to derive accurate motion vectors through block matching. Therefore, for flickering LEDs, this invention employs a motion vector prediction scheme, where the calculation of candidate vectors partially utilizes the principle of block matching, ensuring, to a certain extent, the accuracy of motion vector prediction for the environment surrounding the LED.

[0096] 1.3: Generate the reverse motion vector image for each frame.

[0097] Iterate through each pixel of the current frame and perform the following two steps:

[0098] 1.3.1: Invert the direction of the motion vector of the current pixel to obtain the reverse motion vector.

[0099] 1.3.2: Calculate the mapped coordinates of the current pixel in the next frame using "current pixel coordinates + motion vector", and assign the reverse motion vector to the pixel with the mapped coordinates in the next frame.

[0100] For example: the pixel located at coordinates (1,0) has a motion vector of (0,2); its reverse motion vector (0,-2) is assigned to the pixel in the next frame with mapped coordinates =(1+0, 0+2)=(1,2).

[0101] Handling Abnormal Cases: Because the pixel mapping is not one-to-one, some pixels may not have a reverse motion vector, while others may have multiple reverse motion vectors. For pixels without a reverse motion vector, a default motion vector of zero (0,0) is assigned; for pixels with multiple reverse motion vectors, the motion vector is determined by the last assigned value.

[0102] The reason for generating a reverse motion vector map is that the subsequent pixel blending step requires inferring the pixel's position in the previous frame based on the current frame's pixel position and the reverse motion vector. The reason for not directly inverting the vector is that the starting point of a directly inverted motion vector is not the current pixel, but rather the current pixel's position in the previous frame, such as... Figure 2 As shown, it is impossible to obtain this motion vector from the position of the current pixel.

[0103] Furthermore, in step 1.2.2, when obtaining the nine temporally nearest motion vectors from the previous frame as candidate vectors, directly taking the current pixel position from the previous frame's motion vector image would result in a position mapping error; that is, the current pixel's position in the previous frame should be taken. By inverting the reverse motion vector image from the previous frame, the accurate candidate vectors from the previous frame can be obtained, because the starting point of the motion vector at the current pixel's position in the previous frame has been mapped to the current pixel's position in the current frame. This is another important reason why it is necessary to generate a reverse motion vector image.

[0104] Step 2: Pixel blending.

[0105] The goal of pixel fusion is to identify flickering pixels in an LED image through flicker detection and then mitigate flicker through cross-frame weighted fusion. This step involves performing the operation on every pixel of the frame image, except for the first frame.

[0106] 2.1: Flicker Detection:

[0107] For the current pixel, its pixel value is denoted as C. Based on the motion vector diagram, the mapping coordinates of the current pixel in the next frame are calculated, and the pixel value of the mapping coordinates is taken as the backward mapping pixel value B. Based on the reverse motion vector diagram, the mapping coordinates of the current pixel in the previous frame are obtained, and the pixel value of the mapping coordinates is taken as the forward mapping pixel value F.

[0108] Based on the current pixel value C, the backward mapped pixel value B, and the forward mapped pixel value F, two difference metrics are calculated: forward difference and backward difference. The forward difference is the pixel difference between the current pixel value C and the forward mapped pixel value F, and the backward difference is the pixel difference between the current pixel value C and the backward mapped pixel value B. When both difference metrics are simultaneously greater than a preset flicker threshold, the current pixel is determined to be a flickering pixel. The flicker threshold is generally set between 20 and 50, and can be adjusted according to the scene lighting and LED brightness.

[0109] 2.2: Flicker mitigation:

[0110] If the current pixel is determined to be a flickering pixel, then the pixel value is weighted and fused across frames according to the following formula:

[0111]

[0112] in, The final pixel value after fusion; The average weight is typically set to 0.3~0.7, and can be adjusted according to the flicker intensity to smooth out fluctuations in brightness. The maximum value is the weight, and its value is (1- This is used to preserve LED brightness information and avoid excessive darkness after fusion; it adapts according to flicker intensity (the magnitude of the difference index value), the more severe the flicker, the better. It can be increased appropriately.

[0113] If a pixel is not identified as flickering, its current pixel value is retained directly, meaning there is no need for blending, thus avoiding image blurring.

[0114] Once each pixel of the current frame has been fused, the image after LED flicker mitigation is obtained, and the final output is the processed video frame sequence.

[0115] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described motion-compensated LED flicker mitigation image processing method.

[0116] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method for mitigating LED flicker based on motion compensation, characterized in that, include: Step 1: Calculate the motion vector for each pixel in each frame of the video, and simultaneously generate the inverse motion vector map. Generating the motion vector map for the first frame includes: calculating the motion vector of the first matching block; using a recursive algorithm to obtain the motion vector for each subsequent matching block. Generating the motion vector map for each subsequent frame includes: generating 15 candidate motion vectors for the current prediction block; calculating the weights of the 15 candidate motion vectors; and fusing the 15 candidate motion vectors by weighted average to obtain the final motion vector MV for the current prediction block. pred ; Generate the reverse motion vector map of each frame image, including: traversing each pixel of the current frame and performing the following two operations: inverting the direction of the motion vector of the current pixel to obtain the reverse motion vector; calculating the mapping coordinates of the current pixel in the next frame by "current pixel coordinates + motion vector", and assigning the reverse motion vector to the pixel with the mapping coordinates in the next frame; Step 2: identify LED flickering pixels by flicker detection, and then achieve flicker mitigation by cross-frame weighted fusion.

2. The LED flicker mitigation image processing method based on motion compensation according to claim 1, characterized in that, In step 1, generating the motion vector map of the first frame of the video includes: dividing the frame image into several pixel blocks of size a*a, each pixel block being called a "matching block"; starting from the upper left of the image, traversing each matching block in the image and performing the following steps: Step 1.1.1: Calculate the motion vector of the first matching block: For the first matching block, the SAD algorithm is used to search for the most similar reference matching block in the second frame image, thereby obtaining the motion vector of the current matching block, and assigning this motion vector to all pixels of the current matching block; Step 1.1.2: Use a recursive algorithm to obtain the motion vector of each subsequent matching block. Motion vector of a matching block: Take the motion vectors of the four calculated neighboring matching blocks around the current matching block. The positions of these four neighboring matching blocks are the upper left, upper, upper right, and left of the current matching block, respectively. If a neighboring matching block does not exist at a certain position, the motion vector of that neighboring matching block is defaulted to 0. Several initial reference positions in the second frame are obtained by mapping "current matching block coordinates + neighboring matching block motion vectors". Within a set range centered on each initial reference position, the SAD algorithm is used to search for the most similar reference matching block to obtain the motion vector of the current matching block, and it is assigned to all pixels in the block.

3. The image processing method for mitigating LED flicker based on motion compensation according to claim 1, characterized in that, In step 1, generating the motion vector map for each subsequent frame of the first frame includes: dividing the current frame image into several pixel blocks of size b*b, each block being called a "prediction block"; starting from the upper left of the image, traversing each prediction block in the image and performing the following steps: Step 1.2.1: generating 15 candidate motion vectors for the current prediction block, specifically: 1 zero motion vector; 4 spatially neighboring motion vectors, taken from the motion vector map from the current frame to the next frame, the motion vectors of the 4 calculated spatially neighboring prediction blocks around the current prediction block, the 4 positions being the upper... 1.1.2.2: Calculate the weights of the 15 candidate motion vectors; 9 temporally neighboring motion vectors are taken from the reverse motion vector map from the previous frame to the current frame, specifically the reverse motion vectors of the 9 temporally neighboring pixel blocks surrounding the current predicted block's mapped coordinates in the previous frame. These 9 positions are: center, top, bottom, left, right, top left, bottom left, top right, and bottom right; 1 block matching motion vector is obtained by searching for similar blocks in the reference frame using the SAD algorithm; Step 1.2.2: Calculate the weights of the 15 candidate motion vectors; Motion vector index k=0, zero motion vector weight. ,in The mean of the motion vectors of the surrounding predicted blocks. The base values ​​for static scene weights. The time correlation strength parameter; the motion vector index k = 1~4, and the weight of spatially neighboring motion vectors. ,in This represents the mean absolute pixel difference between the current predicted block and its neighboring predicted blocks. For positional weights, Spatial weight attenuation coefficient; motion vector index k = 5~13, weight of time-nearest motion vectors. ,in A basic priority factor set based on spatial location. For the motion vector of the neighboring predicted block, The current predicted block motion vector, This is the time-weighted decay coefficient. The mean absolute pixel difference between the current predicted block and the corresponding block at the same position in the previous frame; motion vector index k=14, weight of the block matching motion vector. ,in The sum of the absolute pixel differences between the current predicted block and the reference matching block; Step 1.2.3: The final motion vector of the current predicted block is obtained by fusing 15 candidate motion vectors through weighted averaging. ,in Let k be the candidate motion vector; Assign to all pixels in the current prediction block.

4. The LED flicker mitigation image processing method based on motion compensation according to any one of claims 1-3, characterized in that, Step 2 includes: 2.1: Flicker detection: For the current pixel, the pixel value is denoted as C. Based on the motion vector diagram, the mapping coordinates of the current pixel in the next frame are calculated, and the pixel value at this mapping coordinate is taken as the backward mapping pixel value B. Based on the reverse motion vector diagram, the mapping coordinates of the current pixel in the previous frame are obtained, and the pixel value at this mapping coordinate is taken as the forward mapping pixel value F. Based on the current pixel value C, the backward mapping pixel value B, and the forward mapping pixel value F, two difference indicators are calculated: forward difference and backward difference. The forward difference is the pixel difference between the current pixel value C and the forward mapping pixel value F, and the backward difference is the pixel difference between the current pixel value C and the backward mapping pixel value B. When both difference indicators are greater than a preset flicker threshold, the current pixel is determined to be a flickering pixel. 2.2: Flicker mitigation: If the current pixel is determined to be a flickering pixel, the pixel value is weighted and fused across frames according to the following formula: in, The final pixel value after fusion. Weighted by mean, The maximum value is used as the weight; if a pixel is not judged as flickering, the current pixel value is directly retained; after each pixel of the current frame is fused, the image after LED flicker mitigation is obtained.

5. The LED flicker mitigation image processing method based on motion compensation according to any one of claims 1-3, characterized in that, In step 1, the value of b is less than the value of a.

6. The LED flicker mitigation image processing method based on motion compensation according to claim 1, characterized in that, Abnormal case handling when generating the reverse motion vector: For pixels without a reverse motion vector, a zero motion vector (0,0) is assigned by default; for pixels with multiple reverse motion vectors, the motion vector is determined by the last assignment.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the motion-compensated LED flicker mitigation image processing method according to any one of claims 1-6.

Images