An adaptive image enhancement and compression transmission system

Abstract

The present application relates to a kind of adaptive image enhancement and compressed transmission system, specifically relates to the field of image processing and compression encoding, gradient consistency mask is accurately marked gradient distortion area caused by enhancement, with the aid of optical flow guide reference frame pre-alignment and phase correlation correction eliminates motion estimation deviation, again with feature field weighted matching inhibits distortion area interference, finally according to gradient consistency mask average response adaptive adjustment enhancement gain or switch conservative algorithm, to effectively suppress interframe prediction drift and code rate surge in the scene of intense motion, while keeping the effect of detail enhancement, significantly improve compression efficiency and transmission stability.

Description

Technical Field

[0001] This invention relates to the field of image processing and compression coding, and more specifically, to an adaptive image enhancement and compression transmission system. Background Technology

[0002] With the widespread adoption of ultra-high-definition video capture equipment and the rapid growth in demand for real-time video transmission, applications such as high-speed sports broadcasts, drone aerial photography of rapidly passing scenes over the ground, and real-time perception by forward-facing cameras in autonomous vehicles place extremely high demands on the performance of video encoding systems. In these typical scenarios, video content often contains a large number of high-frequency details, such as grass textures, building edges, road markings, and the outlines of moving objects. Simultaneously, objects move rapidly within the frame, with significant displacement changes between adjacent frames. To improve the subjective visual quality for the human eye or the recognition accuracy of subsequent machine vision tasks, many systems introduce image enhancement loops before encoding. The original frame is sharpened and contrast enhanced using algorithms such as multi-scale Laplacian pyramids, guided filtering, or nonlocal means. However, these enhancement operations are not independent of the encoding process, and the spatial domain nonlinear transformations they introduce will profoundly affect the working mechanism of subsequent compression modules. At the same time, the encoder must complete motion estimation, motion compensation, transform quantization, and entropy coding under strict real-time constraints. Any additional computational overhead or degradation of prediction accuracy may lead to excessive encoding latency or uncontrolled output bitrate. Therefore, how to establish a collaborative mechanism between enhancement and compression has become an urgent engineering problem to be solved in the design of current video transmission systems.

[0003] In the context of the aforementioned applications, existing technologies generally employ a processing architecture that separates enhancement and encoding sequentially. This involves performing a complete enhancement operation on the original frame first, and then sending the enhanced image to a standard encoder for compression. However, this simple sequential approach ignores the destructive impact of enhancement operations on the core encoding module—inter-frame prediction. Specifically, multi-scale detail enhancement significantly alters the relative gradient relationships and local statistical characteristics between pixels, making motion vectors that were well-matched in the original frame unreliable in the enhanced frame, leading to a sharp increase in motion estimation errors. Furthermore, inter-frame prediction depends on the reconstruction quality of the reference frame. If the enhancement strategy differs between different frames or different regions of the same frame—for example, using different enhancement intensities for moving and non-moving regions—it will... This leads to a significant increase in the prediction residual between the reference frame and the current enhanced frame, resulting in a spike in bit rate and inter-frame prediction drift. In severe cases, the encoder is forced to frequently insert intra-frame refresh blocks, reducing compression efficiency by 30% to 50% and interrupting real-time transmission. Although some studies have attempted to embed enhancement modules within the coding loop, such as deblocking filters or sample adaptive offsets, these modules only compensate for block boundary artifacts or pixel-level offsets and cannot solve the gradient reconstruction and motion matching failure problems caused by global detail enhancement. Therefore, there is an urgent need for a motion estimation algorithm that can sense the effects of enhancement, enabling the encoder to complete accurate matching in the enhanced feature space or through the mapping relationship between the original frame and the enhanced frame, thereby maintaining the stability of inter-frame prediction and compression efficiency while preserving the enhancement effect. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing an adaptive image enhancement and compression transmission system to solve the problems mentioned in the background art.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: Specifically, it includes: a gradient consistency mask generation module, an optical flow guided reference frame pre-alignment module, a joint feature space motion estimation module, and a motion vector back projection and feedback module connected in sequence, wherein;

[0006] Gradient consistency mask generation module: Receives the current original frame, the reference original frame, the current enhanced frame and the reference enhanced frame after multi-scale Laplacian pyramid enhancement, calculates the gradient magnitude map and gradient direction map of the current original frame and the current enhanced frame, compares the magnitude difference with the first threshold and the direction difference with the second threshold, generates a gradient consistency mask, and pixels with a value of 1 in the gradient consistency mask mark the regions where the enhancement operation significantly changes the gradient structure.

[0007] Optical flow-guided reference frame pre-alignment module: It calls a lightweight optical flow network to calculate the optical flow field from the reference original frame to the current original frame with the current original frame and the reference original frame as input. The optical flow field is applied to the reference enhancement frame and bilinear sampling is performed to output the pre-aligned reference enhancement frame. Then, the gradient consistency mask is used to replace the region marked by the gradient consistency mask in the optical flow field with the block-based phase correlation matching result to obtain the corrected optical flow field and the corresponding corrected aligned reference frame.

[0008] Joint feature space motion estimation module: The current original frame and the current enhanced frame are respectively input into a convolutional neural network with shared weights to extract high-dimensional feature maps. In the feature domain, the gradient consistency mask weighting coefficient is used as the weight of the matching cost function to perform block matching motion estimation. The gradient consistency mask weighting coefficient is negatively correlated with the number of pixels with a value of 1 in the gradient consistency mask. The feature domain motion vector field is output.

[0009] Motion Vector Backprojection and Feedback Module: Backprojects the feature domain motion vector field to the pixel resolution to obtain the pixel domain motion vector. Uses the pixel domain motion vector to perform motion compensation on the corrected alignment reference frame to generate the prediction frame. Calculates the prediction residual and counts the gradient consistency mask average response of the current coding unit. Compares the gradient consistency mask average response with the gain adjustment threshold. If the gradient consistency mask average response exceeds the gain adjustment threshold, the enhancement gain of the next frame is reduced or the conservative enhancement algorithm is switched.

[0010] In a preferred embodiment, the specific operation of calculating the gradient magnitude map and gradient direction map of the current original frame and the current enhanced frame in the gradient consistency mask generation module is as follows:

[0011] The Sobel operator is used to calculate the horizontal and vertical gradients for the current original frame and the current enhanced frame respectively. Then, the gradient magnitude of each pixel is calculated as the arithmetic square root of the sum of the squares of the horizontal and vertical gradients to obtain the gradient magnitude map. The gradient direction of each pixel is calculated as the arctangent of the ratio of the vertical gradient to the horizontal gradient to obtain the gradient direction map.

[0012] Then, a joint gradient structure change significance measure is constructed, which consists of a weighted sum of two terms. The first term is the absolute logarithm of the ratio of the gradient magnitude of the current enhanced frame to the gradient magnitude of the current original frame, and the second term is the cosine of the difference between the gradient direction of the current enhanced frame and the gradient direction of the current original frame. The weighting coefficients of the first and second terms are 0.6 and 0.4, respectively.

[0013] In a preferred embodiment, the specific operation of comparing the magnitude difference with a first threshold and the direction difference with a second threshold to generate a gradient consistency mask is as follows:

[0014] The magnitude difference is defined as the difference between the gradient magnitude of the current enhanced frame and the gradient magnitude of the current original frame, and the direction difference is defined as the difference between the gradient direction of the current enhanced frame and the gradient direction of the current original frame.

[0015] For each pixel, a 7x7 neighborhood window is selected centered on it, and the local mean and local median of the joint gradient structure change significance measure within the neighborhood window are calculated; the first threshold is equal to the scaling factor multiplied by the local mean, and the scaling factor is 0.8; the second threshold is equal to the median weight multiplied by the local median, and the median weight is 1.5; the adaptive threshold is equal to the sum of the first threshold and the second threshold.

[0016] The significance measure of the joint gradient structure change for each pixel is compared with the adaptive threshold corresponding to the pixel. If the significance measure of the joint gradient structure change is greater than the adaptive threshold, the value of the pixel in the gradient consistency mask is set to one, indicating that the gradient structure of the pixel has been significantly changed by the enhancement operation; otherwise, the value of the pixel in the gradient consistency mask is set to zero.

[0017] In a preferred embodiment, the specific operation of calling a lightweight optical flow network to calculate the optical flow field from the reference original frame to the current original frame using the current original frame and the reference original frame as input in the optical flow-guided reference frame pre-alignment module is as follows:

[0018] The lightweight optical flow network adopts the RAFT-lite structure, takes the grayscale images of the current original frame and the reference original frame as input, and outputs the horizontal and vertical displacement components of each pixel through an iterative update method to form an optical flow field.

[0019] The optical flow field is applied to the reference enhancement frame, and the corresponding horizontal and vertical displacement components are added to the coordinates of each pixel in the reference enhancement frame. The pixel values ​​at non-integer coordinates are calculated by bilinear interpolation to obtain the pre-aligned reference enhancement frame.

[0020] Simultaneously, based on the pixels with a value of one in the gradient consistency mask and the value of the joint gradient structure change significance metric, the optical flow correction necessity index for each pixel is calculated. The specific calculation process is as follows:

[0021] Obtain the pixel value corresponding to the gradient consistency mask, multiply it by an exponential function value that is the product of a negative decay constant and a measure of the significance of the joint gradient structure change, and the resulting product is the optical flow correction necessity index.

[0022] In a preferred embodiment, the specific operation of replacing the region marked by the gradient consistency mask in the optical flow field with the block-based phase correlation matching result using the gradient consistency mask is as follows:

[0023] For each pixel, if the optical flow correction necessity index is greater than the preset activation threshold, a local window of size 16 by 16 pixels is selected centered on the pixel. Phase correlation matching based on Fourier transform is performed between the current enhancement frame and the pre-aligned reference enhancement frame to calculate the sub-pixel precision local optical flow correction amount. The local optical flow correction amount is obtained through the following process:

[0024] Discrete Fourier transforms are performed on the image blocks of the current enhanced frame within the local window and the image blocks of the pre-aligned reference enhanced frame at the corresponding positions. The cross power spectrum of the two is calculated, and the phase difference of the relative displacement is obtained by phase expansion of the cross power spectrum. The displacement peak value with sub-pixel precision is obtained by performing inverse Fourier transform on the phase difference. The displacement peak value is the local optical flow correction amount.

[0025] Then, the optical flow field and the local optical flow correction are weighted and fused to obtain the corrected optical flow field;

[0026] Finally, bilinear sampling is performed again on the reference enhancement frame using the corrected optical flow field to obtain the corrected aligned reference frame. The coordinates of each pixel in the corrected aligned reference frame are obtained by adding the horizontal and vertical displacement components of the corrected optical flow field to the corresponding coordinates in the reference enhancement frame and then interpolating them bilinearly.

[0027] In a preferred embodiment, the weighted fusion process specifically includes:

[0028] Multiply the difference obtained by subtracting the optical flow correction necessity index from the optical flow field, then multiply the optical flow correction necessity index by the sum of the optical flow field and the local optical flow correction amount, and finally add the two product results. The sum is the horizontal and vertical displacement components of the corresponding pixel in the corrected optical flow field, thus obtaining the corrected optical flow field.

[0029] In a preferred embodiment, the specific operation of inputting the current original frame and the current enhanced frame into a convolutional neural network with shared weights to extract high-dimensional feature maps in the joint feature space motion estimation module is as follows:

[0030] The weighted convolutional neural network consists of an input layer, four convolutional layers, and a global average pooling layer. The kernel sizes of the four convolutional layers are 7x7, 5x5, 3x3, and 3x3, respectively, and the number of output channels are 32, 64, 128, and 128, respectively. The global average pooling layer outputs a 128-dimensional high-dimensional feature map. In the weighted convolutional neural network, the input layer receives data from the current original frame and the current enhanced frame. The first convolutional layer uses a 7x7 kernel and outputs 32 feature channels. The second convolutional layer uses a 5x5 kernel and outputs 64 feature channels. The third and fourth convolutional layers both use 3x3 kernels and output 128 feature channels, respectively. All convolutional layers use a stride of 1, and each convolutional layer is followed by a non-linear activation function and a local response normalization layer to extract translation-invariant local texture features. The global average pooling layer averages the spatial dimensions of the output feature map of the last convolutional layer, generating a compact high-dimensional feature map.

[0031] A weighted convolutional neural network (CNN) is pre-trained on an image dataset containing various enhancement types, including histogram equalization, Laplacian sharpening, and contrast-limited adaptive histogram equalization. The training objective is to minimize the cosine distance between the current original frame and the current enhanced frame in the feature space. The current original frame, the current enhanced frame, and the corrected alignment reference frame are respectively input into the weighted CNN to obtain the corresponding high-dimensional feature maps of the original frame, enhanced frame, and reference frame. The specific calculation process is as follows: the current original frame is input into the weighted CNN, and after layer-by-layer operations through the input layer, four convolutional layers, and a global average pooling layer, the high-dimensional feature map of the original frame is output. The current enhanced frame is input into the weighted CNN, and after the same layer operations, the high-dimensional feature map of the enhanced frame is output. The corrected alignment reference frame is input into the weighted CNN, and after the same layer operations, the high-dimensional feature map of the reference frame is output.

[0032] In a preferred embodiment, the specific operation of performing block matching motion estimation using gradient consistency mask weighting coefficients as weights in the matching cost function in the feature domain is as follows:

[0033] First, for each pixel, the gradient structure tensor of the current original frame is calculated. The gradient structure tensor is formed by summing the squares of the horizontal and vertical gradient components within a 5x5 neighborhood window centered on each pixel, and summing the products of the horizontal and vertical gradient components. This forms a 3x3 symmetric matrix. Next, the two eigenvalues ​​of the symmetric matrix are solved and arranged in descending order. Then, the anisotropy index is calculated by subtracting the smaller eigenvalue from the larger eigenvalue, dividing by the sum of the two eigenvalues, and adding a constant coefficient to the denominator.

[0034] Secondly, the gradient consistency mask weighting coefficient is calculated based on the gradient consistency mask and the anisotropy index. The calculation process is as follows: obtain the value of the pixel corresponding to the gradient consistency mask, multiply the value by the difference between one and the anisotropy index, and then multiply the resulting product by the penalty intensity coefficient. The result is the gradient consistency mask weighting coefficient.

[0035] Finally, using an 8x8 pixel block in the enhanced frame's high-dimensional feature map as a template, within a search range of ±32 pixels in the reference frame's high-dimensional feature map, the Euclidean distance weighted by gradient consistency mask weighting coefficients is used as the matching cost function. A three-step search method is employed to find the motion vector that minimizes the cost. The step size of each step is adaptively adjusted based on the average gradient consistency mask weighting coefficient of the current block. If the average gradient consistency mask weighting coefficient is less than 0.5, the step size is reduced to 0.75 times the original value. The optimal motion vectors of all blocks are combined to obtain the feature domain motion vector field. The calculation process of the matching cost function is as follows:

[0036] Traverse all pixel positions within the block, multiply the square of the difference between the gradient consistency mask weighting coefficient and the value of the enhanced frame high-dimensional feature map at the current pixel position and the value of the reference frame high-dimensional feature map at the current pixel position after adding the candidate motion vector, and accumulate them to obtain the total cost of the current candidate motion vector. The candidate motion vector contains horizontal displacement components and vertical displacement components. Find the vector that minimizes the total cost among all candidate motion vectors as the optimal motion vector. The optimal motion vector is the motion vector of the current block.

[0037] In a preferred embodiment, the specific operation of back-projecting the feature domain motion vector field to the pixel resolution to obtain the pixel domain motion vector in the motion vector back-projection and feedback module is as follows:

[0038] For each pixel, based on its relative coordinates within its 8x8 block, the distance from the pixel to the block center and the distance from the pixel to the block boundary are calculated. Interpolation weights are then calculated using the gradient consistency mask and the anisotropy exponent. The calculation of the interpolation weights begins by constructing a Gaussian decay function based on the distance from the pixel to the block center. Specifically, this function is an exponential function with a base of the natural constant and an exponent equal to the negative of the square of the distance from the pixel to the block center divided by the square of twice the standard deviation of the Gaussian kernel. The standard deviation of the Gaussian kernel is set to 2.0. Based on this, a distance modulation term and a gradient structure modulation term are introduced. The distance modulation term is an exponential function with a base of the natural constant and an exponent equal to the product of the negative distance modulation decay constant and the distance from the pixel to the block boundary. The gradient structure modulation term is obtained by adding one product of the anisotropy exponent and subtracting one product of the gradient consistency mask, and then multiplying by the modulation coefficient.

[0039] The initial weight of a pixel is obtained by multiplying the value of the Gaussian decay function, the distance modulation term, and the gradient structure modulation term. Finally, the initial weight is normalized by dividing it by the sum of the initial weights of all pixels in the eight-by-eight block to which the pixel belongs, resulting in the final interpolation weight.

[0040] Then, the motion vectors of four adjacent blocks in the feature domain motion vector field are weighted and averaged using interpolation weights to obtain the pixel domain motion vector of each pixel. The specific calculation process is as follows:

[0041] For each 8x8 block in the feature domain motion vector field, the horizontal coordinate of the top-left pixel in the feature domain is taken as the block's horizontal coordinate, and the vertical coordinate is taken as the block's vertical coordinate. The horizontal coordinate of the current pixel to be calculated in the pixel domain is taken as the current pixel's horizontal coordinate, and the vertical coordinate is taken as the current pixel's vertical coordinate. The difference between the current pixel's horizontal coordinate and the block's horizontal coordinate is calculated, and the difference is divided by eight to obtain the decimal offset in the horizontal direction. Similarly, the difference between the current pixel's vertical coordinate and the block's vertical coordinate is calculated, and the difference is divided by eight to obtain the decimal offset in the vertical direction. Using these decimal offsets, bilinear interpolation is used to calculate the interpolation weights corresponding to the centers of four adjacent blocks, located at the top-left, top-right, bottom-left, and right corners, respectively. The horizontal coordinate of the center of the bottom left block is the horizontal coordinate of the block multiplied by eight, and the vertical coordinate is the vertical coordinate of the block multiplied by eight. The horizontal coordinate of the center of the top right block is the horizontal coordinate of the block plus one and multiplied by eight, and the vertical coordinate is the vertical coordinate of the block multiplied by eight. The horizontal coordinate of the center of the bottom left block is the horizontal coordinate of the block multiplied by eight, and the vertical coordinate is the vertical coordinate of the block plus one and multiplied by eight. The horizontal coordinate of the center of the bottom right block is the horizontal coordinate of the block plus one and multiplied by eight, and the vertical coordinate is the vertical coordinate of the block plus one and multiplied by eight. The interpolation weights corresponding to the centers of the four adjacent blocks are inversely proportional to the distances from the centers of the four adjacent blocks to the current pixel. The motion vectors of the four adjacent blocks are weighted and summed using the interpolation weights corresponding to the centers of the four adjacent blocks. The result of the weighted sum is used as the pixel domain motion vector of the current pixel.

[0042] The specific operation of using pixel-domain motion vectors to perform motion compensation on the corrected alignment reference frame is as follows:

[0043] For each pixel in the prediction frame to be generated, obtain the integer coordinate position of the pixel in the corrected alignment reference frame, and the sub-pixel offset composed of the horizontal and vertical components of the pixel domain motion vector at the pixel. Extract the pixel values ​​at the integer coordinate position and its four adjacent integer coordinate positions to the right and bottom. Multiply these four pixel values ​​by their corresponding bilinear interpolation weights, which are determined by the horizontal and vertical offset ratios. Finally, add these four product results to obtain the final pixel value of the pixel, thereby generating the prediction frame.

[0044] In a preferred embodiment, the process of calculating the prediction residual and statistically analyzing the gradient consistency mask average response of the current coding unit specifically includes:

[0045] The difference between the pixel value of the current enhanced frame at each pixel position in the current coding unit and the pixel value of the prediction frame at the corresponding pixel position is calculated to obtain the prediction residual. At the same time, the gradient consistency mask value at each pixel position in the current coding unit is obtained. The gradient consistency mask values ​​of all pixel positions in the current coding unit are accumulated and summed. Then, the summation result is divided by the total number of pixels in the current coding unit to obtain the average gradient consistency mask response.

[0046] Then, the average response of the gradient consistency mask is compared with the gain adjustment threshold, which is set to 0.3. If the average response of the gradient consistency mask is greater than the gain adjustment threshold, the enhancement gain of the next frame is reduced. The enhancement gain of the current frame is the gain coefficient used in the multi-scale Laplacian pyramid enhancement of the current frame. The reduction method is to multiply the enhancement gain of the current frame by an exponential function value with the natural constant as the base and the product of the negative decay rate constant and the difference between the average response of the gradient consistency mask and the gain adjustment threshold. The decay rate constant is set to 2.0.

[0047] If the average response of the gradient consistency mask is greater than the switching threshold, which is set to 0.6, the enhancement algorithm for the next frame will be switched to the conservative enhancement algorithm. The conservative enhancement algorithm only performs basic brightness adjustments such as histogram equalization or gamma correction, and prohibits nonlinear sharpening operations that introduce new edges or significantly change the gradient magnitude.

[0048] The beneficial effects of this invention are as follows: by accurately marking the gradient distortion region caused by enhancement through gradient consistency mask, eliminating motion estimation bias by using optical flow-guided reference frame pre-alignment and phase correlation correction, suppressing interference in the distortion region by feature domain weighted matching, and finally adaptively adjusting the enhancement gain or switching to a conservative algorithm based on the average response of the gradient consistency mask, the invention effectively suppresses inter-frame prediction drift and bit rate spikes in scenes with intense motion, and significantly improves compression efficiency and transmission stability while maintaining the detail enhancement effect. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention;

[0050] Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0053] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0054] Example 1

[0055] This embodiment provides, for example Figure 1-2 An adaptive image enhancement and compression transmission system is shown, specifically comprising: a gradient consistency mask generation module, an optical flow-guided reference frame pre-alignment module, a joint feature space motion estimation module, and a motion vector back-projection and feedback module connected in sequence, wherein;

[0056] Gradient consistency mask generation module: Receives the current original frame, the reference original frame, the current enhanced frame and the reference enhanced frame after multi-scale Laplacian pyramid enhancement, calculates the gradient magnitude map and gradient direction map of the current original frame and the current enhanced frame, compares the magnitude difference with the first threshold and the direction difference with the second threshold, generates a gradient consistency mask, and pixels with a value of 1 in the gradient consistency mask mark the regions where the enhancement operation significantly changes the gradient structure.

[0057] Optical flow-guided reference frame pre-alignment module: It calls a lightweight optical flow network to calculate the optical flow field from the reference original frame to the current original frame with the current original frame and the reference original frame as input. The optical flow field is applied to the reference enhancement frame and bilinear sampling is performed to output the pre-aligned reference enhancement frame. Then, the gradient consistency mask is used to replace the region marked by the gradient consistency mask in the optical flow field with the block-based phase correlation matching result to obtain the corrected optical flow field and the corresponding corrected aligned reference frame.

[0058] Joint feature space motion estimation module: The current original frame and the current enhanced frame are respectively input into a convolutional neural network with shared weights to extract high-dimensional feature maps. In the feature domain, the gradient consistency mask weighting coefficient is used as the weight of the matching cost function to perform block matching motion estimation. The gradient consistency mask weighting coefficient is negatively correlated with the number of pixels with a value of 1 in the gradient consistency mask. The feature domain motion vector field is output.

[0059] Motion Vector Backprojection and Feedback Module: Backprojects the feature domain motion vector field to the pixel resolution to obtain pixel domain motion vectors. Uses the pixel domain motion vectors to perform motion compensation on the corrected alignment reference frame to generate a prediction frame. Calculates the prediction residual and statistically analyzes the gradient consistency mask average response of the current coding unit. Compares the gradient consistency mask average response with the gain adjustment threshold. If the gradient consistency mask average response exceeds the gain adjustment threshold, the enhancement gain of the next frame is reduced or the conservative enhancement algorithm is switched.

[0060] In this embodiment, it is specifically necessary to explain the specific operations of the gradient consistency mask generation module in calculating the gradient magnitude map and gradient direction map of the current original frame and the current enhanced frame as follows:

[0061] The Sobel operator is used to calculate the horizontal and vertical gradients for the current original frame and the current enhanced frame respectively. Then, the gradient magnitude of each pixel is calculated as the arithmetic square root of the sum of the squares of the horizontal and vertical gradients to obtain the gradient magnitude map. The gradient direction of each pixel is calculated as the arctangent of the ratio of the vertical gradient to the horizontal gradient to obtain the gradient direction map.

[0062] The Sobel operator uses a 3x3 convolution kernel, with the horizontal kernel being [[-1,0,1],[-2,0,2],[-1,0,1]] and the vertical kernel being [[-1,-2,-1],[0,0,0],[1,2,1]], to extract the horizontal and vertical edge responses respectively. The gradient magnitude map reflects the absolute magnitude of the edge intensity at each pixel, while the gradient direction map records the normal direction of the edge, with values ​​ranging from negative π to π.

[0063] Then, a joint gradient structure change significance measure is constructed, which consists of a weighted sum of two terms. The first term is the absolute logarithm of the ratio of the gradient magnitude of the current enhanced frame to the gradient magnitude of the current original frame, and the second term is the cosine of the difference between the gradient direction of the current enhanced frame and the gradient direction of the current original frame. The weighting coefficients of the first and second terms are 0.6 and 0.4, respectively.

[0064] The absolute logarithm is used instead of the simple difference because logarithms can symmetrically reflect the amplification and reduction of amplitude and are more sensitive to small changes in the low amplitude region. Cosine distance is used instead of the absolute angle difference because cosine distance is continuously differentiable and has a fixed range of values, which facilitates subsequent threshold comparisons. The weighting coefficients 0.6 and 0.4 were determined through extensive experimental statistics and can balance the contributions of amplitude and direction changes in most natural scenarios. If direction changes are more important in the scenario, the coefficients can be adjusted to 0.5 and 0.5. During the calculation, to prevent the denominator and logarithmic parameter from being zero, a constant coefficient is added for smoothing when calculating the gradient ratio and logarithm. The constant coefficient has a value of one.

[0065] The constant coefficient is set to one because the image pixel value is usually in the range of 0 to 255, and the gradient magnitude is usually in the range of 0 to several hundred. Adding one will not significantly change the ratio and can effectively avoid division by zero error. The joint gradient structure change significance measure is used to quantitatively characterize the combined effect of the relative change of gradient magnitude and the deviation of gradient direction caused by the enhancement operation at each pixel.

[0066] The larger the significance measure of the joint gradient structure change, the more drastic the gradient structure at that pixel is altered by the enhancement operation, and the higher the probability of a matching error in subsequent motion estimation at that location.

[0067] The specific steps for comparing the magnitude difference with the first threshold and the direction difference with the second threshold to generate a gradient consistency mask are as follows:

[0068] The magnitude difference is defined as the difference between the gradient magnitude of the current enhanced frame and the gradient magnitude of the current original frame, and the direction difference is defined as the difference between the gradient direction of the current enhanced frame and the gradient direction of the current original frame.

[0069] Amplitude difference and direction difference quantify the difference before and after enhancement from the two dimensions of amplitude and angle, respectively. However, since the two have different dimensions, it is difficult to adapt to the diverse image content by directly setting a fixed threshold. Therefore, a joint gradient structure change significance measure is introduced as a unified indicator.

[0070] For each pixel, a 7x7 neighborhood window is selected centered on it, and the local mean and local median of the joint gradient structure change significance measure within the neighborhood window are calculated.

[0071] The size of the 7x7 window was determined experimentally: a window that is too small (e.g., 3x3) is susceptible to noise interference and local statistical instability; a window that is too large (e.g., 15x15) blurs edge details and reduces the spatial resolution of the mask. The 7x7 window strikes a good balance between maintaining statistical reliability and preserving details. The local mean reflects the central tendency of significance levels within the neighborhood, while the local median is insensitive to individual extreme values; the two complement each other. The first threshold is equal to the scaling factor multiplied by the local mean, with the scaling factor having a value of 0.8. The second threshold is equal to the median weight multiplied by the local median, with the median weight having a value of 1.5. The adaptive threshold is equal to the sum of the first and second thresholds.

[0072] The scaling factor of 0.8 is used to moderately reduce the influence of the local mean and avoid generating too many labels in texture-dense areas; the median weight of 1.5 is used to amplify the contribution of the median and make the threshold more robust to outliers. The sum of the two is used as an adaptive threshold, which can be dynamically adjusted according to local statistical characteristics. The threshold is lower in flat areas to avoid mislabeling, and higher in texture-rich areas to only label significant changes.

[0073] The significance measure of the joint gradient structure change of each pixel is compared with the adaptive threshold corresponding to the pixel. If the significance measure of the joint gradient structure change is greater than the adaptive threshold, the value of the pixel in the gradient consistency mask is set to one, indicating that the gradient structure of the pixel has been significantly changed by the enhancement operation; otherwise, the value of the pixel in the gradient consistency mask is set to zero.

[0074] The comparison process is performed independently for each pixel, and the generated binary mask can be directly used as the weight basis for subsequent optical flow correction and feature domain motion estimation.

[0075] The adaptive thresholding mechanism uses the local mean to reflect the average significance level in the neighborhood and the local median to resist the interference of outliers. This allows the mask generation process to automatically adjust the judgment sensitivity according to the statistical characteristics of the local texture of the image, avoiding over-labeling of uniform texture areas by the global fixed threshold. Experimental results show that, compared with the global fixed threshold, the adaptive thresholding mechanism reduces the mislabeling rate in uniform sky areas by about 40%, while maintaining a recall rate of over 90% in texture edge areas, effectively improving the reliability of the mask.

[0076] In this embodiment, it is specifically necessary to explain the operation of the optical flow guided reference frame pre-alignment module in which the lightweight optical flow network is called to calculate the optical flow field from the reference original frame to the current original frame using the current original frame and the reference original frame as input.

[0077] The lightweight optical flow network adopts the RAFT-lite structure, takes the grayscale images of the current original frame and the reference original frame as input, and outputs the horizontal and vertical displacement components of each pixel through an iterative update method to form an optical flow field.

[0078] The RAFT-lite network consists of a feature extractor, a correlation layer, and a recurrent iterative update module. The feature extractor uses a six-layer convolutional neural network to extract multi-scale features. The correlation layer calculates the pixel-level correlation volume between feature maps of two frames. The recurrent iterative update module performs twelve iterations based on gated recurrent units, outputting an incremental displacement in each iteration, and finally converges to obtain the sub-pixel precision optical flow field. The grayscale image is obtained by weighting and summing the RGB components of the color frame according to the standard luminance coefficients, with weights of 0.299, 0.587, and 0.114, respectively.

[0079] The optical flow field is applied to the reference enhancement frame, and the corresponding horizontal and vertical displacement components are added to the coordinates of each pixel in the reference enhancement frame. The pixel values ​​at non-integer coordinates are calculated by bilinear interpolation to obtain the pre-aligned reference enhancement frame.

[0080] Simultaneously, based on the pixels with a value of one in the gradient consistency mask and the value of the joint gradient structure change significance measure, the optical flow correction necessity index for each pixel is calculated. The specific calculation process is as follows:

[0081] Obtain the pixel value corresponding to the gradient consistency mask, multiply it by an exponential function value that is the product of a negative attenuation constant and a measure of the significance of the joint gradient structure change, and the resulting product is the optical flow correction necessity index; where the attenuation constant is set to 0.5.

[0082] The attenuation constant of 0.5 was determined through grid search experiments. Under ten typical motion scenarios, the dynamic range of the optical flow correction necessity index covers 0.1 to 0.9, avoiding both premature saturation and excessively low discrimination. The calculation process ensures that when the value of the joint gradient structure change significance measure is large, the exponential function value approaches zero, and 1 minus the exponential function value approaches one. In this case, if the gradient consistency mask value is one, the optical flow correction necessity index value approaches one, indicating an urgent need for correction; if the gradient consistency mask value is zero, the optical flow correction necessity index value is zero, indicating no need for correction. When the value of the joint gradient structure change significance measure is small, the exponential function value is large, and 1 minus the exponential function value is small, resulting in a small optical flow correction necessity index value, indicating that even if the gradient structure changes, strong correction is not required. The optical flow correction necessity index quantifies the urgency of the optical flow field needing correction; the closer its value is to one, the more necessary the optical flow field correction is at the corresponding pixel.

[0083] The specific operation of replacing the regions marked by the gradient consistency mask in the optical flow field with the block-based phase correlation matching results using the gradient consistency mask is as follows:

[0084] For each pixel, if the optical flow correction necessity index is greater than the preset activation threshold, a local window of size 16 by 16 pixels is selected centered on the pixel, and phase correlation matching based on Fourier transform is performed between the current enhancement frame and the pre-aligned reference enhancement frame to calculate the local optical flow correction amount with sub-pixel precision.

[0085] The preset activation threshold was set to 0.3, a value obtained through ROC curve analysis. This ensured that over 90% of the true correction area was activated while keeping the false activation rate below 5%. The 16x16 pixel window size was verified through ablation experiments: when the window size was smaller than 12x12, the peak signal-to-noise ratio of phase correlation decreased; when it was larger than 20x20, the computational load doubled and motion non-uniformity led to accuracy saturation. 16x16 achieved the optimal balance between accuracy and efficiency. The local optical flow correction was obtained through the following process:

[0086] Discrete Fourier transforms are performed on the image blocks of the current enhanced frame within the local window and the image blocks of the pre-aligned reference enhanced frame at the corresponding positions. The cross power spectrum of the two is calculated. The phase difference of the relative displacement is obtained by phase expansion of the cross power spectrum. The displacement peak value with sub-pixel precision is obtained by performing inverse Fourier transform on the phase difference. The displacement peak value is the local optical flow correction amount, which represents the horizontal and vertical offset of the optical flow field that needs to be adjusted within the local window.

[0087] Phase-correlation matching utilizes the translation invariance of Fourier transform to resist local brightness variations and additive noise, and can stably estimate displacement even when there is a contrast difference between the enhanced frame and the original frame; sub-pixel accuracy is achieved by fitting a five-point quadratic surface around the peak of the cross-power spectrum, and the displacement resolution can reach 0.1 pixels.

[0088] Then, the optical flow field and the local optical flow correction are weighted and fused to obtain the corrected optical flow field;

[0089] Finally, bilinear sampling is performed again on the reference enhancement frame using the corrected optical flow field to obtain the corrected aligned reference frame. The coordinates of each pixel in the corrected aligned reference frame are obtained by adding the horizontal and vertical displacement components of the corrected optical flow field to the corresponding coordinates in the reference enhancement frame and then interpolating them using bilinear interpolation. The corrected aligned reference frame serves as the reference input for the subsequent joint feature space motion estimation module.

[0090] The weighted fusion process is as follows:

[0091] Multiply the difference obtained by subtracting the optical flow correction necessity index from the optical flow field, then multiply the optical flow correction necessity index by the sum of the optical flow field and the local optical flow correction amount, and finally add the two product results. The sum is the horizontal and vertical displacement components of the corresponding pixel in the corrected optical flow field, and the corrected optical flow field is obtained.

[0092] This weighted fusion method ensures that when the optical flow correction necessity index is close to one, the corrected optical flow field is almost entirely dominated by the sum of the optical flow field and the local optical flow correction amount, achieving thorough correction of the enhanced distortion region; when the optical flow correction necessity index is close to zero, the corrected optical flow field almost completely retains the original optical flow field, avoiding unnecessary correction to reliable regions; the fusion process guarantees the spatial continuity of the optical flow field, even if adjacent pixels have different optical flow correction necessity indices, the fused optical flow field will not exhibit abrupt changes, because the weights change continuously with the index; the weighted fusion process is mathematically equivalent to the product of the difference obtained by subtracting the optical flow correction necessity index from one and the optical flow field, plus the product of the optical flow correction necessity index and the sum of the optical flow field and the local optical flow correction amount.

[0093] In this embodiment, it is specifically necessary to explain the following operation in the joint feature space motion estimation module: The current original frame and the current enhanced frame are input into a convolutional neural network with shared weights to extract high-dimensional feature maps:

[0094] The weighted convolutional neural network consists of an input layer, four convolutional layers, and a global average pooling layer. The kernel sizes of the four convolutional layers are 7x7, 5x5, 3x3, and 3x3, respectively, and the number of output channels are 32, 64, 128, and 128, respectively. The global average pooling layer outputs a 128-dimensional high-dimensional feature map. In the weighted convolutional neural network, the input layer receives data from the current original frame and the current augmented frame. The first convolutional layer uses a 7x7 kernel and outputs 32 feature channels; the second convolutional layer uses a 5x5 kernel and outputs 64 feature channels; the third and fourth convolutional layers both use 3x3 kernels and output 128 feature channels respectively. All convolutional layers... A stride of one is used, and each convolutional layer is followed by a nonlinear activation function and a local response normalization layer to extract translation-invariant local texture features. The nonlinear activation function uses rectified linear units, whose expression is that the output equals the maximum value between the input and zero. This function is simple to compute and can alleviate the gradient vanishing problem. The local response normalization layer uses cross-channel normalization with a normalization window size of five. The normalization intensity of each channel is determined by the sum of the squares of the five adjacent channels, and the normalization constant is 10 to the power of negative four. This normalization operation can enhance feature contrast and suppress irrelevant responses. The global average pooling layer averages and compresses the spatial dimension of the output feature map of the last convolutional layer to generate a compact high-dimensional feature map.

[0095] A weighted convolutional neural network is pre-trained on an image dataset containing various enhancement types, including histogram equalization, Laplacian sharpening, and contrast-limited adaptive histogram equalization. The training objective is to minimize the cosine distance between the current original frame and the current enhanced frame in the feature space, making the high-dimensional feature map robust to detail sharpening. The training dataset consists of 100,000 pairs of original and enhanced frames, with the enhancement type randomly selected from the three methods mentioned above, each method accounting for one-third of the total. During training, a stochastic gradient descent optimizer is used, with momentum set to 0.9 and an initial learning rate of 0.001, which decays to one-tenth of the original rate every five full epochs, for a total of thirty epochs. The loss function is defined as the negative of the cosine of the angle between the feature vector of the current original frame and the feature vector of the current enhanced frame, i.e., the cosine distance. Minimizing this distance ensures that the feature representation of the same content under different enhancement levels is as close as possible to the original frame's feature vector. The algorithm is as follows: The current original frame, the current enhanced frame, and the corrected alignment reference frame are input into a shared-weight convolutional neural network (CNN), resulting in corresponding high-dimensional feature maps for the original frame, enhanced frame, and reference frame. The specific calculation process is as follows: The current original frame is input into the shared-weight CNN, and after layer-by-layer operations involving the input layer, four convolutional layers, and a global average pooling layer, the high-dimensional feature map of the original frame is output. The current enhanced frame is input into the shared-weight CNN, and after the same layer operations, the high-dimensional feature map of the enhanced frame is output. The corrected alignment reference frame is input into the shared-weight CNN, and after the same layer operations, the high-dimensional feature map of the reference frame is output. The high-dimensional feature maps of the original frame and enhanced frame are used for subsequent calculation of gradient consistency mask weighting coefficients, while the high-dimensional feature maps of the enhanced frame and reference frame are used for feature domain block matching motion estimation.

[0096] The specific operation of block matching motion estimation in the feature domain, using gradient consistency mask weighting coefficients as weights of the matching cost function, is as follows:

[0097] First, for each pixel, the gradient structure tensor of the current original frame is calculated. This tensor is formed by summing the squares of the horizontal and vertical gradient components within a 5x5 neighborhood window centered on each pixel, and summing the products of the horizontal and vertical gradient components. The size of the 5x5 window is chosen empirically: a window that is too small (e.g., 3x3) is more susceptible to noise, leading to unstable tensor estimation; a window that is too large (e.g., 9x9) smooths out fine edges and reduces anisotropy. The discriminant of the numbers is determined; next, the two eigenvalues ​​of the symmetric matrix are solved and arranged in descending order; then, the anisotropy index is calculated. The calculation process is as follows: subtract the smaller value from the larger value of the two eigenvalues, then divide by the sum of the two eigenvalues, and add a constant coefficient to the denominator to prevent division by zero; the anisotropy index ranges from zero to one; the constant coefficient is 10 to the power of negative six, and its value is much smaller than that of typical eigenvalues, so it does not affect the calculation results and is only used to avoid division by zero errors. The closer the value is to one, the more consistent the local gradient direction is.

[0098] Secondly, the gradient consistency mask weighting coefficient is calculated based on the gradient consistency mask and the anisotropy index. The calculation process is as follows: obtain the value of the pixel corresponding to the gradient consistency mask, multiply the value by the difference between 1 and the anisotropy index, and then multiply the resulting product by the penalty strength coefficient. The result is the gradient consistency mask weighting coefficient. Alternatively, it is equivalent to calculating the gradient consistency mask weighting coefficient by subtracting the penalty strength coefficient, the value of the pixel corresponding to the gradient consistency mask, and the product of 1 and the anisotropy index. The penalty strength coefficient ranges from 0.3 to 0.7, with a default value of 0.5. A penalty strength coefficient of 0.5 is used for cross-validation. Determination: On a dataset containing ten test sequences, the search is performed in the range of 0.1 to 0.9 with a step size of 0.1, and 0.5 is finally selected as the value that minimizes the average endpoint error of motion estimation; When the value of the pixel corresponding to the gradient consistency mask is 1 and the anisotropy exponent is small, the weighting coefficient of the gradient consistency mask is significantly reduced to suppress false gradient regions introduced by the enhancement operation; When the value of the pixel corresponding to the gradient consistency mask is 1 and the anisotropy exponent is large, the reduction in the weighting coefficient of the gradient consistency mask is smaller to preserve the effective information of the real edges; When the value of the pixel corresponding to the gradient consistency mask is zero, the weighting coefficient of the gradient consistency mask reaches its maximum value;

[0099] Finally, using an 8x8 pixel block in the enhanced frame's high-dimensional feature map as a template, within a search range of ±32 pixels in the reference frame's high-dimensional feature map, a three-step search method is employed to find the motion vector that minimizes the cost, using the Euclidean distance weighted by the gradient consistency mask weighting coefficient as the matching cost function. The step size of each step is adaptively adjusted based on the average gradient consistency mask weighting coefficient of the current block. If the average gradient consistency mask weighting coefficient is less than 0.5, the step size is reduced to 0.75 times the original value. The optimal motion vectors of all blocks are combined to obtain the feature domain motion vector field. The specific process of the three-step search method is as follows: The first step starts from the center of the search area, with a step size of 16 pixels, and calculates the cost among nine candidate points (the center and eight surrounding directions) to find the point with the minimum cost. The second step uses this point as the center, reducing the step size to eight pixels, and searches for nine candidate points again. The third step reduces the step size to four pixels to finally determine the optimal motion vector. When the average gradient consistency mask weighting coefficient is less than 0.5, the step size is multiplied by 0.75, i.e., the first step becomes 12 pixels, the second step becomes 6 pixels, and the third step becomes 3 pixels, to perform a more refined search in uncertain regions. The calculation process of the matching cost function is as follows:

[0100] Traverse all pixel positions within the block, multiply the square of the difference between the gradient consistency mask weighting coefficient and the value of the enhanced frame high-dimensional feature map at the current pixel position and the value of the reference frame high-dimensional feature map at the current pixel position after adding the candidate motion vector, and accumulate them to obtain the total cost of the current candidate motion vector. The candidate motion vector contains horizontal displacement components and vertical displacement components. Find the vector that minimizes the total cost among all candidate motion vectors as the optimal motion vector. The optimal motion vector is the motion vector of the current block.

[0101] In this embodiment, it is specifically necessary to explain the operation of back-projecting the feature domain motion vector field to the pixel resolution to obtain the pixel domain motion vector in the motion vector back-projection and feedback module as follows:

[0102] For each pixel, based on its relative coordinates within its 8x8 block, the distance from the pixel to the block center and the distance from the pixel to the block boundary are calculated. Interpolation weights are then calculated using a gradient consistency mask and anisotropy exponent. The anisotropy exponent is calculated from the gradient structure tensor eigenvalues ​​of the current original frame, ranging from zero to one; a larger value indicates a more consistent local gradient direction. The interpolation weights are calculated by first constructing a Gaussian decay function based on the distance from the pixel to the block center. Specifically, this function has a base of the natural constant and an exponent of the square of the distance from the pixel to the block center divided by the negative of the square of twice the standard deviation of the Gaussian kernel. The standard deviation of the Gaussian kernel is set to 2.0. The choice of a standard deviation of 2.0 is based on the following considerations: a standard deviation that is too small (e.g., 1.0) will cause the interpolation weights to concentrate near the block center, leaving pixels far from the center almost unaffected by the motion vectors of adjacent blocks, resulting in blocky discontinuities in the motion field; a standard deviation that is too large (e.g., 4.0) will make the weights too smooth, failing to preserve edge sharpness; and a standard deviation of 2.0 maintains the motion... A balance is achieved between field continuity and edge preservation. Experiments have verified that the average endpoint error is reduced by 3.5% across ten test sequences. Building upon this, distance modulation and gradient structure modulation terms are introduced. The distance modulation term is an exponential function with the natural constant as the base and the product of a negative distance modulation attenuation constant and the distance from the pixel to the block boundary as the exponent. The distance modulation attenuation constant is set to 1.0, which allows the distance modulation term to decay rapidly as the pixel approaches the block boundary, thereby reducing the dependence of pixels near the boundary on the motion vectors of adjacent blocks and avoiding cross-boundary aliasing. The gradient structure modulation term is obtained by multiplying the product of the anisotropy exponent and the gradient consistency mask by a modulation coefficient, which is set to 0.5. The modulation coefficient of 0.5 is determined through grid search, and experiments are conducted within the range of 0.3 to 0.7 with a step size of 0.1. 0.5 ensures that the dynamic range of the gradient structure modulation term covers 0.5 to 1.5, effectively enhancing the weight of the real edge region without over-amplifying it.

[0103] The initial weight of a pixel is obtained by multiplying the value of the Gaussian decay function, the distance modulation term, and the gradient structure modulation term. Finally, the initial weight is normalized by dividing it by the sum of the initial weights of all pixels in the eight-by-eight block to ensure that the sum of the weights is 1, thus obtaining the final interpolation weight. The normalization operation ensures that the interpolation weight of each pixel satisfies the probability distribution property within the local block, guaranteeing the unbiasedness of the motion vector after weighted averaging.

[0104] Then, the motion vectors of four adjacent blocks in the feature domain motion vector field are weighted and averaged using interpolation weights to obtain the pixel domain motion vector of each pixel. The specific calculation process is as follows:

[0105] For each 8x8 block in the feature domain motion vector field, the horizontal coordinate of the top-left pixel in the feature domain is taken as the block's horizontal coordinate, and the vertical coordinate is taken as the block's vertical coordinate. The horizontal coordinate of the current pixel to be calculated in the pixel domain is taken as the current pixel's horizontal coordinate, and the vertical coordinate is taken as the current pixel's vertical coordinate. The difference between the current pixel's horizontal coordinate and the block's horizontal coordinate is calculated, and the difference is divided by eight to obtain the decimal offset in the horizontal direction. The difference between the current pixel's vertical coordinate and the block's vertical coordinate is also calculated, and the difference is divided by eight to obtain the decimal offset in the vertical direction. Using the decimal offsets in the horizontal and vertical directions, bilinear interpolation is used to calculate the interpolation weights corresponding to the centers of four adjacent blocks. The four adjacent blocks are located at the top-left, top-right, bottom-left, and bottom-right corners. The horizontal coordinate of the top-left block center is the block's horizontal coordinate multiplied by eight, and the vertical coordinate is the block's vertical coordinate multiplied by eight. The horizontal coordinate of the top-right block center is... The horizontal coordinate of the block is multiplied by 8 and the vertical coordinate is multiplied by 8. The horizontal coordinate of the bottom left block center is multiplied by 8 and the vertical coordinate is multiplied by 8. The horizontal coordinate of the bottom right block center is multiplied by 8 and the vertical coordinate is multiplied by 8. The interpolation weights corresponding to the four adjacent block centers are inversely proportional to the distance from the four adjacent block centers to the current pixel. That is, the closer the block center is to the current pixel, the greater its corresponding interpolation weight. The sum of the interpolation weights corresponding to the four adjacent block centers is one. The motion vectors of the four adjacent blocks are weighted and summed using the interpolation weights corresponding to the four adjacent block centers. The result of the weighted sum is used as the pixel domain motion vector of the current pixel. The weighted averaging process reduces the weight in the enhanced distortion region marked by the gradient consistency mask and increases the weight in the real edge region, thereby maintaining the edge sharpness of the motion field.

[0106] The specific operation of using pixel-domain motion vectors to perform motion compensation on the corrected alignment reference frame is as follows:

[0107] For each pixel in the prediction frame to be generated, obtain the integer coordinate position of the pixel in the corrected alignment reference frame, and the sub-pixel offset composed of the horizontal and vertical components of the pixel domain motion vector at the pixel; extract the pixel values ​​at the integer coordinate position and its four adjacent integer coordinate positions to the right and bottom, and multiply these four pixel values ​​by their corresponding bilinear interpolation weights. The bilinear interpolation weights are determined by the horizontal offset ratio and the vertical offset ratio. Finally, add these four product results to obtain the final pixel value of the pixel, thereby generating the prediction frame. The pixel values ​​at non-integer coordinates are all obtained through bilinear interpolation.

[0108] The specific process of calculating the prediction residual and calculating the gradient consistency mask average response of the current coding unit is as follows:

[0109] The difference between the pixel value of the current enhanced frame at each pixel position within the current coding unit and the pixel value of the predicted frame at the corresponding pixel position is calculated to obtain the prediction residual, which is used for subsequent transform quantization coding. Simultaneously, the gradient consistency mask value at each pixel position within the current coding unit is obtained. The gradient consistency mask values ​​at all pixel positions within the current coding unit are summed, and the sum is divided by the total number of pixels in the current coding unit to obtain the average gradient consistency mask response. The average gradient consistency mask response reflects the proportion of pixels in the current coding unit where the enhancement operation significantly alters the gradient structure. The average gradient consistency mask response is a dimensionless value between zero and one; a higher value indicates more prevalent enhancement distortion within the current coding unit.

[0110] Then, the average response of the gradient consistency mask is compared with the gain adjustment threshold, which is set to 0.3. The gain adjustment threshold of 0.3 was determined through rate-distortion optimization experiments: on ten standard test sequences, scanning the range from 0.15 to 0.5 with a step size of 0.05, 0.3 minimizes the average BD-rate loss, that is, it suppresses enhancement distortion to the maximum extent while ensuring compression efficiency. If the average response of the gradient consistency mask is greater than the gain adjustment threshold, the enhancement gain of the next frame is reduced. The enhancement gain of the current frame is the value used in multi-scale Laplacian pyramid enhancement for the current frame. The gain coefficient is reduced by multiplying the current frame's enhancement gain by an exponential function with a base of the natural constant and an exponential value equal to the product of the negative decay rate constant and the difference between the average response of the gradient consistency mask and the gain adjustment threshold. The decay rate constant is set to 2.0. The decay rate constant of 2.0 ensures that when the average response of the gradient consistency mask exceeds the gain adjustment threshold of 0.1 (i.e., 0.4), the enhancement gain of the next frame is reduced to about 80% of the current frame; when it exceeds 0.2 (i.e., 0.5), the gain is reduced to about 60% of the current frame, thus achieving a smooth and fast response.

[0111] If the average response of the gradient consistency mask is greater than the switching threshold, which is set to 0.6, the enhancement algorithm for the next frame will be switched to a conservative enhancement algorithm. The conservative enhancement algorithm only performs basic brightness adjustments such as histogram equalization or gamma correction, and prohibits nonlinear sharpening operations that introduce new edges or significantly change the gradient magnitude, in order to avoid the accumulation of enhancement distortion leading to inter-frame prediction drift. A switching threshold of 0.6 means that when more than 60% of the pixels in the coding unit are marked as enhancement distortion, the current enhancement algorithm has seriously interfered with inter-frame prediction, and sharpening operations must be completely stopped, retaining only the most basic brightness adjustments to ensure visual acceptability. Experiments show that this threshold setting makes the switching frequency moderate in fast-moving scenes, neither too frequent to cause loss of enhancement effect, nor too sluggish to cause prediction quality collapse.

[0112] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0113] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0118] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An adaptive image enhancement and compression transmission system, characterized in that, Specifically, it includes: The gradient consistency mask generation module, the optical flow-guided reference frame pre-alignment module, the joint feature space motion estimation module, and the motion vector back-projection and feedback module are connected in sequence, wherein; Gradient consistency mask generation module: Receives the current original frame, the reference original frame, the current enhanced frame and the reference enhanced frame after multi-scale Laplacian pyramid enhancement, calculates the gradient magnitude map and gradient direction map of the current original frame and the current enhanced frame, compares the magnitude difference with the first threshold and the direction difference with the second threshold, generates a gradient consistency mask, and pixels with a value of 1 in the gradient consistency mask mark the regions where the enhancement operation significantly changes the gradient structure. Optical flow-guided reference frame pre-alignment module: It calls a lightweight optical flow network to calculate the optical flow field from the reference original frame to the current original frame with the current original frame and the reference original frame as input. The optical flow field is applied to the reference enhancement frame and bilinear sampling is performed to output the pre-aligned reference enhancement frame. Then, the gradient consistency mask is used to replace the region marked by the gradient consistency mask in the optical flow field with the block-based phase correlation matching result to obtain the corrected optical flow field and the corresponding corrected aligned reference frame. Joint feature space motion estimation module: The current original frame and the current enhanced frame are respectively input into a convolutional neural network with shared weights to extract high-dimensional feature maps. In the feature domain, the gradient consistency mask weighting coefficient is used as the weight of the matching cost function to perform block matching motion estimation. The gradient consistency mask weighting coefficient is negatively correlated with the number of pixels with a value of 1 in the gradient consistency mask. The feature domain motion vector field is output. Motion Vector Backprojection and Feedback Module: Backprojects the feature domain motion vector field to the pixel resolution to obtain pixel domain motion vectors. Uses the pixel domain motion vectors to perform motion compensation on the corrected alignment reference frame to generate a prediction frame. Calculates the prediction residual and statistically analyzes the gradient consistency mask average response of the current coding unit. Compares the gradient consistency mask average response with the gain adjustment threshold. If the gradient consistency mask average response exceeds the gain adjustment threshold, the enhancement gain of the next frame is reduced or the conservative enhancement algorithm is switched.

2. The adaptive image enhancement and compression transmission system according to claim 1, characterized in that: In the gradient consistency mask generation module, the specific operations for calculating the gradient magnitude map and gradient direction map of the current original frame and the current enhanced frame are as follows: The Sobel operator is used to calculate the horizontal and vertical gradients for the current original frame and the current enhanced frame respectively. Then, the gradient magnitude of each pixel is calculated as the arithmetic square root of the sum of the squares of the horizontal and vertical gradients to obtain the gradient magnitude map. The gradient direction of each pixel is calculated as the arctangent of the ratio of the vertical gradient to the horizontal gradient to obtain the gradient direction map. Then, a joint gradient structure change significance measure is constructed, which consists of a weighted sum of two terms. The first term is the absolute logarithm of the ratio of the gradient magnitude of the current enhanced frame to the gradient magnitude of the current original frame, and the second term is the cosine of the difference between the gradient direction of the current enhanced frame and the gradient direction of the current original frame. The weighting coefficients of the first and second terms are 0.6 and 0.4, respectively.

3. The adaptive image enhancement and compression transmission system according to claim 2, characterized in that: The specific operation of comparing the magnitude difference with the first threshold and the direction difference with the second threshold to generate a gradient consistency mask is as follows: The magnitude difference is defined as the difference between the gradient magnitude of the current enhanced frame and the gradient magnitude of the current original frame, and the direction difference is defined as the difference between the gradient direction of the current enhanced frame and the gradient direction of the current original frame. For each pixel, a 7x7 neighborhood window is selected centered on it, and the local mean and local median of the joint gradient structure change significance measure within the neighborhood window are calculated; the first threshold is equal to the scaling factor multiplied by the local mean, and the scaling factor is 0.8; the second threshold is equal to the median weight multiplied by the local median, and the median weight is 1.5; the adaptive threshold is equal to the sum of the first threshold and the second threshold. The significance measure of the joint gradient structure change for each pixel is compared with the adaptive threshold corresponding to the pixel. If the significance measure of the joint gradient structure change is greater than the adaptive threshold, the value of the pixel in the gradient consistency mask is set to one, indicating that the gradient structure of the pixel has been significantly changed by the enhancement operation; otherwise, the value of the pixel in the gradient consistency mask is set to zero.

4. The adaptive image enhancement and compression transmission system according to claim 3, characterized in that: In the optical flow-guided reference frame pre-alignment module, the specific operation of calling the lightweight optical flow network to calculate the optical flow field from the reference original frame to the current original frame using the current original frame and the reference original frame as input is as follows: The lightweight optical flow network adopts the RAFT-lite structure, takes the grayscale images of the current original frame and the reference original frame as input, and outputs the horizontal and vertical displacement components of each pixel through an iterative update method to form an optical flow field. The optical flow field is applied to the reference enhancement frame, and the corresponding horizontal and vertical displacement components are added to the coordinates of each pixel in the reference enhancement frame. The pixel values ​​at non-integer coordinates are calculated by bilinear interpolation to obtain the pre-aligned reference enhancement frame. Simultaneously, based on the pixels with a value of one in the gradient consistency mask and the value of the joint gradient structure change significance metric, the optical flow correction necessity index for each pixel is calculated. The specific calculation process is as follows: Obtain the pixel value corresponding to the gradient consistency mask, multiply it by an exponential function value that is the product of a negative decay constant and a measure of the significance of the joint gradient structure change, and the resulting product is the optical flow correction necessity index.

5. The adaptive image enhancement and compression transmission system according to claim 4, characterized in that: The specific operation of replacing the region marked by the gradient consistency mask in the optical flow field with the block-based phase correlation matching result using the gradient consistency mask is as follows: For each pixel, if the optical flow correction necessity index is greater than the preset activation threshold, a local window of size 16 by 16 pixels is selected centered on the pixel. Phase correlation matching based on Fourier transform is performed between the current enhancement frame and the pre-aligned reference enhancement frame to calculate the sub-pixel precision local optical flow correction amount. The local optical flow correction amount is obtained through the following process: Discrete Fourier transforms are performed on the image blocks of the current enhanced frame within the local window and the image blocks of the pre-aligned reference enhanced frame at the corresponding positions. The cross power spectrum of the two is calculated, and the phase difference of the relative displacement is obtained by phase expansion of the cross power spectrum. The displacement peak value with sub-pixel precision is obtained by performing inverse Fourier transform on the phase difference. The displacement peak value is the local optical flow correction amount. Then, the optical flow field and the local optical flow correction are weighted and fused to obtain the corrected optical flow field; Finally, bilinear sampling is performed again on the reference enhancement frame using the corrected optical flow field to obtain the corrected aligned reference frame. The coordinates of each pixel in the corrected aligned reference frame are obtained by adding the horizontal and vertical displacement components of the corrected optical flow field to the corresponding coordinates in the reference enhancement frame and then interpolating them bilinearly.

6. The adaptive image enhancement and compression transmission system according to claim 5, characterized in that: The weighted fusion process is specifically as follows: Multiply the difference obtained by subtracting the optical flow correction necessity index from the optical flow field, then multiply the optical flow correction necessity index by the sum of the optical flow field and the local optical flow correction amount, and finally add the two product results. The sum is the horizontal and vertical displacement components of the corresponding pixel in the corrected optical flow field, thus obtaining the corrected optical flow field.

7. The adaptive image enhancement and compression transmission system according to claim 6, characterized in that: In the joint feature space motion estimation module, the specific operation of inputting the current original frame and the current enhanced frame into a convolutional neural network with shared weights to extract high-dimensional feature maps is as follows: The weighted convolutional neural network consists of an input layer, four convolutional layers, and a global average pooling layer. The kernel sizes of the four convolutional layers are 7x7, 5x5, 3x3, and 3x3, respectively, and the number of output channels are 32, 64, 128, and 128, respectively. The global average pooling layer outputs a 128-dimensional high-dimensional feature map. In the weighted convolutional neural network, the input layer receives data from the current original frame and the current enhanced frame. The first convolutional layer uses a 7x7 kernel and outputs 32 feature channels. The second convolutional layer uses a 5x5 kernel and outputs 64 feature channels. The third and fourth convolutional layers both use 3x3 kernels and output 128 feature channels, respectively. All convolutional layers use a stride of 1, and each convolutional layer is followed by a non-linear activation function and a local response normalization layer to extract translation-invariant local texture features. The global average pooling layer performs spatial dimension compression on the output feature map of the last convolutional layer, generating a compact high-dimensional feature map. A weighted convolutional neural network (CNN) is pre-trained on an image dataset containing various enhancement types, including histogram equalization, Laplacian sharpening, and contrast-limited adaptive histogram equalization. The training objective is to minimize the cosine distance between the current original frame and the current enhanced frame in the feature space. The current original frame, the current enhanced frame, and the corrected alignment reference frame are respectively input into the weighted CNN to obtain the corresponding high-dimensional feature maps of the original frame, enhanced frame, and reference frame. The specific calculation process is as follows: the current original frame is input into the weighted CNN, and after layer-by-layer operations through the input layer, four convolutional layers, and a global average pooling layer, the high-dimensional feature map of the original frame is output. The current enhanced frame is input into the weighted CNN, and after the same layer operations, the high-dimensional feature map of the enhanced frame is output. The corrected alignment reference frame is input into the weighted CNN, and after the same layer operations, the high-dimensional feature map of the reference frame is output.

8. The adaptive image enhancement and compression transmission system according to claim 7, characterized in that: The specific operation of performing block matching motion estimation using gradient consistency mask weighting coefficients as weights in the matching cost function in the feature domain is as follows: First, for each pixel, the gradient structure tensor of the current original frame is calculated. The gradient structure tensor is formed by summing the squares of the horizontal and vertical gradient components within a 5x5 neighborhood window centered on each pixel, and summing the products of the horizontal and vertical gradient components. This forms a 3x3 symmetric matrix. Next, the two eigenvalues ​​of the symmetric matrix are solved and arranged in descending order. Then, the anisotropy index is calculated by subtracting the smaller eigenvalue from the larger eigenvalue, dividing by the sum of the two eigenvalues, and adding a constant coefficient to the denominator. Secondly, the gradient consistency mask weighting coefficient is calculated based on the gradient consistency mask and the anisotropy index. The calculation process is as follows: obtain the value of the pixel corresponding to the gradient consistency mask, multiply the value by the difference between one and the anisotropy index, and then multiply the resulting product by the penalty intensity coefficient. The result is the gradient consistency mask weighting coefficient. Finally, using an 8x8 pixel block in the enhanced frame's high-dimensional feature map as a template, within a search range of ±32 pixels in the reference frame's high-dimensional feature map, the Euclidean distance weighted by gradient consistency mask weighting coefficients is used as the matching cost function. A three-step search method is employed to find the motion vector that minimizes the cost. The step size of each step is adaptively adjusted based on the average gradient consistency mask weighting coefficient of the current block. If the average gradient consistency mask weighting coefficient is less than 0.5, the step size is reduced to 0.75 times the original value. The optimal motion vectors of all blocks are combined to obtain the feature domain motion vector field. The calculation process of the matching cost function is as follows: Traverse all pixel positions within the block, multiply the square of the difference between the gradient consistency mask weighting coefficient and the value of the enhanced frame high-dimensional feature map at the current pixel position and the value of the reference frame high-dimensional feature map at the current pixel position after adding the candidate motion vector, and accumulate them to obtain the total cost of the current candidate motion vector. The candidate motion vector contains horizontal displacement components and vertical displacement components. Find the vector that minimizes the total cost among all candidate motion vectors as the optimal motion vector. The optimal motion vector is the motion vector of the current block.

9. The adaptive image enhancement and compression transmission system according to claim 8, characterized in that: In the motion vector back-projection and feedback module, the specific operation of back-projecting the feature domain motion vector field to the pixel resolution to obtain the pixel domain motion vector is as follows: For each pixel, based on its relative coordinates within its 8x8 block, the distance from the pixel to the block center and the distance from the pixel to the block boundary are calculated. Interpolation weights are then calculated using the gradient consistency mask and the anisotropy exponent. The calculation of the interpolation weights begins by constructing a Gaussian decay function based on the distance from the pixel to the block center. Specifically, this function is an exponential function with a base of the natural constant and an exponent equal to the negative of the square of the distance from the pixel to the block center divided by the square of twice the standard deviation of the Gaussian kernel. The standard deviation of the Gaussian kernel is set to 2.

0. Based on this, a distance modulation term and a gradient structure modulation term are introduced. The distance modulation term is an exponential function with a base of the natural constant and an exponent equal to the product of the negative distance modulation decay constant and the distance from the pixel to the block boundary. The gradient structure modulation term is obtained by adding one product of the anisotropy exponent and subtracting one product of the gradient consistency mask, and then multiplying by the modulation coefficient. The initial weight of a pixel is obtained by multiplying the value of the Gaussian decay function, the distance modulation term, and the gradient structure modulation term. Finally, the initial weight is normalized by dividing the initial weight by the sum of the initial weights of all pixels in the eight-by-eight block to which the pixel belongs, resulting in the final interpolation weight. Then, the motion vectors of four adjacent blocks in the feature domain motion vector field are weighted and averaged using interpolation weights to obtain the pixel domain motion vector of each pixel. The specific calculation process is as follows: For each 8x8 block in the feature domain motion vector field, the horizontal coordinate of the top-left pixel in the feature domain is taken as the block's horizontal coordinate, and the vertical coordinate is taken as the block's vertical coordinate. The horizontal coordinate of the current pixel to be calculated in the pixel domain is taken as the current pixel's horizontal coordinate, and the vertical coordinate is taken as the current pixel's vertical coordinate. The difference between the current pixel's horizontal coordinate and the block's horizontal coordinate is calculated, and the difference is divided by eight to obtain the decimal offset in the horizontal direction. Similarly, the difference between the current pixel's vertical coordinate and the block's vertical coordinate is calculated, and the difference is divided by eight to obtain the decimal offset in the vertical direction. Using these decimal offsets, bilinear interpolation is used to calculate the interpolation weights corresponding to the centers of four adjacent blocks, located at the top-left, top-right, bottom-left, and right corners, respectively. The horizontal coordinate of the center of the bottom left block is the horizontal coordinate of the block multiplied by eight, and the vertical coordinate is the vertical coordinate of the block multiplied by eight. The horizontal coordinate of the center of the top right block is the horizontal coordinate of the block plus one and multiplied by eight, and the vertical coordinate is the vertical coordinate of the block multiplied by eight. The horizontal coordinate of the center of the bottom left block is the horizontal coordinate of the block multiplied by eight, and the vertical coordinate is the vertical coordinate of the block plus one and multiplied by eight. The horizontal coordinate of the center of the bottom right block is the horizontal coordinate of the block plus one and multiplied by eight, and the vertical coordinate is the vertical coordinate of the block plus one and multiplied by eight. The interpolation weights corresponding to the centers of the four adjacent blocks are inversely proportional to the distances from the centers of the four adjacent blocks to the current pixel. The motion vectors of the four adjacent blocks are weighted and summed using the interpolation weights corresponding to the centers of the four adjacent blocks. The result of the weighted sum is used as the pixel domain motion vector of the current pixel. The specific operation of using pixel-domain motion vectors to perform motion compensation on the corrected alignment reference frame is as follows: For each pixel in the prediction frame to be generated, obtain the integer coordinate position of the pixel in the corrected alignment reference frame, and the sub-pixel offset composed of the horizontal and vertical components of the pixel domain motion vector at the pixel. Extract the pixel values ​​at the integer coordinate position and its four adjacent integer coordinate positions to the right and bottom. Multiply these four pixel values ​​by their corresponding bilinear interpolation weights, which are determined by the horizontal and vertical offset ratios. Finally, add these four product results to obtain the final pixel value of the pixel, thereby generating the prediction frame.

10. The adaptive image enhancement and compression transmission system according to claim 9, characterized in that: The process of calculating the predicted residual and calculating the average response of the gradient consistency mask of the current coding unit is as follows: The difference between the pixel value of the current enhanced frame at each pixel position in the current coding unit and the pixel value of the prediction frame at the corresponding pixel position is calculated to obtain the prediction residual. At the same time, the gradient consistency mask value at each pixel position in the current coding unit is obtained. The gradient consistency mask values ​​of all pixel positions in the current coding unit are accumulated and summed. Then, the summation result is divided by the total number of pixels in the current coding unit to obtain the average gradient consistency mask response. Then, the average response of the gradient consistency mask is compared with the gain adjustment threshold, which is set to 0.

3. If the average response of the gradient consistency mask is greater than the gain adjustment threshold, the enhancement gain of the next frame is reduced. The enhancement gain of the current frame is the gain coefficient used in the multi-scale Laplacian pyramid enhancement of the current frame. The reduction method is to multiply the enhancement gain of the current frame by an exponential function value with the natural constant as the base and the product of the negative decay rate constant and the difference between the average response of the gradient consistency mask and the gain adjustment threshold. The decay rate constant is set to 2.

0. If the average response of the gradient consistency mask is greater than the switching threshold, which is set to 0.6, the enhancement algorithm for the next frame will be switched to the conservative enhancement algorithm. The conservative enhancement algorithm only performs basic brightness adjustments such as histogram equalization or gamma correction, and prohibits nonlinear sharpening operations that introduce new edges or significantly change the gradient magnitude.

Images