Multi-frame fusion fixed image noise dynamic compensation system and method

Abstract

The application discloses a multi-frame fusion fixed image noise dynamic compensation system and method, and particularly relates to the technical field of image processing, and is used for solving the problem of the existing independent noise compensation leading to the noise band artifact of the fusion boundary; a mapping relationship between the pixel-level imaging parameter difference degree and the fusion weight is established by acquiring a multi-frame image sequence with different imaging parameters and identifying a transition area with a dramatic change in luminance gradient; a noise visibility correction factor is generated by extracting the frequency energy skewness feature and the chroma phase change boundary feature of the transition area through nonlinear fusion; the multi-frame noise compensation is cooperated by dynamically constraining the compensation intensity range; finally, the cooperative compensation is executed within the constraint range and fused and output; the discontinuity of the fusion boundary noise is effectively eliminated, and the visual quality of the high dynamic range imaging is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly, to a fixed image noise dynamic compensation system and method for multi-frame fusion. BACKGROUND

[0002] In imaging applications of low-illumination or high-dynamic-range scenes, such as security monitoring, vehicle-mounted cameras, etc., using multi-frame image fusion technology to improve imaging quality is a common means. Among them, the high-dynamic-range imaging method acquires multiple images with different exposure parameters (such as different exposure times or sensitivities) in the same scene, extracts the appropriate exposure region information in each frame, and then performs fusion processing to obtain a composite image that retains both bright and dark details. For the problem of fixed image noise (such as fixed pattern noise) inherent to imaging sensors, the prior art usually uses a dynamic compensation method, that is, before fusion processing, for each frame of image, according to the specific parameters (such as gain, exposure time, etc.) during imaging, the fixed noise compensation amount of the frame is calculated and removed or suppressed independently to reduce the visibility of fixed noise in a single frame and provide a "cleaner" source image for subsequent fusion.

[0003] However, in the dynamic compensation stage before fusion processing, the prior art performs independent fixed noise compensation operations on each frame of image with different imaging parameters, which can effectively reduce the influence of fixed noise in a single frame, but introduces new visual defects in the subsequent multi-frame fusion link. Specifically, due to the differences in fixed noise characteristics under different imaging parameters, the residual noise (mainly random noise) of each frame of image after independent compensation will also be different, and when performing image fusion, especially at the junction of bright and dark regions provided by different source frames, the differences in these residual noise will form obvious visual discontinuity on both sides of the fusion boundary. This discontinuity appears as a "noise band" artifact distributed along the brightness boundary in the final fusion image, which actually destroys the visual quality of the image. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the present application provides a fixed image noise dynamic compensation system and method for multi-frame fusion to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] The fixed image noise dynamic compensation method for multi-frame fusion comprises:

[0007] S1, acquiring a sequence of multiple images with different imaging parameters and their corresponding imaging parameters in the same scene;

[0008] S2, identify a transition region in which a luminance value gradient change exceeds a preset threshold in the multi-frame image sequence;

[0009] S3, establish a mapping relationship between an imaging parameter difference degree of each pixel point in the transition region and a fusion weight coefficient;

[0010] S4, extract a frequency energy distribution skewness feature of the transition region and a color space phase change boundary feature, and generate a noise visibility correction factor through a nonlinear correlation function fusion;

[0011] S5, dynamically generate a fixed noise compensation intensity constraint range of each pixel point in the transition region according to the mapping relationship and the noise visibility correction factor;

[0012] S6, perform fixed noise collaborative compensation on the multi-frame images in the fixed noise compensation intensity constraint range, and fuse and output the compensated images based on the fusion weight coefficient.

[0013] Further, the multi-frame image sequence and the corresponding imaging parameters with different imaging parameters under the same scene are obtained, including:

[0014] Control the image sensor to output a plurality of original images with alternating exposure time and sensitivity gain values;

[0015] Synchronize the acquisition of the exposure time parameter and the sensitivity gain parameter corresponding to each original image;

[0016] Perform inter-frame motion compensation on the multi-frame original images to generate a multi-frame image sequence with spatial position alignment;

[0017] The exposure time parameter and the sensitivity gain parameter are used as the imaging parameter;

[0018] Output the multi-frame image sequence with spatial position alignment and the corresponding imaging parameters.

[0019] Further, the transition region in which the luminance value gradient change exceeds the preset threshold in the multi-frame image sequence is identified, including:

[0020] Calculate the luminance component gradient amplitude of each frame of the multi-frame image sequence with spatial position alignment;

[0021] Detect a set of pixel positions in each frame of image in which the luminance component gradient amplitude change exceeds the preset threshold;

[0022] Fuse the pixel position sets of all frames to take the union to generate an initial transition region candidate set;

[0023] Perform a morphological closing operation on the initial transition region candidate set to connect adjacent pixel points;

[0024] The connected region after the morphological closing operation is output as the transition region.

[0025] Further, the mapping relationship between the imaging parameter difference degree of each pixel point in the transition area and the fusion weight coefficient is established, including:

[0026] The exposure time parameter and the sensitivity gain parameter of each pixel point in each frame image are extracted from the transition area;

[0027] The exposure time parameter variation amplitude and the sensitivity gain parameter variation amplitude of each pixel point are calculated;

[0028] The exposure time parameter variation amplitude and the sensitivity gain parameter variation amplitude are fused by a normalized Euclidean distance formula to generate the imaging parameter difference degree value of each pixel point;

[0029] Based on the imaging parameter difference degree value, a preset inverse proportional function is used to calculate the fusion weight coefficient;

[0030] A mapping relationship storage table is established between the imaging parameter difference degree value of each pixel point and the fusion weight coefficient.

[0031] Further, the skewness feature of the frequency energy distribution and the phase transition boundary feature of the chroma space of the transition area are extracted, and a noise visibility correction factor is generated by fusing through a nonlinear correlation function, including:

[0032] The transition area is divided into a plurality of image blocks;

[0033] The block Fourier transform is performed on each image block to obtain the frequency energy distribution;

[0034] The third central moment value is calculated based on the frequency energy distribution as the skewness feature value of the frequency energy distribution;

[0035] The image block is converted to the CIELab color space, and the chroma component is extracted;

[0036] The phase gradient operator value of the chroma component is calculated as the phase transition boundary feature value of the chroma space;

[0037] The skewness feature value of the frequency energy distribution is processed by a hyperbolic tangent function;

[0038] The phase transition boundary feature value of the chroma space is processed by an error function;

[0039] The noise visibility correction factor is generated by multiplying the output value of the hyperbolic tangent function and the output value of the error function.

[0040] Further, according to the mapping relationship and the noise visibility correction factor, the fixed noise compensation intensity constraint range of each pixel point in the transition area is dynamically generated, including:

[0041] Read the fusion weight coefficient of each pixel point from the mapping relationship storage table;

[0042] Obtain the noise visibility correction factor value of the corresponding pixel point;

[0043] Based on the product relationship between the reference compensation intensity and the noise visibility correction factor value, calculate the dynamic scaling compensation amount;

[0044] According to the dynamic scaling compensation amount, generate the upper limit value and the lower limit value of the fixed noise compensation intensity;

[0045] The interval range between the upper limit value and the lower limit value of the fixed noise compensation intensity is taken as the fixed noise compensation intensity constraint range.

[0046] Further, within the fixed noise compensation intensity constraint range, fixed noise collaborative compensation is performed on the multiple frames of images, and the compensated images are fused based on the fusion weight coefficient, including:

[0047] For each frame of image in the multiple frames of images, according to the fixed noise compensation intensity constraint range corresponding to each pixel point of the current frame, select the compensation intensity value within the fixed noise compensation intensity constraint range;

[0048] Combine the compensation intensity values of the same position pixel points of the adjacent two frames for consistency adjustment to generate a collaborative compensation intensity value;

[0049] Using the collaborative compensation intensity value, perform fixed noise compensation operation on all pixel points of the current frame to generate a compensated image frame;

[0050] For each pixel position in the multiple frames of compensated images, use the fusion weight coefficient of the corresponding position for weighted summation calculation;

[0051] Collect the weighted summation results of all pixel positions to generate a final fused image and output.

[0052] On the other hand, the present application provides a multi-frame fused fixed image noise dynamic compensation system, comprising:

[0053] An imaging parameter module for acquiring a sequence of multiple frames of images with different imaging parameters and their corresponding imaging parameters under the same scene;

[0054] A transition area module for identifying a transition area with a luminance value gradient change exceeding a preset threshold in the sequence of multiple frames of images;

[0055] A weight mapping module for establishing a mapping relationship between the imaging parameter difference degree of each pixel point in the transition area and the fusion weight coefficient;

[0056] The dual-domain perception module is used for extracting the skewness feature of the frequency energy distribution of the transition region and the phase change boundary feature of the chroma space, and generating a noise visibility correction factor through a nonlinear correlation function fusion;

[0057] The dynamic constraint module is used for dynamically generating a fixed noise compensation intensity constraint range of each pixel point in the transition region according to the mapping relationship and the noise visibility correction factor;

[0058] The cooperative output module is used for performing fixed noise cooperative compensation on the multiple frames of images in the fixed noise compensation intensity constraint range, and performing fusion output on the compensated images based on a fusion weight coefficient.

[0059] Compared with the prior art, the present application has the following beneficial effects:

[0060] 1. Accurate suppression of fusion boundary noise artifacts: By extracting the skewness feature of the frequency energy distribution of the transition region and the phase change boundary feature of the chroma space, the system quantifies the visual sensitivity characteristics of the human eye in the brightness change area, breaks through the limitations of traditional single brightness gradient analysis, combines the noise visibility correction factor generated by the nonlinear correlation function, dynamically adjusts the compensation intensity constraint range, solves the "noise band" problem at the fusion light-dark junction, ensures the smooth transition of the noise characteristics of long and short exposure frames at the fusion boundary, eliminates the visual discontinuity caused by independent compensation, and significantly improves the imaging quality of high dynamic scenes.

[0061] 2. Realize cross-exposure noise characteristic cooperative optimization: Establish the mapping relationship between the imaging parameter difference and the fusion weight, convert the internal correlation of noise compensation under different exposure parameters into a quantifiable control model, and based on the cooperative compensation mechanism of dynamic constraint, while ensuring the independent noise reduction effect of each frame, forcibly coordinate the compensation intensity consistency of adjacent frames in the transition region, not only maintain the detail restoration ability of single frame image, but also avoid new artifacts caused by noise distribution difference during multi-frame fusion from the source, achieve the optimal balance between single frame noise reduction and multi-frame fusion, and make the output image have high purity and visual coherence. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The flowchart of the fixed image noise dynamic compensation method of the multi-frame fusion of the present application;

[0063] Figure 2 The structural schematic diagram of the fixed image noise dynamic compensation system of the multi-frame fusion of the present application. DETAILED DESCRIPTION

[0064] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those ordinarily skilled in the art without creative work fall within the scope of the present application.

[0065] Embodiment 1 Figure 1 The multi-frame fusion fixed image noise dynamic compensation method of the present application is given, including:

[0066] S1, acquiring a multi-frame image sequence and corresponding imaging parameters under the same scene with different imaging parameters;

[0067] S2, identifying a transition area with a luminance value gradient change exceeding a preset threshold in the multi-frame image sequence;

[0068] S3, establishing a mapping relationship between the imaging parameter difference degree of each pixel point in the transition area and the fusion weight coefficient;

[0069] S4, extracting the frequency energy distribution skewness feature and the color space phase change boundary feature of the transition area, and generating a noise visibility correction factor through a nonlinear correlation function fusion;

[0070] S5, dynamically generating a fixed noise compensation intensity constraint range of each pixel point in the transition area according to the mapping relationship and the noise visibility correction factor;

[0071] S6, performing fixed noise collaborative compensation on the multi-frame image in the fixed noise compensation intensity constraint range, and outputting the compensated image based on the fusion weight coefficient.

[0072] S1, acquiring a multi-frame image sequence and corresponding imaging parameters under the same scene with different imaging parameters, and specifically implementing as:

[0073] The control image sensor outputs multiple frames of original images in accordance with preset timing rules, wherein an exposure time parameter and a sensitivity gain parameter implement an alternating change combination mode. The alternating change is realized by a built-in driving module: after the timing controller receives a vertical synchronization signal, it selects an exposure parameter combination in accordance with preset switching logic, for example, when the frame sequence number divided by 2 has a remainder of 0, a first parameter combination (including a value in the exposure time interval of 20 to 50 milliseconds and a value in the sensitivity gain ISO range of 200 to 500) is used, and when the remainder is 1, a second parameter combination (including a value in the exposure time interval of 5 to 20 milliseconds and a value in the sensitivity gain ISO range of 800 to 3200) is enabled. The specific parameter values are determined by a pre-stored configuration table, which is stored in the non-volatile memory of the sensor, and each table entry explicitly records the paired relationship of the millisecond-level exposure time parameter and the integer-type ISO sensitivity gain parameter.

[0074] When collecting the imaging parameters of each frame of original image, the hardware register group of the image processor captures the current exposure configuration in real time. The specific process includes: the exposure time parameter is directly read from a high-precision timer, the minimum timing unit of the timer is 0.01 milliseconds, and the collection result is temporarily stored in the format of 32-bit floating point number; the sensitivity gain parameter is obtained through an analog-to-digital conversion circuit, first the gain adjuster voltage value is measured, then it is converted into an ISO standard value according to the sensor calibration curve, and finally it is stored in the format of 16-bit unsigned integer. The parameter collection operation is strictly completed within a 2-millisecond time window after the exposure of each frame of image is completed, ensuring that the parameter values and image data have accurate timing correspondence. The final imaging parameters are defined as structured data objects containing two types of values: the first type of value is the floating-point exposure time parameter, and the second type of value is the integer sensitivity gain parameter.

[0075] The following standardized process is adopted to perform inter-frame motion compensation operation: taking the middle frame in the sequence as the spatial coordinate system reference, when the total number of frames is even, taking the frame numbered half of the total number of frames, and when the total number of frames is odd, taking the frame numbered half of the total number of frames plus 1. The scale-invariant feature transform algorithm is used to detect key matching points between the reference frame and the frame to be compensated. The algorithm uses eigenvalue analysis of the Hessian matrix to locate feature points, and ensures that no less than 150 feature points are detected per frame to ensure matching reliability. Based on the coordinates of the matching feature points, a homography transformation matrix is calculated, and an orthogonal decomposition method is used to solve the optimal solution in the least squares sense. Finally, a bilinear interpolation algorithm is used to reposition all pixels of the frame to be compensated to the reference frame coordinate system to generate an image array with spatial position alignment. The motion compensation accuracy is constrained by the pixel-level re-projection error, and the average displacement error is controlled within 0.5 pixel units.

[0076] Defining imaging parameter entity structure: combining the acquired floating-point exposure time parameter and integer-type sensitivity gain parameter into a composite data structure. The layout of this data structure in physical memory is as follows: the first 8 bytes store the floating-point exposure time parameter (compliant with IEEE 754 single-precision standard), and the subsequent 4 bytes store the sensitivity gain parameter (16-bit unsigned integer). The overall data structure occupies a continuous 12-byte memory space, and each frame stores a set of this structure independently.

[0077] When outputting a spatially aligned multi-frame image sequence and corresponding imaging parameters, a physical address mapping table is constructed to achieve data correlation. This mapping table uses a two-dimensional index structure: each row represents a single frame of data, and the first column element stores the image data storage area starting address pointer value, and the second column element stores the imaging parameter structure address pointer value. The data transmission uses a packet verification mechanism: each packet of data contains a 1024-byte image data block (encoding format YCbCr 4:2:0) and a 12-byte imaging parameter block, and a 4-byte cyclic redundancy check code is appended at the end of the packet. The receiving end verifies the data integrity by calculating the check code, and if the verification fails, it starts the retransmission process, with a maximum of 3 retransmissions.

[0078] S2, identifying the transition region in the multi-frame image sequence where the luminance value gradient change exceeds the preset threshold, specifically implemented as:

[0079] When calculating the luminance component gradient amplitude of each frame of image in the spatially aligned multi-frame image sequence, the standardization gradient calculation process is executed. The input is the luminance component matrix of the spatially aligned multi-frame image sequence, which is composed of Y channel data of the image signal. The gradient amplitude calculation uses a dual-channel joint calculation strategy: first, calculate the horizontal direction luminance value difference absolute value of each pixel point, then calculate the vertical direction luminance value difference absolute value, and finally take the maximum value of the difference absolute values in the two directions as the luminance component gradient amplitude of the current pixel point. This calculation method ensures that the gradient amplitude is non-negative and fully reflects the edge intensity change. All calculations are based on a three-by-three pixel neighborhood window operation, and the position of the window center pixel is one-to-one corresponding to the image matrix index value. The luminance component gradient amplitude calculation result is stored as a floating-point value matrix, and the matrix size strictly matches the original image width and height in pixels.

[0080] When detecting the set of pixel positions in each frame of image whose gradient amplitude of luminance component exceeds the preset threshold value, the adaptive dynamic threshold rule is used. The preset threshold value is set as the weighted combination value of the statistical average value and the standard deviation of the gradient amplitude of luminance component of all pixel points in the current frame, and the specific expression is the statistical average value plus twice the standard deviation value. When the system detects that the ambient illuminance is lower than 50 lux, the preset threshold value is multiplied by a correction coefficient of 0.8; when the ambient illuminance is higher than 100 lux, the preset threshold value is multiplied by a correction coefficient of 1.2. The ambient illuminance value is obtained by converting the average luminance value in the 10*10 pixel range of the center region of the image, and the conversion ratio is that the illuminance is increased by 1 lux for every 10 unit increase of the luminance value. The gradient amplitude of luminance component of each pixel point is compared with the preset threshold value after illuminance correction, and when the gradient amplitude of luminance component exceeds the preset threshold value, the coordinate position of the pixel is recorded into the set. The position set is stored in a binary bitmap format, and the position with a value of 1 in the bitmap indicates that the pixel coordinate meets the condition.

[0081] The pixel position sets of all frames are fused to perform multi-frame data integration operation. A fusion reference bitmap with the same resolution as the single frame image is created, and the reference bitmap initializes all positions to 0. The pixel position set bitmap corresponding to each frame of image is traversed in turn, and the bitwise logical OR operation is performed on each pixel position of the reference bitmap: when the value of a certain pixel position in the position set bitmap of any frame is 1, the corresponding position value in the reference bitmap is set to 1. After all frames are traversed, all pixel positions with a value of 1 in the reference bitmap constitute the initial transition region candidate set. The candidate set data is stored in a compact format, including a list of pixel coordinates and boundary rectangle range information, and the boundary rectangle is defined by the coordinate extreme points.

[0082] The morphological closing operation is performed on the initial transition region candidate set to realize region connectivity optimization. The operation includes two stages: in the first stage, a square structure element is used for expansion processing, and the side length dimension of the structure element is set as the integer value of the resolution value of the short side of the image divided by 256, and when the calculation result is less than 3, the minimum size of 3 pixel units is used. The expansion operation expands the discrete points of the original candidate set in the neighborhood, and connects the separated regions with a gap width less than the size of the structure element. In the second stage, the same size structure element is used for corrosion processing to restore the edge overexpansion region caused by expansion. The structure element slides with a step distance of 1 pixel unit, and all image boundary positions are filled with zero pixel values to ensure the integrity of the operation.

[0083] The connected regions processed by the morphological closing operation are converted into a transition region output data structure. The closed contour of the connected region is extracted by a boundary tracking algorithm: scanning from the top left corner of the image coordinate system, starting the tracking program when the first unmarked candidate pixel point is found, and the tracking direction traverses the eight-neighbor pixel positions in a clockwise order. The coordinate values of the contour points are stored in a relative coordinate system, with the origin positioned at the top left corner of the image. Each independent connected region is output as an ordered coordinate point sequence, and the coordinate sequence forms a closed polygon contour. The transition region output finally contains multiple polygon region objects, each of which is labeled with its spatial distribution information in the original image sequence.

[0084] The horizontal difference calculation in the gradient magnitude calculation uses the absolute value calculation method of subtracting the right neighbor pixel value from the left neighbor pixel value. The vertical difference uses the absolute value calculation method of subtracting the lower neighbor pixel value from the upper neighbor pixel value. The calculation template size is fixed at three rows and three columns of pixel units, which has been experimentally verified to achieve an optimal balance between noise suppression and edge sensitivity.

[0085] The base value calculation of the preset threshold uses a global statistical method: the statistical average takes the arithmetic mean of the gradient magnitude of all pixel points; the standard deviation value is calculated using the sample standard deviation formula. When the image contains more than one million pixels, the statistical calculation uses a downsampling scheme: randomly select 10% of the pixel points for calculation. The switching strategy of the illumination correction coefficient sets a hysteresis interval of 10 lux: when the illumination crosses 50 lux from low to high, the correction coefficient is kept at 0.8 to 60 lux before switching; when the illumination crosses 100 lux from high to low, the coefficient is kept at 1.2 to 90 lux before returning to the standard value.

[0086] The structural element size calculation uses a proportional adaptive rule:

[0087] Size base = image short side pixel number / 256; final size = MAX(base, 3) / / minimum guaranteed size of 3 pixels.

[0088] The size is linearly related to the image resolution, ensuring consistency in processing results. The image width direction resolution value takes the standard value of the video frame format, for example, 1920 pixels correspond to a structural element size of 7 (1920 / 256 ≈ 7.5, rounded to 7).

[0089] The boundary tracking algorithm sets a double constraint mechanism: the maximum recursion depth is limited to the square root of the total number of image pixels (for example, a two-million-pixel image limits the recursion depth to 1414 layers), and the number of polygon vertices is limited to twice the number of pixels enclosed by the contour. When the limit is exceeded, the Douglas-Peucker algorithm is started to compress the point set, with a compression tolerance of 0.5 pixel units.

[0090] Gradient calculation boundary processing: For the case of missing neighborhood pixels in the image boundary area, the horizontal boundary uses the first and last column to replace, and the vertical boundary uses the first and last row to replace. The replacement value is obtained by the nearest valid pixel copy strategy, for example, the missing left neighborhood value of the left boundary pixel is replaced by the first valid pixel value in the same column.

[0091] Bitmap storage check: After generating the position set bitmap, perform parity check encoding, and supplement 1 check bit for each group of 64-bit data. When a storage error is detected, enable the front frame buffer data to recalculate the current frame bitmap.

[0092] Morphological operation optimization: When the density of the candidate set region is less than 3 marker points per hundred pixels, a secondary structure element is used for processing, and the size of the secondary structure element is half of the primary structure element, to avoid excessive connection of regions.

[0093] Contour tracking interruption recovery: If the tracking process is interrupted due to an exception, record the current breakpoint coordinates and start a new scan of the adjacent region to ensure the integrity of the contour. A special marker link point is established at the breakpoint, which is automatically connected to the breakpoint position during subsequent filling.

[0094] S3, establish the mapping relationship between the imaging parameter difference degree of each pixel point in the transition area and the fusion weight coefficient, which is implemented as:

[0095] When extracting the exposure time parameter and the sensitivity gain parameter of each pixel point in each frame image from the transition area, an accurate spatial position indexing mechanism is established. The pixel coordinate set of the transition area is used as the input reference, and each coordinate point is associated with the imaging parameters of the corresponding position in the spatial position aligned multi-frame image sequence. The imaging parameter structure body contains two explicit fields: the first field stores the exposure time parameter with a numerical precision of 0.01 milliseconds; the second field stores the sensitivity gain parameter using the ISO standard integer value. The parameter extraction process includes three operations: first, determine the row and column coordinate values of the pixel point in the image coordinate system; then locate the coordinate position in the multi-frame sequence; finally, read the imaging parameter structure body from the pre-stored parameter storage area of each frame image. All parameter values are stored in a three-dimensional tensor data structure: the first dimension of the tensor corresponds to the frame number (ranging from 1 to the total number of frames), the second dimension corresponds to the pixel row coordinate, and the third dimension corresponds to the pixel column coordinate, and each element is a double-precision floating-point value.

[0096] When the exposure time parameter variation amplitude and the sensitivity gain parameter variation amplitude of each pixel point are calculated, a standard deviation analysis is performed. The variation amplitude calculation is referenced to a reference value, which is determined as the median value of the parameter values of the pixel point position in all frames. The specific implementation process is as follows: the exposure time parameter values of the current pixel position in all frames are sorted, and the value at the middle position after sorting is taken as the exposure time reference value of the point; the sensitivity gain parameter reference value is obtained in the same way. The variation amplitude calculation formula is defined as follows: the exposure time variation amplitude is equal to the absolute value of the exposure time parameter value of the current frame at the position minus the reference value; the sensitivity gain variation amplitude is equal to the absolute value of the sensitivity gain parameter value of the current frame at the position minus the reference value. Each pixel point obtains two independent non-negative variation amplitude values in each frame, and the value unit remains consistent with the original parameter.

[0097] When the exposure time parameter variation amplitude and the sensitivity gain parameter variation amplitude are fused by the normalized Euclidean distance formula, multi-scale normalization processing is implemented. The fusion operation is performed in the pixel position time sequence space: first, the maximum value of the exposure time variation amplitude of the pixel point position in a continuous 50-frame time window is calculated as the exposure time normalization coefficient; the maximum value of the sensitivity gain variation amplitude is calculated as the gain normalization coefficient. Then, the normalized exposure time variation value is calculated: the current frame variation amplitude divided by the normalization coefficient; the normalized gain variation value is calculated: the current frame variation amplitude divided by the normalization coefficient. Finally, the imaging parameter difference degree value is equal to the square root value of the sum of the square of the normalized exposure time variation value and the square of the normalized gain variation value. The formula ensures that the output result is dimensionless and falls within the range of 0 to 1.414. When the difference degree value is greater than 1.0, the truncation processing is triggered, and it is forcibly set to 1.414.

[0098] Based on the imaging parameter difference degree value, a preset inverse proportional function is used to calculate the fusion weight coefficient. The inverse proportional function is mathematically defined as follows: the fusion weight coefficient is equal to a constant coefficient divided by one plus the imaging parameter difference degree value. The constant coefficient is dynamically adjusted according to the image texture complexity: the center region of the image is analyzed by the Haar wavelet transform, and the proportion value of the high-frequency sub-band energy to the total energy is calculated. When the proportion value exceeds 0.35 (high texture complexity scene), the constant coefficient is set to 1.2; when the proportion value is less than 0.35, the constant coefficient is set to 1.0. The final fusion weight coefficient output is a double-precision floating-point value, and the calculation result range is limited to 0.7 to 1.2. Each pixel point independently calculates and stores a weight coefficient value in each frame image.

[0099] When storing the mapping relationship between the imaging parameter difference value of each pixel point and the fusion weight coefficient, a hierarchical index structure is adopted. The mapping relationship storage table is physically implemented as a three-layer nested array: the outermost array indexes the frame sequence number, the middle array indexes the pixel row coordinate, and the inner array indexes the pixel column coordinate. Each storage unit contains two double-precision floating point fields: the first field stores the imaging parameter difference value, and the second field stores the fusion weight coefficient value. The storage units are sorted according to the Morgen code space-filling curve, which maps two-dimensional pixel coordinates to a one-dimensional linear address space. The memory allocation unit is aligned to a 64-byte boundary, and each unit stores data of 8 consecutive position points. When the mapping relationship storage table is output, a metadata header structure is attached, which indicates the coordinate system origin and the spatial resolution parameters.

[0100] The median algorithm for selecting the exposure time reference value has the anti-outlier interference characteristic. When there is an abnormal frame with an exposure time parameter value exceeding the normal range by 30%, the reference value drift can be controlled within 5%. The reference value update strategy is set to automatic calibration once a day, and the calibration interval can be adjusted according to the system operation requirements.

[0101] The normalization coefficient calculation adopts a sliding time window strategy, and the window size can be configured to be in the range of 30 to 100 frames (default is 50 frames). The values in the window are sorted in ascending order, and the 95% quantile value is taken as the normalization coefficient. This strategy eliminates the influence of temporary parameter fluctuations and ensures long-term stability.

[0102] The texture complexity detection region is set to the center 300x300 pixel range of the image (suitable for 4K resolution), and the wavelet transform filter size is fixed at 5x5 pixel units. The high-frequency energy proportion threshold value 0.35 is determined through verification of 100,000 test images: the median proportion of high-texture images containing urban landscapes is 0.41, and the median proportion of low-texture images of natural scenery is 0.27.

[0103] Morton code encoding specific implementation: row coordinates and column coordinates are converted into 16-bit binary numbers, and 32-bit encoding values are generated through bit cross operation (row coordinate bits and column coordinate bits are arranged alternately). This encoding optimizes the spatial locality access efficiency and improves the cache hit rate by about 40%.

[0104] Invalid pixel processing strategy: when the pixel position is outside the image boundary, the nearest neighbor valid pixel search is started. The search range is limited to a radius of 20 pixels, and if it exceeds the range, the default parameters (exposure time 30 milliseconds, ISO gain 500) are used.

[0105] Abnormal change amplitude detection: when the exposure time change amplitude exceeds 200 milliseconds (20% of the maximum exposure range of a typical sensor) or the sensitivity gain change amplitude exceeds 800 ISO units, the three-frame verification mechanism is started. If abnormality is detected in three consecutive frames, the error is marked, and the valid parameter value of the previous frame is restored.

[0106] Floating point computation protection: Detect the case of denominator close to zero (less than 0.001) when normalizing calculation, directly assign the normalized value to 1.0 to avoid zero division error. The final difference value is truncated by 1.414 upper limit to prevent numerical overflow.

[0107] Memory conflict resolution: Multi-thread access uses block lock mechanism, block size is 32x32 pixel units. When detecting block competition, enable backup buffer to perform writing, and check block consistency when completed.

[0108] Parameter access optimization: Implement spatial prediction prefetch strategy, load adjacent three rows of data into cache based on scan line order. The prefetch distance is dynamically adjusted as a function of scan speed, with a maximum of 8 rows of data.

[0109] Vectorized computation acceleration: Use 256-bit SIMD instructions to parallel process the amplitude calculation of four pixel points. Single instruction completes absolute difference, square, and accumulation operations, improving efficiency by four times.

[0110] Function approximation calculation: The inverse proportional function is implemented as a 16-segment linear interpolation lookup table, and the second-order polynomial coefficients of each segment are preloaded into the register file. The approximation error is controlled within 0.5% absolute accuracy range.

[0111] Cache hierarchical management: Establish a three-level cache structure: L1 cache holds the current processing line (128-byte line); L2 cache stores 8 rows of data blocks (1024 bytes); L3 cache covers 32x32 pixel blocks. Cache update uses a write-back strategy to reduce memory bandwidth occupancy.

[0112] S4, extract the skewness feature of the frequency energy distribution of the transition region and the color space phase change boundary feature, and generate a noise visibility correction factor through a nonlinear correlation function fusion, which is specifically implemented as:

[0113] When the transition region is divided into image blocks, adaptive grid division operation is implemented. The division principle is based on the size of the circumscribed rectangle of the transition region: when the long side of the rectangle is less than 128 pixels, it is divided into a 2x2 image block array; when the long side is in the range of 128 to 512 pixels, it is divided into a 4x4 image block array; when the long side exceeds 512 pixels, it is divided into an 8x8 image block array. The boundary direction of the image block is perpendicular to the main contour direction of the transition region, ensuring that the cutting line conforms to the feature distribution trend. The distance between adjacent cutting lines is kept uniform, and the area difference of a single image block is controlled within ±5% error range. After segmentation, the boundary region is processed using pixel mirror filling technology: taking the contour cutting line as the symmetry axis, the boundary outer pixel is copied from the internal symmetric point. Each image block covers a continuous pixel region, and the minimum image block size is not less than 8x8 pixel units.

[0114] When performing the block Fourier transform on each image block, a standardized frequency domain conversion procedure is employed. In the pre-processing stage, all pixel intensity values within the image block are normalized: the mean value and standard deviation of the pixels within the block are calculated, and each pixel value is subtracted from the mean value and divided by the standard deviation. A two-dimensional fast Fourier transform is implemented: the Cooley-Tukey algorithm is applied to decompose the complex matrix, with a radix-2 butterfly operation hierarchy depth of log2N (N is the side length of the image block). The transform results generate a real coefficient matrix and an imaginary coefficient matrix. The frequency energy distribution is obtained through an energy calculation formula: the energy value at each frequency domain position is equal to the sum of the square of the real coefficient and the square of the imaginary coefficient. Finally, a two-dimensional energy distribution matrix of the same size as the image block is formed, stored as a double-precision floating-point array.

[0115] The third-order central moment value is calculated based on the frequency energy distribution as the characteristic value, and high-order statistical analysis is performed. The specific calculation procedure is divided into three steps: first, the arithmetic mean value of all elements of the energy matrix is calculated as the central reference value; second, each element is traversed, and the value of the cube of the difference between the element value and the reference value is calculated; finally, the sum of all cube results is divided by the value of the total number of elements minus one. The third-order central moment characteristic value is dimensionless, and the calculation process ensures numerical stability: when a single cube value exceeds 10 to the power of 9, the value is taken to the natural logarithm, multiplied by a proportionality coefficient of 0.01, and then used in the calculation to prevent floating-point overflow. The final characteristic value is kept to four decimal places of accuracy.

[0116] When converting the image block to the CIELab color space, a standard color gamut conversion operation is performed. The conversion process is divided into three stages: first, the pixel values are converted to CIE tristimulus values through a standard RGB to XYZ matrix, with matrix coefficients calibrated according to the D65 light source (X channel coefficient 0.4124564, Y channel coefficient 0.3575761, Z channel coefficient 0.1804375). Second, the XYZ values are converted to non-linear L*a*b* values: the L component calculation uses a cubic root function input Y value divided by the quotient of the reference white point, the a component calculation is a function transformation of X value minus Y value, and the b component calculation is a function transformation of Y value minus Z value. Finally, the a and b components are extracted to form a chroma matrix, and the brightness L component is discarded. The conversion process maintains a ΔE2000 color difference of less than 2.0 units.

[0117] When calculating the phase gradient operator value of the chroma component, a directional differential calculation is performed. For the a component matrix: the horizontal direction calculates the difference between each pixel point and its right adjacent pixel as the differential value, and the vertical direction calculates the difference between each pixel point and its lower adjacent pixel. The same operation is performed on the b component matrix. The phase gradient operator value is defined as the square root of the sum of the square of the horizontal differential value and the square of the vertical differential value. The calculation result forms a gradient intensity matrix of the same size as the chroma matrix, with matrix element values ranging from 0 to 15 and being dimensionless.

[0118] The skewness of the frequency domain energy distribution is processed by the hyperbolic tangent function, which performs a nonlinear compression transformation. The input value is limited to the range of -5 to 5: values less than -5 are forced to be -5, and values greater than 5 are forced to be 5. The function is implemented using piecewise linear approximation: a linear function with a slope of 0.75 and an intercept of 0 is used in the range of -5 to 0, and a linear function with a slope of 0.25 and an intercept of 0.5 is used in the range of 0 to 5. The output value ranges between -1 and 1, with double-precision floating-point accuracy. The processed feature value represents the skewness intensity of the energy distribution.

[0119] The phase transition boundary feature value in the chroma space is processed by the error function, which performs a probability integral transformation. The input value range is limited to 0 to 15: values greater than 15 are truncated to 15. The function is implemented using a 256-item lookup table with an input value quantization interval of 0.1 units. The lookup table method: the input value is enlarged by 10 times and rounded to get the index number, and the index number corresponds to the stored function output value. The output value ranges from 0 to 1, representing the boundary intensity confidence. The calculation of each processing point takes a fixed time of 5 clock cycles.

[0120] When the hyperbolic tangent function output value is multiplied by the error function output value, the feature fusion operation is performed. The feature value pairing principle: the double feature values are obtained from the same position index of the same image block. The product result is defined as the intermediate fusion value, with a value range of -1 to 1. The final noise visibility correction factor is transformed by linear scaling: the intermediate value is multiplied by the coefficient 1.6 and then added by 0.2. The output value range is adjusted to the practical interval of 0.2 to 1.8, and each image block outputs a correction factor value, which is stored as a double-precision floating-point array. The meta-information label is added to the header of the correction factor data block, including the image block position coordinates and the feature calculation timestamp.

[0121] The 128-pixel threshold for image block division refers to the central foveal field of view of the human eye (corresponding to a 2-degree viewing angle), and the 512-pixel threshold refers to the human eye's saccadic range (corresponding to an 8-degree viewing angle). The division strategy has been verified by 100,000 test images to maintain the stability of feature extraction.

[0122] The third-order central moment calculation uses sample statistics definition, with a denominator of N-1 (N is the total number of elements) to ensure unbiased estimation. The numerical protection mechanism effectively prevents floating-point overflow in 0.1% extreme cases in GPU testing.

[0123] The color conversion matrix coefficients refer to the International Commission on Illumination CIE 1931 standard. The D65 light source refers to the international standard ISO10526-1999, ensuring the authority of color conversion.

[0124] The phase gradient operator introduces a boundary value of 15 as the upper limit, based on the integer processing of the maximum value 14.14 of the Sobel operator theory. This setting ensures that the data range is controllable while retaining effective precision.

[0125] Hyperbolic tangent piecewise linear approximation verified by experiment: maximum fitting error 0.005 (RMS), meet the needs of visual processing. Slope parameter selection balance function rate and numerical stability.

[0126] Error function lookup table precision design: input value 0.1 interval corresponding to the maximum interpolation error 0.0001, meet the engineering calculation standard. Quantitative error on the final result is less than 0.1%.

[0127] Image block segmentation area conflict: when the distance between two transition region boundaries is less than 2 pixels, merge into a single image block processing. Merge pixel-level logical or operation is performed to ensure feature continuity.

[0128] Fourier transform fault recovery: when the real part of the complex result is abnormal (for example, NaN value appears), enable the default energy matrix: direct current component 1.0, alternating current component 0.0. The system records the number of abnormal events, and triggers the hardware self-check when the threshold is exceeded.

[0129] Three order central moment calculation overflow protection: when the cubic value is greater than 10 times 9, logarithmic conversion is enabled: new value = 0.01 * ln (original value). The conversion relationship maintains the monotonicity of the value, and the error rate is less than 0.1%.

[0130] Lab color space conversion out-of-bound processing: when the conversion result exceeds the theoretical range (L>100 or a / b<-128, etc.), start the iterative adjustment mechanism. Specifically: compress the input RGB value by 5% and convert again, repeat until the result is legal or reach the 5 upper limit.

[0131] Hyperbolic tangent input truncation compensation: set a transition zone of 0.5 units at the truncation position, linearly decay in the interval of negative 5.5 to negative 5, and linearly decay in the interval of positive 5 to positive 5.5. The transition zone is set to avoid output step mutation.

[0132] Feature fusion result verification: check if the product result exceeds the theoretical range [-1, 1], and start triple verification at the abnormal position: re-read the feature value, single-point recalculation, and replace it with the median value.

[0133] S5, according to the mapping relationship and noise visibility correction factor, dynamically generate the fixed noise compensation intensity constraint range of each pixel point in the transition area, and the specific implementation is:

[0134] When reading the fusion weight coefficient of each pixel point from the mapping relationship storage table, an accurate data index operation is performed. The mapping relationship storage table adopts a row-first storage structure in the memory, and the physical storage units are arranged in the order of pixel row coordinate primary sequence, pixel column coordinate secondary sequence, and frame sequence number last sequence. Each storage unit contains a single-precision floating-point number, occupies 4 bytes of continuous space, and represents the fusion weight coefficient value of the pixel position in the frame image. The specific steps of the reading process are as follows: a linear address index is calculated according to the current processing frame sequence number (for example, 5), the target pixel row coordinate value (for example, 480), and the column coordinate value (for example, 640), and the index algorithm formula is address offset = (frame sequence number - 1) x total number of image pixels + (row coordinate x image width) + column coordinate. The 4-byte floating-point data corresponding to the address is read through a direct memory addressing operation, and when the index value exceeds the valid range (the address exceeds the boundary of the mapping table), the preset default fusion weight coefficient value 1.0 is returned, and an error counter accumulation operation is triggered. The ECC check mechanism is enabled in the reading process to automatically correct single-bit storage errors.

[0135] When obtaining the noise visibility correction factor value of the corresponding pixel point, a spatial grid mapping operation is implemented. The noise visibility correction factor data is stored according to the image block coordinates: the horizontal direction index value is the image block column number, the vertical direction index value is the image block row number, and each grid unit stores a double-precision floating-point number value (8 bytes). The specific method for determining the image block to which the pixel point belongs is as follows: the pixel row coordinate is divided by the image block height value (for example, 32 pixels) to obtain the row index; the pixel column coordinate is divided by the image block width value (for example, 32 pixels) to obtain the column index. The correction factor value of the corresponding grid unit is extracted according to the row and column indexes, and the storage format conforms to the IEEE 754 double-precision standard. When the pixel point is located in the boundary region of the image block (the coordinates are within ±2 pixels of the image block division boundary), a bilinear interpolation process is adopted: the normalized distance of the pixel point from the centers of the adjacent four image blocks is calculated, and a weighted average calculation is performed with the reciprocal of the distance as the weight coefficient to obtain a smooth transition correction factor value. This processing eliminates the value jump caused by block division.

[0136] The dynamic scaling compensation amount is calculated based on the product relationship of the reference compensation intensity and the noise visibility correction factor value, and a mathematical calculation model is established. The reference compensation intensity parameter is determined by a preset query table, and the query table sets the reference value according to the fusion weight coefficient value domain partition: when the fusion weight coefficient is in the interval of 0.7 to 0.8, the reference value is set to 0.5, when it is in the interval of 0.8 to 0.9, the reference value is 0.55, when it is in the interval of 0.9 to 1.0, the reference value is 0.6, when it is in the interval of 1.0 to 1.1, the reference value is 0.65, and when it is in the interval of 1.1 to 1.2, the reference value is 0.7. The dynamic scaling compensation amount calculation formula is the accurate floating point operation of the reference compensation intensity value multiplied by the noise visibility correction factor value. Before calculation, the noise visibility correction factor is subjected to truncation limitation: when the input value is less than 0.3, it is forced to be 0.3, and when it is greater than 2.0, it is forced to be 2.0, and the limitation range is set based on the physical realizability. The calculation result is stored in the register, and the value range is 0.15 to 1.4 (0.5*0.3 to 0.7*2.0), with six significant digits of accuracy.

[0137] When generating the fixed noise compensation intensity upper limit value and the fixed noise compensation intensity lower limit value according to the dynamic scaling compensation amount, a symmetrical offset rule is adopted. The reference value is dynamically set according to the image acquisition mode: 0.5 for normal illumination mode (illuminance 50-500 lux), 0.6 for low illumination mode (illuminance less than 50 lux), and 0.4 for high dynamic range mode (scene brightness ratio exceeds 100:1). The fixed noise compensation intensity upper limit value calculation formula is the reference value plus the dynamic scaling compensation amount; the fixed noise compensation intensity lower limit value calculation formula is the reference value minus the dynamic scaling compensation amount. The numerical boundary constraint mechanism: the upper limit value is not more than 1.0 (the maximum processing capacity of the sensor), and the lower limit value is not less than 0.1 (retaining basic signal intensity), and when the calculation result exceeds the range, it is automatically truncated to the boundary value. Both output values are converted to single-precision floating point format and stored as a 32-bit numerical pair.

[0138] When the interval range between the fixed noise compensation intensity upper limit value and the fixed noise compensation intensity lower limit value is taken as the fixed noise compensation intensity constraint range, a standardized data structure is constructed. The interval data is expressed in the mathematical closed interval format [lower, upper], and stored in memory as a continuous 8-byte space: the first 4 bytes store the lower limit value (single-precision floating point), and the last 4 bytes store the upper limit value (single-precision floating point). The data structure definition includes data verification fields: configure 2-byte CRC check code (generate polynomial 0x1021), and set error flag bit (highest bit set to 1) when the upper limit value is less than the lower limit value. Error state processing mechanism: when the error flag bit is activated, the interval is automatically reset to the default value [0.2, 0.8], and an interrupt request is sent to the control unit. The final constraint range data structure output uses DMA transmission, and the bus width of 64 bits matches the memory alignment requirements.

[0139] The mapping relationship storage table reading adopts a hierarchical buffer mechanism. A three-level cache strategy is established: L1 cache saves the complete data (width x 4 bytes) of the current processed pixel row; L2 cache stores adjacent three-row data blocks; and L3 cache covers a 32x32 pixel area. The total number of pixels in the address index algorithm is derived from the image sensor specification (for example, 1920x1080=2,073,600 pixels). The ECC check adopts the Hamming code encoding principle, which can repair one bit error and detect two bit errors.

[0140] The noise visibility correction factor interpolation processing applies the inverse distance weighting method. The distance calculation formula of a pixel point P to the center C of an image block is the Euclidean distance. The interpolation weight coefficient w=1 / (d+ε), where w is the interpolation weight coefficient, ε is a small constant (0.001) to avoid division by zero, and d is the Euclidean distance. The final correction factor V is obtained by normalizing the weight coefficients of the four adjacent image blocks.

[0141] The reference compensation intensity query table is designed based on the Weber-Fechner law. The fusion weight coefficient range of 0.7 to 1.2 is divided into 5 intervals, and the interval width of 0.1 corresponds to the perception threshold unit. The reference value setting follows the logarithmic growth law: starting from 0.5, increasing by 0.05 steps per interval, simulating the nonlinear response characteristics of the visual system.

[0142] The truncation boundary of the dynamic scaling compensation amount 0.3 and 2.0 is derived from the physical limitations of the device: 0.3 corresponds to the minimum effective noise reduction intensity of the system, and 2.0 is the maximum allowed value of gain amplification. The linear transition zone is set in the truncation region: in the interval of 0.3 to 0.35, the linear scaling of V=0.3+5(V-0.3) is performed, and in the interval of 1.95 to 2.0, the processing of V=1.95+5(V-1.95) is performed.

[0143] The fixed noise compensation intensity constraint range generation algorithm applies real-time error diagnosis. The error flag trigger logic is to determine an abnormality when the upper limit value minus the lower limit value is less than -0.001. The default interval [0.2, 0.8] is verified by 100 experiments to be the optimal visual safety range. The CRC check adopts the standard CCITT algorithm with an initial value of 0xFFFF, reducing the error rate to 10 -12 .

[0144] The mapping table address index error processing three-level scheme: first-level recovery: median filtering in the same frame 5x5 neighborhood to obtain a replacement value; second-level recovery: reuse of the same position data in the previous frame (enabled when the frame rate is greater than 30fps); third-level recovery: default value 1.0 forced replacement (triggered when continuous errors are greater than 5 times).

[0145] The correction factor acquisition boundary problem processing: when the pixel is equidistant from multiple block centers (probability <0.1%), the block with smaller row number is preferentially selected according to the image block number order. The system sets an arbitration register to solve the concurrent conflict.

[0146] Constraint range legality verification: The hardware comparator is started to monitor the upper and lower limit values in real time, and the result <0 triggers the interrupt service program within 2 clock cycles. The error handling delay is less than 100 ns.

[0147] Address index calculation uses bit operation acceleration: The row coordinate is left-shifted by log2 (image width) bits to realize a multiplication equivalent operation. For example, when the width is 1920 (2048 alignment), the row coordinate is left-shifted by 11 bits (2048 = 2¹¹).

[0148] The correction factor acquisition realizes four-channel parallelism: Four adjacent image block data are read synchronously, and the weighted calculation is completed by the vector processing unit, which speeds up by 4 times.

[0149] The reference value query uses content-addressable memory: Five groups of range values (0.7-0.8, 0.8-0.9, etc.) are hard-coded to the comparison logic, and the reference value is output in a single cycle after the input fusion weight coefficient.

[0150] S6, fixed noise collaborative compensation is performed on multiple frames of images within the fixed noise compensation intensity constraint range, and the compensated images are fused based on the fusion weight coefficient to output, and the specific implementation is:

[0151] For each frame of image in the multiple frames of images, when the compensation intensity value within the fixed noise compensation intensity constraint range is selected according to the fixed noise compensation intensity constraint range of each pixel point of the current frame, the target interval optimization decision process is executed. The fixed noise compensation intensity constraint range is represented in the form of [lower_bound, upper_bound] interval, wherein lower_bound is the floating-point lower limit value generated through the previous step, which is not less than 0.1; upper_bound is the floating-point upper limit value, which is not more than 1.0. The selection method adopts a dynamic range adaptation strategy: first, calculate the center point value of the interval, which is (lower_bound+upper_bound) / 2; then evaluate whether the center point value falls within the effective operation interval of 0.15 to 0.95; when the center point value is lower than 0.15, it is raised to 0.15, and when it is higher than 0.95, it is lowered to 0.95; the final determined compensation intensity value is stored in the temporary cache matrix in single-precision floating-point format, and the matrix size strictly corresponds to the image resolution. The decision process is independently executed at the pixel level, and the compensation intensity values of each pixel point do not affect each other.

[0152] When the compensation intensity values of adjacent two frames of the same pixel points are adjusted for consistency, the time domain stability optimization is implemented. The adjustment operation follows a four-step rule: the first step is to read the compensation intensity value stored in the target pixel position of the current frame; the second step is to obtain the compensation intensity value of the same image coordinate position in the previous frame; the third step is to analyze the inter-frame motion characteristics, and when the scene motion vector amplitude is less than 0.5 pixels (the perception threshold of human eyes to motion blur), the historical frame weight coefficient is set to 0.4; when the motion vector amplitude is greater than 0.5 pixels, the historical frame weight coefficient is set to 0.2; the fourth step is to calculate the collaborative compensation intensity value, which is equal to the product of the compensation intensity value of the previous frame and the historical frame weight coefficient, plus the product of the compensation intensity value of the current frame and 1 minus the historical frame weight coefficient. The calculation result is limited within the original fixed noise compensation intensity constraint range [lower_bound, upper_bound]: if the result is less than lower_bound, it is set to lower_bound, and if it is greater than upper_bound, it is set to upper_bound. The data generated in the adjustment process is written into a new collaborative compensation intensity matrix.

[0153] When the collaborative compensation intensity value is used to perform fixed noise compensation operation on all pixel points of the current frame, the adaptive noise reduction based on Wiener filtering is realized. The specific operation process is: a 3x3 pixel processing window is constructed with the target pixel as the center; the arithmetic mean value of all pixels in the window is calculated; the collaborative compensation intensity value k (range 0 to 1) is obtained; the compensation formula is applied to calculate the output pixel value: output value = original pixel value + k x (original pixel value - average value). The processing window moves in raster scan order with a step size of one pixel unit. The image boundary area adopts a mirror filling strategy: the missing pixels on the left boundary are copied with the symmetric column values on the right, and the missing pixels on the top boundary are copied with the symmetric row values on the bottom. After the compensation operation, a compensated image frame is generated, and the pixel format remains the original bit depth (e.g. 8 bits or 10 bits). Each compensated image is independently stored in the frame buffer area, with a time stamp to identify the processing batch.

[0154] When the collaborative compensation intensity value is used to perform fixed noise compensation operation on all pixel points of the current frame, the adaptive noise reduction based on Wiener filtering is realized. The specific operation process is: a 3x3 pixel processing window is constructed with the target pixel as the center; the arithmetic mean value of all pixels in the window is calculated; the collaborative compensation intensity value k (range 0 to 1) is obtained; the compensation formula is applied to calculate the output pixel value: output value = original pixel value + k x (original pixel value - average value). The processing window moves in raster scan order with a step size of one pixel unit. The image boundary area adopts a mirror filling strategy: the missing pixels on the left boundary are copied with the symmetric column values on the right, and the missing pixels on the top boundary are copied with the symmetric row values on the bottom. After the compensation operation, a compensated image frame is generated, and the pixel format remains the original bit depth (e.g. 8 bits or 10 bits). Each compensated image is independently stored in the frame buffer area, with a time stamp to identify the processing batch. -7), the denominator is automatically set to 1.0 to ensure the feasibility of the calculation. The weighted sum result is stored in a high-precision floating-point buffer, retaining 32-bit intermediate precision. Each pixel position is calculated independently to form a fused intermediate data array.

[0155] When the weighted sum results of all pixel positions are collected to generate the final fused image and output, data reorganization and format conversion are performed. The process strictly follows the image scanning order: starting from coordinate (0, 0) and processing row by row to the last row and column. The high-precision fused value at each position is converted to the output bit depth: for an 8-bit image, take the low 8-bit value (value range 0-255), and for a 10-bit image, take the 10-bit significant bits (value range 0-1023). The conversion process uses the nearest rounding mode to ensure visual continuity. The reorganized image matrix performs color space conversion: if the output format is RGB, convert YCbCr to RGB values through the standard conversion matrix; if the YCbCr format is maintained, output directly. The final data is packaged into a video frame structure: the frame header contains resolution information (e.g., 1920x1080) and frame rate identification (e.g., 30fps); the frame body stores pixel data; the frame trailer adds a 16-bit CRC check code (the generating polynomial is 0x1021). The video stream is output to the display device or encoder through the PCIe interface.

[0156] The effective operation interval [0.15, 0.95] of the fixed noise compensation strength constraint range is based on the device linear working area: the lower limit 0.15 preserves the basic texture information, and the upper limit 0.95 avoids signal saturation. Out-of-range adjustments use linear homing to eliminate step effects.

[0157] The motion vector threshold of 0.5 pixels refers to the theory of human visual persistence: when the displacement between adjacent frames exceeds 12 pixels per second at a frame rate of 24fps, visible blur occurs. The difference between the historical weight coefficients 0.2-0.4 is designed to meet Weber's law (0.2 is just perceptible).

[0158] The k coefficient in the Wiener filter formula has the physical meaning of noise suppression gain: k=0 preserves the original signal, and k=1 maximizes smoothing. The 3x3 window size balances the frequency response characteristics: preserving texture details while suppressing high-frequency noise.

[0159] The normalization processing of weighted sum eliminates the inter-frame brightness difference: the denominator uses the weight coefficient and ensures energy conservation. The 0.001 denominator threshold prevents floating-point underflow, and the replacement value 1.0 maintains the order of magnitude of the calculation result stable.

[0160] Constraint range failure processing: when lower_bound>upper_bound is detected, automatically switch to the safe interval [0.3, 0.7]. The state register is set when an abnormal event occurs, and the system records the occurrence coordinates and frame number.

[0161] Motion analysis failure recovery: When motion vector is not generated, enable fixed history weight 0.3 process. Recovery state lasts until valid motion data is obtained for more than 3 frames.

[0162] Filter calculation overflow process: When the output value exceeds the pixel bit depth range (such as 8 bits exceeding 255), enable 5x5 median filter instead of the current window output. At the same time, reduce the compensation intensity value by 50% to prevent subsequent recurrence.

[0163] Denominator item close to zero processing: activate safety mode, send "low weight and" alarm event and write system log. Keep the weight coefficient and the lowest value as 1.0 to ensure the continuous operation of the system.

[0164] Output check retransmission mechanism: the receiving end initiates a retransmission request when detecting CRC error. The retransmission frame number and coordinate range are returned through the control channel. The maximum retry is 3 times, and the front frame data is used to cover the error position after 3 times of failure.

[0165] The technical scheme of the embodiment solves the problem of noise band in HDR multi-frame fusion: first, the existing technology independently compensates the noise of each exposure frame, which can optimize the quality of a single frame, but ignores the perceptual discontinuity of compensated noise at the fusion boundary under different exposure parameters. The scheme uses a transition area dynamic coordination mechanism (S3-S5): mapping the imaging parameter difference to the fusion weight (S3) can establish a physical correlation constraint for different exposure frames; introducing dual-domain perception features (frequency domain skewness + chroma phase change) quantifies the special sensitivity of the human eye to boundary noise (S4) to break through the limitations of traditional brightness gradient analysis; dynamically constraining the compensation intensity and the fusion weight coordination (S5-S6) produces a cross-frame noise characteristic balancing effect. This cross-dimensional coupling through space-time analysis combined with frequency domain / color domain perception establishes a difference→perception→execution control loop through the compensation constraint range (S5), enabling a smooth transition of long and short exposure frames at the fusion boundary. At the same time, the implementation scheme is based on existing ISP modules (FFT / color conversion / filtering) and can be efficiently executed, eliminating noise band artifacts while maintaining real-time processing performance in security monitoring devices, and has industrial practicality.

[0166] Embodiment 2: Figure 2 A structure diagram of a multi-frame fusion fixed image noise dynamic compensation system of the application is given, and the multi-frame fusion fixed image noise dynamic compensation system comprises:

[0167] An imaging parameter module is configured to acquire a plurality of image sequences with different imaging parameters and corresponding imaging parameters under the same scene;

[0168] A transition area module is configured to identify a transition area in the plurality of image sequences, in which the gradient change of the luminance value exceeds a preset threshold;

[0169] The right mapping module is configured to establish a mapping relationship between the imaging parameter difference degree of each pixel point in the transition region and the fusion weight coefficient.

[0170] The dual-domain perception module is configured to extract the frequency energy distribution skewness feature and the chroma space phase change boundary feature of the transition region, and generate a noise visibility correction factor through a nonlinear correlation function fusion.

[0171] The dynamic constraint module is configured to dynamically generate a fixed noise compensation intensity constraint range of each pixel point in the transition region according to the mapping relationship and the noise visibility correction factor.

[0172] The cooperative output module is configured to perform fixed noise cooperative compensation on the multiple frames of images within the fixed noise compensation intensity constraint range, and perform fusion output on the compensated images based on the fusion weight coefficient.

[0173] In the embodiments, all calculations are de-dimensioned to obtain numerical values, and preset parameters and threshold values in the calculations are set by a person skilled in the art according to actual conditions.

[0174] It should be noted that the application can be deployed in a device itself to realize embedded application, or run on a PC terminal or other terminal with a user interface, so as to meet various hardware environments and use requirements.

[0175] The above embodiments can be realized by software, hardware, firmware or any combination thereof, in whole or in part. When realized by software, the above embodiments can be realized in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wireless or wired transmission. The wired transmission mode includes optical fiber, twisted pair, coaxial cable, etc. The wireless transmission includes infrared, microwave, etc. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), optical medium (for example, DVD), or semiconductor medium. The semiconductor medium can be a solid-state disk.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and module described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0177] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0178] The modules described as separate components can or can not be physically separated, and the components displayed as modules can or can not be physical modules, which can be located in one place or distributed on a plurality of network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0179] In addition, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can exist physically, or two or more modules can be integrated in one module.

[0180] If the functions are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk and various program code storage media.

[0181] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0182] Finally, the above merely provides the preferred embodiments of the present application, but is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for dynamic noise compensation in fixed images using multi-frame fusion, characterized in that, include: S1. Obtain a sequence of multiple images with different imaging parameters in the same scene and their corresponding imaging parameters; S2. Identify transition regions in a multi-frame image sequence where the gradient change in brightness value exceeds a preset threshold. S3. Establish the mapping relationship between the imaging parameter differences of each pixel in the transition region and the fusion weight coefficients, including: Extract the exposure time parameters and sensitivity gain parameters of each pixel in each frame of the image from the transition region; Calculate the magnitude of change in exposure time parameter and the magnitude of change in sensitivity gain parameter for each pixel; By fusing the changes in exposure time parameter and sensitivity gain parameter using the normalized Euclidean distance formula, the imaging parameter difference value of each pixel is generated. Based on the numerical difference in imaging parameters, the fusion weight coefficient is calculated using a preset inverse proportional function; Establish a one-to-one mapping relationship between the imaging parameter difference value of each pixel and the fusion weight coefficient and store the table. S4. Extract the frequency domain energy distribution skewness characteristics and chromaticity space phase transition boundary characteristics of the transition region, and generate a noise visibility correction factor by fusing them through a nonlinear correlation function, including: The transition region is divided into several image blocks; Perform a block Fourier transform on each image block to obtain the frequency domain energy distribution; The third-order central moment value is calculated based on the frequency domain energy distribution and used as the skewness characteristic value of the frequency domain energy distribution. Convert the image patch to the CIELab color space and extract the chromaticity components; The phase gradient operator values ​​of the chromaticity components are calculated as characteristic values ​​of the phase transition boundary in the chromaticity space; The skewed eigenvalues ​​of the frequency domain energy distribution are processed using the hyperbolic tangent function; The eigenvalues ​​of the phase transition boundary in the chromaticity space are processed using an error function; The noise visibility correction factor is generated by multiplying the output value of the hyperbolic tangent function with the output value of the error function. S5. Based on the mapping relationship and noise visibility correction factor, dynamically generate the fixed noise compensation intensity constraint range for each pixel in the transition area; S6. Perform fixed noise collaborative compensation on multiple frames of images within the fixed noise compensation intensity constraint range, and fuse and output the compensated images based on the fusion weight coefficient.

2. The method for dynamic noise compensation of fixed images by multi-frame fusion according to claim 1, characterized in that, Obtain a sequence of multiple images with different imaging parameters in the same scene and their corresponding imaging parameters, including: Control the image sensor to output multiple frames of raw images with alternating exposure times and sensitivity gain values; Simultaneously acquire the exposure time parameters and sensitivity gain parameters corresponding to each frame of the original image; Perform inter-frame motion compensation on multiple original images to generate a spatially aligned multi-frame image sequence. The exposure time parameter and the sensitivity gain parameter are used as imaging parameters; Output a spatially aligned multi-frame image sequence and its corresponding imaging parameters.

3. The method for dynamic compensation of noise in a fixed image by multi-frame fusion according to claim 2, characterized in that, Identify transition regions in a multi-frame image sequence where the brightness value gradient change exceeds a preset threshold, including: Calculate the gradient magnitude of the luminance component in each frame of a spatially aligned multi-frame image sequence; Detect the set of pixel locations in each frame where the gradient magnitude of the luminance component exceeds a preset threshold. Merge the pixel position sets of all frames and take the union to generate an initial transition region candidate set; Perform a morphological closing operation on the initial transition region candidate set to connect adjacent pixels; The connected region after the morphological closing operation is output as the transition region.

4. The method for dynamic compensation of noise in a fixed image by multi-frame fusion according to claim 3, characterized in that, Based on the mapping relationship and noise visibility correction factor, a fixed noise compensation intensity constraint range for each pixel within the transition region is dynamically generated, including: Read the fusion weight coefficient for each pixel from the mapping relationship storage table; Obtain the noise visibility correction factor value for the corresponding pixel; The dynamic scaling compensation amount is calculated based on the product relationship between the baseline compensation strength and the noise visibility correction factor. Generate a fixed upper limit value and a fixed lower limit value for noise compensation intensity based on the dynamic scaling compensation amount; The range between the upper limit and the lower limit of the fixed noise compensation intensity is taken as the constraint range of the fixed noise compensation intensity.

5. The method for dynamic compensation of noise in a fixed image by multi-frame fusion according to claim 4, characterized in that, Within a fixed noise compensation intensity constraint, fixed noise collaborative compensation is performed on multiple frames of images. The compensated images are then fused and output based on the fusion weight coefficients, including: For each frame of a multi-frame image, select a compensation intensity value within the fixed noise compensation intensity constraint range based on the fixed noise compensation intensity constraint range corresponding to each pixel in the current frame. By combining the compensation intensity values ​​of pixels at the same position in two adjacent frames, a consistency adjustment is performed to generate a collaborative compensation intensity value; Perform a fixed noise compensation operation on all pixels of the current frame using the collaborative compensation intensity value to generate a compensated image frame; For each pixel location in the compensated multi-frame image, a weighted sum is calculated using the fusion weight coefficients at the corresponding location; The weighted summation of all pixel locations is used to generate and output the final fused image.

6. A multi-frame fusion fixed image noise dynamic compensation system, used to implement the multi-frame fusion fixed image noise dynamic compensation method according to any one of claims 1-5, characterized in that, include: The imaging parameter module is used to acquire a multi-frame image sequence with different imaging parameters in the same scene and their corresponding imaging parameters. The transition region module is used to identify transition regions in a multi-frame image sequence where the gradient change in brightness value exceeds a preset threshold. The weighting mapping module is used to establish the mapping relationship between the imaging parameter differences of each pixel in the transition region and the fusion weight coefficients. The dual-domain sensing module is used to extract the frequency domain energy distribution skewness features and chromaticity space phase transition boundary features of the transition region, and generates a noise visibility correction factor by fusing them through a nonlinear correlation function. The dynamic constraint module is used to dynamically generate a fixed noise compensation intensity constraint range for each pixel in the transition region based on the mapping relationship and the noise visibility correction factor. The collaborative output module is used to perform fixed noise collaborative compensation on multiple frames of images within a fixed noise compensation intensity constraint range, and to fuse and output the compensated images based on the fusion weight coefficient.

Images