A method and system for temporal image background interference suppression based on optical flow guidance

Abstract

The application discloses a time sequence image background interference suppression method and system based on optical flow guidance, and relates to the technical field of image processing. The method comprises the following steps: acquiring a time sequence image sequence, calculating an optical flow field between each two continuous images to obtain a sub-pixel level motion vector field; constructing a space-time deformation field by taking the sub-pixel level motion vector field as a guide; correcting the time sequence image sequence based on the space-time deformation field to eliminate spatial deviation; identifying instantaneous interference pixels from the time sequence image sequence after the spatial deviation is eliminated by using a multi-scale discrimination algorithm; and suppressing the instantaneous interference pixels by using a time domain guided neighborhood repair algorithm to obtain a final time sequence image sequence. The application can effectively overcome non-rigid deformation, significantly reduce a false alarm rate, simultaneously completely retain continuous evolution characteristics of real defects, and greatly improve reliability and precision of time sequence image analysis.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for suppressing background interference in temporal images based on optical flow guidance. Background Technology

[0002] During the time-series image acquisition process, due to the influence of multiple factors, target images acquired at different time periods generally exhibit spatial shifts, specifically manifested as follows:

[0003] Mechanical and structural factors: Vibration of the mounting base of the acquisition equipment (such as industrial cameras and remote sensing sensors), mechanical deviation of the transmission mechanism, or thermal deformation caused by thermal expansion and contraction after the equipment has been working for a long time can cause non-rigid changes in the position and shape of the target in the image.

[0004] Environmental interference factors: Environmental factors such as airflow disturbance, light intensity fluctuations, and slight changes in shooting distance further exacerbate the spatial offset of time-series images, making it difficult to directly match the same target in images from multiple time periods.

[0005] To eliminate the impact of offset on subsequent analysis, time-series images need to be aligned. However, existing technologies have significant limitations: traditional rigid alignment methods (such as translation, rotation, and scaling transformations based on feature points) can only solve simple geometric offsets and cannot handle non-rigid deformations caused by thermal deformation and mechanical deviations. The alignment error is usually above the pixel level, which is difficult to meet the requirements of precision detection. Although some non-rigid alignment methods attempt to optimize the alignment effect through deformation field prediction, they do not make full use of the ability of optical flow information to characterize the spatiotemporal dimension pixel motion correlation. Optical flow can accurately describe the motion trend of pixels between adjacent frames, while existing methods mostly rely on single intra-frame features or simple temporal correlations, which cannot capture the subtle deformation patterns at the pixel level. Ultimately, this makes it difficult to break through the sub-pixel level in alignment accuracy, introducing significant errors for subsequent defect identification and dynamic evolution analysis.

[0006] In the dynamic evolution analysis of defects in time-series images, there are many transient interference factors in the image background that can easily be confused with real defects. These include:

[0007] Transient environmental interference: In natural scenes, birds can block the view and clouds can move rapidly. In industrial scenes, dust can drift by and equipment surfaces can reflect light locally. Such interference is sudden and transient, and can easily be misjudged as "small defects" in a single frame image.

[0008] Dynamic interference of light and shadow: Fluctuations in the gray value of the target caused by changes in the angle of illumination and movement of local shadows can mask the gray value characteristics of the real defect or make the defect-free area appear as a gray value anomaly similar to that of the defect, further increasing the difficulty of identification.

[0009] Existing interference suppression techniques cannot effectively solve the above problems. The core deficiency lies in the fact that most methods focus on spatial domain processing (such as Gaussian filtering for noise reduction and morphological feature filtering) or single-frame feature analysis, which can only eliminate high-frequency noise or regular interference, but cannot distinguish the essential difference between "transient interference" and "real defects" in the temporal dimension. Real defects have temporal stability in time-series images—their morphology and location show a continuous and traceable evolution trend over time (such as crack propagation and increased corrosion area), while transient interference has no continuous evolution pattern and disappears quickly in multiple frames. Existing technologies do not utilize this key feature and cannot establish a "disturbance-defect" distinction criterion from the temporal dimension, ultimately leading to a high false alarm rate and seriously affecting the reliability of time-series analysis results.

[0010] In summary, current time-series image processing technologies suffer from insufficient synergy in the two core aspects of "precise alignment" and "interference suppression," making it difficult for existing methods to meet the demands of "high accuracy and low false alarms" in time-series image analysis for scenarios such as industrial precision inspection and remote sensing dynamic monitoring. Therefore, there is an urgent need for a technical solution that can synergistically address image alignment and background interference suppression, providing an accurate and reliable data foundation for subsequent time-series analysis. Summary of the Invention

[0011] The purpose of this invention is to overcome at least one technical problem existing in the prior art and to provide a method and system for suppressing background interference in time-series images based on optical flow guidance.

[0012] On one hand, embodiments of the present invention provide a method for suppressing background interference in temporal images based on optical flow guidance, comprising: step S1, acquiring an original temporal image sequence, wherein the original temporal image sequence is acquired by the same acquisition device at different time points from the same device under test, and performing multi-scale calculation on the optical flow field between every two consecutive images to obtain a sub-pixel level motion vector field; step S2, using the sub-pixel level motion vector field as a guide, constructing an energy function of the spatiotemporal deformation field in combination with spatiotemporal consistency constraints, and obtaining a spatiotemporally smooth temporal image by minimizing the energy function of the spatiotemporal deformation field. Step S3: Based on the spatiotemporal deformation field, a backward mapping strategy is used to correct the original time-series image sequence to eliminate spatial offsets, including non-rigid offsets caused by equipment vibration, thermal deformation, and airflow disturbances; Step S4: From the time-series image sequence after eliminating spatial offsets, the temporal domain features of pixels are extracted, and based on the fluctuation differences of temporal domain features within different scale time windows, transient interference pixels are identified; Step S5: The transient interference pixels are suppressed using a time-domain guided neighborhood repair algorithm to obtain the final time-series image sequence.

[0013] Further, step S1 includes: step S11, after normalizing the grayscale values ​​of each frame in the initial temporal image sequence, constructing a resolution pyramid, wherein the bottom layer of the resolution pyramid is the original resolution, and the image resolution decreases step by step as the layer increases; step S12, at each layer of the resolution pyramid, based on the variational optimization model to minimize the first energy function, performing optical flow calculation on two consecutive frames to obtain the motion vector field of that layer; step S13, starting from the top layer of the resolution pyramid, refining and correcting the calculated motion vector field layer by layer, and finally obtaining the sub-pixel level motion vector field at the bottom layer.

[0014] Furthermore, step S12 includes: at each level L of the pyramid, processing two consecutive frames of images. and Optical flow calculations are performed to obtain the motion vector field at this level. ;

[0015] in, , They are pixels The calculation of the horizontal and vertical motion components from frame t to frame t+1 is based on minimizing the first energy function using a variational optimization model. In practice, the first energy function is expressed as:

[0016] ;

[0017] in It is the image domain of level L. , , , , It is a constant. For the Charbonnier penalty function, Input variables for the penalty function, It is the smoothing weight parameter. and These are the spatial gradient vectors of the horizontal and vertical motion components, and the energy function. The optimization is performed using the iterative reweighted least squares method. The horizontal motion component of the Lth level of the pyramid. This represents the vertical motion component of the Lth level of the pyramid.

[0018] Furthermore, step S13 includes: step S131, which is the first energy function. Introducing the correction matrix Correction matrix Each correction factor in ;

[0019] in It is an image In position The gradient magnitude, i.e., the image At pixel The magnitude of the spatial gradient vector at that point. It is an adjustable scale parameter;

[0020] Step S132: Starting from the top of the pyramid, for each level L, use bilinear interpolation to interpolate the motion vector field of the previous level. Upsampled to the current layer resolution, the initial motion vector field is obtained. ;

[0021] Step S133, using the motion vector field Using the initial values, the motion vector field of the current level is further optimized by minimizing the first energy function after introducing the correction matrix based on the variational optimization model;

[0022] Step S134: Iterate and optimize N times until the motion vector field at the bottom of the pyramid converges. The convergence condition is that the difference between the motion vectors of two adjacent iterations is less than a preset pixel. Output the sub-pixel level motion vector field at the bottom layer at this time.

[0023] Furthermore, step S2 includes: step S21, using the energy function of the spatiotemporal deformation field constructed based on the sub-pixel level motion vector field and spatiotemporal consistency constraints as the second energy function; step S22, starting from the initial deformation field, using the iterative reweighted least squares method to minimize the second energy function, iterating M times until the energy function value no longer decreases, and solving for the optimal spatiotemporal deformation field as the spatiotemporally smoothed spatiotemporal deformation field.

[0024] Furthermore, the mathematical expression for the second energy function is:

[0025] ;

[0026] ;

[0027] Where t is the temporal index and T is the total number of frames in the image sequence. It is the image domain of the t-th frame. For a spatiotemporal deformation field, for each consecutive frame pair (t, t+1) and each pixel (x, y), the following must be satisfied: ;

[0028] ;

[0029] in , , , , These are spatial smoothing weights and temporal smoothing weights, respectively.

[0030] Furthermore, step S3 includes: step S31, processing each target pixel of the corrected image. Using the target pixel coordinates as the initial guess value, the error function is minimized through iterative optimization, and the sub-pixel level source pixel coordinates of each target pixel in the original image frame are retrieved in reverse. Step S32: Using bilinear interpolation or bicubic interpolation algorithm, the original image frame is resampled according to the source pixel coordinates, and the sampled values ​​are assigned to the corresponding target pixels. Step S33: Steps S31 and S32 are executed repeatedly, traversing all target pixels in the current frame until all target pixels are filled, and the single-frame image correction is completed. Step S34: Steps S31 to S33 are executed one by one for all frames in the temporal image sequence to obtain a spatially aligned temporal image sequence.

[0031] Further, step S4 includes: step S41, extracting the feature vector of each pixel p=(x,y) at different time points t in the temporal image sequence after spatial offset elimination; step S42, based on the extracted feature vector, quantizing the temporal behavior of each pixel in three different scale time windows, including: a short window covering 3-5 adjacent frames, a long window covering 10-20 adjacent frames, and a global window covering the entire temporal image sequence, and calculating the total variance of the feature vector in each window; step S43, inputting the total variance of the feature vector in each window into the core discriminant function, and outputting the discrimination result of each pixel based on the core discriminant function.

[0032] Furthermore, step S5 includes: step S51, for each pixel identified as transient interference... In its location A spatial neighborhood window centered on that pixel is defined on the frame image. Define coverage in the time dimension [ - The time domain window ,exclude In itself, within the spacetime window Step S52: Collect all pixels with a core discrimination function of 1 to form a candidate repair pixel set C; Step S53: Calculate the repair contribution of each candidate repair pixel in the candidate repair pixel set C, and calculate the new gray value of the pixel to be repaired based on the repair contribution; Step S54: Replace the original gray value of the pixel to be repaired with the new gray value, mark the core discrimination function value of the repaired pixel as 1, repeat steps S4 and S5, iterate for a preset number of times until no new transient interference pixels are identified, and output the final repaired time-series image sequence.

[0033] Secondly, embodiments of the present invention provide a temporal image background interference suppression system based on optical flow guidance. This system is applied to the aforementioned optical flow-guided temporal image background interference suppression method. The system includes: a motion vector field construction module, suitable for acquiring an original temporal image sequence, wherein the original temporal image sequence is captured by the same acquisition device at different time points on the same device under test, and multi-scale calculation is performed on the optical flow field between every two consecutive images to obtain a sub-pixel level motion vector field; and a spatiotemporal deformation field construction module, suitable for constructing an energy function based on the sub-pixel level motion vector field and combined with spatiotemporal consistency constraints, and obtaining a sub-pixel level motion vector field by minimizing the energy function. The system comprises: a spatiotemporally smoothed spatiotemporal deformation field; a spatial offset elimination module, suitable for correcting the original time-series image sequence based on the spatiotemporal deformation field using a backward mapping strategy to eliminate spatial offsets, including non-rigid offsets caused by equipment vibration, thermal deformation, and airflow disturbances; a transient interference pixel identification module, suitable for extracting the temporal domain features of pixels from the time-series image sequence after spatial offset elimination, and identifying transient interference pixels based on the fluctuation differences of temporal domain features within different scale time windows; and a final time-series image sequence output module, suitable for suppressing the transient interference pixels using a time-domain guided neighborhood repair algorithm to obtain the final time-series image sequence.

[0034] Thirdly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the above-described optical flow-guided temporal image background interference suppression method.

[0035] Fourthly, embodiments of the present invention also provide a readable storage medium, which, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to execute the above-described optical flow-guided temporal image background interference suppression method.

[0036] The beneficial effects achieved by this invention are as follows: by constructing a high-precision spatiotemporal deformation field guided by optical flow, sub-pixel-level image alignment is achieved, effectively overcoming non-rigid deformation; by combining multi-scale temporal domain discrimination and spatiotemporal collaborative repair, instantaneous interference is accurately distinguished and suppressed, significantly reducing the false alarm rate, while completely preserving the continuous evolution characteristics of real defects, greatly improving the reliability and accuracy of time-series image analysis. Attached Figure Description

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0038] Figure 1 This is a flowchart of a time-series image background interference suppression method based on optical flow guidance, provided in Embodiment 1 of this application.

[0039] Figure 2 This is a schematic diagram of a temporal image background interference suppression system based on optical flow guidance, provided in Embodiment 2 of this application.

[0040] Figure 3 This is a partial block diagram of the electronic device provided in Embodiment 3 of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Example 1

[0043] Before detailing the specific embodiments of the present invention, the overall inventive concept of the present invention is described here for ease of understanding: The core concept of the present invention is to solve the synergistic problem of "precise alignment" and "interference suppression" in temporal image processing through "spatiotemporal collaborative optimization". The core logic revolves around "first solving spatial offset, then distinguishing temporal differences, and finally spatiotemporally collaborative repair", ultimately achieving the goal of "sub-pixel level alignment, low false alarm rate interference suppression, and complete preservation of defect evolution". The specific concept is broken down as follows: The starting point of the concept of the present invention is to identify the two core pain points of temporal image processing, and the two are strongly related: Spatial offset problem: Existing rigid alignment methods cannot handle non-rigid changes such as thermal deformation and mechanical vibration. Non-rigid alignment methods do not fully utilize the pixel-level motion representation capability of optical flow, resulting in difficulty in breaking through the sub-pixel level of alignment accuracy, which lays hidden dangers for subsequent interference and defect differentiation; Interference suppression problem: Existing methods focus on single-frame spatial domain processing and do not utilize the essential difference that "real defects have temporal continuity and instantaneous interference has suddenness", resulting in a high false alarm rate and easy destruction of defect evolution characteristics. Based on this, the core concept of this invention is: first, high-precision spatial alignment must be achieved; then, interference and defects must be distinguished based on temporal dimension differences; and finally, "interference removal and defect preservation" can be achieved through spatiotemporal collaborative repair. The three form a closed loop, and none of them can be omitted.

[0044] like Figure 1 The diagram shown is a flowchart of a time-series image background interference suppression method based on optical flow guidance provided in an embodiment of this application.

[0045] As an example, the method includes: Step S1, acquiring an original temporal image sequence, wherein the original temporal image sequence is acquired by the same acquisition device at different time points from the same device under inspection, and performing multi-scale calculation on the optical flow field between every two consecutive images to obtain a sub-pixel level motion vector field; Step S2, using the sub-pixel level motion vector field as a guide, constructing an energy function of the spatiotemporal deformation field in combination with spatiotemporal consistency constraints, and obtaining a spatiotemporally smooth spatiotemporal deformation field by minimizing the energy function of the spatiotemporal deformation field; Step S3, correcting the original temporal image sequence based on the spatiotemporal deformation field using a backward mapping strategy to eliminate spatial offsets, wherein the spatial offsets include non-rigid offsets caused by device vibration, thermal deformation, and airflow disturbances; Step S4, extracting the temporal domain features of pixels from the temporal image sequence after eliminating spatial offsets, and identifying transient interference pixels based on the fluctuation differences of temporal domain features within different scale time windows; Step S5, suppressing the transient interference pixels using a temporal-guided neighborhood repair algorithm to obtain the final temporal image sequence.

[0046] In some feasible implementations, the core objective of step S1 is to capture the "sub-pixel-level subtle motion trends" of pixels between every two image frames, providing a "fine-grained motion guide" for the subsequent construction of a high-precision deformation field. This solves the problem that traditional optical flow calculations can only achieve pixel-level accuracy and cannot capture the 0.1~1 pixel-level offsets caused by device thermal deformation, minute vibrations, etc., leading to subsequent alignment errors. The final output sub-pixel-level motion vector field (including horizontal / vertical motion components) can accurately describe the motion trajectory of each pixel, providing a reliable data foundation for the construction of the deformation field.

[0047] Step S1 includes: Step S11, after normalizing the grayscale values ​​of each frame in the initial temporal image sequence, constructing a resolution pyramid, wherein the bottom layer of the resolution pyramid is the original resolution, and the image resolution decreases step by step as the layer increases; Step S12, at each layer of the resolution pyramid, based on the variational optimization model to minimize the first energy function, performing optical flow calculation on two consecutive frames to obtain the motion vector field of that layer; Step S13, starting from the top layer of the resolution pyramid, refining and correcting the calculated motion vector field layer by layer, and finally obtaining the sub-pixel level motion vector field at the bottom layer.

[0048] Preferably, step S11 includes: the time-series image sequence is composed of images captured by the same acquisition device at different time points; after normalizing the grayscale values ​​of each frame image, a resolution pyramid is constructed, and the images within the pyramid are represented as... Where L is the hierarchical index and t is the temporal index, the bottom layer of the pyramid represents the original resolution, and the image resolution decreases progressively as L increases (L=0 represents the top layer with the lowest resolution; L= This represents the underlying layer (the original resolution of the image), thus ensuring that optical flow calculations can capture large-scale displacements while also converging to pixel-level fine motion.

[0049] Preferably, step S12 includes: at each level L of the pyramid, processing two consecutive frames of images and Optical flow calculations are performed to obtain the motion vector field at this level. ;in, , They are pixels The calculation of the horizontal and vertical motion components from frame t to frame t+1 is based on minimizing the first energy function using a variational optimization model. In practice, the first energy function is expressed as:

[0050] ;

[0051] in It is the image domain of level L. , , , , It is a constant. For the Charbonnier penalty function, Input variables for the penalty function, It is the smoothing weight parameter. and These are the spatial gradient vectors of the horizontal and vertical motion components, and the energy function. The optimization is performed using the iterative reweighted least squares method. The horizontal motion component of the Lth level of the pyramid. This represents the vertical motion component of the Lth level of the pyramid.

[0052] Specifically, the Iterative Reweighted Least Squares (IRLS) method is used for optimization. The core process includes: Step 1: Initialize the motion vector field. (Usually set to 0, meaning the default pixel has no motion); Step 2: Calculate the energy function value corresponding to the current motion vector field. Step 3: Based on the gradient direction of the energy function, fine-tune the u and v components of each pixel (e.g., update parameters using gradient descent); repeat steps 2-3, iterating 3-5 times until the difference in energy function values ​​between two adjacent iterations is less than a threshold (e.g., 1e-4), or the change in the motion vector is less than 1e-4 pixels; output the optimal motion vector field for the current level L. .

[0053] Preferably, step S13 includes: step S131, which is the first energy function. Introducing the correction matrix Correction matrix Each correction factor in ;

[0054] in It is an image In position The gradient magnitude, i.e., the image At pixel The magnitude of the spatial gradient vector at that point. It is an adjustable scale parameter;

[0055] Step S132: Starting from the top of the pyramid, for each level L, use bilinear interpolation to interpolate the motion vector field of the previous level. Upsampled to the current layer resolution, the initial motion vector field is obtained. ;

[0056] Step S133, using the motion vector field Using the initial values, the motion vector field of the current level is further optimized by minimizing the first energy function after introducing the correction matrix based on the variational optimization model;

[0057] Step S134: Iterate and optimize N times until the motion vector field at the bottom of the pyramid converges. The convergence condition is that the difference between the motion vectors of two adjacent iterations is less than a preset pixel. Output the sub-pixel level motion vector field at the bottom layer at this time.

[0058] Specifically, before refining and correcting the motion vector field, the first energy function is first defined. Introducing the correction matrix ,Right now Correction matrix Each correction factor in ,in It is an image In position The gradient magnitude, i.e., the image At pixel The magnitude of the spatial gradient vector at that point. It is an adjustable scale parameter; the specific correction process is as follows: starting from the top of the pyramid, for each level L, the motion vector field of the previous level is... Upsampled to the current layer resolution, resulting in Upsampling uses bilinear interpolation to maintain the continuity of motion vectors; then... Using the initial values, the motion vector field of the current level is further optimized using the adjusted energy function; the optimization is iterated repeatedly until the bottom layer of the pyramid converges to sub-pixel accuracy; the motion vector field of the bottom layer at this point is then output.

[0059] In some feasible implementations, step S2 includes: step S21, using the energy function of the spatiotemporal deformation field constructed based on the sub-pixel level motion vector field and spatiotemporal consistency constraints as the second energy function; step S22, starting from the initial deformation field, using the iterative reweighted least squares method to minimize the second energy function, iterating M times until the energy function value no longer decreases, and solving for the optimal spatiotemporal deformation field as the spatiotemporally smoothed spatiotemporal deformation field.

[0060] Preferably, the mathematical expression for the second energy function is:

[0061] ;

[0062] ;

[0063] Where t is the temporal index and T is the total number of frames in the image sequence. It is the image domain of the t-th frame. For a spatiotemporal deformation field, for each consecutive frame pair (t, t+1) and each pixel (x, y), the following must be satisfied: ;

[0064] ;

[0065] in , , , , These are spatial smoothing weights and temporal smoothing weights, respectively.

[0066] Preferably, the core objective of step S2 is to transform the discrete motion vectors between frames into a "spatially smooth and globally consistent" deformation field, overcoming the error accumulation problem of traditional frame-by-frame alignment. This solves the problem that traditional frame-by-frame optical flow alignment (frame 1 → frame 2 → frame 3) leads to gradual error accumulation and fails to consider the spatiotemporal continuity of motion, resulting in severe offsets in subsequent frames. The final constructed spatiotemporal deformation field can directly establish a mapping relationship between any frame and the reference frame (first frame), ensuring a unified alignment benchmark for all frames while avoiding abrupt changes in local deformation.

[0067] Specifically, step S21 includes: defining the spatiotemporal deformation field as For each consecutive frame pair (t, t+1) and each pixel (x, y), the following must be satisfied: To ensure optical flow consistency, the fundamental term of the second energy function can be expressed as: Where t is the temporal index and T is the total number of frames in the image sequence. It is the image domain of the t-th frame;

[0068] To eliminate noise and discontinuities in the deformation field and enhance spatiotemporal consistency, this embodiment also includes spatiotemporal constraint terms. ,in , , , , These are spatial smoothing weights and temporal smoothing weights, respectively.

[0069] Combining the above basic terms and spacetime constraint terms yields the complete second energy function. .

[0070] Specifically, step S22 includes: starting from an initial deformation field (except for the first frame, the deformation field of other frames is initially set to 0), and using the principle of minimizing the second energy function, continuously fine-tuning the deformation field using an iterative reweighted least squares method. The value of each point in the equation is considered to be the solution for the optimal deformation field that is both faithful to the optical flow observation and spatiotemporally smooth when the value of the second energy function no longer decreases.

[0071] In some feasible implementations, step S3 includes: step S31, for each target pixel of the corrected image Using the target pixel coordinates as the initial guess value, the error function is minimized through iterative optimization, and the sub-pixel level source pixel coordinates of each target pixel in the original image frame are retrieved in reverse. Step S32: Using bilinear interpolation or bicubic interpolation algorithm, the original image frame is resampled according to the source pixel coordinates, and the sampled values ​​are assigned to the corresponding target pixels. Step S33: Steps S31 and S32 are executed repeatedly, traversing all target pixels in the current frame until all target pixels are filled, and the single-frame image correction is completed. Step S34: Steps S31 to S33 are executed one by one for all frames in the temporal image sequence to obtain a spatially aligned temporal image sequence.

[0072] Preferably, this step utilizes a spatiotemporal deformation field to map each frame of the temporal image sequence onto a reference coordinate system (the first frame), generating a spatially aligned temporal image sequence. The correction process employs a backward mapping strategy, that is, starting from the corrected image pixels, the corresponding pixels in the original image are queried in reverse, and the pixel values ​​are obtained through resampling.

[0073] Specifically, step S31 includes: setting the corrected output image sequence Each frame in It has the same size and coordinate system as the reference frame, and the coordinates of each pixel on it are represented as This is called the target pixel. For each target pixel, it is necessary to look up its value in the original image. The source pixel coordinates (x, y) are currently known from (x, y, t) to... Deformation field ,Right now However, reverse lookup requires a given set of... Find (x,y), and the solution process is as follows: (1) Output the image Each target pixel on As the initial guess value for (x,y); because ideally, when the deformation is small, the source point and the target point should be very close. (2) Iterate the guess value to minimize the error function error(x,y). When the error is less than the threshold, it is considered that the source coordinates (x,y) with sub-pixel precision have been found. The type of iterative optimization algorithm is not limited here. The error function Used to measure the currently hypothesized (x,y) through the deformation field The transformed position, compared with the target position The difference between them. The values ​​at non-integer coordinates (x, y) are calculated using bilinear interpolation.

[0074] Specifically, step S32 includes: for the source pixel coordinates (x, y) obtained in step S31, using a resampling algorithm to extract the source pixel coordinates from the original image. To obtain pixel values, bilinear or bicubic interpolation is preferred to balance computational efficiency and image quality. The resampled pixel values ​​are then assigned to the corrected image. target pixels in To improve efficiency, GPU parallel computing can be used to perform reverse lookup and resampling on multiple target pixels simultaneously.

[0075] Specifically, step S33 includes: for each frame traverse all its target pixels Repeat steps S31 and S32 until... All pixels are filled.

[0076] In some feasible implementations, step S4 includes: step S41, extracting the feature vector of each pixel p=(x,y) at different time points t in the temporal image sequence after eliminating spatial offset; step S42, based on the extracted feature vector, quantizing the temporal behavior of each pixel in three different scale time windows, including: a short window covering 3-5 adjacent frames, a long window covering 10-20 adjacent frames, and a global window covering the entire temporal image sequence, and calculating the total variance of the feature vector in each window; step S43, inputting the total variance of the feature vector in each window into the core discriminant function, and outputting the discrimination result of each pixel based on the core discriminant function.

[0077] The preferred multi-scale discrimination algorithm is to use the essential difference between real defects and transient interference in the temporal dimension to distinguish whether a pixel is a real defect with continuous evolution characteristics or a transient interference that appears briefly.

[0078] Specifically, step S41 includes: for each pixel point p=(x,y) and time point t (where t=1,2,……,T,T is the total number of frames in the image sequence), its feature vector Represented as: ,in It is the gray value of pixel p at time t in the corrected image sequence, normalized to the range [0,1]. It is the spatial gradient magnitude of pixel p at time t, calculated using the Sobel operator. This indicates the grayscale change between adjacent frames. , representing the magnitude of the optical flow vector. , is the amplitude of the deformation field change.

[0079] Specifically, step S42 includes: a short-term window covering 3-5 adjacent frames to capture instantaneous fluctuations; a long-term window covering 10-20 frames to analyze continuous evolution trends; and a global window covering the entire sequence. The sizes of the short-term and long-term windows can be adaptively adjusted according to specific needs. For each pixel, the total variance of its feature vector within the three windows is calculated. , and This serves as the temporal behavior quantization result for that pixel.

[0080] Specifically, step S43 includes: the core discriminant function is represented as:

[0081] ;

[0082] in It is the total variance of the feature vector of pixel p within the long-term window. It is the total variance of the feature vector of pixel p within the global window. It is the total variance of the feature vector of pixel p within the short-term window. It is a small constant to prevent the denominator from being zero. To determine the threshold, when When the condition is met, the function returns 1, indicating that pixel p is judged as a real defect; otherwise, the function returns 0, indicating that pixel p is judged as transient interference.

[0083] In some feasible implementations, step S5 includes: step S51, for each pixel identified as transient interference. In its location A spatial neighborhood window centered on that pixel is defined on the frame image. Define coverage in the time dimension [ - The time domain window ,exclude In itself, within the spacetime window Step S52: Collect all pixels with a core discrimination function of 1 to form a candidate repair pixel set C; Step S53: Calculate the repair contribution of each candidate repair pixel in the candidate repair pixel set C, and calculate the new gray value of the pixel to be repaired based on the repair contribution; Step S54: Replace the original gray value of the pixel to be repaired with the new gray value, mark the core discrimination function value of the repaired pixel as 1, repeat steps S4 and S5, iterate for a preset number of times until no new transient interference pixels are identified, and output the final repaired time-series image sequence.

[0084] Preferably, existing restoration methods (such as mean filtering and image restoration) only utilize the spatial neighborhood information of a single frame, without considering the temporal evolution continuity of pixels, which easily blurs the edges of real defects or introduces unreasonable textures. To solve the above problems, this embodiment proposes a temporal-guided neighborhood restoration algorithm. Its core idea is that for a pixel identified as transient interference, its true pixel value should be consistent with its stable temporal evolution trend and the spatial neighborhood context of the current frame; therefore, the restoration process will simultaneously utilize information from the spatial neighborhood and undisturbed neighboring frames in the temporal domain for collaborative estimation.

[0085] Specifically, step S51 includes: for each pixel identified as transient interference... ,exist Frame image ( Define an M centered on the pixel in the image frame at the current time point. M-space neighborhood window Meanwhile, in the time-series dimension, a coverage is defined [ - The time domain window However, excluding itself, For custom values; in the spacetime window Within the range, all pixels that are not identified as transient interference are collected to form a candidate pixel set C.

[0086] Specifically, step S52 includes: for each candidate repair pixel The formula for calculating a repair contribution is as follows: ,in The time point where the candidate pixel p is located. The spatial attenuation coefficient, The time decay coefficient, It is the candidate repair pixel p in the time domain window The variance of grayscale values ​​within;

[0087] Then, the new grayscale value of the pixel to be repaired is calculated based on the repair contribution. The calculation formula is as follows: ,in It is the pixel p in the corrected image sequence at time point The grayscale values ​​are normalized to the range [0,1].

[0088] Specifically, step S53 includes: since the repair of an interfering pixel may improve the repair environment of its neighboring pixels, after the repair of a pixel marked as interfering, its discrimination result needs to be temporarily updated to 1 (considered as a reliable pixel that has been repaired), and then the repair of other pixels is performed. After all pixels are repaired, discrimination and repair are performed again until the transient interference in the discrimination result disappears, and the final repaired time-series image sequence is output. This sequence has suppressed transient disturbances to the greatest extent possible and preserved the integrity and continuity of the real defects.

[0089] In the above embodiments, a high-precision spatiotemporal deformation field is constructed by optical flow guidance to achieve sub-pixel-level image alignment, effectively overcoming non-rigid deformation; combined with multi-scale temporal domain discrimination and spatiotemporal collaborative repair, instantaneous interference is accurately distinguished and suppressed, significantly reducing the false alarm rate, while fully preserving the continuous evolution characteristics of real defects, greatly improving the reliability and accuracy of time-series image analysis.

[0090] Example 2

[0091] Please see Figure 2 This embodiment provides a structural diagram of a temporal image background interference suppression system based on optical flow guidance.

[0092] As an example, the system is applied to the optical flow-guided temporal image background interference suppression method described in Embodiment 1, and the system includes:

[0093] The motion vector field construction module 21 is suitable for acquiring the original temporal image sequence, which is obtained by the same acquisition device capturing the same device under test at different time points. Multi-scale calculation is performed on the optical flow field between every two consecutive images to obtain the sub-pixel level motion vector field.

[0094] The spatiotemporal deformation field construction module 22 is suitable for constructing a second energy function guided by the sub-pixel level motion vector field and combined with spatiotemporal consistency constraints, and obtaining a spatiotemporally smooth spatiotemporal deformation field by minimizing the second energy function.

[0095] The spatial offset elimination module 23 is suitable for correcting the original time-series image sequence based on the spatiotemporal deformation field using a backward mapping strategy, and eliminating the spatial offset therein, including non-rigid offsets caused by equipment vibration, thermal deformation and airflow disturbance.

[0096] The transient interference pixel identification module 24 is suitable for extracting the temporal features of pixels from a temporal image sequence after spatial offset elimination, and identifying transient interference pixels based on the fluctuation differences of temporal features within different scale time windows.

[0097] The final time-series image sequence output module 25 is suitable for using a time-domain guided neighborhood repair algorithm to suppress the transient interference pixels and obtain the final time-series image sequence.

[0098] It is not difficult to see that this embodiment is a system implementation corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0099] It is worth mentioning that all modules involved in this embodiment are logical units. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.

[0100] Example 3

[0101] Please see Figure 3 The present invention also provides an electronic device, including: a memory and a processor; the memory stores at least one program instruction; the processor loads and executes the at least one program instruction to implement the optical flow-guided temporal image background interference suppression method provided in Embodiment 1.

[0102] The memory 302 and processor 301 are connected via a bus, which may include any number of interconnecting buses and bridges, connecting various circuits of one or more processors 301 and memory 302 together. The bus may also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 301 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 301.

[0103] Processor 301 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory 302 can be used to store data used by processor 301 during operation.

[0104] Example 4

[0105] This invention also proposes a storage medium storing an optical flow-guided temporal image background interference suppression method. When the optical flow-guided temporal image background interference suppression program is executed by a processor, it implements the steps of the optical flow-guided temporal image background interference suppression method described above. Since this storage medium employs all the technical solutions of all the above embodiments, it possesses at least all the beneficial effects brought about by the technical solutions of the above embodiments, which will not be elaborated further here.

[0106] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, based on the guidance provided in this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A method for suppressing background interference in temporal images based on optical flow guidance, characterized in that, include: Step S1: Obtain the original time-series image sequence. The original time-series image sequence is obtained by the same acquisition device capturing the same device under test at different time points. Multi-scale calculation is performed on the optical flow field between every two consecutive images to obtain the sub-pixel level motion vector field. Step S2: Guided by the sub-pixel level motion vector field, construct the energy function of the spatiotemporal deformation field in combination with spatiotemporal consistency constraints, and obtain the spatiotemporally smooth spatiotemporal deformation field by minimizing the energy function of the spatiotemporal deformation field; Step S3: Based on the spatiotemporal deformation field, the original time-series image sequence is corrected using a backward mapping strategy to eliminate spatial offsets, including non-rigid offsets caused by equipment vibration, thermal deformation, and airflow disturbances. Step S4: Extract the temporal features of pixels from the temporal image sequence after eliminating spatial offset, and identify instantaneous interference pixels based on the fluctuation differences of temporal features within different scale time windows. Step S5: Use a time-domain guided neighborhood repair algorithm to suppress the instantaneous interference pixels to obtain the final time-series image sequence.

2. The method for suppressing background interference in temporal images based on optical flow guidance according to claim 1, characterized in that, Step S1 includes: Step S11: After normalizing the grayscale values ​​of each frame in the initial time-series image sequence, a resolution pyramid is constructed. The bottom layer of the resolution pyramid is the original resolution, and the image resolution decreases step by step as the layers increase. Step S12: At each level of the resolution pyramid, minimize the first energy function based on the variational optimization model, perform optical flow calculation on two consecutive frames of images, and obtain the motion vector field of that level; Step S13: Starting from the top of the resolution pyramid, the calculated motion vector field is refined and corrected layer by layer, and finally a sub-pixel level motion vector field is obtained at the bottom layer.

3. The method for suppressing background interference in temporal images based on optical flow guidance according to claim 2, characterized in that, Step S12 includes: At each level L of the pyramid, for two consecutive frames of images and Optical flow calculations are performed to obtain the motion vector field at this level. ; in, , They are pixels The calculation of the horizontal and vertical motion components from frame t to frame t+1 is based on minimizing the first energy function using a variational optimization model. In practice, the first energy function is expressed as: ； in It is the image domain of level L. , , , , It is a constant. For the Charbonnier penalty function, Input variables for the penalty function, It is the smoothing weight parameter. and These are the spatial gradient vectors of the horizontal and vertical motion components, and the energy function. The optimization is performed using the iterative reweighted least squares method. The horizontal motion component of the Lth level of the pyramid. This represents the vertical motion component of the Lth level of the pyramid.

4. The optical flow-guided temporal image background interference suppression method according to claim 3, characterized in that, Step S13 includes: Step S131, is the first energy function. Introducing the correction matrix Correction matrix Each correction factor in is ; in It is an image In position The gradient magnitude, i.e., the image At pixel The magnitude of the spatial gradient vector at that point. It is an adjustable scale parameter; Step S132: Starting from the top of the pyramid, for each level L, use bilinear interpolation to interpolate the motion vector field of the previous level. Upsampled to the current layer resolution, the initial motion vector field is obtained. ; Step S133, using the motion vector field Using the initial values, the motion vector field of the current level is further optimized by minimizing the first energy function after introducing the correction matrix based on the variational optimization model; Step S134: Iterate and optimize N times until the motion vector field at the bottom of the pyramid converges. The convergence condition is that the difference between the motion vectors of two adjacent iterations is less than a preset pixel. Output the sub-pixel level motion vector field at the bottom layer at this time.

5. The optical flow-guided temporal image background interference suppression method according to claim 4, characterized in that, Step S2 includes: Step S21: Use the energy function of the spatiotemporal deformation field constructed based on the sub-pixel level motion vector field and spatiotemporal consistency constraints as the second energy function; Step S22: Starting from the initial deformation field, the second energy function is minimized using the iterative reweighted least squares method. The number of iterations is M, until the energy function value no longer decreases. The optimal spatiotemporal deformation field is then solved as the spatiotemporally smoothed spatiotemporal deformation field.

6. The method for suppressing background interference in temporal images based on optical flow guidance according to claim 5, characterized in that, The mathematical expression for the second energy function is: ； ； in, For the second energy function, It is a fundamental term of the second energy function. Here, t represents the spatiotemporal constraint term, t is the temporal index, and T is the total number of frames in the image sequence. It is the image domain of the t-th frame. For a spatiotemporal deformation field, for each consecutive frame pair (t, t+1) and each pixel (x, y), the following must be satisfied: , , They are pixels Horizontal and vertical motion components from frame t to frame t+1; ； in , , , , These are spatial smoothing weights and temporal smoothing weights, respectively.

7. The method for suppressing background interference in temporal images based on optical flow guidance according to claim 1, characterized in that, Step S3 includes: Step S31: For each target pixel of the corrected image Using the target pixel coordinates as the initial guess, the error function is minimized through iterative optimization, and the sub-pixel level source pixel coordinates of each target pixel in the original image frame are retrieved in reverse. Step S32: Using bilinear interpolation or bicubic interpolation algorithm, the original image frame is resampled according to the source pixel coordinates, and the sampled value is assigned to the corresponding target pixel; Step S33: Repeat steps S31 and S32 to traverse all target pixels in the current frame until all target pixels are filled, thus completing the single-frame image correction. Step S34: Perform steps S31 to S33 one by one on all frames in the original temporal image sequence to obtain a spatially aligned temporal image sequence.

8. The method for suppressing background interference in temporal images based on optical flow guidance according to claim 1, characterized in that, Step S4 includes: Step S41: Extract the feature vector of each pixel p=(x,y) at different time points t in the temporal image sequence after eliminating spatial offset; Step S42: Based on the extracted feature vectors, quantify the temporal behavior of each pixel in three different time windows, including: short window covering 3-5 adjacent frames, long window covering 10-20 adjacent frames, and global window covering the entire temporal image sequence, and calculate the total variance of the feature vector in each window. Step S43: Input the total variance of the feature vector within each window into the core discriminant function, and output the discrimination result of each pixel based on the core discriminant function.

9. The method for suppressing background interference in temporal images based on optical flow guidance according to claim 8, characterized in that, Step S5 includes: Step S51: For each pixel identified as transient interference... In its location A spatial neighborhood window centered on that pixel is defined on the frame image. Define coverage in the time dimension [ - The time domain window ,exclude In itself, within the spacetime window The internal collection of all pixels with a core discrimination function of 1 forms a candidate repair pixel set C; Step S52: Calculate the repair contribution of each candidate repair pixel in the candidate repair pixel set C, and calculate the new gray value of the pixel to be repaired based on the repair contribution. Step S53: Replace the original gray value of the pixel to be repaired with the new gray value, mark the core discrimination function value of the repaired pixel as 1, repeat steps S4 and S5, iterate for a preset number of times until no new transient interference pixels are identified, and output the final repaired time-series image sequence.

10. A temporal image background interference suppression system based on optical flow guidance, the system being used to implement the temporal image background interference suppression method based on optical flow guidance according to any one of claims 1-9, characterized in that, The system includes: The motion vector field construction module is suitable for acquiring raw temporal image sequences, which are obtained by the same acquisition device capturing the same device under test at different time points. Multi-scale calculation is performed on the optical flow field between every two consecutive images to obtain a sub-pixel level motion vector field. The spatiotemporal deformation field construction module is suitable for constructing a second energy function guided by the sub-pixel level motion vector field and combined with spatiotemporal consistency constraints, and obtaining a spatiotemporally smooth spatiotemporal deformation field by minimizing the second energy function; The spatial offset elimination module is suitable for correcting the original time-series image sequence based on the spatiotemporal deformation field using a backward mapping strategy, and eliminating the spatial offset therein, including non-rigid offsets caused by equipment vibration, thermal deformation and airflow disturbance. The transient interference pixel identification module is suitable for extracting the temporal features of pixels from temporal image sequences after spatial offset elimination, and identifying transient interference pixels based on the fluctuation differences of temporal features within different scale time windows. The final time-series image sequence output module is suitable for using a time-domain guided neighborhood repair algorithm to suppress the instantaneous interference pixels and obtain the final time-series image sequence.

Images