Event-image bimodal fusion video turbulence correction method, medium and system

Abstract

This invention discloses a video turbulence correction method, medium, and system based on event-image dual-mode fusion, belonging to the field of digital image processing. The method includes: simultaneously acquiring event voxels of the target and the image sequence to be corrected; inputting the event voxels into a pre-trained encoding / decoding structure to obtain features; projecting the features along the optical flow direction and performing 3D reconstruction to obtain target motion features; extracting background scene representations; fusing the target motion features and background scene representations in the channel dimension to obtain edge-guided features; and inputting the image sequence and background scene representations together into a pre-trained video restoration network to obtain a turbulence-corrected video. The training loss of the encoding / decoding structure includes: representing the optical flow field estimated from the image sequence without turbulence disturbance, and representing the target motion field obtained by projecting the features along the optical flow direction. This invention can accelerate the restoration process and improve restoration quality.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing, and more specifically, relates to a video turbulence correction method, medium, and system based on event-image dual-mode fusion. Background Technology

[0002] Atmospheric turbulence is one of the major interference factors in long-range optical imaging. It is caused by random changes in the air refractive index due to temperature and airflow fluctuations, and usually manifests as geometric distortion that changes rapidly over time and spatially uneven blurring degradation. This type of degradation is characterized by randomness and can significantly damage image structure and texture information, leading to a sharp decline in image quality and severely weakening the robustness of downstream vision tasks such as target detection, recognition, and tracking.

[0003] To suppress turbulence degradation, traditional techniques generally employ pixel-level registration combined with reference frame construction to achieve turbulence correction (such as CN117078538B), thereby enabling video restoration. However, these methods rely on complex geometric alignment and quality screening of a large number of images, resulting in high computational costs and poor adaptability to dynamic scenes, making them prone to failure when the target is moving rapidly.

[0004] With the development of deep learning technology, researchers have proposed turbulence restoration networks based on single-frame images and multi-frame image sequences. Single-frame image-based methods (such as CN111738953A) offer high convenience, as they do not rely on multi-frame image sequences. However, due to the highly ill-conditioned nature of turbulence degradation, the information provided by a single frame image is insufficient to reliably restore the true structure, limiting the stability and restoration quality of the method. Multi-frame image sequence-based methods (such as CN119784648B) utilize the temporal information provided by a large number of consecutive images to stabilize the restoration process. However, these methods generally require longer input sequences to enhance the restoration effect, leading to significantly increased system latency, data transmission, and storage costs, making them unsuitable for deployment in scenarios with extremely high timeliness requirements, such as real-time monitoring, UAV platforms, and ground-based telescopes.

[0005] Besides efficiency issues, turbulence correction methods based on single-frame images and multi-frame image sequences all rely on images captured by traditional frame cameras. The low temporal sampling rate of traditional frame cameras also poses a key challenge, namely, the inability to obtain sufficient and dense temporal information within a short time window. This makes it difficult to effectively distinguish between the real target motion and turbulent disturbances, easily leading to motion aliasing, which further affects the restoration quality.

[0006] Overall, how to simultaneously improve the efficiency and quality of turbulence correction remains a technical challenge that urgently needs to be addressed. Summary of the Invention

[0007] To address the shortcomings and improvement needs of existing technologies, this invention provides a video turbulence correction method, medium, and system based on event-image dual-mode fusion. The aim is to overcome the bottlenecks in restoration efficiency and quality of traditional turbulence correction methods, accelerate the restoration process, and improve restoration accuracy.

[0008] To achieve the above objectives, according to one aspect of the present invention, a video turbulence correction method based on event-image dual-mode fusion is provided, comprising:

[0009] Synchronously acquire the target's event voxels and the image sequence to be corrected ;

[0010] event voxels Input a pre-trained encoder-decoder structure to obtain features , will feature After projection along the optical flow direction, three-dimensional reconstruction is performed to obtain the target motion characteristics. ;

[0011] According to event voxels and image sequences Extracting background scene representation ;

[0012] target motion characteristics and background scene representation By fusing along the channel dimension, edge-guided features are obtained. ;

[0013] Image sequence and edge guidance features The video is input into a pre-trained video restoration network to obtain a turbulence-corrected video. ;

[0014] The encoding and decoding structure includes an encoding structure and a decoding structure. The encoding structure is used to extract multi-scale spatiotemporal features of event voxels. The decoding structure is used for multi-scale spatiotemporal features. Decode to obtain features The training loss of the encoding / decoding structure includes , This represents the optical flow field estimated from an image sequence free of turbulence disturbances. Representation of features The target motion field obtained by projecting along the direction of optical flow , It represents the 1-norm.

[0015] Furthermore, based on event voxels and image sequences Extracting background scene representation ,include:

[0016] Get the background mask covering the background area ;

[0017] Background mask Acting on image sequences The background pixels in each frame of the image are extracted, and the average value of the background pixels in each frame is calculated to obtain the background pixel value after multi-frame average padding. ;

[0018] Calculate background pixels Spatial gradient;

[0019] Background mask Acting on event voxels The background pixels at each time point are obtained, and the number of event polarity alternations at each pixel is counted. ; Represents the pixel position coordinates;

[0020] Number of event polarity alternations at each pixel Normalization is performed to obtain the weight matrix. ;

[0021] Spatial gradient and weight matrix Multiply to obtain the background scene representation. .

[0022] Further, obtain the background mask covering the background area. ,include:

[0023] according to The target region is optimized to obtain the target region mask. ; The preset motion amplitude threshold, Represents pixel position coordinates, express The amplitude of movement of the pixel at that location;

[0024] Mask the target region sequentially Perform morphological dilation and morphological fill operations to obtain the target mask. ;

[0025] target mask Perform a logical inversion to obtain the background mask that covers the background area. .

[0026] Furthermore, both morphological dilation and morphological filling operations are performed using a convolution kernel size of [missing value]. The convolution operation is complete.

[0027] Furthermore, in the encoding and decoding structure, the encoding structure is a three-dimensional residual convolutional module, and the decoding structure is a three-dimensional channel attention layer. Features of corresponding scales in the encoding and decoding structures are merged through skip connections and cross-scale fusion operations, and the merged features are used as the input of the next layer of the decoding structure.

[0028] Furthermore, the video restoration network is a video restoration network based on the Mamba architecture.

[0029] According to another aspect of the present invention, a computer program product is provided, including a stored computer program that, when executed by a processor, implements the video turbulence correction method for event-image dual-mode fusion provided by the present invention.

[0030] According to another aspect of the present invention, a computer-readable storage medium is provided, including a stored computer program that, when executed by a processor, implements the video turbulence correction method for event-image dual-mode fusion provided by the present invention.

[0031] According to another aspect of the present invention, a video turbulence correction system based on event-image dual-mode fusion is provided, comprising:

[0032] A computer-readable storage medium for storing computer programs;

[0033] And a processor for reading a computer program stored in a computer-readable storage medium to implement the video turbulence correction method for event-image dual-mode fusion provided by the present invention.

[0034] This invention discovers that event cameras asynchronously record pixel brightness changes with microsecond-level precision, enabling them to capture dense temporal cues in an extremely short time. They respond only to changing regions, exhibiting high temporal resolution sampling and sparse spatial response. In turbulent environments, event data can capture crucial instantaneous information about target motion, forming a more reliable dynamic representation. Furthermore, since event cameras are inherently insensitive to static backgrounds, their data distribution emphasizes moving parts, facilitating the differentiation between real motion and turbulent disturbances in the temporal dimension, achieving motion decoupling. Based on this consideration, the event-image dual-mode fusion video turbulence correction method provided by this invention achieves video turbulence correction by leveraging event information acquired by event cameras on top of image sequences acquired by frame cameras. In the turbulence correction method provided by this invention, event information effectively compensates for the shortcomings of traditional frame cameras, providing a novel technical approach to overcome the long-standing efficiency bottlenecks and spatiotemporal decoupling difficulties in turbulence restoration.

[0035] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0036] (1) This invention utilizes the optical flow field estimated from a clean image sequence free from turbulence disturbances to construct a dimension reduction constraint. This is used to guide the training process of the encoding / decoding structure, enabling the extraction of features from target event voxels using this structure. At the same time, the dense structure of the target event is preserved and optimized, while the sparse and irregular response of the background event is suppressed, and the features are optimized. The target motion field obtained by projecting along the direction of optical flow That is, a clean target sports field, for the target sports field Target motion features obtained after dimensional reconstruction Turbulent disturbances were effectively removed. This resulted in the generation of clean target motion features with turbulent disturbances removed. Based on this, the present invention further extracts background scene representation. and the target motion characteristics Representation of background scene Fusion yields stable edge-guided features This invention provides high temporal resolution motion priors for video restoration networks, enhancing their ability to recover target details and improving video restoration quality. Simultaneously, it fully utilizes the high temporal resolution motion priors provided by events. On one hand, high-quality restoration can be achieved with only a small number of images, avoiding the dependence on long sequence frames in traditional methods, thus significantly reducing system latency and data volume. It also possesses good real-time performance and adaptability, making it particularly suitable for scenarios such as long-distance monitoring. On the other hand, by constructing the target motion field using the high temporal resolution of events, it achieves spatiotemporal decoupling between real motion and random turbulence, overcoming the motion decoupling difficulties caused by insufficient temporal sampling rates in traditional frame-dependent methods. Furthermore, the event-driven structural representation further overcomes the motion decoupling difficulties caused by insufficient temporal sampling rates in traditional frame-dependent methods.

[0037] Overall, this invention leverages the high temporal resolution and motion sensing capabilities of event cameras to overcome the bottlenecks in restoration efficiency and quality of traditional turbulence correction methods, effectively accelerating the restoration process and improving restoration accuracy.

[0038] (2) The present invention further discovers that turbulent motion has an approximate harmonic motion pattern, exhibiting frequent polarity alternation in event response; pixels with a high number of polarity alternations are close to sharp edges of the image, and the spatial gradient reflects the image edges. Based on this, in the preferred embodiment of the present invention, the spatial gradient after multi-frame averaging based on event polarity statistics is used to obtain the background scene representation. It effectively suppresses spatial edge distortion and maintains background structure stability, improves the background preservation ability of the restoration network, and ultimately effectively improves the quality of video restoration.

[0039] (3) In a preferred embodiment of the present invention, a clean target motion field is selected based on a preset motion amplitude threshold. After optimizing the target region, morphological dilation and morphological filling operations are performed on it to obtain the target mask. Then, the background mask covering the background area is obtained by logical inversion. The background mask constructed from this Continuous and accurate, it provides stable edge information for high-quality background restoration, effectively avoiding breakage and missed detection problems.

[0040] (4) In the preferred embodiment of the present invention, the encoding structure is a three-dimensional residual convolutional module, the decoding structure is a three-dimensional channel attention layer, and the features of the corresponding scales in the encoding structure and the decoding structure are merged through skip connections and cross-scale fusion operations. The merged features are used as the input of the next layer of the decoding structure. The encoding and decoding structure is a lightweight network that can effectively store and compute overhead while ensuring the implementation of functions. Attached Figure Description

[0041] Figure 1 A flowchart of a video turbulence correction method based on event-image dual-mode fusion provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of clean moving target extraction and reconstruction in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of background mask generation in an embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram of the stable background characterization results in this embodiment.

[0045] Figure 5 This is a schematic diagram illustrating the effects of turbulence correction before and after in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0047] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0048] To simultaneously accelerate the restoration process of video turbulence correction and improve restoration quality, this invention provides a video turbulence correction method, medium, and system based on event-image dual-mode fusion. The overall concept is to provide prior information for the video restoration network using event information in a turbulent environment, thereby leveraging the high temporal resolution and motion perception advantages of event cameras to effectively compensate for the shortcomings of traditional frame cameras. This overcomes the bottlenecks of traditional methods that rely solely on image sequences in terms of low efficiency and difficulty in motion decoupling, while simultaneously accelerating the restoration process and improving restoration quality.

[0049] The following is an example.

[0050] Example 1:

[0051] A video turbulence correction method based on event-image dual-mode fusion, such as Figure 1 As shown, it includes:

[0052] S0: Synchronously acquire the target's event voxels and the image sequence to be corrected ;

[0053] S1: Transfer event voxels Input a pre-trained encoder-decoder structure to obtain features , will feature After projection along the optical flow direction, three-dimensional reconstruction is performed to obtain the target motion characteristics. ;

[0054] S2: Based on event voxels and image sequences Extracting background scene representation ;

[0055] S3: Target motion characteristics and background scene representation By fusing along the channel dimension, edge-guided features are obtained. ;

[0056] S4: Image sequence and edge guidance features The video is input into a pre-trained video restoration network to obtain a turbulence-corrected video. .

[0057] The following provides a detailed explanation of each step.

[0058] An event camera, also known as a neuromorphic vision sensor or dynamic vision sensor, is a novel type of vision sensor inspired by the working principle of the biological retina. Unlike traditional frame cameras that capture static "snapshots" (frames) of the entire scene at fixed time intervals, an event camera continuously and independently detects changes in the light intensity (brightness) received by each pixel. When a pixel detects a logarithmic change in light intensity exceeding a preset threshold, it immediately triggers an event. This event is a minimal data packet containing information about time, location, and direction of change. Therefore, the output of an event camera is not a series of complete images, but rather an asynchronous "event stream."

[0059] An event voxel is a three-dimensional space-time grid structure used to represent event camera data. It transforms asynchronous, sparse event streams into a continuous tensor form suitable for deep learning models. Each event voxel grid is a three-dimensional array with dimensions corresponding to spatial coordinates (x, y) and time (t). Each event voxel cell stores the number of events or the cumulative intensity at a specific spatial location within a specific time window.

[0060] In this embodiment, step S0 involves synchronously acquiring the event voxels of the target. and image sequences This provides a foundation for utilizing prior information about events.

[0061] In step S1 of this embodiment, the encoding and decoding structure used includes an encoding structure and a decoding structure. The encoding structure is used to extract the multi-scale spatiotemporal features of event voxels. The decoding structure is used for multi-scale spatiotemporal features. Decode to obtain features .

[0062] Optionally, in this embodiment, the encoding structure is a three-dimensional residual convolutional module, the decoding structure is a three-dimensional channel attention layer, and the features of corresponding scales in the encoding and decoding structures are merged through skip connections and cross-scale fusion operations. The merged features are used as the input of the next layer of the decoding structure.

[0063] Event voxels ( For timestamp information, It is polar. Indicates high, In the input encoding structure (representing a wide range of inputs), multi-scale 3D residual convolution feature extraction will be performed.

[0064] Specifically, the encoding structure includes multiple cascaded convolutional blocks, each using three-scale convolutional kernels, namely ( ), ( )and( By using residual blocks, the response to temporal and spatial gradients can be enhanced. Taking two residuals as an example, the multi-scale 3D residual convolution module outputs features. It can be represented as:

[0065] ,

[0066] in, It is the ReLU activation function; This represents a multi-scale 3D residual convolution module. and Both represent convolutional blocks.

[0067] Subsequently, features A three-dimensional channel attention mechanism (3D-CA) will be introduced, which obtains channel importance weights through global pooling and MLP activation. This process can be represented as:

[0068] ,

[0069] in, and Indicates the weights of the MLP; This represents the GeLU activation function.

[0070] To highlight the continuous motion edge structure of the target event, this embodiment constructs a masked optical flow-guided dimensionality reduction constraint, such as... Figure 2 As shown. During the training process of the encoding / decoding structure, a clean image sequence without turbulence disturbance is input into a pre-trained optical flow estimation network to obtain a clean motion field. Simultaneously, the spatial gradient (edge ​​information) of the clean image sequence is calculated and masked. The gradient mask is then multiplied by the corresponding clean motion field to obtain the optical flow field estimated by the clean gradient mask. Based on this optical flow field Construct motion constraints to constrain the features output by the decoding structure. The dimensionality-reduced features retain the dense target response consistent with the gradient mask motion field while suppressing the sparse background response inconsistent with the mask motion field. The loss corresponding to this motion constraint is:

[0071] ,

[0072] in, This represents the optical flow field estimated from an image sequence free of turbulence disturbances. Representation of features The target motion field obtained by projecting along the direction of optical flow , It represents the 1-norm.

[0073] Based on the above motion constraints, the target event trajectory can be kept dense and coherent, while reducing the discrete response of background events. This applies to a clean target motion field. High-dimensional fine-grained structures can be recovered by performing 3D reconstruction. Optionally, in this embodiment, reconstruction is performed through 3D upsampling and deconvolution:

[0074] ,

[0075] This yields the target motion characteristics that are free from turbulence disturbances and possess structural consistency. .

[0076] This embodiment uses event information to extract target motion features. This approach can fully utilize the high temporal resolution motion priors provided by events. On the one hand, high-quality restoration can be obtained with only a small number of images, avoiding the dependence of traditional methods on long sequence frames, thereby significantly reducing system latency and data volume. It also has good real-time performance and adaptability, making it particularly suitable for scenarios such as long-distance monitoring. On the other hand, by using the high temporal resolution of events to construct the target motion field, it can achieve spatiotemporal decoupling between real motion and random turbulence, overcoming the motion decoupling difficulties caused by insufficient time sampling rate in traditional frame-dependent methods. At the same time, event-driven structural representation can also overcome the motion decoupling difficulties caused by insufficient time sampling rate in traditional frame-dependent methods.

[0077] Step S2 in this embodiment specifically includes:

[0078] Utilize the acquired clean target sports field By setting a threshold for motion amplitude Optimize the target region, avoid additional noise, and obtain the target region mask. :

[0079] ,

[0080] in, Optionally, in this embodiment, a preset motion amplitude threshold is used. ; Represents pixel position coordinates, express The amplitude of movement of the pixel.

[0081] Mask the target region sequentially Perform morphological dilation and morphological fill operations to obtain the target mask. ,like Figure 3 As shown, morphological dilation operations can expand the boundaries of the target region to fill in discontinuous areas; morphological filling operations can fill in the voids within the region, creating a complete and coherent target mask. Optionally, in this embodiment, both the morphological dilation and morphological filling operations are performed using a convolution kernel with a size of [missing information]. The convolution operation is complete.

[0082] target mask Perform a logical inversion to obtain the background mask that covers the background area. :

[0083] ,

[0084] background mask Will be used subsequently from image sequences Event voxels The background portion is extracted independently to ensure that the background modeling is not affected by target occlusion or motion interference.

[0085] Through the above operations, this embodiment generates a continuous and accurate background mask, providing stable edge information for high-quality background restoration and effectively avoiding the breakage and missed detection problems of traditional methods.

[0086] Based on the extracted background mask Step S2 in this embodiment further includes:

[0087] Background mask Acting on image sequences The images in each frame are used to extract the background pixels from each frame:

[0088] ,

[0089] in, Represents image sequence In time The captured images, This represents the dot product. express The background pixels in.

[0090] To restore the missing background caused by the target motion, this embodiment performs multi-frame average imputation in the time dimension. That is, it calculates the average value of background pixels in each frame of the image to obtain the background pixels after multi-frame average imputation. :

[0091] ,

[0092] Calculate background pixels Spatial gradient:

[0093] ,

[0094] in, and These represent the gradients in the x and y directions, respectively. The term "Sobel operator" is used here, but it should be noted that the Sobel operator is only one optional calculation method. In other embodiments of the present invention, other methods can also be used to calculate the spatial gradient.

[0095] At the same time, the background mask Acting on event voxels The background pixels at each time point are obtained, and the number of event polarity alternations at each pixel is counted. ; Represents the pixel position coordinates;

[0096] The number of event polarity alternations, i.e., the number of times the event polarity at the same pixel changes from positive to negative and vice versa, can be formally expressed as:

[0097] ,

[0098] in, for Momentary event polarity for Polarity of events at any given moment.

[0099] Number of event polarity alternations at each pixel Normalization is performed to obtain the weight matrix. :

[0100] ,

[0101] in, This represents the Softmax function. This indicates a convolution operation.

[0102] Spatial gradient and weight matrix Multiply to obtain the background scene representation. :

[0103] ,

[0104] Turbulent motion exhibits near-resonant motion characteristics, displaying frequent polarity alternations in event responses. Pixels with a high frequency of polarity alternations are located near sharp image edges, and the spatial gradient reflects these edges. Therefore, this embodiment utilizes the spatial gradient obtained by statistically weighting multiple frames based on event polarity to suppress spatial edge distortion and maintain background structure stability, thereby enhancing the background preservation capability of the restoration network. Thus, the background scene representation extracted in this embodiment using the above method... It exhibits high stability and is insensitive to turbulent disturbances.

[0105] In step S3 of this embodiment, clean target motion features are fused. and stable background scene representation The obtained edge guidance features It can provide stable edge references for subsequent video restoration, and edge-guided features It can be represented as follows:

[0106] ,

[0107] in, This indicates fusion along the channel dimension. Edge-guided features. This provides a stable edge reference for subsequent video restoration.

[0108] As an optional implementation, in step S4 of this embodiment, the video restoration network used is a video restoration network based on the Mamba architecture.

[0109] like Figure 5 As shown, to fully utilize Mamba's long sequence modeling capabilities while preserving the spatiotemporal neighborhood structure, this invention employs a three-dimensional Hilbert curve to scan and unfold the image sequence and stable edge guidance, transforming the 3D spatiotemporal volume into a single continuous sequence:

[0110] , ,

[0111] , ,

[0112] in, This indicates the calculation of a three-dimensional Hilbert curve. This represents a pre-trained sparse encoder. This represents a pre-trained dense encoder. This space-filling method exhibits good local continuity, allowing Mamba to maintain linear complexity. Under the premise of simultaneously capturing long-term dependencies and local spatial relationships.

[0113] Then the sequence and Input the Mamba base network layer. The core discrete state of a single-layer Mamba is updated as follows:

[0114] , ,

[0115] in, Represents the discrete time step index, indicating the first... At that moment Indicates the system at time... The hidden state vector is used to describe the potential dynamic structure information of the scene; The state transition matrix represents the dynamic evolution of the system state between adjacent time steps. This represents the input mapping matrix, used to map external inputs to the state space; Indicates the system at time... The output observation vector; The observation matrix represents the hidden states and is used to map them to the observation space; the input selection gate... Used to dynamically suppress turbulence and enhance the effective structural response; This represents the gating weight matrix, used to learn the importance of input features in different dimensions.

[0116] Typical parameters in the model are set as input channels. State Dimension The Mamba Block has a stacking depth of 4 layers; each layer contains an input linear projection (64→96), an SSM backbone (96-dimensional state), and a 1D depthwise separable convolution (kernel size=3), maintaining a lightweight design while possessing strong spatiotemporal modeling capabilities.

[0117] The sequence after processing by the Mamba video restoration network is as follows:

[0118] ,

[0119] in, This indicates a video restoration network based on the Mamba architecture. This indicates a feature splicing operation along the channel dimension.

[0120] sequence Recovered as a 3D tensor using Hilbert mapping :

[0121] ,

[0122] The final restored video is then generated using a 3×3 convolutional reconstruction head.

[0123] ,

[0124] in, This represents the convolution operation. This represents the first-layer convolutional mapping operator, used for local feature reconstruction and nonlinear mapping of intermediate features. This represents the second-layer convolutional mapping operator, used to map the feature space to the image space and generate the final restoration result.

[0125] This structure achieves global modeling and efficient correction of turbulent distortion through space-time preserved Hilbert expansion and linear complexity Mamba, achieving high-quality restoration results even with low latency and lightweight networks.

[0126] By constructing edge-guided features and edge-guided features With the image sequence to be restored By inputting the video into the network, this embodiment achieves joint guidance from moving target features and background features, comprehensively improving the structural consistency and overall accuracy of turbulence restoration. This embodiment provides the network with a comprehensive and stable prior that combines target motion and background information by fusing clean target motion features with stable background representations, enhancing the restoration model's ability to recover target details and background textures.

[0127] Example 2:

[0128] A computer program product includes a stored computer program that, when executed by a processor, implements the video turbulence correction method for event-image dual-mode fusion provided in Embodiment 1 above.

[0129] Example 3:

[0130] A computer-readable storage medium includes a stored computer program that, when executed by a processor, implements the video turbulence correction method for event-image dual-mode fusion provided by the present invention.

[0131] Example 4:

[0132] A video turbulence correction system based on event-image dual-mode fusion includes:

[0133] A computer-readable storage medium for storing computer programs;

[0134] And a processor for reading a computer program stored in a computer-readable storage medium to implement the video turbulence correction method for event-image dual-mode fusion provided in Embodiment 1 above.

[0135] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video turbulence correction method using event-image dual-mode fusion, characterized in that, include: Synchronously acquire the target's event voxels and the image sequence to be corrected ; The event voxels Input a pre-trained encoding / decoding structure to obtain features , the features After projection along the optical flow direction, three-dimensional reconstruction is performed to obtain the target motion characteristics. ; According to the event voxels and the image sequence Extracting background scene representation ; The target motion features and the background scene representation By fusing along the channel dimension, edge-guided features are obtained. ; The image sequence and the edge guiding features The video is input into a pre-trained video restoration network to obtain a turbulence-corrected video. ; The encoding / decoding structure includes an encoding structure and a decoding structure, wherein the encoding structure is used to extract multi-scale spatiotemporal features of event voxels. The decoding structure is used for the multi-scale spatiotemporal features. Decode to obtain features The training loss of the encoding / decoding structure includes... , This represents the optical flow field estimated from an image sequence free of turbulence disturbances. Representation of features The target motion field obtained by projecting along the direction of optical flow , It represents the 1-norm.

2. The video turbulence correction method based on event-image dual-mode fusion as described in claim 1, characterized in that, According to the event voxels and the image sequence Extracting background scene representation ,include: Get the background mask covering the background area ; The background mask Acting on the image sequence The background pixels in each frame of the image are extracted, and the average value of the background pixels in each frame is calculated to obtain the background pixel value after multi-frame average padding. ; Calculate the background pixels Spatial gradient; The background mask Acting on the event voxel The background pixels at each time point are obtained, and the number of event polarity alternations at each pixel is counted. ; Represents the pixel position coordinates; Number of event polarity alternations at each pixel Normalization is performed to obtain the weight matrix. ; Combine the spatial gradient with the weight matrix Multiply to obtain the background scene representation. .

3. The video turbulence correction method based on event-image dual-mode fusion as described in claim 2, characterized in that, Get the background mask covering the background area ,include: according to The target region is optimized to obtain the target region mask. ; The preset motion amplitude threshold, Represents pixel position coordinates, express The amplitude of movement of the pixel at that location; Sequentially mask the target region Perform morphological dilation and morphological fill operations to obtain the target mask. ; For the target mask Perform a logical inversion to obtain the background mask that covers the background area. .

4. The video turbulence correction method based on event-image dual-mode fusion as described in claim 3, characterized in that, Both the morphological dilation and morphological filling operations are performed using a convolution kernel with a size of [missing value]. The convolution operation is complete.

5. The video turbulence correction method based on event-image dual-mode fusion as described in any one of claims 1 to 4, characterized in that, In the encoding and decoding structure, the encoding structure is a three-dimensional residual convolutional module, the decoding structure is a three-dimensional channel attention layer, and the features of corresponding scales in the encoding and decoding structures are merged through skip connections and cross-scale fusion operations. The merged features are used as the input of the next layer of the decoding structure.

6. The video turbulence correction method based on event-image dual-mode fusion as described in any one of claims 1 to 4, characterized in that, The video restoration network is a video restoration network based on the Mamba architecture.

7. A computer program product, characterized in that, The method includes a stored computer program, which, when executed by a processor, implements the video turbulence correction method of event-image dual-mode fusion as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The method includes a stored computer program, which, when executed by a processor, implements the video turbulence correction method of event-image dual-mode fusion as described in any one of claims 1 to 6.

9. A video turbulence correction system using event-image dual-mode fusion, characterized in that, include: A computer-readable storage medium for storing computer programs; And a processor for reading a computer program stored in the computer-readable storage medium to implement the video turbulence correction method of event-image dual-mode fusion as described in any one of claims 1 to 6.

Images