A near-repetitive infrared video cleaning system and method based on spatiotemporal characteristics

By using a spatiotemporal feature-based near-repeating infrared video cleaning system, the COLLNet-LSTM model is employed to extract the spatiotemporal joint features of infrared videos and delete similar video segments, thus solving the problems of low efficiency and low accuracy in infrared video cleaning and achieving efficient infrared video data processing.

CN117992630BActive Publication Date: 2026-11-17SHENYANG LIGONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410283244.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-13
Publication Date
2026-11-17
Estimated Expiration
2044-03-13

AI Technical Summary

Technical Problem

Existing infrared video cleaning methods are inefficient and inaccurate when processing large-scale near-repeating infrared video data, and cannot effectively solve the problem of poor cleaning effect caused by weak texture information and low resolution in infrared video.

Method used

A near-repeating infrared video cleaning system based on spatiotemporal features is adopted, including a keyframe extraction module, a spatiotemporal joint feature extraction module, a cosine similarity detection module, and a near-repeating infrared video cleaning module. The spatiotemporal joint features of infrared videos are extracted using the COLLNet-LSTM model, and similar video segments are deleted through cosine similarity detection.

Benefits of technology

It improves the cleaning efficiency and quality of infrared video data, effectively solves the problems of weak texture information and low resolution in infrared video, and enhances the performance and robustness of the model in recognition and classification tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117992630B_ABST
    Figure CN117992630B_ABST
Patent Text Reader

Abstract

This invention provides a near-repeating infrared video cleaning system and method based on spatiotemporal features, relating to the field of infrared video data cleaning technology. The system includes a keyframe extraction module for extracting keyframes from infrared video; a spatiotemporal joint feature extraction module for extracting spatiotemporal features from infrared video keyframes; a cosine similarity detection module for calculating the similarity of the extracted spatiotemporal features of the infrared video keyframes to identify and detect near-repeating content between video frames; and a near-repeating infrared video cleaning module for comparing the similarity calculated by the cosine similarity detection module with a set threshold. This system and method fully consider the inherent characteristics of infrared video and can effectively solve problems such as weak texture information and low resolution in infrared video, leading to poor infrared video data cleaning results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of infrared video data cleaning technology, and in particular to a near-repeating infrared video cleaning system and method based on spatiotemporal characteristics. Background Technology

[0002] Common video modalities include visible light video data and infrared video data. Visible light video data has rich texture information, but it cannot obtain high-quality video data in complex environments such as dust, fog, rain, and night. Infrared video data, on the other hand, senses the differences between different targets and the background based on the different surface temperatures of objects, thus solving the problem of visible light video data being difficult to use in complex environments such as rain and night. The imaging characteristics of infrared video make infrared target detection systems more adaptable to complex environments, and compared to visible light video, it is more suitable for all-weather applications such as nighttime video surveillance, military, and aerospace, which are susceptible to environmental interference. However, compared with visible light video data, infrared video data has characteristics such as lower image contrast, lower resolution, and lower signal-to-noise ratio. Moreover, all-weather infrared video shooting generates a large amount of similar and repetitive infrared video data, causing problems such as information redundancy and storage pressure. These are problems that urgently need to be solved for applications such as target detection, tracking, and strategic early warning using infrared video.

[0003] Currently, the most commonly used video data cleaning methods are feature-based. However, existing feature extraction models focus primarily on spatial feature extraction, with insufficient research on temporal feature extraction. Furthermore, due to the inherent characteristics of infrared video, cleaning efficiency and accuracy are low when processing large-scale near-repeating infrared video data, significantly impacting infrared video processing. To address these issues and improve the efficiency and quality of video data processing, an effective near-repeating infrared video cleaning method is urgently needed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a near-repeating infrared video cleaning system and method based on spatiotemporal characteristics. This system fully considers the characteristics of infrared video itself and cleans near-repeating infrared video, effectively solving the problems of poor cleaning effect caused by weak texture information and low resolution of infrared video data.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: On the one hand, the present invention provides a near-repeating infrared video cleaning system based on spatiotemporal features, including a keyframe extraction module, a spatiotemporal joint feature extraction module, a cosine similarity detection module, and a near-repeating infrared video cleaning module; the keyframe extraction module extracts keyframes of infrared video; the spatiotemporal joint feature extraction module uses a convolutional neural network to extract spatial features of infrared video keyframes, flattens and increases the dimensionality of the extracted spatial features, inputs them into a recurrent neural network and associates them with temporal features, and outputs a spatiotemporal multidimensional feature vector; the cosine similarity detection module uses cosine similarity to calculate the similarity of the extracted spatiotemporal features of infrared video keyframes to identify and detect near-repeating content between video frames; the near-repeating infrared video cleaning module compares the similarity calculated by the cosine similarity detection module with a set threshold. If the similarity is greater than the threshold, the similar infrared video segments are deleted; otherwise, the infrared video segments are retained.

[0006] Preferably, the keyframe extraction module includes an infrared image information enhancement unit, a noise removal unit, and a frame difference calculation unit; the infrared image information enhancement unit is used to enhance the contrast of infrared video image features; the noise removal unit is used to remove noise from the infrared video image to obtain clear infrared image features; and the frame difference calculation unit is used to obtain the inter-frame difference of the infrared video and extract keyframes.

[0007] Preferably, the spatiotemporal joint feature extraction module adopts a spatiotemporal joint feature extraction method to perform feature extraction processing on two infrared video frame sequences respectively, including a spatial feature extraction unit and a temporal feature extraction unit;

[0008] The spatial feature extraction unit uses the COLLNet deep network model to extract spatial features from near-repeating infrared video.

[0009] The temporal feature extraction unit uses COLLNet-LSTM to associate temporal information features between infrared video frames, combining spatial and temporal features.

[0010] Preferably, the spatiotemporal joint feature extraction module uses the COLLNet-LSTM model to perform spatiotemporal joint feature extraction on near-repeating infrared videos. The COLLNet-LSTM model integrates large kernel convolution, structure reparameterization, dynamic convolution ODconv model with the lightweight convolutional network model COLLNet to extract spatial features from infrared video data, and associates with the LSTM model to extract temporal features from infrared video data. In the COLLNet-LSTM model, a lightweight convolutional neural network model COLLNet is designed. This network consists of shallow residual blocks, which reduces feature loss and improves the model speed during spatial feature extraction of infrared video data. To ensure that the shallow network has good information perception capabilities, the COLLNet deep network model uses large kernel convolution to extract infrared video frame features, improves the network's receptive field, and enhances the model's ability to represent fine features. At the same time, the COLLNet deep network model adopts a full-dimensional dynamic convolution ODConv model, which considers the dynamics of the spatial domain, input channel, output channel, and convolutional kernel. Through attention in four dimensions, it achieves complementary features extraction capabilities in multiple aspects and improves the ability to extract contextual information features.

[0011] Preferably, the COLLNet convolutional network model is composed of a lightweight network that combines large convolutional kernels and dynamic convolutional structures, including a 7×7 Conv, a max pooling layer MaxPool, seven lightweight residual blocks, a 3×3 convolutional layer with 512 channels, a LeakyReLU activation function layer, and an average pooling layer.

[0012] The cropped image is first fed into a 7×7 2D convolutional layer with a stride of 2 for dimensionality reduction; the 7×7 Conv output features are then fed into a max pooling layer to reduce the number of model parameters and improve model training speed.

[0013] Each of the seven lightweight residual blocks consists of a large kernel convolutional block and a LeakyReLU activation layer; the large kernel convolutional block in the first two lightweight residual blocks has 64 channels and a size of 13×13; the large kernel convolutional block in the middle three lightweight residual blocks has 128 channels and a size of 13×13; and the large kernel convolutional block in the last two lightweight residual blocks has 256 channels and a size of 13×13.

[0014] Preferably, the large kernel convolutional block structure consists of the following five layers:

[0015] The first layer is the batch normalization layer;

[0016] The second layer is a 1×1 dynamic convolutional layer structure, used to dynamically process the features of the channel and spatial domain;

[0017] The third layer consists of a 13×13 large kernel convolutional layer and a 5×5 convolutional layer for structural parameter reparameterization;

[0018] The fourth layer is the same as the first layer, and it is a batch normalization layer.

[0019] The fifth layer is a 1×1 dynamic convolutional layer.

[0020] Preferably, the dynamic convolutional structure includes a global average pooling operation (GAP), a fully connected layer (FC), and activation layers (ReLU, Sigmoid, and Softmax classifiers); global average pooling is used to shrink the dimensions, and fully connected layers are used to generate different attention values.

[0021] Preferably, the cosine similarity detection module uses cosine similarity as the similarity of near-repeating infrared videos.

[0022] On the other hand, the present invention provides a near-repeating infrared video cleaning method based on spatiotemporal features, comprising the following steps:

[0023] Step 1: Obtain the infrared video frame sequences of the first and second videos;

[0024] Step 2: Use a multi-stage processing method to extract keyframes from the two infrared video frame sequences;

[0025] 1) Enhance the infrared video frame sequence to improve the contrast of infrared video image features;

[0026] The local histogram calculation method is used to redistribute brightness to change image contrast, making object features more obvious and enhancing recognition.

[0027] 2) Use the inter-frame difference method to calculate the pixel difference between adjacent frames of the video, and select the n frames with the largest difference by using the local maximum method;

[0028] 3) Perform noise reduction processing on the selected video frames;

[0029] 4) Calculate the pixel differences between adjacent frames of the video again using the inter-frame difference method, and select the n / 2 frame with the largest difference as the key frame by using the local maximum method;

[0030] 5) Repeat steps 1)-4) to perform multi-stage keyframe extraction;

[0031] Step 3: Use the COLLNet-LSTM model to associate the temporal information features between key frames of infrared video images, combine spatial and temporal features, and extract the spatiotemporal joint features of two infrared video image frames;

[0032] Step 4: After processing the spatiotemporal joint features of the two infrared video image frames through a fully connected layer, calculate the cosine similarity between the two infrared videos.

[0033] Step 5: Compare the cosine similarity of the two infrared videos with the set threshold. If the cosine similarity is greater than the threshold, the two infrared videos are considered to be nearly identical and the comparison video is deleted; otherwise, it is not deleted.

[0034] The beneficial effects of adopting the above technical solution are as follows: The near-repeating infrared video cleaning system and method based on spatiotemporal features provided by this invention fully considers the characteristics of infrared video itself and can effectively solve the problems of poor cleaning effect of infrared video data due to weak texture information and low resolution. The designed convolutional neural network model COLLNet, through the ingenious integration of lightweight design, large convolutional kernels and dynamic convolution technology, achieves efficient extraction and optimized representation of image features.

[0035] Specifically:

[0036] Efficient preprocessing stage: First, the cropped image undergoes preliminary feature extraction and spatial dimension compression using a 7×7 2D convolutional layer with a stride of 2, effectively reducing the complexity of the input data while preserving important global information. Subsequent max pooling further reduces the number of model parameters, accelerating training convergence.

[0037] Multi-level feature learning: The core of the model consists of seven lightweight residual blocks that progressively deepen and increase in the number of channels. Each residual block employs a large-kernel convolution combined with a dynamic convolution structure to fully exploit spatial features at different levels. This ensures that the model can capture fine-grained features at lower levels and construct more abstract and complex representations at higher levels.

[0038] Dynamic convolution mechanism: In the large kernel convolutional blocks, the dynamic convolutional layer introduces global average pooling (GAP), fully connected layers and various activation functions (such as ReLU, Sigmoid, etc.) to achieve adaptive adjustment and attention allocation of input features in the channel dimension, enabling the model to dynamically allocate resources according to the input content, thereby improving the effectiveness of feature selection and the generalization ability of the model.

[0039] Structural reparameterization and normalization: Structural parameters are reparameterized within each large kernel convolutional block through a combination of 13×13 and 5×5 convolutional layers, and batch normalization layers are used to ensure that the output of each layer has stable distribution characteristics, which is beneficial to gradient propagation and network training efficiency.

[0040] The COLLNet convolutional network model not only improves the performance of deep learning models in recognition and classification tasks, but also achieves excellent results with limited computing resources due to its lightweight characteristics and dynamic adaptability. The model's structural design helps reduce the risk of overfitting and enhances the model's robustness and generalization ability on various visual tasks. Attached Figure Description

[0041] Figure 1 A structural block diagram of a near-repetitive infrared video cleaning system based on spatiotemporal features provided in an embodiment of the present invention;

[0042] Figure 2 This is a structural diagram of the COLLNet deep network model provided in an embodiment of the present invention;

[0043] Figure 3 This is a diagram of a large kernel convolutional block structure provided in an embodiment of the present invention;

[0044] Figure 4 A diagram of a dynamic convolutional structure provided in an embodiment of the present invention;

[0045] Figure 5 This is a diagram of the COLLNet-LSTM model structure provided in an embodiment of the present invention;

[0046] Figure 6 This is a schematic diagram of cosine similarity provided for an embodiment of the present invention.

[0047] Figure 7 A flowchart illustrating a near-repeating infrared video cleaning method based on spatiotemporal features, provided for an embodiment of the present invention;

[0048] Figure 8 This is a schematic diagram of the inter-frame difference method provided in an embodiment of the present invention. Detailed Implementation

[0049] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0050] In this embodiment, a near-repeating infrared video cleaning system based on spatiotemporal features includes a keyframe extraction module, a spatiotemporal joint feature extraction module, a cosine similarity detection module, and a near-repeating infrared video cleaning module, as follows: Figure 1As shown; the keyframe extraction module extracts keyframes from the infrared video; the spatiotemporal joint feature extraction module uses a convolutional neural network to extract spatial features from the infrared video keyframes, flattens the extracted spatial features and increases their dimensionality, inputs them into a recurrent neural network and associates them with temporal features, outputting a spatiotemporal multidimensional feature vector to solve the problem of difficult feature extraction between infrared video frames; the cosine similarity detection module uses cosine similarity to calculate the similarity of the extracted spatiotemporal features of the infrared video keyframes to identify and detect nearly duplicate content between video frames; the nearly duplicate infrared video cleaning module compares the similarity calculated by the cosine similarity detection module with a set threshold. If the similarity is greater than the threshold, the similar infrared video segments are deleted; otherwise, the infrared video segments are retained.

[0051] In this embodiment, the keyframe extraction module includes an infrared image information enhancement unit, a noise removal unit, and a frame difference calculation unit; the infrared image information enhancement unit is used to enhance the contrast of infrared video image features; the noise removal unit is used to remove noise from the infrared video image to obtain clear infrared image features; and the frame difference calculation unit is used to obtain the inter-frame difference of the infrared video and extract keyframes.

[0052] The spatiotemporal joint feature extraction module uses a spatiotemporal joint feature extraction method to perform feature extraction processing on two infrared video frame sequences, including a spatial feature extraction unit and a temporal feature extraction unit.

[0053] The spatial feature extraction unit uses the COLLNet deep network model to extract spatial features from near-repeating infrared video.

[0054] The temporal feature extraction unit uses COLLNet-LSTM to associate temporal information features between infrared video frames, combining spatial and temporal features to solve the problem of extracting temporal features from infrared video image sequences, thereby improving the model's ability to understand and recognize complex scenes, its robustness, and its classification accuracy.

[0055] In this embodiment, the spatiotemporal joint feature extraction module uses the COLLNet-LSTM model to perform spatiotemporal joint feature extraction on near-repeating infrared videos. The COLLNet-LSTM model integrates large kernel convolution, structure reparameterization, dynamic convolution (ODConv) model, and the lightweight convolutional network model COLLNet to extract spatial features from infrared video data, and associates it with the LSTM model to extract temporal features from the infrared video data. Considering the characteristics of low resolution and low contrast in infrared video images, using a deeper model would lead to the loss of detailed features. Therefore, a lightweight convolutional network model C is designed within the COLLNet-LSTM model. OLLNet, a network composed of shallow residual blocks, reduces feature loss and improves model speed during spatial feature extraction from infrared video data. To ensure good information perception capabilities in the shallow network, the COLLNet deep network model uses large-kernel convolutions to extract features from infrared video frames, enhancing the network's receptive field and improving its ability to represent fine features. Simultaneously, the COLLNet deep network model employs a full-dimensional dynamic convolution ODConv model, considering the dynamics of the spatial domain, input channel, output channel, and convolution kernel. Through attention across these four dimensions, it achieves complementary feature extraction capabilities across multiple dimensions, enhancing the ability to extract contextual information features.

[0056] In this embodiment, the convolutional network model COLLNet is a self-built spatial feature extraction model, such as... Figure 2 As shown, the model consists of a lightweight network that combines large convolutional kernels and dynamic convolutional structures, including a 7×7 Conv, a max pooling layer MaxPool, seven lightweight residual blocks, a 3×3 convolutional layer with 512 channels, a LeakyReLU activation function layer, and an average pooling layer.

[0057] The cropped image is first fed into a 7×7 2D convolutional layer with a stride of 2 for dimensionality reduction; the output features of the 7×7 Conv layer are then fed into a max pooling layer MaxPool to reduce the number of model parameters and improve the training speed.

[0058] Each of the seven lightweight residual blocks consists of a large kernel convolutional block and a LeakyReLU activation layer; the large kernel convolutional blocks in the first two lightweight residual blocks have 64 channels and a size of 13×13; the large kernel convolutional blocks in the middle three lightweight residual blocks have 128 channels and a size of 13×13; and the large kernel convolutional blocks in the last two lightweight residual blocks have 256 channels and a size of 13×13.

[0059] The structure of each kernel convolutional block consists of the following five layers:

[0060] The first layer is the batch normalization layer (BatchNorm);

[0061] The second layer is a 1×1 dynamic convolutional layer structure, used to dynamically process the features of the channel and spatial domain;

[0062] The third layer consists of a 13×13 large kernel convolutional layer and a 5×5 convolutional layer for structural parameter reparameterization;

[0063] The fourth layer is the same as the first layer, and it is a batch normalization layer.

[0064] The fifth layer is a 1×1 dynamic convolutional layer.

[0065] Large kernel convolutions are used to ensure that lightweight network models have good information perception capabilities. The COLLNet model uses large kernel convolutions to enhance the receptive field of the lightweight network and improve the model's ability to represent fine features. The large kernel convolution structure is as follows: Figure 3 As shown, in the residual structure, two 1×1 convolutional layers and a 13×13 large kernel convolutional layer are used to form a block, and the entire block is nonlinearly activated using the residual structure and the LeakyReLU activation function.

[0066] The COLLNet model employs a full-dimensional dynamic convolutional structure to achieve multi-dimensional, multi-faceted, and multi-level feature extraction. The dynamic convolutional structure, such as... Figure 4 As shown, the model includes a global average pooling (GAP) operation, a fully connected (FC) layer, and activation layers such as ReLU, Sigmoid, and Softmax classifiers. In the figure, k represents the kernel size, and C... input W represents the number of input channels. i α represents the weight corresponding to the i-th convolutional kernel. s The attention value, α, represents the spatial dimension. c α represents the attention value along the input channel dimension. f α represents the attention value in the output channel dimension. w This represents the attention value of the entire convolutional kernel. Simultaneously, global flat pooling is used to shrink the dimension, and fully connected layers are used to generate different attention values.

[0067] In this embodiment, the specific steps for the COLLNet deep network model to extract spatial features from infrared video are as follows:

[0068] Step A1: Perform a size change operation on the infrared video frames after frame capture processing to obtain a 256×256 infrared image;

[0069] Step A2: The infrared image obtained in step A1 is fed into a 7×7 two-dimensional convolution with a stride of 2 for dimensionality reduction, and then fed into a max pooling layer to reduce the number of model parameters and improve the training speed of the model.

[0070] Step A3: Enter the Block of large kernel convolution, perform convolution and batch normalization layer transformation, and then send it into ODConv dynamic convolution to dynamically process the channels and spatial domains, and enhance the feature extraction of the channels and spatial domains.

[0071] Step A4: Extract the features of interest using a multi-dimensional attention mechanism in parallel processing to enhance the points of interest in the infrared image. Then, pass them into a large kernel convolutional layer to improve the receptive field of the model. Finally, output them to the next block through ODConv dynamic convolution.

[0072] In this embodiment, the COLLNet-LSTM model integrates the COLLNet and LSTM models to extract spatiotemporal joint features from infrared video images, such as... Figure 5 As shown, the specific steps are as follows:

[0073] Step C1: Input the image sequence into the lightweight model to extract spatial features;

[0074] Step C2: Use local self-attention to enhance local feature extraction capabilities, use large kernel convolution to improve the receptive field of the image, and use full-dimensional dynamic convolution to associate the contextual information of the image.

[0075] Step C3: Flatten the extracted spatial features and increase their dimensionality, then input them into the LSTM model to correlate the temporal features;

[0076] Step C4: Output the feature vectors; where A and B are two feature vectors.

[0077] The cosine similarity detection module uses cosine similarity as the similarity metric for near-repeating infrared videos. Cosine similarity is characterized by its independence from vector size, sparsity friendliness, effectiveness in similarity measurement, and fast computation. Since cosine similarity is independent of vector size, it is only affected by the angle between the vectors. It can be used in various fields such as video similarity comparison, document similarity search, collaborative filtering recommendation, cluster analysis, and natural language processing tasks to help measure the similarity and relevance between data objects.

[0078] In this embodiment, cosine similarity is used to measure the similarity of infrared video frame sequences, such as... Figure 6As shown, A and B are two vectors, and θ is the angle between vectors A and B. The cosine similarity of the vectors ranges from -1 to 1. A cosine similarity of 1 indicates that the two vectors point in exactly the same direction; they are identical. A cosine similarity of -1 indicates that the two vectors point in completely opposite directions; they are completely different. The closer the cosine similarity value is to 1, the more similar the two vectors are; the closer it is to -1, the greater the difference between the two vectors. The cosine similarity of N-dimensional vectors in N-dimensional space is shown in the formula.

[0079]

[0080] In this embodiment, a near-repeating infrared video cleaning method based on spatiotemporal features is described, such as... Figure 7 As shown, it includes the following steps:

[0081] Step 1: Obtain the infrared video frame sequences of the first and second videos;

[0082] Step 2: Use a multi-stage processing method to extract keyframes from the two infrared video frame sequences;

[0083] 1) Enhance the infrared video frame sequence to improve the contrast of infrared video image features;

[0084] The local histogram calculation method is used to redistribute brightness to change image contrast, making object features more obvious and enhancing recognition.

[0085] Local histograms are a method of enhancing image features by performing histogram analysis on local regions of an image. By dividing the image into multiple small blocks and calculating the gray-level distribution of each block, local feature information can be obtained. In infrared image processing, local histograms can be used to enhance the contrast of local areas and extract features of specific targets.

[0086] 2) Use the inter-frame difference method to calculate the pixel difference between adjacent frames of the video, and select the n frames with the largest difference by using the local maximum method;

[0087] Inter-frame differencing is a video processing technique used to compare the differences between adjacent frames in a video sequence. By calculating the differences between pixels in adjacent frames, motion or changes in the video can be detected. In multi-stage methods, inter-frame differencing can be used to reduce redundant data and select the frame with the greatest difference as the keyframe. Figure 8 As shown. The specific steps are as follows:

[0088] Step S1: Frame Difference Calculation; For consecutive video frames, calculate the pixel difference between adjacent frames. Typically, the difference between pixels is calculated using the formula |frame 1 - frame 2|. This formula can be adjusted according to actual needs; for example, color difference or other specific difference metrics can be used. The calculation method is shown in the following formula:

[0089] D n (x, y) = |f n (x, y)-f n-1 (x, y)|

[0090] Among them, D n f represents the difference between the video frame at time n and the image at time n-1. n f is the grayscale value of a pixel in the video frame at time n. n-1 is the grayscale value of the pixel in the video frame at time n-1.

[0091] Step S2: Smoothing; Smoothing is an image processing technique used to reduce noise or irregularities in an image, making it smoother and more continuous. During smoothing, convolution is used to smooth the input signal through a window to reduce noise and improve signal readability. In multi-stage keyframe extraction, smoothing can be applied to inter-frame difference calculation to reduce the impact of noise on the accuracy of keyframe extraction. The convolution calculation is shown in the following formula:

[0092]

[0093] Where a and v are number sequences, and n is the number of bits to be shifted.

[0094] Step S3: Local Maximum; The smoothed data is processed using local maximums to select keyframes. This method effectively preserves the frames with the greatest inter-frame differences, reducing data redundancy. The n frames with the greatest differences are selected. Then, image enhancement is performed, and n / 2 frames are extracted using frame differencing. This achieves better richness, with the extracted video frames evenly distributed throughout the video.

[0095] 3) Perform noise reduction processing on the selected video frames; noise reduction processing refers to using noise reduction methods to remove noise signals in the image or reduce the impact of noise on image quality, to restore the real details and information of the image as much as possible, while maintaining the smoothness and natural effect of the image.

[0096] 4) Calculate the pixel differences between adjacent frames of the video again using the inter-frame difference method, and select the n / 2 frame with the largest difference as the key frame by using the local maximum method;

[0097] 5) Repeat steps 1)-4) to perform multi-stage keyframe extraction;

[0098] In this embodiment, a two-stage processing method is used to extract keyframes from infrared video images, including the following steps:

[0099] Acquire infrared video frames;

[0100] Infrared video image enhancement: An adaptive histogram equalization algorithm is used to enhance the infrared video frames obtained in step 1. The adaptive histogram equalization algorithm redistributes brightness by calculating the local histogram of the infrared video image, thereby changing the contrast of the infrared video image and enhancing the contrast of its features, thus improving the weakness of infrared image information. After enhancement, the infrared image features are more obvious and have better recognizability. The enhanced infrared video frame images help the model extract features, distinguish the types of infrared video frames, and improve the detection accuracy of similar repeating videos.

[0101] Extract the first stage key frames; use the inter-frame difference method to calculate the pixel difference between adjacent frames of the enhanced infrared video, process the local maximum value of the pixel difference values ​​between all adjacent frames after calculation, and select n frames as the first stage key frames.

[0102] Denoising processing; The first-stage keyframes are processed using image denoising algorithms to reduce noise in the infrared video image and improve signal readability;

[0103] Extract the second-stage keyframes; use the inter-frame difference method to calculate the pixel difference between the first-stage keyframes after denoising in step 4, and use the local maximum method to select n / 2 frames as the second-stage keyframes.

[0104] Step 3: Use the COLLNet-LSTM model to associate the temporal information features between key frames of infrared video images, combine spatial and temporal features, and extract the spatiotemporal joint features of two infrared video image frames;

[0105] Step 4: After processing the spatiotemporal joint features of the two infrared video image frames through a fully connected layer, calculate the cosine similarity between the two infrared videos.

[0106] Step 5: Compare the cosine similarity of the two infrared videos with the set threshold. If the cosine similarity is greater than the threshold, the two infrared videos are considered to be nearly identical and the comparison video is deleted; otherwise, it is not deleted.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A spatio-temporal feature based near-duplicate infrared video cleaning system, characterized in that: The system includes a keyframe extraction module, a spatiotemporal joint feature extraction module, a cosine similarity detection module, and a near-repeating infrared video cleaning module. The keyframe extraction module extracts keyframes from the infrared video. The spatiotemporal joint feature extraction module uses a convolutional neural network to extract spatial features from the infrared video keyframes, flattens and increases the dimensionality of the extracted spatial features, inputs them into a recurrent neural network to correlate with temporal features, and outputs a spatiotemporal multidimensional feature vector. The cosine similarity detection module uses cosine similarity to calculate the similarity of the extracted spatiotemporal features of the infrared video keyframes to identify and detect near-repeating content between video frames. The near-repeating infrared video cleaning module compares the similarity calculated by the cosine similarity detection module with a set threshold; if the similarity is greater than the threshold, the similar infrared video segments are deleted; otherwise, the infrared video segments are retained. The spatiotemporal joint feature extraction module uses a spatiotemporal joint feature extraction method to perform feature extraction processing on two infrared video frame sequences, including a spatial feature extraction unit and a temporal feature extraction unit. The spatial feature extraction unit uses the COLLNet deep network model to extract spatial features from near-repeating infrared video. The temporal feature extraction unit uses COLLNet-LSTM to associate temporal information features between infrared video frames, combining spatial and temporal features. The spatiotemporal joint feature extraction module employs the COLLNet-LSTM model to extract spatiotemporal joint features from near-repeating infrared videos. The COLLNet-LSTM model integrates large-kernel convolution, structure reparameterization, dynamic convolution ODconv, and the lightweight convolutional network model COLLNet to extract spatial features from infrared video data, and associates with the LSTM model for temporal feature extraction. Within the COLLNet-LSTM model, a lightweight convolutional neural network model, COLLNet, is designed, consisting of shallow residual blocks. This reduces feature loss and improves model speed during spatial feature extraction from infrared video data. To ensure good information perception capabilities in the shallow network, the COLLNet deep network model uses large-kernel convolution to extract infrared video frame features, enhancing the network's receptive field and improving its ability to represent fine features. Simultaneously, the COLLNet deep network model employs a full-dimensional dynamic convolution ODConv model, considering the dynamics across four dimensions: spatial domain, input channel, output channel, and convolutional kernel. Through attention across these four dimensions, it achieves complementary feature extraction capabilities across multiple dimensions, enhancing the ability to extract contextual information features.

2. A near-duplicate infrared video cleaning system based on spatio-temporal features according to claim 1, characterized in that: The keyframe extraction module includes an infrared image information enhancement unit, a noise removal unit, and a frame difference calculation unit; the infrared image information enhancement unit is used to enhance the contrast of infrared video image features. The noise removal unit is used to remove noise from the infrared video image to obtain clear infrared image features; the frame difference calculation unit is used to obtain the inter-frame difference of the infrared video and extract key frames.

3. The method according to claim 2, wherein: The COLLNet convolutional network model consists of a lightweight network that combines large convolutional kernels and dynamic convolutional structures, including a 7×7 Conv, a max pooling layer MaxPool, seven lightweight residual blocks, a 3×3 convolutional layer with 512 channels, a LeakyReLU activation function layer, and an average pooling layer. The cropped image is first fed into a 7×7 2D convolutional layer with a stride of 2 for dimensionality reduction; the 7×7 Conv output features are then fed into a max pooling layer to reduce the number of model parameters and improve model training speed. Each of the seven lightweight residual blocks consists of a large kernel convolutional block and a LeakyReLU activation layer; the large kernel convolutional block in the first two lightweight residual blocks has 64 channels and a size of 13×13; the large kernel convolutional block in the middle three lightweight residual blocks has 128 channels and a size of 13×13; and the large kernel convolutional block in the last two lightweight residual blocks has 256 channels and a size of 13×13.

4. The method of claim 3, wherein the method is based on spatio-temporal features. The large kernel convolutional block structure consists of the following five layers: The first layer is the batch normalization layer; The second layer is a 1×1 dynamic convolutional layer structure, used to dynamically process the features of the channel and spatial domain; The third layer consists of a 13×13 large kernel convolutional layer and a 5×5 convolutional layer for structural parameter reparameterization; The fourth layer is the same as the first layer, and it is a batch normalization layer. The fifth layer is a 1×1 dynamic convolutional layer.

5. The near-repeating infrared video cleaning method based on spatiotemporal features according to claim 4, characterized in that: The dynamic convolutional structure includes a global average pooling operation (GAP), a fully connected layer (FC), and activation layers ReLU, Sigmoid, and a Softmax classifier. We shrink the dimensions using global average pooling and generate different attention values ​​using fully connected layers.

6. A near-repeating infrared video cleaning system based on spatiotemporal characteristics according to claim 5, characterized in that: The cosine similarity detection module uses cosine similarity as the similarity of near-repeating infrared videos.

7. A method for near-duplicate infrared video cleaning based on spatio-temporal features, implemented based on the system of claim 1, characterized in that: Includes the following steps: Step 1: Obtain the infrared video frame sequences of the first and second videos; Step 2: Use a multi-stage processing method to extract keyframes from the two infrared video frame sequences; Step 3: Use the COLLNet-LSTM model to associate the temporal information features between key frames of infrared video images, combine spatial and temporal features, and extract the spatiotemporal joint features of two infrared video image frames; Step 4: After processing the spatiotemporal joint features of the two infrared video image frames through a fully connected layer, calculate the cosine similarity between the two infrared videos. Step 5: Compare the cosine similarity of the two infrared videos with the set threshold. If the cosine similarity is greater than the threshold, the two infrared videos are considered to be nearly identical and the comparison video is deleted; otherwise, it is not deleted.

8. The method according to claim 7, wherein the method is characterized by: The specific method for step 2 is as follows: 1) Enhance the infrared video frame sequence to improve the contrast of infrared video image features; The local histogram calculation method is used to redistribute brightness to change image contrast, making object features more obvious and enhancing recognition. 2) Calculate the pixel differences between adjacent frames of the video using the inter-frame difference method, and select the n frames with the largest differences by using the local maximum method; 3) Perform noise reduction processing on the selected video frames; 4) Calculate the pixel differences between adjacent frames of the video again using the inter-frame difference method, and select the n / 2 frames with the largest difference as key frames by using the local maximum method; 5) Repeat steps 1)-4) to perform multi-stage keyframe extraction.