Infrared video small target detection method and system based on temporal coding and decoding structure
Patent Information
- Application Number
- CN202411055444.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-02
AI Technical Summary
The existing infrared object detection algorithms have poor detection effects in low-contrast and high-noise infrared images, especially in complex scenarios, which are difficult to identify small objects, and have high computational costs, making it difficult to meet the needs of infrared video object detection.
Using the infrared video weak object detection method based on YOLOv5 and time codec structure, the spatial and temporal characteristics of image frame sequence are extracted through lightweight feature extraction network, time codec structure and time factor algorithm, which reduces calculation costs and improves detection Accuracy and robustness.
It effectively reduces the calculation cost of infrared video object detection, improves detection accuracy and robustness, and can more accurately identify weak targets in infrared video, and is suitable for infrared object detection in complex scenarios.
Smart Images

Figure CN118570709B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing detection, and in particular relates to an infrared video dim small target detection method and system based on YOLOv5 and a temporal coding and decoding structure. Background Art
[0002] In the modern society moving towards automation and intelligence, infrared target detection technology plays a vital role in many fields such as military, security monitoring, intelligent transportation, environmental monitoring, etc. Infrared target detection uses the difference in radiation characteristics between the target and the background in the infrared band to achieve automatic recognition and positioning of the target; compared with traditional visible light detection technology, infrared detection technology can effectively monitor under all-weather and all-day conditions, especially at night or in harsh environments with low visibility, showing stronger adaptability and superiority.
[0003] With the rapid development of artificial intelligence technology, especially the widespread application of deep learning technology in the field of image processing, infrared target detection algorithms have also made significant progress. Traditional rule-based algorithms are gradually unable to meet the needs of highly complex and highly dynamic scenes, while infrared target detection algorithms based on deep learning can automatically extract features by learning a large amount of data, thereby achieving more accurate and faster target detection.
[0004] Existing deep learning-based object detection algorithms (such as YOLO, Fast-RCNN, DETR, etc.) have outstanding detection capabilities on visible light images, but when applied to infrared images, their detection effects are greatly reduced due to the low contrast and high noise of infrared images. The reason is that the existing object detection algorithms only perform detection on single-frame images, and in single-frame infrared images, the target to be detected is difficult to distinguish from the background. At present, some studies use image frame sequences for object detection in videos. For example, the object detection algorithm C3D proposed in the literature [D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning Spatiotemporal Features with 3D Convolutional Networks” ICCV, pp. 4489–4497, Dec. 2015] uses a 3D convolutional network to extract features of image frame sequences and performs pixel-level object detection on feature maps, but its feature extraction capability is limited, resulting in unsatisfactory detection results. For example, the target detection algorithm I3D proposed in the literature [J.Carreira and A. Zisserman, “Quo Vadis, Action Recognition A New Model and theKinetics Dataset” CVPR, Honolulu, HI: IEEE, Jul. 2017, pp. 4724–4733] is based on a dual-stream network structure and uses optical flow and image frame sequences to detect targets in the video. However, the computational cost of obtaining optical flow information is high.
[0005] In summary, due to the particularity of its imaging principle, infrared images usually have low contrast and more noise. At the same time, the small size, various shapes, and high similarity of radiation characteristics of infrared targets with the background all bring many difficulties to the design and optimization of algorithms. In addition, it is difficult to achieve ideal detection effects by relying on the information of a single frame image when detecting moving objects with low signal-to-noise ratio in extremely complex scenes. In order to overcome these problems and further improve detection accuracy and robustness, the development of new infrared target detection algorithms has become a hot topic and trend in this field. Summary of the invention
[0006] In view of the problems of insufficient single-frame image information in infrared scenes and high computational cost of existing video target detection algorithms, the present invention provides an infrared video weak small target detection method and system based on YOLOv5 and a temporal codec structure. An algorithm combining a lightweight feature extraction network, a temporal codec structure and a time factor is used to reduce the computational cost of infrared video target detection and improve the detection accuracy and robustness of infrared video target detection.
[0007] A method for detecting small infrared target in video based on YOLOv5 and temporal codec structure includes the following steps:
[0008] (1) Obtain infrared video and manually annotate the targets in each frame of the video, while performing data enhancement on the image;
[0009] (2) Divide the infrared video into multiple continuous image frame sequences and construct samples based on the time factors obtained by random sampling, and then divide all samples into training sets and test sets;
[0010] (3) Construct an infrared small target detection model based on YOLOv5 and temporal codec structure, which includes:
[0011] Backbone extraction network, used to extract spatial information features in images;
[0012] The encoder network is used to fuse the spatial information features of all images in the sequence to obtain a spatiotemporal feature map with temporal information;
[0013] The decoder network determines the time encoding according to the time factor, so that it acts on the spatiotemporal feature map as an offset in the time dimension, thereby highlighting the features of one frame of the image to predict the location and category of the target;
[0014] (4) Using the training set to train the above infrared dim target detection model to determine the optimal model parameters and save them;
[0015] (5) By inputting the test set samples into the trained model, the weak infrared targets and their locations can be detected.
[0016] Furthermore, the specific implementation method of step (1) is as follows: for each frame of the infrared video, the target in the image is manually labeled in the form of a rectangular frame using LabelImage software and the position coordinates of the upper left corner and the lower right corner of the rectangular frame are recorded, and then the normalized height, width and center point coordinates of the rectangular frame are calculated together with the category of the target (the infrared weak target or interference object to be detected) as the image annotation information; then the image is subjected to random noise data enhancement processing based on normal distribution, that is, the image is injected with randomly sampled noise, so as to simulate the detection environment containing infrared weak targets under complex background conditions.
[0017] Furthermore, the normalized height, width and center coordinates of the rectangle are calculated using the following expressions:
[0018] ;
[0019]
[0020] in: and They are the position coordinates of the upper left corner and lower right corner of the rectangular box, width and height are the width and height of the image, respectively. is the normalized center coordinate of the rectangular frame, w and h are the normalized width and height of the rectangular frame respectively.
[0021] Furthermore, the specific implementation of step (2) is as follows:
[0022] 2.1 Divide the infrared video into multiple continuous image frame sequences;
[0023] 2.2 For any image frame sequence, randomly select an image from it p , to ensure the image p There are T-1 frames of images before and after, where T is the set image sequence length;
[0024] 2.3 Get image p And the subsequent T-1 frame images constitute the image sequence P;
[0025] 2.4 Randomly sample a time factor t (a natural number) from the interval (1, T), and use the annotation information of the t-th frame image in the image sequence P as the true value label of the sample;
[0026] 2.5 The true value label, time factor t and image sequence P are combined into a sample, and then all samples are divided into a training set and a test set in proportion.
[0027] Furthermore, the backbone extraction network is based on a lightweight Darknet of YOLOv5, which uses 3D convolution, 3D pooling and 3D batch normalization operations. The convolution kernel size of the 3D convolution in the time dimension is 1, and only spatial information of the image is extracted without fusing the time information. The convolution kernel size of the 3D convolution used by the encoder network in the time dimension is 3. The decoder network selectively highlights the features of the t-th frame image from the spatiotemporal feature map and performs target detection on the t-th frame image, where t is the time factor corresponding to the image sequence.
[0028] Furthermore, the specific implementation of step (4) is as follows:
[0029] 4.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer;
[0030] 4.2 Input the image sequence P and time factor t in the training set samples into the model, forward propagate the model output to obtain the corresponding prediction result, i.e. the location and category of the target, and calculate the loss function between the prediction result and the true value label;
[0031] 4.3 According to the loss function, the optimizer is used to iteratively update the model parameters through the gradient descent method until the loss function converges or the maximum number of iterations is reached, and the training is completed.
[0032] An infrared video small target detection system based on YOLOv5 and temporal codec structure, comprising:
[0033] The acquisition module is used to acquire infrared video and manually mark the targets in each frame of the video, and perform data enhancement processing on the image;
[0034] A partitioning module is used to divide the infrared video into multiple continuous image frame sequences and construct samples based on the time factor obtained by random sampling, and then divide all samples into a training set and a test set;
[0035] The detection network module stores an infrared small target detection model based on YOLOv5 and a temporal codec structure. The model includes a backbone extraction network, an encoder network, and a decoder network. The backbone extraction network is used to extract spatial information features in the image; the encoder network is used to fuse the spatial information features of all images in the sequence to obtain a spatiotemporal feature map with temporal information; the decoder network determines the time coding according to the time factor, so that it acts on the spatiotemporal feature map as an offset in the time dimension, thereby highlighting the features of one frame of the image to predict the location and category of the target;
[0036] A training module uses a training set to train the infrared small target detection model to determine and save the optimal model parameters;
[0037] The detection module is used to input the test set samples into the trained model to detect the infrared weak targets and their positions.
[0038] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned infrared video small target detection method.
[0039] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the infrared video small target detection method is implemented.
[0040] The present invention designs a method for detecting small infrared video targets based on YOLOv5 and a temporal codec structure, which includes an image frame sequence training strategy, a temporal factor sampling strategy, and a codec training mechanism, so as to utilize the temporal and spatial information between images for more accurate target detection. On the one hand, the image frame sequence training strategy captures the temporal and spatial relationship between image frame sequences in the video and utilizes temporal context information to convert the training samples from a traditional single frame to multiple frames; on the other hand, the temporal factor sampling strategy and the codec training mechanism specify the temporal factor and perform training according to the corresponding annotation value, so that the network model has temporal correlation; at the same time, the codec structure has a strong coding capability and can effectively perform feature extraction and target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 The figure is a flow chart of a method for detecting small targets in infrared video based on YOLOv5 and a temporal codec structure.
[0042] Figure 2 for Figure 1 Detailed expansion flow chart of step S11 in FIG.
[0043] Figure 3 for Figure 1 Detailed expansion flow chart of step S12 in FIG.
[0044] Figure 4 for Figure 1 Detailed expansion flow chart of step S13 in FIG.
[0045] Figure 5 for Figure 1 Detailed expansion flow chart of step S14 in FIG.
[0046] Figure 6 for Figure 1 Detailed expansion flow chart of step S15 in FIG.
[0047] Figure 7It is a structural block diagram of the infrared video small target detection system of the present invention.
[0048] Figure 8 The present invention is a schematic diagram of the structure of an electronic device implemented according to the method of the present invention. DETAILED DESCRIPTION
[0049] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0050] like Figure 1 As shown, the infrared video weak small target detection method based on YOLOv5 and time codec structure of the present invention comprises the following steps:
[0051] Step S1: Obtain an infrared small target video, cut the infrared small target video into an image frame sequence and save it, manually annotate each frame, normalize the annotated information, and perform data enhancement operations on the image frame.
[0052] like Figure 2 As shown, the specific implementation process of this step is as follows:
[0053] S11: Obtain an infrared video of a small target to be detected. The infrared video is composed of a series of continuous image frames. The entire infrared video is cut into several independent image frames and saved.
[0054] S12: According to the obtained image frame, the infrared weak small target in the image frame is manually labeled and normalized using LabelImage software. According to the different labeled information, it can be divided into two categories: target and interference. The labeled information includes the position of the upper left corner and the lower right corner of the object. The specific normalization formula is:
[0055] ;
[0056]
[0057] in: and Respectively represent the position coordinates of the upper left corner and lower right corner of the annotation box, width and height is the width and height of the image frame, is the normalized coordinate of the center point of the annotation box, w and h is the normalized width and height of the annotation box.
[0058] S13: According to the acquired image frame, a random noise data enhancement method based on normal distribution is implemented, and the specific algorithm formula is:
[0059]
[0060] in: p is the original image, q is the normally distributed noise, is the enhanced image; under this strategy, randomly sampled noise is injected into each image to simulate the infrared weak target detection environment under complex background conditions.
[0061] Step S2: According to the image frame sequence, a training set and a test set are divided, and a sample division strategy is designed so that each sample contains a continuous image frame sequence, a time factor is sampled based on the length of the image frame sequence, and standard information of the current sample is obtained according to the sampled time factor.
[0062] like Figure 3 As shown, the specific implementation process of this step is as follows:
[0063] S21: Randomly select an image from an image frame sequence, and ensure that there are T-1 frames of images before this image, where T is a manually set hyperparameter, representing the length of the image frame sequence in a training sample; the resulting image frame sequence is represented as (S1, S2, S3, …, S T ), where this T frame sequence is a continuous T frame image sequence in a video.
[0064] S22: Take out the image selected in S21 and the subsequent T-1 frame image to form an image frame sequence with a length of T.
[0065] S23: Randomly select a number t from the uniform distribution of (1, T) as the time factor, that is, t~u(1, T), and convert the image frame sequence (S1, S2, S3, …, S T )'s tth frame S t The annotation information is used as the true value of this image sequence.
[0066] S24: Transform the image sequence (S1, S2, S3, …, S T ) and the tth frame S t The labeled information is used as a training sample and input into the subsequent network.
[0067] Step S3: According to the divided training set, an infrared dim small target detection network model is designed. The model is based on YOLOv5 and the temporal codec structure, and can detect the infrared image frame sequence in the training sample to obtain the location and category information of the target.
[0068] like Figure 4 As shown, the specific implementation process of this step is as follows:
[0069] S31: The infrared small target detection network model consists of three modules: backbone extraction network BackBone, encoder network Encoder and decoder network Decoder.
[0070] S32: The backbone feature extraction network Backbone of the infrared small target detection network model is based on the lightweight Darknet of YOLOv5. The network uses 3D convolution operations and 3D pooling operations to extract spatial information features. Unlike the Darknet in YOLOv5, the Darknet in this module uses 3D convolution, 3D pooling and 3D batch normalization operations, and the convolution kernel shape in the time dimension is 1, that is, the time information is not fused in this step, only the spatial information is extracted.
[0071] S33: The encoder network of the infrared small target detection network model, Encoder, fuses the spatial information features of the image frame sequence to obtain a feature map with temporal information; in the 3D convolution kernel used in the Encoder, the convolution kernel shape of the time dimension is 3, which fuses the time and space information to obtain a fused spatiotemporal feature map of the T frame image.
[0072] S34: Decoder network of infrared small target detection network model, obtains time code according to the time factor obtained by random sampling, superimposes the feature map obtained in S33 with the time code and predicts the target position and category. The spatiotemporal feature map is the fusion feature map of T frame images. The time factor t is added, and the features of the tth frame image are selectively highlighted from the spatiotemporal feature map in the decoder, and the target detection is performed on the tth frame image.
[0073] Step S4: Use the training set to train the infrared small target detection network model, obtain various parameters of the network model, and obtain a trained network detection model.
[0074] like Figure 5 As shown, the specific implementation process of this step is as follows:
[0075] S41: extract training samples from the training set to obtain image frame sequences, time factors and true values for subsequent training;
[0076] S42: The image frame sequence and the time factor are sent as input to the infrared small target detection network model, and the obtained predicted value and the true value in S41 are used as input of the loss function.
[0077] Step S5: Use the test data set to test the trained detection network model and evaluate the test results.
[0078] like Figure 6As shown, the specific implementation process of this step is as follows:
[0079] S51: Take a test sample from the test set to obtain a test sample image frame sequence, manually set the time factor to the last frame image in the image frame sequence, and the true value to the labeled value of the last frame image;
[0080] S52: The image frame sequence and the time factor are sent as input to the trained infrared small target detection network model to obtain detection information and calculate the mAP value.
[0081] Corresponding to the aforementioned infrared video small target detection method embodiment, the present invention also provides an infrared video small target detection system embodiment.
[0082] like Figure 7 As shown, the infrared video small target detection system of this embodiment includes:
[0083] The acquisition module 21 acquires the infrared weak small target video, divides the infrared weak small target video into an image frame sequence and saves it, manually annotates each frame, normalizes the annotated information, and performs data enhancement operations on the image frame;
[0084] A partitioning module 22 is configured to partition a training set and a test set according to the image frame sequence, design a sample partitioning strategy so that each sample contains a continuous image frame sequence, sample a time factor based on the length of the image frame sequence, and obtain standard information of the current sample according to the sampled time factor;
[0085] The detection network module 23 designs an infrared dim target detection network model according to the divided training set. The model is based on YOLOv5 and a temporal codec structure and can detect infrared image frame sequences in the training samples to obtain the location and category information of the target.
[0086] The training module 24 uses the training set to train the infrared small target detection network model, obtains various parameters of the network model, and obtains the trained network detection model;
[0087] The detection module 25 uses the test data set to test the trained detection network model and evaluate the test results.
[0088] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0089] Accordingly, the present invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-mentioned infrared video small target detection method. Figure 8 The figure shows a hardware structure of any electronic device with data processing capability in which the infrared video small target detection system provided by the embodiment of the present invention is located. Figure 8 In addition to the processor, memory and network interface shown, any electronic device with data processing capability in which the system in the embodiments is located may also include other hardware according to the actual functions of the electronic device, which will not be described in detail.
[0090] Accordingly, the present invention also provides a computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned infrared video weak small target detection method is implemented. The computer-readable storage medium can be an internal storage unit of the aforementioned electronic device with data processing capability, such as a hard disk or memory, or it can be other external storage devices, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. The computer-readable storage medium can also include both an internal storage unit of an electronic device with data processing capability and an external storage device, which is used to store the computer program and other programs and data required by the electronic device with data processing capability, and can also be used to temporarily store data that has been output or is to be output.
[0091] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.
Claims
1. A method for detecting small infrared target in video based on YOLOv5 and temporal codec structure, comprising the following steps: (1) Obtain infrared video and manually annotate the targets in each frame of the video, and perform data enhancement processing on the image. Specifically, for each frame of the infrared video, use LabelImage software to manually annotate the targets in the image in the form of a rectangular frame and record the position coordinates of the upper left corner and lower right corner of the rectangular frame. Then, use the following expression to calculate the normalized height, width and center point coordinates of the rectangular frame together with the target category as the image annotation information; then, perform random noise data enhancement processing based on normal distribution on the image, that is, inject randomly sampled noise into the image to simulate the detection environment containing infrared weak targets under complex background conditions; ; ; in: and They are the position coordinates of the upper left corner and lower right corner of the rectangular box, width and height are the width and height of the image, respectively. is the normalized center coordinate of the rectangular frame, w and h They are the normalized width and height of the rectangular frame respectively; (2) The infrared video is divided into multiple continuous image frame sequences and samples are constructed based on the time factor obtained by random sampling, and then all samples are divided into training sets and test sets. The specific implementation method is as follows: 2.1 Divide the infrared video into multiple continuous image frame sequences; 2.2 For any image frame sequence, randomly select an image from it p , to ensure the image p There are T-1 frames of images before and after, where T is the set image sequence length; 2.3 Get image p And the subsequent T-1 frame images constitute the image sequence P; 2.4 Randomly sample a time factor t from the interval (1, T), and use the annotation information of the t-th frame image in the image sequence P as the true value label of the sample; 2.5 The true value label, the time factor t and the image sequence P are combined into a sample, and then all samples are divided into a training set and a test set according to proportion; (3) Construct an infrared small target detection model based on YOLOv5 and temporal codec structure, which includes: Backbone extraction network, used to extract spatial information features in images; The encoder network is used to fuse the spatial information features of all images in the sequence to obtain a spatiotemporal feature map with temporal information; The decoder network determines the time encoding according to the time factor, so that it acts on the spatiotemporal feature map as an offset in the time dimension, thereby highlighting the features of one frame of the image to predict the location and category of the target; The backbone extraction network is based on a lightweight Darknet of YOLOv5, which uses 3D convolution, 3D pooling and 3D batch normalization operations. The convolution kernel size of the 3D convolution in the time dimension is 1, and only the spatial information of the image is extracted without fusing the time information. The convolution kernel size of the 3D convolution used by the encoder network in the time dimension is 3. The decoder network selectively highlights the features of the t-th frame image from the spatiotemporal feature map and performs target detection on the t-th frame image, where t is the time factor corresponding to the image sequence. (4) Using the training set to train the above infrared dim target detection model to determine the optimal model parameters and save them; (5) By inputting the test set samples into the trained model, the weak infrared targets and their locations can be detected.
2. According to claim 1, a method for detecting small infrared target in video based on YOLOv5 and temporal codec structure, Features: The specific implementation of step (4) is as follows: 4.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer; 4.2 Input the image sequence P and time factor t in the training set samples into the model, forward propagate the model output to obtain the corresponding prediction result, i.e. the location and category of the target, and calculate the loss function between the prediction result and the true value label; 4.3 According to the loss function, the optimizer is used to iteratively update the model parameters through the gradient descent method until the loss function converges or the maximum number of iterations is reached, and the training is completed.
3. An infrared video small target detection system based on YOLOv5 and temporal codec structure, Features: The system implements the infrared video small target detection method as claimed in claim 1 or 2, and the system comprises: The acquisition module is used to acquire infrared video and manually mark the targets in each frame of the video, and perform data enhancement processing on the image; A partitioning module is used to divide the infrared video into multiple continuous image frame sequences and construct samples based on the time factor obtained by random sampling, and then divide all samples into a training set and a test set; The detection network module stores an infrared small target detection model based on YOLOv5 and a temporal codec structure. The model includes a backbone extraction network, an encoder network, and a decoder network. The backbone extraction network is used to extract spatial information features in the image; the encoder network is used to fuse the spatial information features of all images in the sequence to obtain a spatiotemporal feature map with temporal information; the decoder network determines the time coding according to the time factor, so that it acts on the spatiotemporal feature map as an offset in the time dimension, thereby highlighting the features of one frame of the image to predict the location and category of the target; A training module uses a training set to train the infrared small target detection model to determine and save the optimal model parameters; The detection module is used to input the test set samples into the trained model to detect the infrared weak targets and their positions.
