Instantaneous target recognition method based on small sample of video

Abstract

The present invention provides a method for instantaneous target recognition based on small video samples, comprising: extracting continuous image information from a historical video sample containing a target; dividing the continuous image of each frame sample into n identical small images according to the proportion of the target pixels to the total image pixels, and inputting them into a positive sample set and a negative sample set into a binary classification neural network model for pre-training; when the loss function value of the binary classification neural network model is less than or equal to a threshold, the current binary classification neural network model is used as the final binary classification neural network model; deploying the final binary classification neural network model to a tf-serving server, starting multiple target recognition threads in parallel, extracting a frame of image information from the real-time video signal, dividing the frame of image information into n small images, and assigning them to multiple target recognition threads for image classification; and immediately triggering a target alarm once the target recognition thread emits a positive sample signal.

Description

Technical Field

[0001] This invention belongs to the field of target recognition in artificial intelligence technology, specifically relating to a method for instantaneous target recognition based on small video samples. Background Technology

[0002] In recent years, high-speed cameras have become increasingly common in scientific research, military testing, and industrial production assessment. Compared to the efficiency of ordinary cameras, which captures around 24 frames per second, high-speed cameras can achieve 1,000 to 10,000 frames per second. Some military-specific high-speed cameras can even reach 1 million to 10 million frames per second. The higher the frame rate, the better it is at capturing the motion trajectories of fast-moving objects, making it easier to capture instantaneous dynamic processes invisible to the human eye in real life. Therefore, high-speed cameras can achieve rapid, multiple sampling of high-speed targets in a very short time. The advantages of high-speed cameras are obvious, especially in the military field. For capturing low-altitude, fast-moving, hidden, and small targets appearing in the camera's view, high-speed cameras can be used to capture images and then utilize target recognition algorithms to provide alerts for the captured targets. With the large-scale development and application of deep learning in artificial intelligence, supervised learning, which relies on training with large amounts of labeled data, is widely used in the field of video target recognition. Deep learning models such as Fast-RCNN and the YOLO series have achieved significant success. However, in the military field, due to the industry's specific nature, it is often impossible to obtain a sufficient amount of labeled data to support the training of deep learning neural network models. Furthermore, the military's need for target identification and alerting typically requires high timeliness, sometimes even millisecond-level instantaneous target identification tasks. Most current target identification methods rely on extremely large datasets, but in the military, methods such as inter-frame differencing and background differencing are commonly used for target identification. The key to these methods is extracting target contour features, but they are greatly affected by the background environment and often cannot adapt to target identification in dynamic backgrounds, exhibiting poor adaptability and robustness in real-world applications. This patent proposes a few-shot learning and instantaneous target identification method based on a small number of labeled video samples, which can specifically address the issues of sample size, adaptability, and robustness. Therefore, a few-shot learning and instantaneous target identification method based on a small number of labeled video samples is one of the key solutions to the problem of artificial intelligence not being able to truly function effectively in certain military tasks. Summary of the Invention

[0003] The purpose of this invention is to address the aforementioned problems in the prior art by providing a method for instantaneous target recognition based on small video samples. This invention achieves good algorithmic performance for instantaneous target recognition and alerting in small-sample videos, realizes target dataset expansion based on single-video small samples, and instantaneous target recognition based on parallel algorithm calls, thereby improving the algorithm's generalization ability and the timeliness of target alert prompts.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] The present invention provides a method for instantaneous target recognition based on small video samples, wherein it includes the following steps:

[0006] (i) Extract at least 125 consecutive frames of image information from a historical video sample containing the target;

[0007] (ii) Based on the proportion of the target pixels in each frame of the image information to the total number of pixels in the image, design the size of the sliding window to ensure that the target is complete within the sliding window. Based on the size of the sliding window, divide the continuous image of each frame of the sample into n identical small images.

[0008] (iii) Based on the small images segmented in step (ii), the small images containing the target are included in the positive sample set, and the small images not containing the target are included in the negative sample set. The positive sample set and the negative sample set together form the target dataset for training the neural network.

[0009] (iv) Expand the positive sample set from step (iii) so that the ratio of the number of positive samples to the number of negative samples is at least 2:3. Combine the negative sample set and the expanded positive sample set to form the target dataset for neural network training. Input the target dataset into the binary classification neural network model for pre-training. Adjust the weight parameters of the binary classification neural network model. When the loss function value of the binary classification neural network model is greater than the threshold, repeat step (i) and continue training the binary classification neural network model by increasing the number of frame samples. At the same time, iteratively calculate the loss function value of the binary classification neural network model. When the loss function value of the binary classification neural network model is less than or equal to the threshold, it proves that the binary classification neural network model has reached the fitting state. Save the weight parameters of the binary classification neural network model at this time and use the current binary classification neural network model as the final binary classification neural network model.

[0010] (v) Deploy the final binary classification neural network model to the tf-serving server, start multiple target recognition threads in parallel, and deploy a binary classification neural network model for each target recognition thread;

[0011] (vi) Extract a frame of image information from the real-time video signal, and use the sliding window size in step (ii) to divide the frame of image information into n small images. Assign the n small images to the binary classification neural network models of multiple target recognition threads for image classification. Once two or more binary classification neural network models in the target recognition threads emit positive sample signals, or one of the binary classification neural network models in the target recognition threads emits two or more positive sample signals, a target alarm is immediately triggered; otherwise, continue monitoring until the end and exit.

[0012] The present invention provides a method for instantaneous target recognition based on video small samples, wherein: in steps (ii) and (vi), each frame of image is segmented using the following steps:

[0013] (a) Define the target within a rectangle with pixel size w×h. Set the length of the sliding window to w′≈w+ξ and the width to h′≈h+ξ, where ξ is set to 5 pixels.

[0014] (b) Extract each frame image. Based on the sliding window size and the set overlap rate (iou) between adjacent small images (the overlap rate is the proportion of overlap between adjacent small images), segment each frame image from top to bottom and from left to right and save it as a total of n small image samples p of the same size. i This forms a sample set P = {p1, p2, ..., p...} N};

[0015] Expressed by formula (1)

[0016] n=(W*H) / (w'*h'*iou 2 )-----Formula (1)

[0017] Where: w' is the width of the sliding window; h' is the length of the sliding window; iou is the overlap ratio; W is the width of each frame; H is the length of each frame; and n is the total number of small images that each frame is divided into.

[0018] The present invention provides a method for instantaneous target recognition based on video small samples, wherein: when training a binary classification neural network model, the negative sample set and the expanded positive sample set from step (iv) are input, and the output is 1 or 0, where 1 represents a positive sample and 0 represents a negative sample; in step (vi), n small images segmented by real-time video signals are input to the final binary classification neural network model, and when the output of the final binary classification neural network model is 1, the binary classification neural network model in the target recognition thread emits a positive sample signal.

[0019] The present invention provides a method for instantaneous target recognition based on video small samples, wherein: in steps (i) and (vi), continuous image information is extracted and saved using the OpenCV tool library.

[0020] The present invention provides a method for instantaneous target recognition based on small video samples, wherein the plurality of target recognition threads are at least two or more target recognition threads.

[0021] The present invention provides a method for instantaneous target recognition based on small video samples, wherein: in step (iii), the threshold is 10. -3 .

[0022] The present invention provides a method for instantaneous target recognition based on small video samples, wherein the overlap rate is 0.5-0.8.

[0023] The present invention provides a method for instantaneous target recognition based on small video samples, wherein: in step (iv), the positive sample set expansion includes: rotation, scaling, translation transformation, Gaussian blur or motion blur processing.

[0024] The present invention provides a method for instantaneous target recognition based on small video samples, wherein: the binary classification neural network is ResNet50, and the binary classification neural network includes: an input layer, a pooling layer, a convolutional layer, a fully connected layer, a classification layer, and an output layer.

[0025] The present invention provides a method for instantaneous target recognition based on small video samples, wherein: the positive sample set is a small image containing the target, and the small image includes at least 5% of the complete small image.

[0026] Beneficial effects

[0027] The instantaneous target recognition method based on small video samples of the present invention has the following advantages compared with traditional target recognition methods:

[0028] 1. Traditional video target detection and recognition models rely on a large amount of data for training deep learning models. However, the target classification model based on small video samples proposed in this patent only needs to obtain a few or even a single video sample. Furthermore, it uses data augmentation and other sample expansion methods only for target images, which can greatly reduce the impact of background noise on the model accuracy and effectively solve the bottleneck problem of limited data samples available for training in the military field.

[0029] 2. Deep learning-based object detection models often have complex network structures. Replacing object detection models with deep learning object classification models can effectively reduce computational difficulty and reduce the problem-solving approach from the original complex regression problem to a simple classification problem. This effectively solves the problems of high time and space complexity and poor algorithm efficiency in object detection algorithms.

[0030] 3. Targets that need to be detected and alerted via video are often small or fast-moving and may disappear in an instant. The multi-threaded algorithm recognition program deployed by tf-serving proposed in this patent effectively enhances the timeliness of target detection compared to the previous single-threaded algorithm recognition. A single thread only needs to process one-tenth of the original amount of data, which greatly reduces the model inference time. Finally, setting multiple threads to return a value of 1 before triggering an alarm effectively reduces the missed detection and false detection rates, and has a significant effect on improving the robustness of the algorithm. Attached Figure Description

[0031] Figure 1 This is a flowchart of the instantaneous target recognition method based on small video samples according to the present invention. Detailed Implementation

[0032] Here is an implementation case, taking the recognition of an underwater weapon firing action as an example. The specific implementation steps of a method for instantaneous target recognition based on a single video sample are as follows:

[0033] The present invention provides a method for instantaneous target recognition based on small video samples, wherein it includes the following steps:

[0034] (i) Extract at least 125 frames of continuous image information from a historical video sample containing the target, and use the OpenCV tool library to extract and save the continuous image information;

[0035] (II) Based on the proportion of target pixels to the total number of pixels in each frame of the image, design the size of the sliding window to ensure that the target is complete within the sliding window. According to the size of the sliding window, divide the continuous image of each frame of the sample into n identical small images. Perform the following steps to segment each frame of the image:

[0036] (a) Define the target within a rectangle with pixel size w×h. Set the length of the sliding window to w′≈w+ξ and the width to h′≈h+ξ, where ξ is set to 5 pixels.

[0037] (b) Extract each frame image. Based on the aforementioned sliding window size and the set overlap rate (iou) between adjacent small images (overlap rate is the proportion of overlap between adjacent small images, ranging from 0.5 to 0.8, with an overlap rate of 0.7), segment each frame image from top to bottom and from left to right, and save it as a total of n small image samples of the same size p. i This forms a sample set P = {p1, p2, ..., p...} N};

[0038] Expressed by formula (1)

[0039] n=(W*H) / (w'*h'*iou2 )-----Formula (1)

[0040] Where: w' is the width of the sliding window; h' is the length of the sliding window; iou is the overlap ratio; W is the width of each frame; H is the length of each frame; n is the total number of small images that each frame is divided into;

[0041] The positive sample set consists of small images containing the target, which must include at least 5% of the complete small image.

[0042] (iii) Based on the small images segmented in step (ii), the small images containing the target are included in the positive sample set, and the small images not containing the target are included in the negative sample set. The positive sample set and the negative sample set together form the target dataset for training the neural network.

[0043] (iv) Because the number of positive samples is much smaller than the number of negative samples, the positive sample set from step (iii) needs to be expanded. This expansion can be achieved through rotation, scaling, translation, Gaussian blur, or motion blur, ensuring that the ratio of positive to negative samples is at least 2:3. The target classification algorithm model is then obtained, as shown in Table 1. A classification accuracy of over 98% was achieved on the test samples.

[0044] Dataset Sample size Positive sample size negative sample number resolution Before data augmentation 4046 532 3541 320×320 After data augmentation 6032 2491 3541 320×320

[0045] The negative sample set and the expanded positive sample set are combined to form the target dataset for training the neural network. This target dataset is then input into a binary classification neural network model (ResNet50) for pre-training. The binary classification neural network includes an input layer, pooling layer, convolutional layer, fully connected layer, classification layer, and output layer. Its output is either 1 or 0, where 1 represents a positive sample and 0 represents a negative sample. The weight parameters of the binary classification neural network model are adjusted until the loss function value of the model is greater than 10. -3 Then, repeat step (1), increasing the number of frame samples mentioned above to continue training the binary classification neural network model, while iteratively calculating the loss function value of the binary classification neural network model; when the loss function value of the binary classification neural network model is less than or equal to 10... -3 When the binary classification neural network model has reached the fitting state, the weight parameters of the corresponding binary classification neural network model are saved, and the current binary classification neural network model is used as the final binary classification neural network model.

[0046] (v) Deploy the final binary classification neural network model to the tf-serving server, and start 10 target recognition threads in parallel, with each target recognition thread deploying a binary classification neural network model;

[0047] (vi) Extract a frame of image information from the real-time video signal, and use the sliding window size in step (ii) to divide the frame of image information into n small images. Distribute the n small images equally to the binary classification neural network models of 10 target recognition threads for image classification. When the final output of the binary classification neural network model is 1, the binary classification neural network model in the target recognition thread emits a positive sample signal. Once two or more binary classification neural network models in the target recognition threads emit positive sample signals, or one of the binary classification neural network models in the target recognition thread emits two or more positive sample signals, a target alarm is immediately triggered; otherwise, continue monitoring until the end and exit.

[0048] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

Claims

1. A method for video-based on small sample instantaneous target recognition, characterized in that: It comprises the following steps: (I) extracting at least 125 frames of continuous image information from a historical video sample containing a target; (II) designing the size of the sliding window according to the proportion of the target pixels in the entire image in each frame of image information, ensuring that the target is complete in the sliding window, and dividing each frame of sample image into n identical small images according to the size of the sliding window; (III) listing the small images containing the target in the positive sample set and the small images not containing the target in the negative sample set according to the small images divided in step (II), and the positive sample set and the negative sample set together form a target data set for neural network training; (IV) expanding the positive sample set in step (III) so that the ratio of the number of positive sample set to the number of negative sample set is at least 2:3, and the negative sample set and the expanded positive sample set together form a target data set for neural network training, inputting the target data set into a binary classification neural network model for pre-training, adjusting the weight parameters of the binary classification neural network model, when the loss function value of the binary classification neural network model is greater than the threshold value, repeating step (I) to continue training the binary classification neural network model by increasing the number of frames of sample, and iteratively calculating the loss function value of the binary classification neural network model; when the loss function value of the binary classification neural network model is less than or equal to the threshold value, it is proved that the binary classification neural network model has reached the fitting state, and the weight parameters of the binary classification neural network model at this time are saved, and the current binary classification neural network model is used as the final binary classification neural network model; (V) deploying the final binary classification neural network model into a tf-serving server, and starting multiple target recognition threads in parallel, each of which deploys a binary classification neural network model; (VI) extracting a frame of image information from a real-time video signal, dividing the frame of image information into n small images using the sliding window size in step (II), and assigning the n small images to the binary classification neural network models of multiple target recognition threads for image classification, and once two or more binary classification neural network models in the target recognition threads send positive sample signals, or one binary classification neural network model in the target recognition thread sends two or more positive sample signals, target alarm is performed; otherwise, continue to listen until the end and exit.

2. The method of claim 1, wherein: In steps (II) and (VI), each frame of image is segmented by the following steps: (a) frame the target in a rectangle, and the pixel size of the rectangle is w x h, and the length of the sliding window is w' ≈ w + ξ and the width is h' ≈ h + ξ, wherein ξ is set to 5 pixels; (b) taking out each frame image, according to the sliding window size and the set overlap rate (iou) between adjacent small images, the overlap rate being the overlap ratio between adjacent small images, dividing and saving each frame image into n small image samples p of the same size from top to bottom and from left to right i , forming a sample set P = {p1, p2, …, p N} which is expressed by formula (1) n = (W*H) / (w'*h'*iou 2 ) ----- Equation (1) wherein: w' is the width of the sliding window; h' is the length of the sliding window; iou is the overlap rate; W is the width of each frame of image; H is the length of each frame of image; n is the total number of small images divided from each frame of image.

3. The method of claim 2, wherein: In the training of the binary classification neural network model, the negative sample set and the expanded positive sample set of step (four) are input, and the output is 1 or 0, 1 representing a positive sample and 0 representing a negative sample; in step (six), the final binary classification neural network model is input with n small images segmented from real-time video signals, and the binary classification neural network model in the target recognition thread sends a positive sample signal when the final binary classification neural network model outputs 1.

4. The method of claim 3, wherein: In steps (one) and (six), continuous image information is extracted and saved by using the OpenCV tool library.

5. The method of claim 4, wherein: The plurality of target recognition threads are at least two or more target recognition threads.

6. The method of claim 5, wherein: In step (three), the threshold value is 10 -3 .

7. The method of claim 6, wherein: The overlap rate is 0.5-0.

8.

8. The method of claim 7, wherein: In step (four), the positive sample set expansion includes rotation, scaling, translation transformation, Gaussian blur or motion blur processing.

9. The method of claim 8, wherein: The binary classification neural network is ResNet50, and the binary classification neural network comprises an input layer, a pooling layer, a convolutional layer, a fully connected layer, a classification layer and an output layer.

10. The method of claim 9, wherein: The positive sample set is a small image containing a target, which includes at least 5% of the complete small image.

Images