A real-time target detection method for a video stream of a discontinuous sampling sequence

Abstract

A real-time target detection method for a video stream of a discontinuous sampling sequence, comprising the following steps: analyzing a discontinuous sequence of video frames by a target detection network, the target detection network using data features of a previous frame collected before a current analyzed frame, supplementing data features of the current frame, and finally the target detection network obtaining a detection result through the supplemented features; a network training framework is used to collect and generate discontinuous video frame sequence data from a video as training data, and the target detection network is trained to obtain model parameters of a neural network for target detection; a video stream processing analysis framework adopts a pipeline operation mode, divides the trained target detection network into multiple parallel parts, and performs pipeline target detection on the video stream; and a limited-depth queue is used to retain features of frames that have been detected to supplement subsequent data. The application can better and more accurately perform target detection.

Description

Technical Field

[0001] This invention belongs to the field of real-time target detection technology for video streams, specifically relating to a real-time target detection method for video streams with discontinuous sampling sequences. Background Technology

[0002] With the advancement of computer vision technologies and the development of current social science and technology, computer vision has gradually played an important role in social life. Computer vision algorithms, including object detection, have played a crucial role in fields such as security, autonomous driving, intelligent transportation, and field inspection. Current object detection algorithms can perform near real-time object detection on trained targets with high accuracy. However, current object detection algorithms, such as patent CN115063669A, only perform independent detection on each frame of a video, failing to effectively utilize the correlation between frames. In many scenarios, the target being detected is very similar to the background, causing the target to be undetectable. In such cases, such as in the wild, in cities, and in non-visible light fields, two problems often arise: firstly, the target cannot be correctly detected; secondly, the background is incorrectly identified as the target. Since targets or backgrounds usually move, humans can identify targets from consecutive video frames.

[0003] Traditional object detection algorithms are based on detecting single-frame images. They typically detect all images in a video stream, but when the processing speed cannot keep up with the video transmission speed, some video frames have to be discarded.

[0004] Traditional video processing methods typically use all videos within a certain time period for analysis. However, these videos often have high similarity when the selected time period is short, or the required computing resources increase when the selected time period is long.

[0005] Existing deep learning-based object detection algorithms rarely utilize pipelined implementations. Current algorithms, in pursuit of a balance between performance and computational speed, suffer from two limitations: firstly, the computational load of the object detection task cannot fully utilize all computational resources; secondly, because the next frame can only be detected after the current video frame is completed, the frame rate for video detection is limited by the frame detection speed. Furthermore, this trade-off leads to model size constraints imposed by factors such as detection speed, resulting in reduced model detection accuracy.

[0006] Current solutions involve introducing additional data modalities for processing in specific scenarios. For example, in a field where theft of fruit is being prevented, infrared sensors are typically added to avoid detecting thieves who are dressed in clothing similar to the environment. These sensors utilize the heat generated by the human body. However, this increases equipment and maintenance costs, which can be a burden for the target group of the system. Summary of the Invention

[0007] In order to overcome the shortcomings of the existing technology, the present invention aims to provide a real-time target detection method for video streams with discontinuous sampling sequences. By introducing historical image information and comparing the current image with previous image information as a supplement, the target detection can be performed better and more accurately.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A real-time target detection method for video streams with discontinuous sampling sequences includes the following steps;

[0010] Step (1): The discontinuous video frame sequence is analyzed by the target detection network. The target detection network uses the data features of the previous frame collected before the current frame being analyzed to supplement the data features of the current frame with the features and information contained in the previous frame. Finally, the target detection network obtains the detection result through the supplemented features.

[0011] Wherein, the currently analyzed frame refers to the video frame from which the target is currently analyzed, and the preceding frame refers to the video frame that was analyzed by a neural network before the currently analyzed frame;

[0012] Step (2): Collect and generate discontinuous video frame sequence data from the video as training data through the network training framework. The last frame of the sequence will be used as the detected frame, and the remaining frames will be used as the preceding frames. Train the target detection network in step (1) to obtain the model parameters of the neural network used for target detection.

[0013] Step (3): The video stream processing and analysis framework adopts a pipeline operation method, which divides the target detection network trained in step (2) into multiple parallel parts to perform pipelined target detection on the video stream; at the same time, a queue with a limited depth is used to retain the features of the detected frames to supplement the subsequent data.

[0014] In step (1), the specific steps for supplementing the data features of the currently analyzed frame are as follows:

[0015] First, differential information is used to analyze the interpolation between the current frame and the previous frame, and this information is used as relevant features for target detection analysis.

[0016] Then, the information contained in the previous frame is added to the features of each layer of the neural network corresponding to the currently analyzed frame through the feature supplementation module;

[0017] Secondly, the current frame is analyzed by combining the feature analysis module with the historical information of the previous frame, and the output of each neural network layer is used as information for the analysis of subsequent frames.

[0018] Finally, the features extracted by the feature analysis module are analyzed through the object detection network to obtain the final result of object detection.

[0019] The specific steps (1) are as follows:

[0020] (1a) Analyze the difference between the current frame being analyzed and the previous frame to obtain the difference feature, the calculation formula of which is:

[0021] z = F([RR′,GG′,BB′])

[0022] Here, F represents the neural network for extracting features from the data, z represents the extracted features, and R-R', G-G', B-B' indicate the normalized differences between the current detection frame and a previous frame in the red, green, and blue channels. The RGB colors are first normalized, and then the difference is calculated. For the target detection network and the supplementary feature prediction network, the differential feature analysis module and the feature extraction neural network F with the same structure but different parameters are used to extract the current frame data and the previous frame data to obtain the image features analyzed by the feature analysis module in step (1c).

[0023] z=concat[F([RR′,GG′,BB′]),F([R,G,B])];

[0024] (1b) The features generated by the previous frame (i.e. the discontinuous sampled video frame) of the current frame being analyzed are fused into the features of the current frame being analyzed as supplementary information; here, the features are the outputs of each neural network module in the feature analysis module in step (1c);

[0025] Feature Supplement Module F C The specific formula is as follows:

[0026] z = F C (x0,x1,…)=Net FC (concat([x0,x1,…]))

[0027] The `concat` operation represents merging multiple features x0, x1, ... along the feature channel dimension, and then passing them through a neural network (Net).FC To perform fusion and supplementation; here Net FC The bottleneck module in a multi-layer ResNet has n times the original number of channels n×C as input channels and C as output; x0, x1, ... represent the feature maps corresponding to the current frame being analyzed and the previous discontinuous frames, which have the same length, width and number of feature channels.

[0028] The feature completion module F C It is used to be inserted between each feature extraction convolutional layer to extract information from the previous video frame and supplement the features of the current analyzed frame;

[0029] (1c) The input image (i.e., the image features described in step (1a)) and other relevant information (i.e., the features extracted from the previous frame in step (1c) as described in step (1b)) are analyzed and processed based on a fully convolutional neural network, and the features required for the current image target detection task are obtained in combination.

[0030] The calculation formula for the feature analysis module is as follows:

[0031] [z1,z2,z3,…,z n ]=F(x ij )

[0032] Where F is a feature extraction network based on a convolutional neural network. A convolutional network typically consists of several convolutional layers, with batch normalization layers and non-linear activation layers between them. Optional pooling layers may also be included to reduce the image size, [z1, z2, z3, ..., z...]. n In the network, the features output by several pooling or convolutional layers are used as the input to the object detection module in step (1d).

[0033] (1d) Based on the features described in the feature analysis module, analyze and detect the target information contained therein, and provide the target's confidence level, category, and detection box information (i.e., the coordinate position of the detected target and the size of the detection box); the calculation formula is as follows:

[0034] [(p1,c1,b1),(p2,c2,b2),…,(p m ,c m ,b m )]=H([z1,z2,z3,…,z n ])

[0035] Wherein, H is the object detection network, which consists of several convolutional layers, fully connected layers, and upsampling layers. The convolutional layers and fully connected layers are followed by batch normalization layers and non-linear activation layers; the input is a set of features [z1, z2, z3, ..., z] extracted by the feature analysis module described in step (1c). n, as the input of one or more convolutional layers and fully connected layers in the network, the output is multiple groups of detection results, including the confidence (p) of the detection results, the category (c) of the detected target, and the coordinates (b = [x, y, h, w]) of the bounding box of the detected target.

[0036] The specific steps of step (2) are as follows:

[0037] (2a) A training data generator is used to generate the data for the object detection network in step (1); first, the video data is passed through a feature prediction network to generate the information as the input of the feature supplement module, and then, by means of random sampling, discontinuous video frames and their corresponding prediction information are selected as training data, and a batch of data used for a single training is obtained;

[0038] First, select the video data x for learning from the video dataset X for training i , and obtain the supplementary feature c p corresponding to each frame x of the video data through the supplementary feature prediction network F ij ; ij c p = F (x ij )

[0039] Then, randomly select a frame x ij as the detected frame, and then, within the range of [0, j), select a specific number N of video frames by random sampling as the supplement to the current detected frame, where N < j / 3; N < j / 3 is used to ensure the sparsity of the previous frame information during the network learning process, so as to avoid the object detection network relying too much on the information contained in the previous frame during the learning process;

[0040] Finally, the selected video frames and the related data are used as a sample of data for training the object detection network described in step (1);

[0041] (2b) The network training framework includes a feature storage dictionary, which is used to save the feature information generated during the forward propagation process when the object detection network described in step (1) uses the data generated by the training data generator described in step (2a) for training, including the feature information output by each neural network module in the object detection network and the features predicted by the prediction supplement feature network, and randomly provides one of the two features during training; at the same time, it includes a feature prediction network based on gradient cutoff, which is used to predict the features of the previous frame as the pseudo-supplementary features of the current detected frame, and the details are as follows:

[0042] The feature storage dictionary stores the relevant features z corresponding to a specific frame x ij of a specific video during the previous M training processes. ijThe predicted feature dictionary stores the predictions for a specific video frame x during the first M training iterations. ij Predicted video features

[0043] The features stored in the feature dictionary are the feature information of the corresponding video frame propagating forward in the target detection network, or pseudo-features, i.e. the output of the feature prediction network, which are used by the feature supplementation module in step (1a) to supplement the features of the current detected frame.

[0044] When the training framework samples supplementary features for a specific frame, it will do so according to the hyperparameter p. s The corresponding probability is selected from the feature information generated by the feature prediction network and the feature information during the forward propagation of the network; where the hyperparameter p s It is a pre-set probability used to determine the sampling source;

[0045] The feature prediction network based on gradient cutoff is the same as the main architecture of the target detection network described in step (1), but does not include the supplementary module described in step (1a); the network parameters are learned from the target detection network by the target detection network based on the exponential average moving method, and gradient cutoff is used to block the gradient in the backpropagation from propagating to the feature prediction network; that is, it does not participate in gradient descent and learning during the learning process of the target detection network.

[0046] (2c) A hybrid loss function, comprising a conventional object detection loss function and a target feature contrast loss function based on contrastive learning, is used to measure the difference between the detection results of the object detection network described in step (1) on the training data generated in step (2a) and the annotations; this loss function can be expressed as:

[0047]

[0048] The general object detection loss function is:

[0049]

[0050] Among them l iou The boundary regression loss function is specifically defined as follows:

[0051]

[0052] Here, A represents the predicted boundary, B represents the true boundary, and C represents the minimum convex closed boundary between the two boundaries; p The confidence loss function is obtained during the calculation of the model inference process, and is specifically defined as follows:

[0053] l p =BCE(p o,p iou )

[0054] Where, p o p represents the confidence level in predicting the target, while p iou This represents the Interchange of Union (IOU) metric between the predicted target and the actual bounding box.

[0055] l c The loss function for class prediction is defined as follows:

[0056] l c =BCE(c p ,c gt )

[0057] Among them, c p Let c represent the probability that the detected target is classified as c. gt This represents the probability that the true classification of the detected target is c;

[0058] The contrastive learning loss function is defined as follows:

[0059]

[0060] Here, q∈obj represents the target detected in the current analyzed frame, k∈pos(q) means that k is the same target in the sampled frame as a positive sample, and k≠q means that among all targets, the target is not q itself; τ represents the temperature coefficient of the contrastive learning loss function.

[0061] Step (3) specifically involves:

[0062] (3a) Real-time video frame skipping acquisition: For the currently processed video frame, during the reprocessing of its preceding video frame, based on the speed of analysis and processing by the current algorithm, image frames to be discarded are intelligently selected, and pseudo-target detection results are generated for the discarded image frames based on the target detection results of the previously processed frames; First, the prediction algorithm for the number of frames to be skipped is as follows:

[0063] First, it is necessary to estimate the processing latency of the pipeline in the target detection processing framework described in step (3b). For the pipeline, assuming it contains n tasks, the time consumed by these n tasks in each processing step is t1, t2, ..., t3 respectively. n Use a queue of depth m to record the time t of the most recent m processed tasks. 1 =max(t1,t2,…,t) n Then, among the durations recorded in the queue, the minimum value T = min(t) 1 ,t 2 …,t nThis serves as an estimate for the current processing speed. Since the frame rate of a video stream or video is usually a fixed value r, the time difference between frames in the video stream or video stream can be calculated. Seconds. The number of frames that need to be skipped is... (Round up) where k is the offset constant, usually 0;

[0064] For predicting the detection results of skipped frames, an exponential average moving average is used as the prediction method. First, a first-in, first-out queue of depth n is used to record the output z of the object detection network and the time interval t from the current skipped frame. Then, the time interval is converted into a weight value of the result. Based on the formula: w = exp(-t), the detection result of that frame is calculated.

[0065]

[0066] Then, the final result is obtained by solving the object detection results. Finally, the result is stored in a temporary queue, and the detection results are output in the order of the video or video stream.

[0067] (3b) The target detection network obtained by training in step (2) and the data processing process before and after the algorithm are divided into multiple sub-algorithms according to the deployment environment and computing power, so as to make reasonable use of computing resources and maximize the use of the storage and computing performance of the device.

[0068] The aforementioned pipeline processing framework consists of the following components:

[0069] (31) Image preprocessing module, used to convert video data into the data format set by the network and perform normalization operation;

[0070] (32) (Several) Feature extraction modules, which break down the feature extraction described in step (1c) into several computational tasks with similar computational complexity;

[0071] (33) (Several) Target detection modules, the target detection modules described in step (1d) are divided into several computational tasks with equal computational load;

[0072] (34) Target detection result calculation module, used to calculate the target detection result.

[0073] The framework execution flow is roughly as follows: after acquiring the detection frame, it goes through the above pipeline in sequence, and finally stores the result. Then, it outputs the skipped frames in the same order as the input frames.

[0074] The beneficial effects of this invention are:

[0075] First, this invention utilizes discontinuous video frame data to perform target detection on real-time images. It effectively leverages the image correlation between video frames, enhancing the detection efficiency of targets that are difficult to detect using conventional single-frame image-based target detection algorithms. For small targets or targets similar to the background, it is often difficult to spot them in a single frame using conventional deep learning algorithms or even the human eye. However, in a continuous video stream, the movement of such targets makes them much easier to detect. Therefore, in indoor, urban, and outdoor scenarios, this invention can more effectively identify smaller targets and targets similar to the background.

[0076] Second, this invention employs discontinuous frame detection. For skipped frames, a method similar to interpolation is used to predict and generate pseudo-detection results. Simultaneously, an algorithm is used to predict and skip frames evenly, ensuring that more frames are detected while maintaining video smoothness.

[0077] Third, the present invention adopts a discontinuous frame selection scheme, which balances the information carried by the frame with the computing resources, thereby improving the target detection effect while reducing the computing resources required.

[0078] This invention utilizes pipeline technology to divide the target detection algorithm into multiple parts, making full use of computing resources and reducing the latency of data processing between frames. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of a real-time target detection algorithm for video streams with discontinuous sampling sequences.

[0080] Figure 2 This is a schematic diagram of a real-time object detection network for discontinuous video stream sequences.

[0081] Figure 3 This is a schematic diagram of a neural network training framework.

[0082] Figure 4 This is a schematic diagram of a real-time target detection framework. Detailed Implementation

[0083] The present invention will now be described in further detail with reference to the accompanying drawings.

[0084] Reference Appendix Figure 1 The present invention relates to a real-time target detection method for video streams with discontinuous sampling sequences, comprising:

[0085] Step 1. Perform discontinuous sampling on the video stream, and select frames from the video stream for analysis based on the computational load and relevant settings;

[0086] Step 2. For the frame being analyzed, combine the information from the previous frame with the frame's visual characteristics to perform analysis and extract features;

[0087] Step 3. Extract the final target detection results from the features obtained from the analysis.

[0088] The method is implemented through the following system, which includes:

[0089] ① A target detection network for discontinuous frames, ② A training framework for neural networks, ③ A real-time processing framework.

[0090] The described neural network training framework is used to train an object detection network (i.e., an object detection network for video streams with discontinuous sampling sequences) based on video data. The framework first samples video data to train the model, and then uses a feature prediction network to replace some of the features. After the neural network learns through gradient descent, the features of each layer are saved as historical frame information, and the neural network in the training framework, which has the same architecture as the feature extraction network, is updated using an exponential moving average algorithm.

[0091] The target detection network for video streams with discontinuous sampling sequences is the core of this application. It is used to analyze input video frame images and identify the targets to be detected. This network is trained using the aforementioned training framework and transformed and implemented into a detection algorithm (program) that can be used in real-world scenarios through a fast-output real-time processing framework.

[0092] The aforementioned real-time processing framework transforms the target detection network into a pipelined detection algorithm (program) to effectively utilize hardware resources and reduce data processing latency. Simultaneously, the processing framework dynamically adjusts the number of skipped frames based on the algorithm's execution speed. Furthermore, the framework supplements the skipped video frames with target detection results through interpolation.

[0093] Component 1. As attached Figure 2 As shown, a deep neural network for target detection in discontinuous video sequences consists of a feature supplementation module, a differential feature analysis module, a feature analysis module, and a target detection module.

[0094] The feature supplementation module is used to fuse the feature information corresponding to the previous frame into the features of the current detected frame, so as to effectively detect targets with small differences from the background using multi-frame data. This module exists between the various neural network sub-modules in the feature analysis network.

[0095] The differential analysis module is used to analyze the image numerical differences between the current frame and the previous frame, in order to more directly obtain the differences between frames. In this example, this module replaces the first convolutional layer in the feature analysis network to process the data.

[0096] The feature analysis module is used to analyze the features of the detected video frames and extract information from them for target detection.

[0097] The target detection module is used to analyze the output of the feature extraction module to find relevant targets and represent them as bounding box coordinates, target confidence, category and other information.

[0098] Component 1a. As attached Figure 2 As shown, feature completion module F C For the deep neural network module, the specific formula is as follows:

[0099] z = F C (x0,x1,…)=Net FC (concat([x0,x1,…]))

[0100] The `concat` operation represents merging multiple features x0, x1, ... along the feature channel dimension. Then, it is processed through a neural network (Net). FC To perform integration and supplementation. Here, Net... FC The bottleneck module in a multi-layer ResNet has n times the original number of channels (n×C) as input and C as output. x0, x1, ... represent the feature maps of the current frame being analyzed and the previous discontinuous frames, with the same length, width, and number of feature channels.

[0101] A feature supplementation module can be inserted between each feature extraction convolutional layer to extract information from previous video frames and supplement the features of the currently analyzed frame. In this embodiment, we use the convolutional network of ResNet 101 excluding the first convolutional layer as the feature analysis network. Therefore, for the feature supplementation module, the feature outputs of the 33 Bottleneck Block modules will be used as the object of feature supplementation. The outputs of the 33 Bottleneck Block modules corresponding to the 1st, 2nd, 4th, 8th, 16th, 32nd, 64th, 128th, 256th, 512th, 1024th, 2048th, and 4096th frames preceding the analyzed frame will be used as the source of feature supplementation to carefully supplement the features.

[0102] Component 1b. As attached Figure 2 As shown, the calculation formula for the differentiated feature analysis module is as follows:

[0103] z = F([RR′,GG′,BB′])

[0104] Here, F represents the neural network that extracts features from the data, z represents the extracted features, and R-R', G-G', and B-B' indicate the normalized differences between the current detection frame and a previous frame in the red, green, and blue channels. In other words, the RGB colors are first normalized, and then the difference is calculated. For the object detection network and the supplementary feature prediction network, a differential feature analysis module and a feature extraction neural network F with the same structure but different parameters are used to extract features from the current frame data and the previous frame data.

[0105] z=concat[F([RR′,GG′,BB′]),F([R,G,B])]

[0106] In this embodiment, we compare the current frame being analyzed with the data information of a frame collected one minute ago, subtract the differences between the frames, and use the 3-layer Bottleneck Block in ResNet as the feature area neural network F to analyze the current frame and the differences, and extract features.

[0107] Component 1c. As attached Figure 2 As shown, the calculation formula for the feature analysis module is as follows:

[0108] [z1,z2,z3,…,z n ]=F(x i j)

[0109] Where F represents a feature extraction network based on a convolutional neural network. A convolutional network typically consists of several convolutional layers, with batch normalization layers and non-linear activation layers between them. Optional pooling layers may also be included to reduce image size. [z1,z2,z3,…,z] n [z1, z2, z3, ..., z] represents the features output by several pooling or convolutional layers in the ResNet 101 network, which are then used as the input to the object detection module in step (2d). In this embodiment, the convolutional network of ResNet 101 excluding the first convolutional layer is used as the feature analysis network, and the outputs of its 33 Bottleneck Block modules are used as the results calculated by the feature analysis module: [z1, z2, z3, ..., z 33 ].

[0110] Component 1d. As attached Figure 2 As shown, the target detection module uses the following calculation formula:

[0111] [(p1,c1,b1),(p2,c2,b2),…,(p m ,c m ,b m)]=H([z1,z2,z3,…,z n ])

[0112] Where H is the object detection network, consisting of several convolutional layers, fully connected layers, and upsampling layers. The convolutional and fully connected layers are followed by batch normalization layers and non-linear activation layers. The input is a set of features [z1, z2, z3, ..., z] extracted by the feature analysis module described in step (2c). n ], which serves as the input to one or more convolutional and fully connected layers in the network, outputs multiple sets of detection results, including the confidence score (p) of the detection results, the category of the detected target (c), and the coordinates of the bounding box of the detected target (b = [x, y, h, w]).

[0113] In this embodiment, the YoloHead, with the same structure as in YOLOv5, is used as one of the detection networks in the object detection network H. The object detection network H in this embodiment includes four detection networks. The inputs of these four detection networks use Layer 1 to Layer 4 of ResNet 101. The input and output features of these four modules serve as the inputs to the object detection modules. If the input and output sizes are inconsistent, they are unified by scaling and then concatenated along the feature dimension.

[0114] In addition, in this embodiment, the detected targets include two categories: pedestrians and vehicles.

[0115] Taking a traffic intersection in a city as an example, video frames are accessed from road surveillance cameras to detect pedestrians and vehicles. The network reads images from the video stream and merges information from several previously read images to analyze the targets in the images and detect pedestrians and vehicles on the road.

[0116] Component 2. As attached Figure 3 As shown, a training framework for neural networks includes: a network training framework comprising a training data generator, a feature prediction network, a predicted feature and forward propagation feature storage component, and a hybrid loss function.

[0117] The training data generator is a sampler that selects videos from the dataset and uses the features of the feature prediction module or features retained during the previous training of the video as the image input and supplementary features in the object detection network. In this example, in order to reduce the number of calculations, when analyzing the current frame, the previous frames are not forward propagated. Instead, the features of each neural network layer saved in the previous forward propagation process or the features generated by the feature prediction network are used.

[0118] As described above, the feature prediction network generates "pseudo" features of the preceding frame to reduce unnecessary computations. Here, the feature prediction network avoids unnecessary backpropagation computations through gradient cutoff and employs an exponential moving average algorithm to update the prediction network.

[0119] The aforementioned predictive feature and intermediate feature storage component, as described above, is used to store the features generated during the forward propagation or feature prediction process of a specific frame. These features are the forward propagation features and the predictive features, respectively. During the training process, these features are fused into the features of the detected frame as features of the preceding frame for target detection.

[0120] The hybrid loss function is used to train the object detection network described in this application. As part of the training framework, this loss function measures the quality of the results during each forward propagation of the network and optimizes the model parameters through backpropagation.

[0121] Component 2a. As attached Figure 3 As shown, the training data generator mainly includes a feature prediction network that generates predicted features to supplement the input information of the feature module; a random sampling component that selects discontinuous frames and related data from the video for training through random sampling; and a loss function used to train the network.

[0122] For the training data generator, firstly, in the video dataset X = {x} used for training... i Select video data x for learning from} i The video data contains several frames of data x i ={x ij Then, the supplementary feature prediction network F... p Obtain each frame x of the video ij Supplementary feature c corresponding to the data ij =F p (x ij ).

[0123] Next, select any frame x from all video frames. ij As one frame detected in the current training data, N frames of video are then randomly sampled within the range [0, k). As for frame x ijSupplementary frames. Furthermore, the number of selected frames N ≤ j / 3. The supplementary features corresponding to the supplementary frames, along with the currently detected frame and its associated annotations, are used as training data for training. In this embodiment, the dataset contains a uniform 18,000 video frames, comprising 300 seconds of video, with 60 frames per second. During sampling, when randomly sampling N frames, N is set to 128. Simultaneously, a sparse-to-dense random sampling method is used; specifically, for time intervals x... ij For closer video frames, the number of sampling times increases; for video frames with larger intervals, the number of sampling times decreases.

[0124] For the feature prediction module, the following method is used: Figure 2 The feature analysis module uses the same neural network architecture as the feature prediction module described later. Its network weights are learned from the object detection network using the exponential moving average (EMA) algorithm. The network output is the feature {z} of a specific node. i In this embodiment, the convolutional network of ResNet 101 with the first convolutional layer removed is used as the feature prediction module, and also as the feature analysis module.

[0125] Component 2b. As attached Figure 3 As shown, the feature prediction network and intermediate feature storage component include a dictionary- and queue-based feature storage module. The network-predicted and actually extracted features are saved to a dictionary. The feature storage dictionary stores the features from the most recent M training iterations for a specific frame x in a specific video. ij The relevant features and the predicted feature dictionary store the relevant features for a specific video frame x during the first M training iterations. ij The predicted video features. In this embodiment, we use a dictionary and a queue of depth 128 to store the outputs of each of the 33 Bottleneck Blocks in the feature analysis network (the convolutional network in ResNet 101 excluding the first convolutional layer) during the first 128 training sessions.

[0126] The feature dictionary stores the features as the results of the forward propagation of the corresponding video frames in the object detection network, supplemented by the supplementary module described in component 1a later, through specific video frames. The prediction feature dictionary, on the other hand, stores the direct output features of the supplementary feature prediction model.

[0127] During training, when sampling additional features for specific frames, it will be done according to the hyperparameter p. s The corresponding probability is used to select features as supplementary features from the actual feature dictionary and the predicted feature dictionary. In this embodiment, the hyperparameter p sIt is set to 0.5, which means that the probability of selecting the actual feature and the predicted feature as supplementary features is set to 0.5-0.5.

[0128] For the feature prediction network, a gradient cutoff approach is used to prevent modifications to the model during training. The main architecture is the same as the object detection network, but it does not include the supplementary module described in (2a). The network parameters are learned from the object detection network using an exponential average moving average approach, and gradient cutoff is used to block gradients, preventing them from participating in gradient descent and learning during the object detection network's learning process.

[0129] Component 2c. As attached Figure 3 As shown, the hybrid loss function includes a general object detection loss function and a contrastive learning-based loss function.

[0130]

[0131] The general object detection loss function is:

[0132]

[0133] Among them l iou The boundary regression loss function is specifically defined as follows:

[0134]

[0135] Here, A marks the predicted boundary, B marks the actual boundary, and C marks the minimum convex closed boundary between the two boundaries. p For the confidence loss function, it is specifically defined as follows:

[0136]

[0137] l c The loss function for class prediction is defined as follows:

[0138]

[0139] The contrastive learning loss function is defined as follows:

[0140]

[0141] Here, q represents the target detected in the currently analyzed frame, k∈pos(q) means that k is the same target in the sampled frame as a positive sample, and k≠q means that among all targets, there are targets that are not q itself. τ represents the temperature coefficient of the contrastive learning loss function.

[0142] In this embodiment, the temperature coefficient τ in the contrastive learning loss function is set to 0.05.

[0143] Component 3. As attached Figure 4 As shown, a video stream processing and analysis framework for real-time target detection of discontinuous sequences is presented. The framework employs a pipelined approach, dividing the target detection algorithm into multiple parallelizable parts. Simultaneously, a queue of finite depth is used to preserve the features of already detected frames to supplement subsequent data. This analysis framework includes a real-time video frame skipping acquisition module and a pipelined target detection framework.

[0144] The real-time video frame-skipping acquisition described herein is used to acquire video frames from a video stream by skipping frames. Unlike traditional frame-by-frame acquisition of video streams, the acquisition method described in this application skips some video frames in the video stream. Furthermore, unlike traditional frame-skipping acquisition, the frame-skipping acquisition method described in this application dynamically adjusts according to the processing speed of the target detection pipeline. In addition, for the skipped video frames, we use interpolation and other methods to supplement the video frame results.

[0145] The pipelined target detection framework described above is used to pipeline the target detection algorithm, namely the differential feature analysis module, feature analysis module, feature supplementation module, and target detection module mentioned in component 1, and to analyze and process the images acquired by the frame skipping acquisition module. In other words, the target detection algorithm is divided into several independently executed sub-algorithms, and through a pipelined approach, hardware resources are maximized while the target detection frame rate is improved.

[0146] Component 3a. As attached Figure 4 As shown, real-time video frame skipping acquisition includes estimating the number of frames to be skipped, i.e., predicting the target detection time, and predicting the outcome of the skipped frames. First, the algorithm for predicting the number of frames to be skipped is as follows:

[0147] First, it is necessary to estimate the processing latency of the pipeline in the object detection processing framework described in step (3b). For the pipeline, assuming it contains n tasks, the time consumed by these n tasks in each processing iteration is t1, t2, ..., t... n We use a queue of depth m to record the time t of the most recent m processed tasks. 1 =max(t1,t2,…,t) n Then, among the durations recorded in the queue, the minimum value T = min(t) 1 ,t 2 …,t n This serves as an estimate for the current processing speed. Since the frame rate of a video stream or video is usually a fixed value r, the time difference between frames in the video stream or video stream can be calculated. Seconds. The number of frames that need to be skipped is... (Round up) where k is the offset constant.

[0148] In this embodiment, the target detection algorithm mentioned in component 1 is divided into three pipeline tasks to process the data. Simultaneously, the queue depth for recording the time consumed by each task is 32, and the frame rate of the analyzed video data is 120 frames per second. The offset constant k is set to 0.

[0149] For predicting the detection results of skipped frames, an exponential average moving average method is used. First, a first-in, first-out queue of depth n is used to record the output z of the object detection network and the time interval t from the current skipped frame. Then, the time interval is converted into a weight value of the result. Based on the formula: w = exp(-t), the detection result of that frame is...

[0150]

[0151] The final result is then obtained by calculating the target detection results. Finally, the result is stored in a temporary queue, and the detection results are output in the order of the video or video stream. In this embodiment, a first-in-first-out queue with a depth of 64 is set up for the image whose frames are skipped.

[0152] Component 3b. As attached Figure 4 As shown, the pipeline-based target detection processing framework involved consists of the following components:

[0153] i. Image preprocessing module, used to convert video data into the data format set by the network and perform normalization operations;

[0154] ii. (Several) Feature extraction modules, which break down the feature extraction described in step (2c) into several computational tasks with similar computational complexity;

[0155] iii. (Several) Target detection modules, the target detection modules described in step (2d) are divided into several computational tasks with equal computational load;

[0156] iv. Target detection result calculation module, used to calculate the target detection results.

[0157] The framework execution flow is roughly as follows: after acquiring the detection frame, it goes through the above pipeline in sequence, and finally stores the result. Then, it outputs the skipped frames in the same order as the input frames.

Claims

1. A real-time target detection method for video streams with discontinuous sampling sequences, characterized in that, Includes the following steps; Step (1): The discontinuous video frame sequence is analyzed by the target detection network. The target detection network uses the data features of the previous frame collected before the current frame being analyzed to supplement the data features of the current frame with the features and information contained in the previous frame. Finally, the target detection network obtains the detection result through the supplemented features. Wherein, the currently analyzed frame refers to the video frame from which the target is currently analyzed, and the preceding frame refers to the video frame that was analyzed by a neural network before the currently analyzed frame; Step (2): Collect and generate discontinuous video frame sequence data from the video as training data through the network training framework. The last frame of the sequence will be used as the detected frame, and the remaining frames will be used as the preceding frames. Train the target detection network in step (1) to obtain the model parameters of the neural network used for target detection. Step (3): The video stream processing and analysis framework adopts a pipeline operation method, which divides the target detection network trained in step (2) into multiple parallel parts to perform pipelined target detection on the video stream; at the same time, a queue with a limited depth is used to retain the features of the detected frames to supplement the subsequent data. In step (1), the specific steps for supplementing the data features of the currently analyzed frame are as follows: First, differential information is used to analyze the interpolation between the current frame and the previous frame, and this information is used as relevant features for target detection analysis. Then, the information contained in the previous frame is added to the features of each layer of the neural network corresponding to the currently analyzed frame through the feature supplementation module; Secondly, the current frame is analyzed by combining the feature analysis module with the historical information of the previous frame, and the output of each neural network layer is used as information for the analysis of subsequent frames. Finally, the features extracted by the feature analysis module are analyzed through the object detection network to obtain the final result of object detection.

2. The real-time target detection method for video streams with discontinuous sampling sequences according to claim 1, characterized in that, The specific steps (1) are as follows: (1a) Analyze the difference between the current frame being analyzed and the previous frame to obtain the difference feature, the calculation formula of which is: z = F([RR′,GG′,BB′]) Here, F represents the neural network for extracting features from the data, z represents the extracted features, and R-R', G-G', B-B' indicate the normalized difference between the current detection frame and a previous frame in the red, green, and blue channels. The RGB colors are first normalized, and then the difference is calculated to obtain the image features analyzed by the feature analysis module in step (1c). z=concat[F([RR′,GG′,BB′]),F([R,G,B])]; (1b) The features generated by the discontinuously sampled video frames are fused into the features of the current frame being analyzed as supplementary information; the features are the outputs of each neural network module in the feature analysis module in step (1c); Feature Supplement Module F C The specific formula is as follows: z=F C (x0,x1,…)=Net FC (concat([x0,x1,…])) The `concat` operation represents merging multiple features x0, x1, ... along the feature channel dimension, and then passing them through a neural network (Net). FC To perform fusion and supplementation; here Net FC The bottleneck module in a multi-layer ResNet has n times the original number of channels n×C as input channels and C as output; x0, x1, ... represent the feature maps corresponding to the current frame being analyzed and the previous discontinuous frames, which have the same length, width and number of feature channels. (1c) Based on the image features described in step (1a) and the features extracted from the previous frame in step (1b) as described in step (1c), perform analysis and processing, and synthesize the features required for the current image target detection task; The calculation formula for the feature analysis module is as follows: [z1,z2,z3,…,z n ]=F(x ij ) Where F is a feature extraction network based on a convolutional neural network. A convolutional network typically consists of several convolutional layers, with batch normalization layers and non-linear activation layers between them. Optional pooling layers may also be included to reduce the image size, [z1, z2, z3, ..., z...]. n In the network, the features output by several pooling or convolutional layers are used as the input to the object detection module in step (1d). (1d) Based on the features described in the feature analysis module, analyze and detect the target information contained therein, and provide the coordinate position of the detected target and the size of the detection box; the calculation formula is as follows: [(p1,c1,b1),(p2,c2,b2),…,(p m ,c m ,b m )]=H([z1,z2,z3,…,z n ]) Wherein, H is the object detection network, which consists of several convolutional layers, fully connected layers, and upsampling layers. The convolutional layers and fully connected layers are followed by batch normalization layers and non-linear activation layers; the input is a set of features [z1, z2, z3, ..., z] extracted by the feature analysis module described in step (1c). n ], which serves as the input to one or more convolutional and fully connected layers in the network, outputs multiple sets of detection results, including the confidence score (p) of the detection results, the category of the detected target (c), and the coordinates of the bounding box of the detected target (b = [x, y, h, w]).

3. The real-time target detection method for video streams with discontinuous sampling sequences according to claim 2, characterized in that, The feature completion module F C It is used to insert between each feature extraction convolutional layer to extract information from the previous video frame and supplement the features of the current analyzed frame.

4. The real-time target detection method for video streams with discontinuous sampling sequences according to claim 1, characterized in that, Step (2) specifically involves: (2a) Training data generator, used to generate data for training the target detection network described in step (1); First, select video data x for learning from the video dataset X used for training. i And by supplementing the feature prediction network F p Obtain each frame x of the video ij Supplementary feature c corresponding to the data ij =F p (x ij ); Then, randomly select a frame x ij , as the detected frame, and then within the range of [0, j), randomly sample a specific number N of video frames as a supplement to the current detected frame, where N < j / 3; N < j / 3 is used to ensure the network learning process; Finally, the selected video frames and related data are used as a sample of data to train the target detection network described in step (1); (2b) The network training framework includes a feature storage dictionary, which is used to store the feature information generated by the target detection network in step (1) during the forward propagation process using the data generated by the training data generator in step (2a) during training. The feature information includes the feature information output by each neural network module in the target detection network and the features predicted by the prediction supplementary feature network. During training, one of the two features is randomly provided. It also includes a gradient-based feature prediction network, which is used to predict the features of the previous frame as pseudo-supplementary features for the current detected frame. (2c) A hybrid loss function, comprising a conventional object detection loss function and a target feature contrast loss function based on contrastive learning, is used to measure the difference between the detection results of the object detection network described in step (1) on the training data generated in step (2a) and the annotations; this loss function can be expressed as: The general object detection loss function is: Among them l iou The boundary regression loss function is specifically defined as follows: Here, A represents the predicted boundary, B represents the true boundary, and C represents the minimum convex closed boundary between the two boundaries; p The confidence loss function is obtained during the calculation of the model inference process, and is specifically defined as follows: l p =BCE(p o ,p iou ) Where, p o p represents the confidence level in predicting the target, while p iou This represents the Interchange of Union (IOU) metric between the predicted target and the actual bounding box. l c The loss function for class prediction is defined as follows: l c =BCE(c p ,c gt ) Among them, c p Let c represent the probability that the detected target is classified as c. gt This represents the probability that the true classification of the detected target is c; The contrastive learning loss function is defined as follows: Here, q∈obj represents the target detected in the current analyzed frame, k∈pos(q) means that k is the same target in the sampled frame as a positive sample, and k≠q means that among all targets, the target is not q itself; τ represents the temperature coefficient of the contrastive learning loss function.

5. The real-time target detection method for video streams with discontinuous sampling sequences according to claim 4, characterized in that, The details of step (2b) are as follows: The feature storage dictionary stores the specific frames x for a specific video during the first M training iterations. ij Relevant features z ij The predicted feature dictionary stores the predictions for a specific video frame x during the first M training iterations. ij Predicted video features The features stored in the feature dictionary are the feature information of the corresponding video frame propagating forward in the target detection network, or pseudo-features, i.e. the output of the feature prediction network, which are used by the feature supplementation module in step (1a) to supplement the features of the current detected frame. When the training framework samples supplementary features for a specific frame, it will do so according to the hyperparameter p. s The corresponding probability is selected from the feature information generated by the feature prediction network and the feature information during the forward propagation of the network; where the hyperparameter p s It is a pre-set probability used to determine the sampling source; The feature prediction network based on gradient cutoff is the same as the main architecture of the target detection network described in step (1), but does not include the supplementary module described in step (1a); the network parameters are learned from the target detection network by the target detection network based on the exponential average movement, and gradient cutoff is used to block the gradient in the backpropagation from propagating to the feature prediction network.

6. The real-time target detection method for video streams with discontinuous sampling sequences according to claim 1, characterized in that, Step (3) specifically involves: (3a) Real-time video frame skipping acquisition: For the currently processed video frame, during the reprocessing of its previous video frame, based on the speed of the current algorithm analysis and processing, intelligently select and discard image frames, and generate pseudo target detection results for the discarded image frames based on the target detection results of the previous processed frames. (3b) The target detection network obtained by training in step (2) and the data processing process before and after the algorithm are divided into multiple sub-algorithms according to the deployment environment and computing power, so as to make reasonable use of computing resources and maximize the use of the storage and computing performance of the device.

7. A real-time target detection method for video streams with discontinuous sampling sequences according to claim 6, characterized in that, The prediction algorithm for the number of frames to be skipped in (3a) is as follows: First, it is necessary to estimate the processing latency of the pipeline in the target detection processing framework described in step (3b). For the pipeline, assuming it contains n tasks, the time consumed by these n tasks in each processing step is t1, t2, ..., t3 respectively. n Use a queue of depth m to record the time t of the most recent m processed tasks. 1 =max(t1,t2,…,t) n ); Then, among the durations recorded in the queue, the minimum value T = min(t) 1 ,t 2 …,t n As an estimate of the current processing speed; the frame rate of the acquired video stream or video is a fixed value r, and the time difference between frames of the video stream or video stream is calculated. seconds; the number of frames to be skipped is (rounded up), where k is the offset constant, usually 0.

8. A real-time target detection method for video streams with discontinuous sampling sequences according to claim 7, characterized in that, For the prediction of the frame skipping detection results, an exponential average shift is used as the prediction method. First, a first-in, first-out queue of depth n is used to record the output z of the object detection network and the time interval t between the current skipped frame. Then, the time interval is converted into a weight value of the result. Based on the formula: w = exp(-t), the detection result of the frame is calculated. Then, the final result is obtained by solving the object detection results. Finally, the result is stored in a temporary queue, and the detection results are output in the order of the video or video stream.

9. A real-time target detection method for video streams with discontinuous sampling sequences according to claim 7, characterized in that, The aforementioned pipeline processing framework consists of the following components: (31) Image preprocessing module, used to convert video data into the data format set by the network and perform normalization operation; (32) Feature extraction module, which breaks down the feature extraction described in step (1c) into several computational tasks with similar computational complexity; (33) Target detection module: The target detection module described in step (1d) is divided into several computational tasks with equal computational load. (34) Target detection result calculation module, used to calculate the target detection result.

Images