Automatic generation method for language datasets of surveillance videos with controllable deviations in subject and behavior

Abstract

This invention relates to the field of computer vision and focuses on an automatic method for generating surveillance video language datasets with controllable subject and behavior deviations. Based on deep learning techniques and cross-validation methods in mathematics, this method first constructs an enhanced surveillance video captioning model using a target tracking module to annotate the descriptive text of the generated surveillance video, controlling the degree of subject deviation in the descriptive text. Then, a data filtering model based on iterative deviation cross-validation is used to filter the descriptive text, controlling the degree of behavior deviation in the text description, ultimately obtaining a video language dataset with controllable subject and behavior deviations. The dataset produced by this invention has known degrees of subject and behavior deviation, thus providing effective assistance for training tasks such as multimodal surveillance video language understanding models.

Description

Technical Field

[0001] This invention, based on deep learning technology and cross-validation methods in mathematics, studies an automatic generation method for surveillance video language datasets with controllable deviations in subject and behavior. First, an enhanced surveillance video caption model is constructed using a target tracking module to annotate the descriptive text of the generated surveillance video, controlling the degree of subject deviation in the descriptive text. Then, a data filtering model based on iterative deviation cross-validation is used to filter the descriptive text, controlling the degree of behavioral deviation in the text description, ultimately obtaining a video language dataset with controllable subject and deviation. This invention belongs to the field of computer vision, specifically involving deep learning, target detection, and cross-validation techniques. Background Technology

[0002] Intelligent surveillance plays a crucial role in public safety, serving as a vital cornerstone for maintaining social order and protecting people's lives and property. Multimodal video language understanding technology focuses on understanding the relationship between surveillance video and language modalities. It can automatically understand the content of surveillance videos and generate detailed text descriptions, thus providing strong support for video language understanding applications in the surveillance field, such as video question answering, video subtitle generation, and dense video subtitle generation. Furthermore, it can significantly improve the efficiency of investigating and analyzing surveillance videos in real-world scenarios and provide important technical support for the construction of next-generation intelligent security systems.

[0003] However, at the data level, research on multimodal video language understanding technology relies on surveillance video datasets with well-defined text descriptions. Currently, commonly available public video text description datasets are geared towards general video domains, and most existing surveillance video data only includes labels for abnormal events and time information, lacking substantial sentence-level text descriptions relevant to the video content. This severely limits research on multimodal surveillance video language understanding. To address this issue, considering the high cost and low efficiency of manual annotation and the low quality of machine annotation, a combination of manual and machine annotation should be adopted to construct and expand surveillance video datasets with well-defined text descriptions. Previous research has shown that although the quality of machine-annotated data is not high, it can still serve as an important source of training data when the deviation from manually annotated data is controllable.

[0004] Therefore, researching how to use machines to automatically generate text annotations for surveillance videos and controlling the deviation between machine and human annotations is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] This invention differs from existing dataset generation methods by utilizing deep learning technology and cross-validation methods in mathematics. Targeting the visual characteristics of surveillance videos, it divides them into two main visual components: subject and behavior, to study controllable deviations. This invention proposes an automatic generation method for surveillance video language datasets with controllable subject and behavior deviations. First, it proposes integrating a target tracking model as a new branch into the video caption generation model to improve the existing caption generation model's ability to identify and describe the target subject engaging in behavior, thereby achieving high-quality video caption generation. Then, it proposes using a low-quality data filtering algorithm to control behavioral deviations in the dataset's text descriptions, thereby estimating the behavioral deviation degree. Finally, it estimates the overall deviation of the machine-annotated data based on the subject and behavior deviation degrees. Since the deviation degrees of the generated dataset's subject and behavior are known, it can serve as an important supplementary source of training data for multimodal surveillance video language understanding models.

[0006] The main process of this method is shown in the appendix. Figure 1 As shown, it can be divided into the following four steps: generating monitoring video text data with controllable subject deviation, estimating the subject deviation degree of automatically labeled text data, filtering low-quality behavior deviation data based on iterative deviation cross-validation, and generating a dataset with controllable subject and behavior deviation.

[0007] 1. Generation of text data from surveillance videos with controllable subject deviation.

[0008] An enhanced surveillance video caption generation model is built using a target tracking module to generate descriptive text for surveillance videos. This model controls the degree of subject deviation in the descriptive text to improve the existing caption generation model's ability to identify and describe the target subject engaging in behavior, thereby achieving high-quality video caption generation.

[0009] 2. Estimation of subject deviation in automatically labeled text data

[0010] The existing dataset is used to evaluate the subject deviation of the text generated by the surveillance video captioning model. The Euclidean distance between the feature vectors of the subject words is calculated to obtain the current subject deviation. Based on this deviation, the subject deviation of the automatically labeled text data is estimated, thereby ensuring that the subject deviation of the automatically labeled text is known.

[0011] 3. Low-quality data filtering based on iterative deviation cross-validation

[0012] For the generated video subtitles, a low-quality data filtering model based on iterative deviation cross-validation is used to filter the descriptive text, control the degree of behavioral deviation in the text description, and further improve the quality of the generated video subtitles.

[0013] 4. Generation of datasets with controllable deviations from the subject and behavior

[0014] The deviation of sample behavior is calculated based on the data in the final generated dataset, and a known dataset of noise with controllable deviation between the subject and behavior is finally obtained, laying the data foundation for downstream tasks such as multimodal surveillance video language understanding models.

[0015] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0016] First, traditional manual annotation methods are extremely expensive and time-consuming, and their annotation quality varies greatly, making them unsuitable for training large-scale and effective multimodal surveillance video language understanding models. Furthermore, most existing datasets related to video language understanding tasks are geared towards general domains, and the labels on most existing surveillance video data only include abnormal event category labels and time information, with a significant lack of sentence-level textual descriptions related to the video content. This severely limits research on multimodal surveillance video language understanding.

[0017] To address these issues, this invention proposes an automatic generation method for surveillance video language datasets with controllable subject and behavior deviations. This method can automatically generate high-quality video language annotation datasets, saving the cost and time of manual annotation. Simultaneously, this invention can also obtain the degree of subject and behavior deviation in the generated dataset, i.e., obtain a surveillance video language dataset with known noise. Therefore, it can effectively assist in training tasks such as multimodal surveillance video language understanding models. Attached Figure Description

[0018] Figure 1 A flowchart of a method for automatically generating language datasets from surveillance videos with controllable deviations in subject and behavior.

[0019] Figure 2 A video caption generation model that integrates video coding strategies that combine global and local features.

[0020] Figure 3 A dual-path model for quality inspection of machine-annotated text data behavior description.

[0021] Figure 4 Lightweight temporal network model Detailed Implementation

[0022] Based on the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.

[0023] Step 1: Generating text data from surveillance video showing controllable deviation of the main subject.

[0024] Step 1.1: Use the target tracking module to build an enhanced surveillance video caption model

[0025] The enhanced surveillance video captioning model, built using a target tracking module, employs a video coding strategy that integrates global and local features. The model architecture is shown in the attached figure. Figure 2 As shown in the diagram, the model first performs fixed sampling on the input raw video, extracting several video frame segments, and then extracts global video features using a visual encoder. Simultaneously, a pre-trained object detector is used to identify and locate objects in each video frame. A target filter then tracks and analyzes the motion trajectory of the detected objects within the video frames, further filtering out objects that change in consecutive frames. The parts outside the detection box of these objects are then occluded to obtain an image containing only the tracked target, which is then input into the visual encoder to obtain local video features. Finally, the global and local video features output from the visual encoder are fused together and input into a multimodal decoder along with text features for predicting the text output. Overall, the caption generation model designed in this section can construct an enhanced visual feature representation that focuses on the active target subject by fusing global video features with filtered local features, enabling the generation of text annotations with low deviation from the subject description.

[0026] For example, SwinBERT and SORT can be selected as the main branch for video captioning and the branch for object tracking, respectively. This model can be trained using existing video language datasets in the surveillance domain. With enhanced object recognition capabilities, compared to the original SwinBERT, this model can generate more accurate object-related text descriptions. During model training, the enhanced visual features and text features are input into the multimodal decoder. The model uses the enhanced visual features and the generated text sequence to jointly predict the next word. The loss function for model training continues to use the original cross-entropy loss, i.e., given the correct indices of the previous t-1 words. And visual representation ν, we can obtain the current t-th word. The prediction, using p y Let l represent the probability, and the training loss is calculated as shown in formula (1), where l1 is the word length.

[0027]

[0028] Step 1.2: Generation of Monitoring Video Text Data

[0029] The collected surveillance videos are input into the aforementioned surveillance video captioning model to obtain descriptive text for the videos. The video input consists of the surveillance video data to be labeled, and the text input begins with a start character to predict the next text. The model then continues predicting the next text until a stop character appears. In this way, a corresponding descriptive text label is generated for each piece of surveillance video data.

[0030] Step 2: Estimating the subject deviation of automatically labeled text data

[0031] Because the differences in target subject categories are small across various types of surveillance videos, making identification relatively easy, the subject deviation of the trained surveillance video captioning model will be relatively consistent when generating text annotations for any surveillance video. In other words, the subject deviation of the generated text dataset can be directly estimated from the subject deviation on existing surveillance domain video language datasets. Specifically, firstly, the machine-annotated subject words and manually annotated subject words automatically generated by the surveillance video captioning model on existing surveillance domain video language datasets are represented in pairs. Then, these words are input into BERT to extract feature vectors, and the deviation between these word feature vectors is calculated.

[0032] To further improve the accuracy and robustness of subject deviation estimation, this invention proposes a subject deviation measurement method based on semantic distribution alignment. Compared to traditional distance calculation between word pairs, this method evaluates the overall deviation of machine-annotated and human-annotated text in the semantic space from the perspective of overall distribution.

[0033] Specifically, the set of main words in machine-annotated text data is represented as S. m The set of main words in manually annotated text data is represented as S. a First, pre-trained language models (such as BERT) are used to map each subject's vocabulary to a unified feature vector space to obtain a feature set.

[0034] Subsequently, the machine was used to label the subject set S. m With manually labeled subject set S a The eigenmean vector μ m μ a and the characteristic covariance matrix Σ m , Σ a Based on mean and covariance information, the semantic distribution alignment deviation ε1 is defined as follows:

[0035]

[0036] Where, ||μ m -μ a ||2 represents the Euclidean distance between the mean values ​​of the main features labeled by machines and those labeled by humans; Tr(·) represents the trace operation of the matrix, used to measure the overall difference between the covariance matrices; It represents the square root of the product of covariance matrices, used to estimate the similarity of characteristic distributions.

[0037] By comprehensively considering both mean deviation and covariance structure deviation, this method can effectively capture the consistency differences in the overall semantic space distribution between machine-annotated text and human-annotated text, avoiding the amplification of local errors caused by matching based solely on a single word pair. Finally, the subject deviation evaluation result based on ε1 can serve as an important basis for the overall quality control of machine-annotated text data generated by the surveillance video caption model.

[0038] Step 3: Low-quality data filtering based on iterative deviation cross-validation

[0039] Step 3.1: Train a dual-path model that can verify the quality of machine-defined text behavior.

[0040] Train a two-channel model to estimate the accuracy of behavioral descriptions in input machine-labeled text data. (See attached image) Figure 3 As shown, the model includes an anomaly detection model based on the C3D feature video branch (this model consists of a lightweight temporal network; see step 3.1.1 for details), and an anomaly detection model based on the GPT3 large language model and cue words. The cue words inform the large language model to analyze whether the input text sentence contains abnormal behavior. The training data uses existing surveillance video language datasets. The C3D feature video branch anomaly detection model takes videos from the dataset as input, while the GPT3 large language model takes descriptive text and cue words from the dataset as input. Both outputs determine whether an anomaly exists, and cross-entropy loss is used to adjust the results. Figure 3 The trainable neural network module can be used for training.

[0041] Subsequently, by inputting a pair of video and machine-annotated text descriptions into the trained model, the accuracy of the automatically labeled behavioral descriptions can be determined by judging whether the outputs of the two branches in anomaly detection are consistent.

[0042] Step 3.1.1: Lightweight Temporal Network Construction

[0043] Video segment features extracted by C3D are input into a lightweight multi-scale temporal module for anomaly detection. This temporal module consists of two connected models, with the input features passing through both modules, such as... Figure 4As shown: one is a single-layer 1D dilated convolutional layer (kernel size 3, dilation rate r = 3), and the other is a lightweight Transformer layer: a single-layer Transformer (attention mechanism and feedforward network FFN) using an improved window attention mechanism. The improved attention mechanism is as follows: the time axis is divided into blocks (e.g., window size = 5), and self-attention is computed within each window. Skip connections across windows: a hop global connection is added every two windows (e.g., window 1→4→7…) to preserve key long-range dependencies. The output features of the dilated convolution and the Transformer are fused through residual connections, and then the fused output features are... Figure 3 The anomaly prediction head (single-layer MLP) outputs the anomaly probability.

[0044] The specific parameter settings of the model are as follows: the dilated convolutional layer uses 256 convolutional kernels (number of output channels), the convolutional kernel time span is 3, the dilation rate is 3, padding of 6 ensures temporal alignment of input and output, and is equipped with the ReLU activation function; the lightweight Transformer layer projects the input features to 256 dimensions, adopts an improved attention mechanism with a window size of 5, sets 4 attention heads (64 dimensions each), establishes cross-window skip connections every 2 windows (e.g., window 1→4→7), the feedforward network hidden layer has a dimension of 512 and uses GELU activation, and layer normalization and residual connections are introduced; in the feature fusion stage, the dilated convolution and the 256-dimensional features output by the Transformer are added element-wise, and finally anomaly probabilities are generated through a single-layer MLP prediction head (256-dimensional input, 1-dimensional output, Sigmoid activation). The design achieves lightweighting with a unified dimension (256), complements local perception through dilated convolution with long-range modeling of Transformer, and balances computational efficiency with key temporal dependency capture by combining cross-window skip connections (global interaction every 15 frames), making it suitable for real-time video anomaly detection scenarios.

[0045] Step 3.2: Use cross-validation to filter the datasets that ultimately deviate from controllable parameters.

[0046] An iterative-validation loop is adopted, repeatedly filtering out data samples with high behavioral deviations from the machine-labeled data in multiple iterations, retaining samples with low behavioral deviations. These samples are used to train the dual-path detection model initialized in step 3.1, and the confidence level of the data samples is tested. Specifically, in each iteration, the unfiltered candidate machine-labeled set C (initially all data in the entire machine-labeled dataset) is divided into two disjoint parts. One part is used as the training set to train the dual-path detection model until the model converges (i.e., the loss hardly decreases), and the other part is used as the test set for sample filtering. During the training of the dual-path model, the anomaly detection and temporal network parts of the video branch are updated with parameters, while the parameters of other parts do not need to be updated. During testing, the algorithm stores the samples in the test set that are accurately judged by the dual-path model into the controllable behavioral deviation data sample set S1, and removes samples with low confidence according to a pre-set removal ratio γ, thereby further improving the cleanliness of S1. Subsequently, the test and training sets are swapped, and the re-initialized dual-path model is used for training and testing. This process is repeated to obtain another set of controllable behavioral deviation data, S2. After this, S1, S2, and data where the dual-path model made inaccurate judgments are removed from the candidate machine-labeled set C, and S1 and S2 are added to the sample set S. Then, the accuracy ρ of set S is tested using a dual-path detection model pre-trained on a manually labeled dataset. Finally, according to the formula ρ = (1-ε2)... 2 +ε2 2 The behavioral deviation ε2 of the data samples in S is calculated from the accuracy ρ. If the deviation is too high (greater than 50%), the low-quality data in the dataset can be filtered more aggressively by adjusting the removal ratio γ.

[0047] Finally, after multiple iterations to meet the deviation requirement, the sample set S becomes the machine-annotated text dataset output in this step.

[0048] Step 4: Generation of a dataset with controllable deviations in subject and behavior

[0049] Based on the data obtained in the final sample set S, the formula ρ=(1-ε2) is used. 2 +ε2 2 The behavioral deviation ε2 of the data samples is calculated, and based on the subject deviation ε1 obtained in step 2, the overall deviation array of the dataset can be obtained as ε = [ε1, ε2]. This dataset has known noise; therefore, this part of the data can serve as an important source of training data for the multimodal surveillance video language understanding model. Subsequently, a weakly supervised neural network training scheme that considers data noise can be designed to enhance the language understanding capability of multimodal surveillance videos.

Claims

1. A method for automatically generating a video language dataset with controllable subject and behavior deviation, comprising the following steps: For any surveillance video set, first, an enhanced surveillance video captioning model is constructed using a target tracking module to automatically label the description text of the surveillance video and control the deviation degree of the subject in the description text; then, a data filtering model based on iterative deviation cross-validation is used to filter the description text to control the deviation degree of the behavior in the text description, and finally, a video language dataset with controllable subject and deviation can be obtained; The target tracking module is used to construct an enhanced surveillance video captioning model, which adopts a video coding strategy that fuses global and local features. First, the input raw video is fixedly sampled to extract a plurality of video frame segments, and global video features are extracted through a visual encoder. At the same time, a pre-trained object detector is used to identify and locate the objects in each video frame, and a target filter is used to track and analyze the motion trajectory of the detected objects in the video frame, so as to further filter out the objects that change in consecutive frames and cover other parts outside the object detection frame to obtain images containing only the tracked target, which are then input into the visual encoder to obtain local video features. Finally, the global video features and local video features output by the visual encoder are fused, and the text features are input into a multi-modal decoder together to predict the text output and generate text labels with low subject description deviation; The data filtering model based on iterative deviation cross-validation is used to filter the description text, which is implemented as follows: A two-way model is trained to estimate whether the behavior description in the input machine-labeled text data is accurate. The model includes an anomaly detection model based on a C3D feature video branch and an anomaly detection model based on a GPT3 large language model and a prompt word, wherein the prompt word is used to inform the large language model to analyze whether the input text sentence contains abnormal behavior. The training data uses an existing surveillance video language dataset, wherein the input of the C3D feature video branch anomaly detection model is the video in the dataset, and the input of the GPT3 large language model is the description text and the prompt word in the dataset. The output of both branches is whether there is an anomaly, and the cross-entropy loss is used to train the trainable neural network module. Then, a pair of video and machine-labeled text description is input into the trained model, and whether the automatically labeled behavior description is accurate can be determined by judging whether the outputs of the two branches in the anomaly detection are consistent. In the manner of an iterative-verification cycle, repeatedly filter out high behavior deviation data samples and retain low behavior deviation samples from the machine-labeled data in multiple iterations, use the samples to train the initialized two-path detection model and test the confidence of the data samples; specifically, in each iteration, divide the candidate machine-labeled set C that has not been filtered into two non-intersecting parts, one part is used as a training set to train the two-path detection model until the model converges, and the other part is used as a test set for sample filtering; during the training of the two-path model, the anomaly detection and timing network part of the video branch updates the parameters, and the parameters of other parts do not need to be updated; during the testing process, the algorithm stores the samples in the test set that are accurately judged by the two-path model into the sample set S1 of controllable behavior deviation data, and removes the samples with low confidence according to the pre-set removal ratio γ, thereby further improving the cleanliness of S1; then, exchange the above test set and training set, use the reinitialized two-path model for training and testing, and repeat the above process, to obtain another sample set S2 of controllable behavior deviation data; after that, remove S1, S2 and the data that is not accurately judged by the two-path model from the candidate machine-labeled set C, and add S1 and S2 to the sample set S, and then use the two-path detection model pre-trained on the manually labeled data set to test the accuracy ρ of the set S; Subsequently, according to the formula p = (1 - ε2) 2 + ε2 2 The behavior deviation degree ε2 of the data samples in S is calculated from the accuracy ρ; if the deviation degree is too high, i.e. greater than 50%, the value of the removal ratio γ is adjusted to more aggressively filter the low-quality data of the data set.

2. The method of claim 1, wherein: During training of the model, an existing video language dataset is used; the model uses visual features augmented by a target detector, target filter, and generated text sequence to jointly predict the next word; the loss function for training of the model continues to use cross-entropy loss, i.e., the correct index given the previous t-1 words and visual representation v, to get the prediction, p of the current tthword y denoted as a probability, whose training loss is computed as l1 is the word length.

3. The method of claim 1, wherein: The description text is used to describe the event occurring in the monitoring video.

4. The method of claim 1, wherein: The subject deviation degree in the description text is calculated by inputting the pair of machine automatically labeled subject words and manually labeled subject words for the same video in the existing video language data set into BERT to extract feature vectors, and then calculating the deviation degree based on the semantic distribution alignment method, and the value is denoted as ε1; specifically, A set of subject words in the machine-labeled text data is represented as S m A set of subject words in the human-labeled text data is represented as S a First, each subject word is mapped to a unified feature vector space using a pre-trained language model to obtain a feature set; the feature mean vectors μ m a and the feature covariance matrices Σ, Σ of the machine-labeled subject set S m a Based on the mean and covariance information, the semantic distribution alignment deviation ε1 is defined as follows:​​ 5. The method of claim 1, wherein: The timing network branch in the anomaly detection model is specifically: The video segment features extracted by C3D are input into the lightweight multi-scale timing module for anomaly detection. This part of the timing module is connected by two paths. The input features pass through two modules, one is a 1D dilated convolution layer with a convolution kernel size of 3 and an expansion rate r=3, and the other is a lightweight Transformer layer: only one layer of Transformer, i.e. attention mechanism and feedforward network FFN; the dilated convolution output features and the Transformer output features are fused through residual connection, and then the fused output features are output through the anomaly prediction head to output the anomaly probability.

6. The method of claim 5, wherein, The attention mechanism is specifically: divide the time axis into blocks, set the window size = 5, and calculate the self-attention in each window; cross-window skip connection: add a skip global connection every 2 windows to establish key long-range dependencies.

7. The method of claim 1, wherein: The subject and deviation controllable video language data set is characterized in that: The sample set S of the data filtering model based on the iterative deviation cross-validation is a controllable video language data set, the subject deviation degree of the machine-labeled text in the data set is ε1, the behavior deviation degree is ε2, and the array of the overall deviation degree of the data set is ε=[ε1, ε2].

Images