A method and system for pseudo-label denoising of video action detection based on self-training

Abstract

The present invention discloses a method and system for denoising pseudo labels for video action detection based on self-training, which relates to the field of intelligent safety monitoring of power system videos and is used to improve the ability of video action detection models to utilize unlabeled video data, including: using a semi-supervised framework based on a teacher-student model to reduce the noise of low-quality pseudo labels without adding additional parameter restrictions, thereby improving the efficiency of utilizing unlabeled video data. The pseudo labels are sorted from high to low according to the evaluation quality to ensure that the model learns differentially for pseudo labels of different qualities; the accuracy of low-quality pseudo labels is improved through a self-fine-tuning module; based on the improved pseudo labels, a training method based on time series prediction invariance is further proposed to help the model learn more robust spatiotemporal features. Compared with general semi-supervised methods, the present invention introduces active learning and pseudo-label self-fine-tuning technology in the field of semi-supervised video action detection for the first time, which can generate pseudo labels with higher accuracy than the default self-training settings.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent video safety monitoring for power systems, and in particular to a method and system for denoising pseudo-labels for video action detection based on self-training, which is applicable to power grid operation scenarios. Background Art

[0002] Training temporal action detection models relies on large amounts of manually annotated data. Video annotation is more complex and time-consuming than image annotation. Furthermore, in the context of intelligent power safety monitoring, video content annotation requires specialized knowledge. This makes it challenging to label video data from power operation scenes, necessitating the integration of pseudo-label learning techniques. However, learning effective task representations from unlabeled videos is a challenging task. Existing methods are mostly based on self-learning and pseudo-labeling. However, due to the presence of categorical errors and temporal boundary deviations in pseudo-labels, learning from unlabeled videos using pseudo-labels has been less than ideal. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by the present invention is: how to solve the problem that the prior art has unsatisfactory results when using pseudo-label training containing noise.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for pseudo-label denoising of video action detection based on self-training, which comprises the following steps:

[0006] Extract the spatiotemporal semantic features of the video. Through the pre-trained visual encoder, extract the visual features containing temporal information and spatial information to obtain the temporal feature vector and spatial feature vector;

[0007] The video feature vector is mapped to a subspace suitable for temporal action detection tasks. Temporal feature vectors and spatial feature vectors containing clear semantic information are introduced. A feature fusion and mapping module is designed to obtain spatiotemporal feature vectors corresponding to the video length.

[0008] According to the spatiotemporal feature vector corresponding to the video length, two identical detection models are initialized, one as a supervision model for generating pseudo labels, and the other as a training model for updating weights.

[0009] The detection model includes a multi-layer feature encoding converter, a multi-layer group normalization layer, an action category prediction branch and a time series prediction branch, and obtains preliminary action predictions through reasoning of the supervised model;

[0010] The preliminary action predictions are ranked from high to low quality according to the pseudo-label evaluation ranking method;

[0011] The self-tuning module is used to denoise 50% of the pseudo labels after quality ranking. The self-tuning module reuses the inference module of the supervision model, performs several inferences and aggregates the inference results to obtain the denoised pseudo labels.

[0012] As a preferred solution of the method for pseudo-label denoising of video action detection based on self-training described in the present invention, wherein: the spatiotemporal semantic features of the video are extracted by pre-training a visual encoder to extract visual features containing temporal information and spatial information, and the temporal feature vector and the spatial feature vector obtained include:

[0013] Input action video sequence , where M represents the total number of frames, Represents the first Video frames;

[0014] Process the action video sequence into a sequence of T video clips :

[0015] ;

[0016] in, is the total number of video clips, each clip contains 16 frames, and the interval between clips is 4 frames;

[0017] Based on the obtained fragment sequence, features are extracted, RGB frames and optical flow frames are extracted from the fragments, and d-dimensional vectors are obtained by passing them through 3D convolutional neural networks. 、 , the video feature vector sequence expression is:

[0018] ,

[0019] , ,

[0020] in, is the RGB feature of the t-th segment, is the optical flow feature of the t-th segment, concat represents splicing in the channel dimension, is the spatiotemporal feature of the tth segment, is the video semantic vector space, The time series length of the feature, is the number of channels of the feature.

[0021] As a preferred solution of the method for pseudo-label denoising of video action detection based on self-training described in the present invention, the video feature vector is mapped to a subspace suitable for the task of temporal action detection, temporal feature vectors and spatial feature vectors containing clear semantic information are introduced, and a feature fusion and mapping module is designed to obtain a spatiotemporal feature vector corresponding to the video length, including:

[0022] Based on the transposition of the obtained video feature vector sequence F, the local context information of the temporal dimension is aggregated through a feature fusion and mapping module composed of a stacked one-dimensional convolutional neural network to obtain the visual feature vector ,in, is the D-dimensional feature vector of the t-th segment.

[0023] As a preferred solution of the method for pseudo-label denoising of video action detection based on self-training described in the present invention, the video feature vector is mapped to a subspace suitable for the task of temporal action detection, temporal feature vectors and spatial feature vectors containing clear semantic information are introduced, and a feature fusion and mapping module is designed to obtain a spatiotemporal feature vector corresponding to the video length, including:

[0024] Based on the transposition of the obtained video feature vector sequence F, the local context information of the temporal dimension is aggregated through a feature fusion and mapping module composed of a stacked one-dimensional convolutional neural network to obtain the visual feature vector ,in, is the D-dimensional feature vector of the t-th segment.

[0025] As a preferred solution of the method for denoising pseudo labels for video action detection based on self-training described in the present invention, wherein: the method initializes two identical detection models according to the spatiotemporal feature vector corresponding to the video length, one as a supervision model for generating pseudo labels, and the other as a training model for updating weights, including:

[0026] The detection model consists of a multi-layer feature encoding converter, each layer of which adds a converter module with one-dimensional convolution modules of different sizes. The input of the first layer is the visual feature vector The output of the first layer is the visual feature vector of the further processed time series double downsampling , the current visual feature vector is used as the input of the second layer, and is forward propagated layer by layer, and the output of each layer is merged into , where L is the number of layers of the multi-layer feature encoding converter, Output of layer l , For the L-th layer output, if T cannot be Divisible by The time dimension is expanded to , is greater than or equal to T and can be The smallest integer that is divisible;

[0027] The detection model contains a multi-layer group normalization layer, which includes L layers of group normalization layers. The output of each layer is passed through the group normalization layer to obtain a deep representation of the video features. ,in, is the output of the lth group normalization layer, , Normalize the output of the Lth layer group;

[0028] The detection model includes an action category prediction branch, which consists of two layers of one-dimensional convolutional neural networks and a fully connected network. Each layer is connected through a normalization layer and a ReLU activation layer. For each layer, the output of the normalization layer is , the output of the action category prediction branch is , where C is the number of action categories that need to be predicted;

[0029] The detection model includes a time series prediction branch, which consists of two layers of one-dimensional convolutional neural networks and a fully connected network. Each layer is connected through a normalization layer and a ReLU activation layer. For each layer, the output of the normalization layer is , the output result of the branch predicted by the timing boundary is ,Of the two dimensions, the first dimension represents the action start time point predicted by the current ,segment, and the second dimension represents the action end time point predicted by the ,current segment;

[0030] A single video gets several prediction results, where the j-th action prediction is expressed as ,in, is the action category, is the start time of the action in the video, The end time of the action in the video;

[0031] The output results of the action category prediction branches of each layer Splicing ,in is the action category output by the first layer, is the action category output by the second layer, The action category output by the Lth layer is clear. ;

[0032] To get and , the output results of the timing prediction branches of each layer Splicing ,in is the timing boundary of the first layer output, is the timing boundary of the second layer output, For the temporal boundary output by the Lth layer, select the N action predictions with the highest confidence according to A in the temporal dimension, and select N corresponding temporal intervals. Use the confidence threshold to filter to obtain a preliminary action prediction and clarify the start time of the action in the video. and when the action ends in the video , all action predictions contained in a video are regarded as a pseudo label.

[0033] As a preferred solution of the method for pseudo-label denoising of video action detection based on self-training described in the present invention, the preliminary action prediction is sorted from high to low quality according to the pseudo-label evaluation sorting method, including:

[0034] Get several pseudo labels from several videos and calculate the category uncertainty and sample difficulty of each pseudo label.

[0035] Several pseudo labels and corresponding A and B are obtained from several videos. For an action prediction j, all time intervals of the same type and overlapping time intervals are obtained from A and B. , calculate the temporal bound uncertainty of each pseudo label, where represents the action start time in the pseudo-label, represents the end time of the action in the pseudo-label, and K represents the number of times a random offset is added to an action interval;

[0036] A comprehensive evaluation is completed based on the category uncertainty of each pseudo-label, the sample difficulty of each pseudo-label, and the temporal boundary uncertainty of each pseudo-label.

[0037] As a preferred embodiment of the method for denoising pseudo labels for video action detection based on self-training described in the present invention, the denoising of the 50% pseudo labels after quality sorting is performed using a self-tuning module, the self-tuning module reuses the inference module of the supervision model, performs several inferences and aggregates the inference results to obtain denoised pseudo labels, including:

[0038] According to the pseudo-label, a random offset is added to each predicted time series interval in the 50% pseudo-labels after confidence ranking, and a new offset interval is obtained by adding the offset based on the prediction. ,in is the start time of the new offset interval, The end time of the new offset interval;

[0039] Repeat the operation K times to obtain a set of adjacent intervals with random offsets ,in is the starting time of the ith adjacent interval of the jth pseudo label, For the end time of the i-th adjacent interval of the j-th pseudo-label, expand the interval to twice the length of the interval, generate a set of expanded intervals and extract the features of the corresponding time series positions, add noise and re-infer through the time series prediction branch of the supervision model to obtain K new time series predictions, average the K new time series predictions to obtain the final denoised time series interval, and update each prediction of the pseudo-label.

[0040] As a preferred embodiment of the method for denoising pseudo labels for video action detection based on self-training described in the present invention, the method further comprises: denoising 50% of the pseudo labels after quality sorting using a self-fine-tuning module, the self-fine-tuning module reuses the inference module of the supervision model, performs several inferences, and aggregates the inference results to obtain denoised pseudo labels,

[0041] A Gaussian sampling-based temporal scaling enhancement strategy is applied to all extracted temporal features. The action category prediction branch of the training model is trained using the classification loss function, and the temporal interval loss function is used to train the temporal prediction branch of the training model. The training model is ranked based on pseudo-label evaluation, and the 50% pseudo-labels with comprehensive evaluation scores above the threshold are first trained for E rounds.

[0042] The updated low-scoring pseudo-labels are added to the training process, and the training model is trained for E rounds. The weights of the training model are updated to the supervision model through exponential smoothing averaging.

[0043] Another object of the present invention is to provide a self-training-based video action detection pseudo-label denoising system, which can extract visual features containing temporal information and spatial information in the video by introducing a pre-trained visual encoder module, optimize the features in the temporal and spatial dimensions in combination with the feature fusion and mapping modules, and use the inference module to generate action prediction results, and then generate high-quality pseudo-labels through the detection model module, and at the same time use the self-fine-tuning module to denoise and optimize low-quality pseudo-labels, thereby solving the problems of unstable pseudo-label quality, insufficient feature processing and limited model dynamic optimization capabilities in the prior art, thereby effectively improving the accuracy and robustness of video action detection.

[0044] To solve the above technical problems, the present invention provides the following technical solutions: a pseudo-label denoising system for video action detection based on self-training, comprising: a pre-trained visual encoder module, a feature fusion and mapping module, an inference module, a detection model module, and a self-fine-tuning module;

[0045] The pre-trained visual encoder module is to divide the video into multiple segments by inputting the action video sequence, each segment contains temporal information and spatial information.

[0046] Extract RGB frames and optical flow frames from the clips, and encode them through a pre-trained 3D convolutional neural network to embed the temporal and spatial information of the video into feature vectors. This generates temporal and spatial feature vectors, provides feature expressions with rich semantics, and outputs a sequence of visual feature vectors.

[0047] The feature fusion and mapping module maps the visual feature sequence of the video to a subspace suitable for temporal action detection, performs feature fusion based on the different characteristics of temporal and spatial information, designs a one-dimensional convolutional neural network as the feature fusion component, processes the local context information of the temporal dimension layer by layer, aggregates the multi-layer convolution results, and outputs a spatiotemporal feature vector consistent with the length of the video;

[0048] The inference module generates preliminary action predictions by inputting spatiotemporal features, including action categories, time intervals, and confidence levels, and uses a layer-by-layer calculation method to perform prediction reasoning, combining the outputs of the action category prediction branch and the time series prediction branch;

[0049] The detection model module works collaboratively with two identical detection models, responsible for pseudo-label generation and model weight update respectively. It combines all features to generate pseudo-labels, including action categories and their corresponding time intervals, and then sorts and evaluates them.

[0050] The self-tuning module uses the reasoning ability of the supervised model to perform denoising on pseudo labels whose quality ranking is lower than a threshold, adds random offsets to low-quality pseudo labels, generates multiple perturbed time series intervals, obtains different prediction results through reasoning, aggregates the prediction results, generates denoised time series intervals, and uses the denoised pseudo labels to further optimize the training model. Through the time series scaling enhancement strategy based on Gaussian sampling and the classification loss and time series interval loss functions, the dynamic improvement of model performance is achieved.

[0051] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the above-mentioned method for pseudo-label denoising of video action detection based on self-training.

[0052] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for pseudo-label denoising of video action detection based on self-training.

[0053] The beneficial effects of the present invention are as follows: the present invention extracts the temporal and spatial features of the video through a pre-trained visual encoder module, optimizes the features in the temporal and spatial dimensions in combination with the feature fusion and mapping modules, generates action prediction results using the inference module, and generates high-quality pseudo labels through the detection model module. At the same time, a self-tuning module is used to denoise and optimize low-quality pseudo labels, thereby solving the problems of unstable pseudo label quality, insufficient feature processing and low denoising efficiency in the prior art, thereby effectively improving the accuracy and robustness of video action detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 This is an overall flow chart of a method for pseudo-label denoising for video action detection based on self-training provided by the first embodiment of the present invention.

[0056] Figure 2 A flowchart of a pseudo-label evaluation and ranking method in a pseudo-label denoising method for video action detection based on self-training is provided in accordance with the second embodiment of the present invention.

[0057] Figure 3 A flowchart of a pseudo-label self-tuning method for a pseudo-label denoising method for video action detection based on self-training is provided in accordance with the second embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0059] Example 1, with reference to Figure 1 According to one embodiment of the present invention, a method for pseudo-label denoising of video action detection based on self-training is provided, comprising:

[0060] Step 1: Extract the spatiotemporal semantic features of the video. Through the pre-trained visual encoder, extract the visual features containing temporal information and spatial information to obtain the temporal feature vector and the spatial feature vector.

[0061] Input action video sequence , where M represents the total number of frames, Represents the first Video frames. The action sequence is further processed into a sequence of T video clips , ,in represents the t-th video segment, each segment contains 16 frames, and the interval between segments is 4 frames.

[0062] Extract features from the obtained fragment sequence, extract RGB (Red Green Blue original video frame) frames and optical flow frames from the fragment, and pass them through the 3D convolutional neural network to obtain d-dimensional vectors 、 , the video feature vector sequence is ,in , , is the video semantic vector space.

[0063] Step 2: Map the video feature vector to a subspace suitable for the temporal action detection task. Use the obtained temporal feature vector and spatial feature vector containing clear semantic information to design a feature fusion and mapping module to obtain the spatiotemporal feature vector corresponding to the video length.

[0064] Transpose the obtained video feature vector sequence F to obtain , through the feature fusion and mapping module composed of stacked one-dimensional convolutional neural networks, the local context information of the time series dimension is aggregated to obtain the visual feature vector ,in is the D-dimensional feature vector of the t-th segment.

[0065] Step 3. Based on the obtained spatiotemporal feature vector corresponding to the video length, two identical detection models are initialized. One is used as a supervision model to generate pseudo labels, and the other is used as a training model to update weights. The model contains a multi-layer feature encoding converter, a multi-layer group normalization layer, an action category prediction branch, and a time series prediction branch. Preliminary action predictions are obtained through reasoning of the supervision model.

[0066] First, the detection model consists of a multi-layer feature encoding converter, each layer of which adds a converter module with different-sized one-dimensional convolution modules. The input of the first layer is the visual feature vector The output of the first layer is the visual feature vector of the further processed time series double downsampling , the visual feature vector is used as the input of the second layer and propagated forward layer by layer, and the output of each layer is combined into , where L is the number of layers of the multi-layer feature encoding converter, Output of layer l If T cannot be Divisible by The time dimension is expanded to , is greater than or equal to T and can be The smallest integer that is divisible;

[0067] Subsequently, the detection model includes a multi-layer group normalization layer, which includes L layers of group normalization layers. The outputs of each layer in step 3.1) are passed through the group normalization layer to obtain the deep representation of the video features. ,in ;

[0068] Secondly, the detection model contains an action category prediction branch, which consists of two layers of one-dimensional convolutional neural networks and a fully connected network, with a normalization layer and a ReLU activation layer between each layer. , the output of the action category prediction branch is , where C is the number of action categories that need to be predicted;

[0069] Again, the detection model contains a time series prediction branch, which consists of two layers of one-dimensional convolutional neural networks and a fully connected network, with a normalization layer and a ReLU activation layer between each layer. For each layer, the output of the normalization layer is , the output of the action category prediction branch is ,Of the two dimensions, the first dimension represents the action start time point predicted by the current segment, and the second dimension represents the action end time point predicted by the current segment;

[0070] Finally, a single video may get multiple prediction results, where the j-th action prediction is expressed as , the three elements are action category, start time in the video, and end time in the video, where , it is necessary to splice the output results of the action category prediction branches of each layer , the output results of the timing prediction branches of each layer are spliced ​​together to obtain , select the topK action predictions with the largest confidence based on A in the time series dimension, and select the topK corresponding time series intervals, using the confidence threshold The preliminary action predictions are obtained by filtering, and all action predictions contained in a video are regarded as a pseudo label.

[0071] Among them, Topk is the number of action predictions obtained by filtering based on confidence.

[0072] Step 4: Reference Figure 2Based on the obtained preliminary action predictions, the preliminary action predictions are sorted from high to low quality according to the pseudo-label evaluation ranking method, where the pseudo-label evaluation ranking method includes three evaluation criteria, namely category uncertainty, sample difficulty, and temporal boundary uncertainty.

[0073] First, based on the N pseudo labels obtained from N videos, the category uncertainty of each pseudo label is calculated:

[0074] ,

[0075] Among them, J is the number of predictions in pseudo labels, C is the number of actions, is the confidence score of the i-th action in the j-th prediction;

[0076] Then, based on the N pseudo labels obtained from N videos, the sample difficulty of each pseudo label is calculated:

[0077] ,

[0078] in, is the confidence threshold of the prediction, is the confidence score vector of the j-th prediction, ;

[0079] Then, based on the N pseudo labels obtained from N videos and the corresponding A and B, for an action prediction j, all time intervals of the same type and overlapping time intervals are obtained from A and B. ,in is the start time of the time series interval, For the end time of the time series interval, calculate the time series boundary uncertainty of each pseudo label:

[0080] ,

[0081] in, is the starting time point of the j-th action prediction, is the predicted end time point of the j-th action, Represents the standard deviation of a vector;

[0082] Finally, according to the calculated , calculate the comprehensive evaluation score:

[0083] ,

[0084] in, is the category uncertainty, is the pseudo-label sample difficulty, represents the boundary uncertainty, 、 、 They are The weight of For all pseudo labels The maximum value of , M is the comprehensive score of each pseudo label, all pseudo labels are sorted in descending order according to the calculated M, the first 50% are regarded as high-quality pseudo labels, and the last 50% are regarded as low-quality pseudo labels.

[0085] Step 5: Reference Figure 3 According to the obtained pseudo-label sorting, 50% of the pseudo-labels after quality sorting are denoised using the self-tuning module. The self-tuning module reuses the inference module of the supervision model, performs several inferences and aggregates the inference results to obtain the denoised pseudo-labels.

[0086] First, based on the obtained pseudo-labels, a random offset is added to each predicted time series interval in the 50% pseudo-labels ranked by confidence:

[0087] ,

[0088] in, is the random offset intensity of the interval start time, The random offset strength of the interval end time. Add the offset based on the prediction to get the new offset interval Repeat the operation of adding offsets K times to obtain a set of adjacent intervals of random offsets ;

[0089] Then, according to the obtained set of random offset adjacent intervals , Represents the i-th adjacent interval generated by the j-th pseudo-label and expands the interval to twice the length of the interval, thus generating a set of expanded intervals:

[0090] ,

[0091] in, To expand the start time of the interval, is the end time of the expanded interval. Extract the features of the corresponding time series position for this group of expanded intervals:

[0092] ,

[0093] in, Represents the corresponding characteristics of the first expanded interval, Represents the corresponding characteristics of the second expanded interval, Represents the corresponding features of the K-th expanded interval, respectively In the corresponding time interval 、 、 Obtained above.

[0094] Finally, based on the features of the K expanded intervals obtained for an action prediction, these features are added with noise and re-inferred through the time series prediction branch of the supervised model to obtain K new time series predictions. These K new time series predictions are averaged to obtain the final denoised time series interval:

[0095] ,

[0096] in, is the start time of the denoising time series interval, is the end time of the denoising time series interval, and model is the time series prediction branch of the supervision model.

[0097] After obtaining the final time series interval, update the predictions of the pseudo labels:

[0098] ,

[0099] in, They are the category of pseudo label, action start time, and action end time respectively.

[0100] Step 6. In the final model training stage, the updated pseudo-labels are used to supervise the training model and update the weights.

[0101] First, a time series scaling enhancement strategy based on Gaussian sampling is used for all extracted time series features, and a classification loss function is used. Train the action category prediction branch of the training model, where , and whether it is a positive or negative sample, γ is the tempering factor for focusing on difficult samples, the larger γ is, the smaller the impact of simple samples; use the time series interval loss function Train the time series prediction branch of the training model, where ρ is the Euclidean distance between the prediction interval of the training model and the time series interval of the corresponding pseudo label, and c is the length of the union of the prediction interval of the training model and the time series interval of the corresponding pseudo label;

[0102] Then, according to the obtained pseudo-label evaluation ranking, the training model is first trained for E rounds using 50% of pseudo-labels with comprehensive evaluation scores higher than the threshold;

[0103] Then, the updated low-scoring pseudo-labels are added to the training process, and the training model is trained for E rounds.

[0104] Finally, the weights of the training model are updated to the supervised model through exponential smoothing averaging:

[0105] ,

[0106] in, is the weight of the training model, is the weight of the supervision model, The weights updated for the training model, is the updated weight coefficient.

[0107] This invention is mainly used to make full use of labeled videos and unlabeled videos to improve the accuracy of action detection models when there is a shortage of professional labelers.

[0108] Example 2, an embodiment of the present invention, provides a system for pseudo-label denoising method for video action detection based on self-training, comprising: a pre-trained visual encoder module, a feature fusion and mapping module, an inference module, a detection model module, and a self-tuning module;

[0109] The pre-trained visual encoder module is to divide the video into multiple segments by inputting the action video sequence, each segment contains temporal information and spatial information.

[0110] Extract RGB frames and optical flow frames from the clips, and encode them through a pre-trained 3D convolutional neural network to embed the temporal and spatial information of the video into feature vectors. This generates temporal and spatial feature vectors, provides feature expressions with rich semantics, and outputs a sequence of visual feature vectors.

[0111] The feature fusion and mapping module maps the visual feature sequence of the video to a subspace suitable for temporal action detection, performs feature fusion based on the different characteristics of temporal and spatial information, designs a one-dimensional convolutional neural network as the feature fusion component, processes the local context information of the temporal dimension layer by layer, aggregates the multi-layer convolution results, and outputs a spatiotemporal feature vector consistent with the length of the video;

[0112] The inference module generates preliminary action predictions by inputting spatiotemporal features, including action categories, time intervals, and confidence levels, and uses a layer-by-layer calculation method to perform prediction reasoning, combining the outputs of the action category prediction branch and the time series prediction branch;

[0113] The detection model module works collaboratively with two identical detection models, responsible for pseudo-label generation and model weight update respectively. It combines all features to generate pseudo-labels, including action categories and their corresponding time intervals, and then sorts and evaluates them.

[0114] The self-tuning module uses the reasoning ability of the supervised model to perform denoising on pseudo labels whose quality ranking is lower than a threshold, adds random offsets to low-quality pseudo labels, generates multiple perturbed time series intervals, obtains different prediction results through reasoning, aggregates the prediction results, generates denoised time series intervals, and uses the denoised pseudo labels to further optimize the training model. Through the time series scaling enhancement strategy based on Gaussian sampling and the classification loss and time series interval loss functions, the dynamic improvement of model performance is achieved.

[0115] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0116] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0117] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting, or processing in another suitable manner as necessary, and then storing it in a computer memory.

[0118] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0119] Example 3: In this example, in order to verify the beneficial effects of the present invention, economic benefit calculation and simulation experiments were conducted to scientifically demonstrate the effectiveness of the present invention. This example conducted experiments on the existing traditional method and the method of this example.

[0120] This embodiment proposes a self-training model. Compared with existing methods, PLD can generate higher quality pseudo labels.

[0121] This paper uses two commonly used video action detection datasets: THUMOS14 and ActivityNet v1.3. Furthermore, it utilizes 4,019 videos collected from real power operation scenarios. These videos comprise a training set of 3,319, a validation set of 700, and a test set of 697, covering five action categories.

[0122] The video preprocessing steps are as follows: First, RGB and optical flow data are extracted from the video frame by frame. Next, features are extracted from the RGB and optical flow frames using an I3D model pre-trained on the ImageNet dataset. Feature extraction is performed in segments of 16 frames, with a 4-frame interval between segments. For example, the first three segments extract information from frames [0-15], [4-19], and [8-23], respectively. The I3D output has 1024 feature channels. Finally, the RGB and optical flow features are concatenated to produce a 2048-dimensional feature tensor.

[0123] The proposed model is configured as follows: 1. The visual encoder uses a two-stream I3D model; 2. The feature fusion and mapping modules of the detection model use two layers of one-dimensional convolutional blocks, and the feature encoder uses a seven-layer stack of feedforward transformers for multi-scale feature extraction. The action category prediction branch and the time series prediction branch each contain two layers of one-dimensional convolutional neural networks and one layer of fully connected networks; 4. The PLD model is optimized using the Adam optimizer with an initial learning rate of 1 × 10−4 and a weight drop rate of 5 × 10−2 during training and inference.

[0124] This paper uses mean average precision (mAP) to evaluate the performance of the PLD model. mAP is calculated as follows: First, for each prediction result, its intersection over union (IoU) with all ground truths is calculated. Then, multiple IoU thresholds are set based on task requirements. The IoU thresholds for THUMOS14 are [0.3:0.7:0.1], and for ActivityNet v1.3 are [0.5:0.95:0.05]. Next, for each prediction result, if its IoU with a ground truth is greater than or equal to the set threshold and the ground truth has not been matched by a previous prediction result, it is classified as a prediction problem. If the IoU between the prediction result and the ground truth is less than the threshold, or if the IoU is greater than the threshold but the ground truth has been matched by a previous prediction result, it is classified as a prediction problem. Based on the number of prediction problems, prediction problems, and prediction problems, the precision at the current IoU threshold is calculated. The category precisions at all IoU thresholds are averaged to obtain the precision (AP) for each category. The APs for all categories are averaged to obtain the mean average precision (mAP). mAP is between 0 and 1, and the larger the mAP is, the higher the detection accuracy is.

[0125] In the three datasets, the PLD model proposed in this paper significantly improves the mAP compared with the state-of-the-art methods, which shows that the PLD model of this invention can generate higher quality speech.

[0126] Table 1: Mean Average Precision of the Models

[0127] ,

[0128] Referring to Table 1, the results show that: through verification using three datasets: THUMOS14, ActivityNet v1.3, and real power scenarios, the mAP results of the PLD model proposed in this paper are all better than those of other models, proving that the PLD model can effectively generate high-quality pseudo labels.

[0129] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A pseudo-label denoising method for video action detection based on self-training, characterized in that: include: Extract the spatiotemporal semantic features of the video. Use a pre-trained visual encoder to extract visual features containing temporal and spatial information, and then concatenate the two types of visual features to obtain a video feature vector. The video feature vector is mapped to a subspace suitable for temporal action detection tasks. Temporal feature vectors and spatial feature vectors containing clear semantic information are introduced. A feature fusion and mapping module is designed to obtain spatiotemporal feature vectors corresponding to the video length. Based on the spatiotemporal feature vector corresponding to the video length, two identical detection models are initialized, one as a supervision model to generate pseudo labels, and the other as a training model to update weights; The detection model includes a multi-layer feature encoder converter, a multi-layer group normalization layer, an action category prediction branch and a time series prediction branch, and obtains preliminary action predictions through reasoning of the supervised model; The preliminary action predictions are ranked from high to low quality according to the pseudo-label evaluation ranking method; The self-tuning module is used to denoise the 50% of pseudo labels after quality ranking. The self-tuning module reuses the inference module of the supervision model, performs several inferences, and aggregates the inference results to obtain the denoised pseudo labels. The method initializes two identical detection models based on the spatiotemporal feature vector corresponding to the video length, one as a supervision model for generating pseudo labels, and the other as a training model for updating weights, including: The detection model consists of a multi-layer feature encoding converter, each layer adds a converter module with one-dimensional convolution modules of different sizes, and the input of the first layer is the spatiotemporal feature vector The output of the first layer is the temporal feature vector of the further processed time series double downsampling The current spatiotemporal feature vector is used as the input of the second layer and propagated forward layer by layer, and the output of each layer is merged into {F1, F2, ... F l ,...,F L }, where L is the number of layers of the multi-layer feature encoding converter, F l Output of layer l F L For the L-th layer output, if T cannot be 2 l Divisible by The time dimension is extended to T by filling 0 ' , T ' Is greater than or equal to T and can be 2 L The smallest integer that is divisible; The detection model includes a multi-layer group normalization layer, which includes L layers of group normalization layers. The output of each layer is passed through the group normalization layer to obtain the deep representation of the spatiotemporal feature vector video features {F'1,F'2,…,F' l ,...,F' L }, where F' l is the output of the lth group normalization layer, F' L Normalize the output of the Lth layer group; The detection model includes an action category prediction branch, which consists of two layers of one-dimensional convolutional neural networks and a fully connected network. Each layer is connected through a normalization layer and a ReLU activation layer. For each layer, the output F' of the normalization layer is l , the output of the action category prediction branch is Among them, C is the number of action categories that need to be predicted; The detection model includes a time series prediction branch, which includes two layers of one-dimensional convolutional neural networks and a fully connected network. Each layer is connected through a normalization layer and a ReLU activation layer. For each layer, the output F' of the normalization layer is l , the output result of the branch predicted by the timing boundary is Of the two dimensions, the first dimension represents the predicted action start time point of the current segment, and the second dimension represents the predicted action end time point of the current segment; A single video gets several prediction results, where the j-th action prediction is expressed as in, is the action category, is the start time of the action in the video, The end time of the action in the video; The output results of the action category prediction branch of each layer A1, A2, ..., A L Splicing in is the action category output by the first layer, is the action category output by the second layer, The action category output by the Lth layer is clear. To get and The output results B1, B2, ..., B of each layer's timing prediction branch are L Splicing in is the timing boundary of the first layer output, is the timing boundary of the second layer output, For the temporal boundary output by the Lth layer, select the N action predictions with the highest confidence according to A in the temporal dimension, and select N corresponding temporal intervals. Use the confidence threshold to filter to obtain a preliminary action prediction and clarify the start time of the action in the video. and when the action ends in the video All action predictions contained in a video are regarded as a pseudo label.

2. The method for pseudo-label denoising of video action detection based on self-training according to claim 1, characterized in that: The spatiotemporal semantic features of the video are extracted by pre-training the visual encoder to extract visual features containing temporal information and spatial information, and then the two types of visual features are spliced ​​to obtain the video feature vector including: Input action video sequence Frames={fr1,fr2,…fr t ,…,fr M }, where M represents the total number of frames, fr t represents the tth video frame of the action video sequence; Process the action video sequence into a sequence of T video clips Clips: Clips={c1,c2,…c t ,...,c T }; Where T is the total number of video clips, each clip contains 16 frames, and the interval between clips is 4 frames; Based on the obtained fragment sequence, features are extracted, RGB frames and optical flow frames are extracted from the fragments, and d-dimensional vectors are obtained by passing them through 3D convolutional neural networks. The video feature vector sequence expression is: F={f1,f2,…f t ,…,f T } in, is the RGB feature of the t-th segment, is the optical flow feature of the tth segment, concat represents splicing in the channel dimension, f t is the spatiotemporal feature of the t-th segment, R is the video semantic vector space, T is the temporal length of the feature, and d is the number of channels of the feature.

3. The method for denoising pseudo labels for video action detection based on self-training according to claim 2, characterized in that: The video feature vector is mapped to a subspace suitable for the temporal action detection task, the temporal feature vector and the spatial feature vector containing clear semantic information are introduced, and the feature fusion and mapping module is designed to obtain the spatiotemporal feature vector corresponding to the video length. Based on the transposition of the obtained video feature vector sequence F, the local context information of the time series dimension is aggregated through the feature fusion and mapping module composed of stacked one-dimensional convolutional neural networks to obtain the spatiotemporal feature vector in, is the D-dimensional feature vector of the t-th segment.

4. The method for denoising pseudo labels for video action detection based on self-training according to claim 3, characterized in that: The preliminary action predictions are sorted from high to low quality according to the pseudo label evaluation ranking method, including: Get several pseudo labels from several videos and calculate the category uncertainty and sample difficulty of each pseudo label. Several pseudo labels and corresponding A and B are obtained from several videos. For an action prediction j, all time intervals of the same type and overlapping time intervals are obtained from A and B. Calculate the temporal bound uncertainty of each pseudo label, where represents the action start time in the pseudo-label, represents the end time of the action in the pseudo-label, and K represents the number of times a random offset is added to an action interval; A comprehensive evaluation is completed based on the category uncertainty of each pseudo-label, the sample difficulty of each pseudo-label, and the temporal boundary uncertainty of each pseudo-label.

5. The method for denoising pseudo labels for video action detection based on self-training according to claim 4, characterized in that: The 50% pseudo labels after quality ranking are denoised using the self-tuning module. The self-tuning module reuses the reasoning module of the supervision model, performs several reasonings and aggregates the reasoning results to obtain the denoised pseudo labels, including: According to the pseudo-label, a random offset is added to each predicted time series interval in the 50% pseudo-labels after confidence ranking, and a new offset interval is obtained by adding the offset based on the prediction. in is the start time of the new offset interval, The end time of the new offset interval; Repeat the operation K times to obtain a set of adjacent intervals with random offsets in is the starting time of the ith adjacent interval of the jth pseudo label, For the end time of the i-th adjacent interval of the j-th pseudo-label, expand the interval to twice the length of the interval, generate a set of expanded intervals and extract the features of the corresponding time series positions, add noise and re-infer through the time series prediction branch of the supervision model to obtain K new time series predictions, average the K new time series predictions to obtain the final denoised time series interval, and update each prediction of the pseudo-label.

6. The method for pseudo-label denoising of video action detection based on self-training according to claim 5, characterized in that: The 50% pseudo labels after quality ranking are denoised using a self-tuning module, the self-tuning module reuses the reasoning module of the supervision model, performs several inferences and aggregates the inference results to obtain denoised pseudo labels, and further includes: A Gaussian sampling-based temporal scaling enhancement strategy is applied to all extracted temporal features. The action category prediction branch of the training model is trained using the classification loss function, and the temporal interval loss function is used to train the temporal prediction branch of the training model. The training model is first trained for E rounds using 50% of the pseudo labels with a comprehensive evaluation score higher than the threshold, sorted by pseudo label evaluation. The updated low-scoring pseudo-labels are added to the training process, and the training model is trained for E rounds. The weights of the training model are updated to the supervision model through exponential smoothing averaging.

7. A system for pseudo-label denoising of video action detection based on self-training, applying the pseudo-label denoising method of video action detection based on self-training according to any one of claims 1 to 6, characterized in that: Includes pre-trained visual encoder module, feature fusion and mapping module, reasoning module, detection model module and self-tuning module; The pre-trained visual encoder module is to divide the video into multiple segments by inputting the action video sequence, each segment contains temporal information and spatial information. Extract RGB frames and optical flow frames from the clips, and encode them through a pre-trained 3D convolutional neural network to embed the temporal and spatial information of the video into feature vectors. This generates temporal and spatial feature vectors, provides feature expressions with rich semantics, and outputs a video feature vector sequence. The feature fusion and mapping module maps the video feature vector sequence of the video to a subspace suitable for temporal action detection, performs feature fusion based on the different characteristics of temporal and spatial information, designs a one-dimensional convolutional neural network as a feature fusion component, processes the local context information of the temporal dimension layer by layer, aggregates the multi-layer convolution results, and outputs a spatiotemporal feature vector consistent with the length of the video; The inference module generates preliminary action predictions by inputting spatiotemporal feature vectors, including action categories, time intervals, and confidence levels, and performs prediction reasoning using a layer-by-layer calculation method, combining the outputs of the action category prediction branch and the time series prediction branch; The detection model module works collaboratively with two identical detection models, responsible for pseudo-label generation and model weight update respectively. It combines all features to generate pseudo-labels, including action categories and their corresponding time intervals, and then sorts and evaluates them. The self-tuning module uses the reasoning ability of the supervised model to perform denoising on pseudo labels whose quality ranking is lower than a threshold, adds random offsets to low-quality pseudo labels, generates multiple perturbed time series intervals, obtains different prediction results through reasoning, aggregates the prediction results, generates denoised time series intervals, and uses the denoised pseudo labels to further optimize the training model. Through the time series scaling enhancement strategy based on Gaussian sampling and the classification loss and time series interval loss functions, the dynamic improvement of model performance is achieved.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the processor implements the steps of a method for pseudo-label denoising of video action detection based on self-training according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for pseudo-label denoising of video action detection based on self-training described in any one of claims 1 to 6 are implemented.

Images