Weakly supervised video temporal action detection and classification method and system

Abstract

The present invention provides a weakly supervised video temporal action detection and classification method and system, comprising: extracting a unimodal feature map based on a long input video; mapping the unimodal feature map into an action classification probability and an action localization probability map; distilling background pseudo-labels based on the action localization probability map; extracting a visual feature map and a text feature map based on the long input video and a list of action names; calculating an action detection probability map; distilling foreground pseudo-labels based on the action detection probability map; performing background co-optimization on action localization based on the background pseudo-labels; performing foreground co-optimization on action detection based on the foreground pseudo-labels; generating video classification results based on the action classification probability, and generating video detection results based on the action localization probability map. The present invention adopts a distillation collaboration strategy to promote the complementary advantages of unimodal and cross-modal frameworks, achieving more complete and accurate temporal action detection and classification.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a method and system for weakly supervised video temporal action detection and classification. Background Art

[0002] The rapid development of 5G communication technology has significantly fueled the explosion of multimedia video, thereby increasing research interest in video content understanding and retrieval. Human-centered action analysis is central to video content analysis. Therefore, temporal action detection, which detects and classifies action positions in a temporal dimension, has become crucial for video editing, recommendation, and retrieval, and has been widely researched and applied in recent years.

[0003] For a large number of long, unedited videos, we first divide them into two groups: one serves as a training set, annotated with action categories and start and end positions, while the other serves as a test set, without any annotation. The goal of temporal action detection is to use the training set to train a generalizable model that can detect the start and end time positions of all actions in the test set videos and classify and predict the detected actions. Videos may contain multiple action categories and action instances, requiring as complete detection and accurate classification as possible.

[0004] More specifically, temporal action detection can be divided into two settings: strong supervision and weak supervision. The strong supervision setting requires labeling each video in the training set with the precise start and end time locations of the action instance and the action category. With sufficient temporal boundary labels, this setting has achieved excellent detection performance. However, temporal boundary labeling is often time-consuming and labor-intensive, and it is difficult to obtain data on a large scale, thus limiting its application in practical scenarios. In contrast, the weak supervision setting only requires labeling the action category for each video in the training set. This labeling cost is low, easy to obtain on a large scale, and therefore has greater practical application value.

[0005] Existing weakly supervised temporal action detection methods based on deep neural networks all employ a unimodal multi-instance learning approach, achieving temporal detection through an action classification optimization objective. On the one hand, since the classification objective focuses on a small number of discriminative action frames, while the detection objective focuses on complete action frames, this difference in optimization objectives can confuse the model. On the other hand, the limited amount of information a single modality captures leads to a serious problem of missed detections in existing weakly supervised methods.

[0006] Patent document CN111652083A application number (202010403823.4) discloses a weakly supervised temporal action detection method and system based on adaptive sampling. It uses an adaptive sampling module and a category activation sequence mutual supervision module to alleviate the detection omission problem faced by weakly supervised tasks. However, the above patent does not consider the introduction of visual text pre-training cross-modal information to supplement the missing information of a single modality. Relying solely on a single modality and limited action category labels is not enough to provide sufficient position supervision information, making it difficult to achieve complete and accurate temporal action boundary detection.

[0007] Therefore, it is necessary to propose a new technical solution to improve the above technical problems. Summary of the Invention

[0008] In view of the defects in the prior art, the purpose of the present invention is to provide a weakly supervised video temporal action detection and classification method and system.

[0009] According to the present invention, a weakly supervised video temporal action detection and classification method is provided, the method comprising the following steps:

[0010] Step S1: Use a 3D deep convolutional feature encoding network to extract a single-modal feature map of a preset dimension from the input long video;

[0011] Step S2: Use the action classification network composed of fully convolutional layers to map the unimodal feature map into action classification probabilities of preset dimensions, and calculate the loss function with the given action classification labels;

[0012] Step S3: Use the action localization network composed of full convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions;

[0013] Step S4: For the action localization probability map, use the threshold method to convert it into background pseudo label;

[0014] Step S5: For the input action name list, use the learnable word vector to map it into an action name vector;

[0015] Step S6: For the input long video and action name vector, use the visual text pre-trained encoding network to extract the visual feature map and text feature map of preset dimensions;

[0016] Step S7: Using a temporal network composed of fully convolutional layers to enhance the visual feature map, performing matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map;

[0017] Step S8: For the action detection probability map, use the threshold method to convert it into a foreground pseudo label;

[0018] Step S9: Using the background pseudo-labels to guide the collaborative optimization of the action detection network;

[0019] Step S10: using the foreground pseudo-label to guide the collaborative optimization of the action localization network;

[0020] Step S11: for the action classification probability and action location probability map, use the threshold method to generate action category prediction and detection position prediction respectively.

[0021] Preferably, the step S1 calculates the optical flow motion information based on the RGB data, and uses a feature encoding network composed of deep convolution to map the optical flow data and the RGB data into feature maps of T*D dimensions respectively; wherein T represents the time length of the video and D represents the feature dimension of the video; the RGB features and the optical flow features are cascaded to generate a fused single-modal feature map F s , whose dimension is T*2D;

[0022] The step S2 uses the action classification network composed of full convolutional layers to transform the single modal feature map F s Mapping the action classification probability K to a preset dimension indicates the probability that each video contains different action categories; using the given action classification label to supervise the action classification probability, calculating the loss function to train the action classification network until the loss function converges;

[0023] The loss function is calculated as follows:

[0024]

[0025] Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, φ cls represents the action classification network, and H represents the cross entropy function.

[0026] Preferably, the step S3 uses an action localization network φ composed of full convolutional layers loc The unimodal feature map F s Mapped to the action localization probability map L of the preset dimension loc , represents the probability of each frame belonging to different action categories, its dimension is T*C, C represents the total number of action categories;

[0027] The step S4 is to determine the action location probability map L loc, and use the threshold method to convert it into a three-element background pseudo label P b , its dimension is T*C, including a large number of background frames and a small number of foreground frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground;

[0028] The step S5 uses the learnable word vector φ pro Map the input action name list to an action name vector.

[0029] Preferably, in step S6, for the input long video RGB and action name vector, a visual text pre-trained encoding network is used to extract a visual feature map F v and text feature map F t , whose dimensions are all T*D;

[0030] The step S7 uses a temporal network φ composed of a full convolutional layer tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability of each frame belonging to different action categories, and its dimension is T*C;

[0031] The step S8 is to calculate the motion detection probability map L dec , using the threshold method to convert it into a three-element foreground pseudo label P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

[0032] Preferably, the step S9 uses the background pseudo label P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct the loss function to train the word vector φ pro and timing network φ tem , until the loss function converges;

[0033] The loss function is calculated as follows:

[0034]

[0035] Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (XI ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, x i represents a video instance, p i,b is its background pseudo label, n i is the name of the action, · represents matrix multiplication, and H represents the cross entropy function;

[0036] The step S10 uses the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges;

[0037] The loss function is calculated as follows:

[0038]

[0039] where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo label, and H represents the cross entropy function.

[0040] The present invention also provides a weakly supervised video temporal action detection and classification system, which includes the following modules:

[0041] Module M1: Uses a 3D deep convolutional feature encoding network to extract a single-modal feature map of a preset dimension from the input long video;

[0042] Module M2: Uses an action classification network composed of fully convolutional layers to map the unimodal feature map into action classification probabilities of a preset dimension, and calculates the loss function with the given action classification label;

[0043] Module M3: Use the action localization network composed of fully convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions;

[0044] Module M4: For the action localization probability map, use the threshold method to convert it into background pseudo label;

[0045] Module M5: For the input action name list, use the learned word vector to map it into the action name vector;

[0046] Module M6: For the input long video and action name vector, use the visual text pre-trained encoding network to extract visual feature maps and text feature maps of preset dimensions;

[0047] Module M7: Use a temporal network composed of fully convolutional layers to enhance the visual feature map, perform matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map;

[0048] Module M8: For the action detection probability map, use the threshold method to convert it into a foreground pseudo label;

[0049] Module M9: Using background pseudo-labels to guide collaborative optimization of action detection networks;

[0050] Module M10: Use foreground pseudo labels to guide the collaborative optimization of the action localization network;

[0051] Module M11: For the action classification probability and action location probability map, the threshold method is used to generate action category prediction and detection position prediction respectively.

[0052] Preferably, the module M1 calculates the optical flow motion information based on the RGB data, and uses a feature encoding network composed of deep convolution to map the optical flow data and RGB data into feature maps of T*D dimensions respectively; wherein T represents the time length of the video and D represents the feature dimension of the video; RGB features and optical flow features are cascaded to generate a fused single-modal feature map F s , whose dimension is T*2D;

[0053] The module M2 uses the action classification network composed of full convolutional layers to transform the single modal feature map F s Mapping the action classification probability K to a preset dimension indicates the probability that each video contains different action categories; using the given action classification label to supervise the action classification probability, calculating the loss function to train the action classification network until the loss function converges;

[0054] The loss function is calculated as follows:

[0055]

[0056] Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, φ clsrepresents the action classification network, and H represents the cross entropy function.

[0057] Preferably, the module M3 uses an action localization network φ composed of full convolutional layers loc The unimodal feature map F s Mapped to the action localization probability map L of the preset dimension loc , represents the probability of each frame belonging to different action categories, its dimension is T*C, C represents the total number of action categories;

[0058] The module M4 is used to locate the action probability map L loc , and use the threshold method to convert it into a three-element background pseudo label P b , its dimension is T*C, including a large number of background frames and a small number of foreground frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground;

[0059] The module M5 uses the learned word vector φ pro Map the input action name list to an action name vector.

[0060] Preferably, the module M6 uses the visual text pre-trained encoding network to extract the visual feature map F for the input long video RGB and action name vector v and text feature map F t , whose dimensions are all T*D;

[0061] The module M7 uses a temporal network φ composed of full convolutional layers tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability of each frame belonging to different action categories, and its dimension is T*C;

[0062] The module M8 is used to determine the probability map L of the action detection. dec , using the threshold method to convert it into a three-element foreground pseudo label P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

[0063] Preferably, the module M9 uses the background pseudo label P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct the loss function to train the word vector φ pro and timing network φ tem , until the loss function converges;

[0064] The loss function is calculated as follows:

[0065]

[0066] Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, x i represents a video instance, p i,b is its background pseudo label, n i is the name of the action, · represents matrix multiplication, and H represents the cross entropy function;

[0067] The module M10 uses the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network Φ loc , until the loss function converges;

[0068] The loss function is calculated as follows:

[0069]

[0070] where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo label, and H represents the cross entropy function.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. This paper specifically designs a distillation collaboration framework for weak supervision. By distilling knowledge from both unimodal and multimodal branches, we leverage their complementary strengths to better address the task of temporal action detection and classification. Specifically, we use unimodal and cross-modal branches to model the video from multiple perspectives, and provide each branch with confidence position labels to facilitate collaborative training. This powerful combination enables the final framework to produce more complete and accurate classification and detection results.

[0073] 2. To distill complementary information from unimodal and cross-modal approaches, this paper exploits their respective strengths, encouraging the unimodal branch to output background priors and the cross-modal branch to output foreground priors. To reduce noise in the distilled information, this paper leverages a thresholding method to filter confident pseudo-labels, effectively improving the robustness of the distilled knowledge.

[0074] 3. In order to promote the strong combination of unimodality and cross-modality, the present invention designs a collaborative optimization strategy, using the confident background prior to supervise the cross-modal branch and the confident foreground prior to supervise the unimodal branch, fusing complementary position information during the alternating training process, thereby obtaining powerful temporal action detection and classification effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0076] Figure 1 A flow chart of a method in an embodiment of the present invention;

[0077] Figure 2 Schematic diagram of the system in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0079] Example 1:

[0080] According to the present invention, a weakly supervised video temporal action detection and classification method is provided, which comprises the following steps:

[0081] Step S1: Use a 3D deep convolutional feature encoding network to extract a single-modal feature map of a preset dimension for the input long video; calculate the optical flow motion information based on the RGB data, and use the feature encoding network composed of deep convolution to map the optical flow data and RGB data into feature maps of T*D dimensions respectively; where T represents the time length of the video and D represents the feature dimension of the video; cascade the RGB features and optical flow features to generate a fused single-modal feature map F s , whose dimension is T*2D.

[0082] Step S2: Use the action classification network composed of full convolutional layers to map the unimodal feature map to the action classification probability of the preset dimension, and calculate the loss function with the given action classification label; Use the action classification network composed of full convolutional layers to map the unimodal feature map F s The action classification probability K is mapped to a preset dimension, indicating the probability that each video contains different action categories. The action classification probability is supervised using the given action classification label, and the loss function is calculated to train the action classification network until the loss function converges.

[0083] The loss function is calculated as follows:

[0084]

[0085] Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, φ cls represents the action classification network, and H represents the cross entropy function.

[0086] Step S3: Use the action localization network composed of full convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions; Use the action localization network composed of full convolutional layers φ loc The unimodal feature map F s Mapped to the action localization probability map L of the preset dimension loc , represents the probability that each frame belongs to different action categories, its dimension is T*C, and C represents the total number of action categories.

[0087] Step S4: For the action positioning probability map, use the threshold method to convert it into a background pseudo label; for the action positioning probability map L loc , and use the threshold method to convert it into a three-element background pseudo label P b , whose dimension is T*C, including a large number of background frames and a small number of foreground frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

[0088] Step S5: For the input action name list, use the learnable word vector to map it into the action name vector; use the learnable word vector φ pro Map the input action name list to an action name vector.

[0089] Step S6: For the input long video and action name vector, use the visual text pre-trained encoding network to extract the visual feature map and text feature map of the preset dimension; for the input long video RGB and action name vector, use the visual text pre-trained encoding network to extract the visual feature map F v and text feature map F t , whose dimensions are all T*D.

[0090] Step S7: Use the temporal network composed of full convolutional layers to enhance the visual feature map, perform matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map; use the temporal network composed of full convolutional layers φ tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability that each frame belongs to different action categories, and its dimension is T*C.

[0091] Step S8: For the action detection probability map, use the threshold method to convert it into a foreground pseudo label; for the action detection probability map L dec , using the threshold method to convert it into a three-element foreground pseudo label P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

[0092] Step S9: Use the background pseudo label to guide the collaborative optimization of the action detection network; use the background pseudo label P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct the loss function to train the word vector φ pro and timing network φ tem , until the loss function converges;

[0093] The loss function is calculated as follows:

[0094]

[0095] Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, xi represents a video instance, p i,b is its background pseudo label, n i is the name of the action, · represents matrix multiplication, and H represents the cross entropy function.

[0096] Step S10: Use the foreground pseudo label to guide the collaborative optimization of the action localization network; use the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges;

[0097] The loss function is calculated as follows:

[0098]

[0099] where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo label, and H represents the cross entropy function.

[0100] Step S11: for the action classification probability and action location probability map, use the threshold method to generate action category prediction and detection position prediction respectively.

[0101] Example 2:

[0102] Example 2 is a preferred example of Example 1 and is used to illustrate the present invention in more detail.

[0103] The present invention also provides a weakly supervised video temporal action detection and classification system, which includes the following modules:

[0104] Module M1: Use 3D deep convolutional feature encoding network to extract a single-modal feature map of preset dimensions for the input long video; calculate optical flow motion information based on RGB data, and use the feature encoding network composed of deep convolution to map the optical flow data and RGB data into feature maps of T*D dimensions respectively; where T represents the time length of the video and D represents the feature dimension of the video; Cascade the RGB features and optical flow features to generate a fused single-modal feature map F s , whose dimension is T*2D.

[0105] Module M2: Use the action classification network composed of full convolutional layers to map the unimodal feature map to the action classification probability of the preset dimension, and calculate the loss function with the given action classification label; Use the action classification network composed of full convolutional layers to map the unimodal feature map F s The action classification probability K is mapped to a preset dimension, indicating the probability that each video contains different action categories. The action classification probability is supervised using the given action classification label, and the loss function is calculated to train the action classification network until the loss function converges.

[0106] The loss function is calculated as follows:

[0107]

[0108] Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, φ cls represents the action classification network, and H represents the cross entropy function.

[0109] Module M3: The action localization network composed of full convolutional layers is used to map the unimodal feature map into an action localization probability map of preset dimensions; the action localization network φ composed of full convolutional layers is used to map the unimodal feature map into an action localization probability map of preset dimensions; loc The unimodal feature map F s Mapped to the action localization probability map L of the preset dimension loc , represents the probability that each frame belongs to different action categories, its dimension is T*C, and C represents the total number of action categories.

[0110] Module M4: For the action positioning probability map, use the threshold method to convert it into a background pseudo label; for the action positioning probability map L loc , and use the threshold method to convert it into a three-element background pseudo label P b , whose dimension is T*C, including a large number of background frames and a small number of foreground frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

[0111] Module M5: For the input action name list, use the learned word vector to map it to the action name vector; use the learned word vector φ pro Map the input action name list to an action name vector.

[0112] Module M6: For the input long video and action name vector, use the visual text pre-trained encoding network to extract the visual feature map and text feature map of the preset dimension; for the input long video RGB and action name vector, use the visual text pre-trained encoding network to extract the visual feature map F v and text feature map F t , whose dimensions are all T*D.

[0113] Module M7: Use the temporal network composed of full convolutional layers to enhance the visual feature map, perform matrix multiplication on the enhanced visual feature map and the text feature map to generate the action detection probability map; use the temporal network φ composed of full convolutional layers to enhance the visual feature map and the text feature map to generate the action detection probability map. tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability that each frame belongs to different action categories, and its dimension is T*C.

[0114] Module M8: For the action detection probability map, use the threshold method to convert it into a foreground pseudo label; for the action detection probability map L dec , using the threshold method to convert it into a three-element foreground pseudo label P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

[0115] Module M9: Use background pseudo labels to guide the collaborative optimization of action detection network; use background pseudo labels P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct the loss function to train the word vector φ pro and timing network φ tem , until the loss function converges;

[0116] The loss function is calculated as follows:

[0117]

[0118] Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, xi represents a video instance, p i,b is its background pseudo label, n i is the name of the action, · represents matrix multiplication, and H represents the cross entropy function.

[0119] Module M10: Use foreground pseudo labels to guide the collaborative optimization of the action localization network; use foreground pseudo labels P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges;

[0120] The loss function is calculated as follows:

[0121]

[0122] where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo label, and H represents the cross entropy function.

[0123] Module M11: For the action classification probability and action location probability map, the threshold method is used to generate action category prediction and detection position prediction respectively.

[0124] Example 3:

[0125] Example 3 is a preferred example of Example 1 and is used to illustrate the present invention in more detail.

[0126] In response to the defects in the existing technology, the purpose of the present invention is to provide a method and system for complementary weakly supervised video temporal action detection and category prediction based on a distillation collaboration strategy. Based on the original long video with only action category labels, a weakly supervised model is trained to accurately detect the start and end time positions of action instances in the video and accurately classify the detected actions.

[0127] like Figure 1Shown is a flowchart of an embodiment of a method for complementary weakly supervised video temporal action detection and category prediction based on a distillation collaboration strategy of the present invention. The present invention specifically sets weak supervision and carefully designs a distillation collaboration framework. By distilling knowledge from both unimodal and multimodal branches, their advantages are complemented to better solve the task of temporal action detection and classification. Specifically, the present invention uses unimodal branches and cross-modal branches to perform multi-perspective modeling of the video, and provides each other with confidence position labels to facilitate collaborative training. The strong combination enables the final framework to produce more complete and accurate classification and detection results; in order to distill complementary information from unimodal and cross-modal, the present invention exploits their respective advantages, encourages unimodal branches to output background priors, and encourages cross-modal branches to output foreground priors. To reduce the noise in the distilled information, the present invention makes full use of the threshold method to screen confident pseudo-labels, effectively improving the robustness of the distilled knowledge; to promote the strong combination of unimodality and cross-modality, the present invention designs a collaborative optimization strategy, using confident background prior to supervise the cross-modal branch and confident foreground prior to supervise the unimodal branch, fusing complementary position information during the alternating training process, thereby obtaining powerful temporal action detection and classification effects.

[0128] According to the present invention, a weakly supervised video temporal action detection and classification method is provided, comprising:

[0129] Unimodal feature map extraction step: Use a 3D deep convolutional feature encoding network to extract a unimodal feature map of preset dimensions from the input long video;

[0130] Action classification step: Use the action classification network composed of fully convolutional layers to map the unimodal feature map into action classification probabilities of preset dimensions, and calculate the loss function with the given action classification label;

[0131] Action localization step: Use the action localization network composed of fully convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions;

[0132] Background pseudo-label distillation step: For the action localization probability map, use the threshold method to convert it into a background pseudo-label;

[0133] Prompt step: For the input action name list, use the learnable prompt vector to map it into the action name vector;

[0134] Cross-modal feature map extraction step: For the input long video and action name vector, use the visual text pre-trained encoding network to extract visual feature maps and text feature maps of preset dimensions;

[0135] Action detection step: Use a temporal network composed of fully convolutional layers to enhance the visual feature map, perform matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map;

[0136] Foreground pseudo-label distillation step: For the action detection probability map, use the threshold method to convert it into a foreground pseudo-label;

[0137] Background collaboration step: Background pseudo-labels are used to guide the collaborative optimization of the action detection network;

[0138] Foreground collaboration step: Foreground pseudo labels are used to guide the collaborative optimization of the action localization network;

[0139] Detection and classification result step: For the action classification probability and action positioning probability map, use the threshold method to generate action category prediction and detection position prediction respectively.

[0140] Specifically, the unimodal feature map extraction step includes: calculating optical flow motion information based on RGB data, and using a feature encoding network composed of deep convolution to map the optical flow data and RGB data into feature maps of T*D dimensions. Where T represents the time length of the video and D represents the feature dimension of the video. Then, the RGB features and optical flow features are cascaded to generate a fused unimodal feature map F s , whose dimension is T*2D.

[0141] Specifically, the action classification step includes: using an action classification network composed of a full convolution layer to classify the single-modal feature map F s The action classification probability K is mapped to a preset dimension, indicating the probability that each video contains different action categories. The action classification probability is supervised using the given action classification label, and the loss function can be calculated to train the action classification network until the loss function converges. The loss function is calculated as follows:

[0142]

[0143] Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, φ cls Represents the action classification network, and H represents the cross entropy function. Given the action classification label, based on sufficient training data, the action classification network can accurately predict the action category contained in the video.

[0144] Specifically, the action localization step includes: using an action localization network Φ composed of a full convolutional layer loc The unimodal feature map F sMapped to the action localization probability map L of the preset dimension loc , represents the probability of each frame belonging to different action categories. Its dimension is T*C, where C represents the total number of action categories. In the weakly supervised setting, due to the lack of accurate position labels for supervision, there is a certain amount of noise in the action localization probability map.

[0145] Specifically, the background pseudo-label distillation step includes: for the action positioning probability map L loc In order to reduce the impact of label noise on subsequent training, the threshold method is used to filter the confidence pseudo labels and convert them into ternary background pseudo labels P b , whose dimension is T*C, containing a large number of background frames and a small number of foreground frames. A frame value of 0 indicates that the frame belongs to the background, a frame value of -1 indicates that the frame label is uncertain, and a frame value of 1 indicates that the frame belongs to the foreground, thereby effectively improving the robustness of the distilled knowledge.

[0146] Specifically, the word prompting step includes: using a learnable word prompting vector Φ pro Map the input action name list to an action name vector.

[0147] Specifically, the cross-modal feature map extraction step includes: for the input long video RGB frame and the given action name vector, using the visual text pre-trained encoding network to extract the visual feature map F v and text feature map F t , whose dimensions are all T*D.

[0148] Specifically, the action detection step includes: using a temporal network Φ composed of a full convolutional layer tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability of each frame belonging to different action categories, and its dimension is T*C. In the weak supervision setting, due to the lack of accurate position labels for supervision, there is a certain amount of noise in the action detection probability map.

[0149] Specifically, the foreground pseudo label distillation step includes: for the action detection probability map L dec In order to reduce the impact of label noise on subsequent training, the threshold method is used to filter the confidence pseudo labels and convert them into ternary foreground pseudo labels P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. A frame value of 0 indicates that the frame belongs to the background, a frame value of -1 indicates that the frame label is uncertain, and a frame value of 1 indicates that the frame belongs to the foreground, thereby effectively improving the robustness of the distilled knowledge.

[0150] Specifically, the background collaboration step includes: using background pseudo label P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct the loss function to train the word vector φ pro and timing network φ tem , until the loss function converges. The loss function is calculated as follows:

[0151]

[0152] Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, x i represents a video instance, p i,b is its background pseudo label, n i is the action name, · denotes matrix multiplication, and H represents the cross-entropy function. The unimodal and cross-modal branches are considered as multi-view modeling of the video. By providing the action detection network with unimodal background confidence pseudo-labels and co-training them, the advantages of the unimodal branch in detecting background are integrated into the cross-modal branch, resulting in more complete and accurate action detection results.

[0153] Specifically, the foreground collaboration step includes: using the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges. The loss function is calculated as follows:

[0154]

[0155] where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,fis the foreground pseudo-label, and H represents the cross-entropy function. The unimodal and cross-modal branches are considered to model the multi-view of the video. By providing the action localization network with cross-modal foreground confidence pseudo-labels and co-training them, the advantages of the cross-modal branch in foreground detection are integrated into the unimodal branch, resulting in more complete and accurate action detection results.

[0156] A weakly supervised video temporal action detection and classification system provided by the present invention includes:

[0157] Unimodal feature map extraction module: uses a 3D deep convolutional feature encoding network to extract a unimodal feature map of preset dimensions from the input long video;

[0158] Action classification module: This module uses an action classification network composed of fully convolutional layers to map unimodal feature maps into action classification probabilities of a preset dimension, and calculates the loss function with the given action classification labels.

[0159] Action localization module: Uses an action localization network composed of fully convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions;

[0160] Background pseudo-label distillation module: For the action localization probability map, a threshold method is used to convert it into a background pseudo-label;

[0161] Prompt module: For the input action name list, use the learnable prompt vector to map it into the action name vector;

[0162] Cross-modal feature map extraction module: For the input long video and action name vector, a visual text pre-trained encoding network is used to extract visual feature maps and text feature maps of preset dimensions;

[0163] Action detection module: Use a temporal network composed of fully convolutional layers to enhance the visual feature map, perform matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map;

[0164] Foreground pseudo-label distillation module: For the action detection probability map, a threshold method is used to convert it into a foreground pseudo-label;

[0165] Background collaboration module: uses background pseudo-labels to guide collaborative optimization of action detection networks;

[0166] Foreground collaboration module: uses foreground pseudo-labels to guide the collaborative optimization of the action localization network;

[0167] Detection and classification result module: For the action classification probability and action positioning probability map, the threshold method is used to generate action category prediction and detection position prediction respectively.

[0168] Specifically, the unimodal feature map extraction module includes: calculating optical flow motion information based on RGB data, using a feature encoding network composed of deep convolution to map the optical flow data and RGB data into feature maps of T*D dimensions. Where T represents the time length of the video and D represents the feature dimension of the video. Then, the RGB features and optical flow features are cascaded to generate a fused unimodal feature map F s , whose dimension is T*2D.

[0169] Specifically, the action classification module includes: using an action classification network composed of a full convolution layer to transform the single modal feature map F s The action classification probability K is mapped to a preset dimension, indicating the probability that each video contains different action categories. The action classification probability is supervised using the given action classification label, and the loss function can be calculated to train the action classification network until the loss function converges. The loss function is calculated as follows:

[0170]

[0171] Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, Φ cls Represents the action classification network, and H represents the cross entropy function. Given the action classification label, based on sufficient training data, the action classification network can accurately predict the action category contained in the video.

[0172] Specifically, the action localization module includes: an action localization network Φ composed of a full convolutional layer loc The unimodal feature map F s Mapped to the action localization probability map L of the preset dimension loc , represents the probability of each frame belonging to different action categories. Its dimension is T*C, where C represents the total number of action categories. In the weakly supervised setting, due to the lack of accurate position labels for supervision, there is a certain amount of noise in the action localization probability map.

[0173] Specifically, the background pseudo-label distillation module includes: for the action positioning probability map L loc In order to reduce the impact of label noise on subsequent training, the threshold method is used to filter the confidence pseudo labels and convert them into ternary background pseudo labels P b, whose dimension is T*C, containing a large number of background frames and a small number of foreground frames. A frame value of 0 indicates that the frame belongs to the background, a frame value of -1 indicates that the frame label is uncertain, and a frame value of 1 indicates that the frame belongs to the foreground, thereby effectively improving the robustness of the distilled knowledge.

[0174] Specifically, the word prompting module includes: using a learnable word prompting vector Φ pro Map the input action name list to an action name vector.

[0175] Specifically, the cross-modal feature map extraction module includes: for the input long video RGB frame and the given action name vector, using the visual text pre-trained encoding network to extract the visual feature map F v and text feature map F t , whose dimensions are all T*D.

[0176] Specifically, the action detection module includes: a temporal network Φ composed of a full convolutional layer tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability of each frame belonging to different action categories, and its dimension is T*C. In the weak supervision setting, due to the lack of accurate position labels for supervision, there is a certain amount of noise in the action detection probability map.

[0177] Specifically, the foreground pseudo label distillation module includes: for the action detection probability map L dec In order to reduce the impact of label noise on subsequent training, the threshold method is used to filter the confidence pseudo labels and convert them into ternary foreground pseudo labels P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. A frame value of 0 indicates that the frame belongs to the background, a frame value of -1 indicates that the frame label is uncertain, and a frame value of 1 indicates that the frame belongs to the foreground, thereby effectively improving the robustness of the distilled knowledge.

[0178] Specifically, the background collaboration module includes: using background pseudo labels P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and build a loss function to train the word vector Φ pro and the sequential network Φ tem , until the loss function converges. The loss function is calculated as follows:

[0179]

[0180] Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, x i represents a video instance, p i,b is its background pseudo label, n i is the action name, · denotes matrix multiplication, and H represents the cross-entropy function. The unimodal and cross-modal branches are considered as multi-view modeling of the video. By providing the action detection network with unimodal background confidence pseudo-labels and co-training them, the advantages of the unimodal branch in detecting background are integrated into the cross-modal branch, resulting in more complete and accurate action detection results.

[0181] Specifically, the foreground collaboration module includes: using the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges. The loss function is calculated as follows:

[0182]

[0183] where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo-label, and H represents the cross-entropy function. The unimodal and cross-modal branches are considered to model the multi-view of the video. By providing the action localization network with cross-modal foreground confidence pseudo-labels and co-training them, the advantages of the cross-modal branch in foreground detection are integrated into the unimodal branch, resulting in more complete and accurate action detection results.

[0184] Example 4

[0185] Example 4 is a variation of Example 1.

[0186] Unimodal feature map extraction step: Using a 3D deep convolutional feature encoding network, a unimodal feature map of a preset dimension is extracted from the input long video. Each video to be detected has only the action category annotation, without the precise action boundary position annotation;

[0187] Action classification step: Use the action classification network composed of fully convolutional layers to map the unimodal feature map into action classification probabilities of preset dimensions, and calculate the loss function with the given action classification label;

[0188] Action localization step: Use the action localization network composed of fully convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions;

[0189] Background pseudo-label distillation step: For the action localization probability map, use the threshold method to convert it into a background pseudo-label;

[0190] Prompt step: For the input action name list, use the learnable prompt vector to map it into the action name vector;

[0191] Cross-modal feature map extraction step: For the input long video and action name vector, a visual text pre-trained encoding network is used to extract visual feature maps and text feature maps of preset dimensions. Each video to be tested is only annotated with the action category, without the precise action boundary location. The action name vector is pre-given and fixed, corresponding to the names of all action categories;

[0192] Action detection step: Use a temporal network composed of fully convolutional layers to enhance the visual feature map, perform matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map;

[0193] Foreground pseudo-label distillation step: For the action detection probability map, use the threshold method to convert it into a foreground pseudo-label;

[0194] Background collaboration step: Background pseudo-labels are used to guide the collaborative optimization of the action detection network;

[0195] Foreground collaboration step: Foreground pseudo labels are used to guide the collaborative optimization of the action localization network;

[0196] Detection and classification result step: For the action classification probability and action positioning probability map, use the threshold method to generate action category prediction and detection position prediction respectively.

[0197] Specifically, the weakly supervised video temporal action detection network framework consists of a single-modal feature map extraction module, an action classification module, an action localization module, a background pseudo-label distillation module, a word prompting module, a cross-modal feature map extraction module, an action detection module, a foreground pseudo-label distillation module, a background collaboration module, a foreground collaboration module, and a detection and classification result module. Figure 2 shown.

[0198] In such Figure 2 In the system framework of the embodiment shown, on the one hand, the RGB and optical flow data corresponding to the video to be detected are input into the single-modal feature map extraction module, and the single-modal feature map F is output.s , its dimension is T*2D, T represents the time length of the video, and 2D represents the feature dimension of the video. The video feature map extraction module is a downsampling encoding network composed of a series of 3D convolutional layers (+batchnorm layer + relu layer). Existing network structures such as two-stream I3D, TSN, C3D, etc. can be used. The unimodal feature map F of the video to be detected s This is input into the action classification module and mapped to an action classification probability K of dimension C*1, representing the probability that each video contains different action categories. Using the given action classification labels to supervise the action classification probabilities, a loss function can be calculated to train the action classification network until the loss function converges. The loss function is calculated as follows:

[0199]

[0200] Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, Φ cls Represents the action classification network, and H represents the cross entropy function. Given the action classification label, based on sufficient training data, the action classification network can accurately predict the action category contained in the video.

[0201] Unimodal feature map F of the video to be detected s It will also be input into the action localization module and mapped into an action localization probability map L with a dimension of T*C loc , represents the probability of each frame belonging to different action categories, and C represents the total number of action categories. In the weakly supervised setting, due to the lack of accurate position labels for supervision, there is a certain amount of noise in the action localization probability map. In order to distill complementary information from single modality and cross-modality and tap into their respective advantages, the single modality branch is encouraged to generate confident background pseudo labels. For the action localization probability map L loc In order to reduce the impact of label noise on subsequent training, the threshold method is used to filter the confidence pseudo labels and convert them into ternary background pseudo labels P b , whose dimension is T*C, containing a large number of background frames and a small number of foreground frames. A frame value of 0 indicates that the frame belongs to the background, a frame value of -1 indicates that the frame label is uncertain, and a frame value of 1 indicates that the frame belongs to the foreground, thereby effectively improving the robustness of the distilled knowledge.

[0202] On the other hand, the RGB data and action name list corresponding to the video to be detected are input into the cross-modal feature map extraction module, and the visual feature map F is output. v and text feature map F t , whose dimensions are T*D. Visual feature map F v Will pass through the timing network φ tem Enhance, and enhance the visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability of each frame belonging to different action categories, and its dimension is T*C. In the weak supervision setting, due to the lack of accurate position labels for supervision, there is a certain amount of noise in the action detection probability map. In order to distill complementary information from single modality and cross-modality, explore their respective advantages, and encourage the cross-modal branch to generate confident foreground pseudo labels. For the action detection probability map L dec In order to reduce the impact of label noise on subsequent training, the threshold method is used to filter the confidence pseudo labels and convert them into ternary foreground pseudo labels P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. A frame value of 0 indicates that the frame belongs to the background, a frame value of -1 indicates that the frame label is uncertain, and a frame value of 1 indicates that the frame belongs to the foreground, thereby effectively improving the robustness of the distilled knowledge.

[0203] In order to promote the strong combination of single modality and cross-modality, a collaborative optimization strategy is designed. The cross-modal branch is supervised by the confidence background prior, and the single modal branch is supervised by the confidence foreground prior. The complementary position information is integrated in the alternating training process to obtain a strong temporal action detection and classification effect. For the background collaborative module, the background pseudo label P is used. b Optimize the action detection network, supervise the background or foreground label frames, ignore the label uncertainty frames, and build a loss function to train the word vector φ pro and timing network φ tem , until the loss function converges. The calculation formula is as follows:

[0204]

[0205] Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, x i represents a video instance, p i,bis its background pseudo label, n i is the action name, · denotes matrix multiplication, and H represents the cross-entropy function. The unimodal and cross-modal branches are considered as multi-view modeling of the video. By providing the action detection network with unimodal background confidence pseudo-labels and co-training them, the advantages of the unimodal branch in detecting background are integrated into the cross-modal branch, resulting in more complete and accurate action detection results.

[0206] For the foreground collaboration module, use the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges. The loss function is calculated as follows:

[0207]

[0208] where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo-label, and H represents the cross-entropy function. The unimodal and cross-modal branches are considered to model the multi-view of the video. By providing the action localization network with cross-modal foreground confidence pseudo-labels and co-training them, the advantages of the cross-modal branch in foreground detection are integrated into the unimodal branch, resulting in more complete and accurate action detection results.

[0209] After the overall framework training is completed, the predicted action classification probability and action location probability map are input into the detection and classification result module, and the threshold method is used to generate action category prediction and detection location prediction respectively.

[0210] In summary, the present invention has carefully designed a framework for distillation collaboration, which distills knowledge from both unimodal and multimodal branches to promote their complementary advantages to better solve the task of temporal action detection and classification. Specifically, the present invention uses unimodal branches and cross-modal branches to perform multi-perspective modeling of videos, and provides each other with confident position labels to facilitate collaborative training, combining the strong forces to enable the final framework to produce more complete and accurate classification and detection results; in order to distill complementary information from unimodal and cross-modal, the present invention exploits their respective advantages, encourages unimodal branches to output background priors, and encourages cross-modal branches to output foreground priors. In order to reduce the noise in the distilled information, the present invention makes full use of the threshold method to screen confident pseudo-labels, effectively improving the robustness of the distilled knowledge; in order to promote the strong combination of unimodal and cross-modal, the present invention designs a collaborative optimization strategy, uses the confident background prior to supervise the cross-modal branch, and uses the confident foreground prior to supervise the unimodal branch, and fuses the complementary position information in the alternating training process, thereby obtaining a powerful temporal action detection and classification effect.

[0211] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1, Embodiment 2, and Embodiment 3.

[0212] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0213] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A weakly supervised video temporal action detection and classification method, characterized by: The method comprises the following steps: Step S1: Use a 3D deep convolutional feature encoding network to extract a single-modal feature map of a preset dimension from the input long video; Step S2: Use the action classification network composed of fully convolutional layers to map the unimodal feature map into action classification probabilities of preset dimensions, and calculate the loss function with the given action classification labels; Step S3: Use the action localization network composed of full convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions; Step S4: For the action localization probability map, use the threshold method to convert it into background pseudo label; Step S5: For the input action name list, use the learnable word vector to map it into an action name vector; Step S6: For the input long video and action name vector, use the visual text pre-trained encoding network to extract the visual feature map and text feature map of preset dimensions; Step S7: Using a temporal network composed of fully convolutional layers to enhance the visual feature map, performing matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map; Step S8: For the action detection probability map, use the threshold method to convert it into a foreground pseudo label; Step S9: Using the background pseudo-labels to guide the collaborative optimization of the action detection network; Step S10: using the foreground pseudo-label to guide the collaborative optimization of the action localization network; Step S11: using the threshold method to generate action category prediction and detection position prediction for the action classification probability and action location probability map respectively; The step S9 uses the background pseudo label P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct the loss function to train the word vector φ pro and timing network φ tem , until the loss function converges; The loss function is calculated as follows: Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, x i represents a video instance, p i,b is its background pseudo label, n i is the name of the action, · represents matrix multiplication, and H represents the cross entropy function; The step S10 uses the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges; The loss function is calculated as follows: where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo label, and H represents the cross entropy function.

2. The weakly supervised video temporal action detection and classification method according to claim 1 is characterized in that The step S1 calculates the optical flow motion information based on the RGB data, and uses a feature encoding network composed of deep convolution to map the optical flow data and RGB data into feature maps of T*D dimensions respectively; wherein T represents the time length of the video and D represents the feature dimension of the video; RGB features and optical flow features are cascaded to generate a fused single-modal feature map F s , whose dimension is T*2D; The step S2 uses the action classification network composed of full convolutional layers to transform the single modal feature map F s Mapping the action classification probability K to a preset dimension indicates the probability that each video contains different action categories; using the given action classification label to supervise the action classification probability, calculating the loss function to train the action classification network until the loss function converges; The loss function is calculated as follows: Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, φ S represents the feature encoding network, φ cls represents the action classification network, and H represents the cross entropy function.

3. The weakly supervised video temporal action detection and classification method according to claim 1 is characterized in that The step S3 uses the action localization network φ composed of full convolutional layers loc The unimodal feature map F s Mapped to the action localization probability map L of the preset dimension loc , represents the probability of each frame belonging to different action categories, its dimension is T*C, C represents the total number of action categories; The step S4 is to determine the action location probability map L loc , and use the threshold method to convert it into a three-element background pseudo label P b , its dimension is T*C, including a large number of background frames and a small number of foreground frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground; The step S5 uses the learnable word vector φ pro Map the input action name list to an action name vector.

4. The weakly supervised video temporal action detection and classification method according to claim 1 is characterized in that In step S6, for the input long video RGB and action name vector, a visual text pre-trained encoding network is used to extract the visual feature map F v and text feature map F t , whose dimensions are all T*D; The step S7 uses a temporal network φ composed of a full convolutional layer tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability of each frame belonging to different action categories, and its dimension is T*C; The step S8 is to calculate the motion detection probability map L dec , using the threshold method to convert it into a three-element foreground pseudo label P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

5. A weakly supervised video temporal action detection and classification system, characterized by: The system includes the following modules: Module M1: Uses a 3D deep convolutional feature encoding network to extract a single-modal feature map of a preset dimension from the input long video; Module M2: Uses an action classification network composed of fully convolutional layers to map unimodal feature maps into action classification probabilities of preset dimensions, and calculates the loss function with the given action classification labels; Module M3: Use the action localization network composed of fully convolutional layers to map the unimodal feature map into an action localization probability map of preset dimensions; Module M4: For the action localization probability map, use the threshold method to convert it into background pseudo label; Module M5: For the input action name list, use the learned word vector to map it into the action name vector; Module M6: For the input long video and action name vector, use the visual text pre-trained encoding network to extract visual feature maps and text feature maps of preset dimensions; Module M7: Use a temporal network composed of fully convolutional layers to enhance the visual feature map, perform matrix multiplication on the enhanced visual feature map and the text feature map to generate an action detection probability map; Module M8: For the action detection probability map, use the threshold method to convert it into a foreground pseudo label; Module M9: Using background pseudo-labels to guide collaborative optimization of action detection networks; Module M10: Use foreground pseudo labels to guide the collaborative optimization of the action localization network; Module M11: Based on the action classification probability and action location probability map, the threshold method is used to generate action category prediction and detection position prediction respectively; The module M9 uses the background pseudo label P b Guide the collaborative optimization of the action detection network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct the loss function to train the word vector φ pro and timing network φ tem , until the loss function converges; The loss function is calculated as follows: Among them, θ SV is the parameter of the cross-modal visual feature encoding network, θ ST is the parameter of the cross-modal text feature encoding network, θ pro is the parameter of the word vector, θ tem are the parameters of the temporal network, (X I ,N C ,P I,b ) represents the distribution of input video, action name list and background pseudo label, x i represents a video instance, p i,b is its background pseudo label, n i is the name of the action, · represents matrix multiplication, and H represents the cross entropy function; The module M10 uses the foreground pseudo label P f Guide the collaborative optimization of the action localization network, perform supervision on the background or foreground label frames, ignore the label uncertainty frames, and construct a loss function to train the action localization network φ loc , until the loss function converges; The loss function is calculated as follows: where θ S is the parameter of the unimodal feature encoding network, θ loc are the parameters of the action localization network, (X I ,P I,f ) is the input video, the distribution of foreground pseudo labels, x i represents a video instance, p i,f is the foreground pseudo label, and H represents the cross entropy function.

6. The weakly supervised video temporal action detection and classification system according to claim 5, characterized in that The module M1 calculates the optical flow motion information based on the RGB data, and uses the feature encoding network composed of deep convolution to map the optical flow data and RGB data into feature maps of T*D dimensions respectively; where T represents the time length of the video and D represents the feature dimension of the video; RGB features and optical flow features are cascaded to generate a fused single-modal feature map F s , whose dimension is T*2D; The module M2 uses the action classification network composed of full convolutional layers to transform the single modal feature map F s Mapping the action classification probability K to a preset dimension indicates the probability that each video contains different action categories; using the given action classification label to supervise the action classification probability, calculating the loss function to train the action classification network until the loss function converges; The loss function is calculated as follows: Among them, θ S is the parameter of the unimodal feature encoding network, θ cls are the parameters of the action classification network, (X I ,Y I ) represents the distribution of input video and action classification labels, x i Represents a video instance, y i is its action classification label, Φ S represents the feature encoding network, Φ cls represents the action classification network, and H represents the cross entropy function.

7. The weakly supervised video temporal action detection and classification system according to claim 5, characterized in that The module M3 uses the action localization network Φ composed of full convolutional layers loc The unimodal feature map F s Mapped to the action localization probability map L of the preset dimension loc , represents the probability of each frame belonging to different action categories, its dimension is T*C, C represents the total number of action categories; The module M4 is used to locate the action probability map L loc , and use the threshold method to convert it into a three-element background pseudo label P b , its dimension is T*C, including a large number of background frames and a small number of foreground frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground; The module M5 uses the learned word vector Φ pro Map the input action name list to an action name vector.

8. The weakly supervised video temporal action detection and classification system according to claim 5, characterized in that: The module M6 uses the visual text pre-trained encoding network to extract the visual feature map F for the input long video RGB and action name vector v and text feature map F t , whose dimensions are all T*D; The module M7 uses a temporal network Φ composed of full convolutional layers tem Enhanced visual feature map F v , the enhanced visual feature map F v ' and text feature map F t Perform matrix multiplication to generate the action detection probability map L dec , represents the probability of each frame belonging to different action categories, and its dimension is T*C; The module M8 is used to determine the probability map L of the action detection. dec , using the threshold method to convert it into a three-element foreground pseudo label P f , whose dimension is T*C, containing a large number of foreground frames and a small number of background frames. The frame value of 0 indicates that the frame belongs to the background, the frame value of -1 indicates that the frame label is uncertain, and the frame value of 1 indicates that the frame belongs to the foreground.

Images