Online Action Detection Method and System for Progressive Privilege Information Distillation

Abstract

The present invention provides an online action detection method and system for progressive privileged information distillation, comprising: step S1: extracting features from an uncropped video using a video feature extraction network to obtain a video feature sequence that can express video semantic information; step S2: capturing videos of different lengths after a current moment, and using corresponding video feature sequences to respectively train a plurality of different offline teacher models; step S3: using the extracted video feature sequence to train an online student model to obtain a trained online student model; step S4: aligning features between the offline teacher model and the online student model at different levels of the model layer; step S5: after feature alignment, the online student model learns from a plurality of different offline teacher models by progressive model distillation; step S6: using the extracted video feature sequence to predict the result of action detection at the current moment according to the online student model after distillation learning.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and image processing, and specifically, to an online action detection method and system for progressive privileged information distillation. Background Art

[0002] With the development of technologies such as transmission and storage, videos have become an important carrier for information transmission, and hundreds of millions of videos are generated every day. The online action detection task, that is, to locate the time position where an action occurs in real time from a video, has wide application values, such as real-time video monitoring and autonomous driving, etc. With the development of deep learning, the methods for online action detection can be classified into the following several types.

[0003] One is to use a recurrent neural network model to model the video sequence, so as to output a judgment on the action at the current moment. Specifically, first, video data is subjected to feature extraction to obtain a video feature sequence, and then a recurrent neural network model, such as LSTM or GRU, etc., is used to model the video feature sequence, so that it outputs a judgment on the action prediction at the current moment at each moment. Since online videos are unidirectional on the time scale, the recurrent neural network model often also adopts a unidirectional structure. Therefore, its disadvantage is that the judgment on the action at the current moment only utilizes the current moment and the previous video frames. In addition to using recurrent neural networks to model online videos, another method is to predict the video after the current moment, and then judge the action at the current moment based on the predicted result and the observed video data. This type of method models the implicit relationship between what may happen at the current moment and in the future, thereby improving the performance of the online action detection model.

[0004] Patent document CN111050116A (application number: 201910872679.6) discloses a system and method for online action detection using a temporal recurrent network. This method obtains the image data at the current moment and the past image frames, models the data using a recurrent neural network, and outputs at least one target-oriented action during the current image frame. This method only models and utilizes the relationship between the current moment and the previous image content for online action detection, and it ignores the implicit information between what may happen in the future, and this implicit information is very helpful for online action detection. For example, humans often predict what may happen in the future when making judgment decisions while driving. Summary of the Invention

[0005] Aiming at the defects in the prior art, the purpose of the present invention is to provide an online action detection method and system for progressive privileged information distillation.

[0006] An online action detection method for progressive privileged information distillation according to the present invention includes:

[0007] Step S1: extracting features from the uncut video using a video feature extraction network to obtain a video feature sequence that can express the semantic information of the video;

[0008] Step S2: intercept videos of different lengths after the current moment, and use corresponding video feature sequences to train multiple different offline teacher models respectively;

[0009] Step S3: training the online student model using the extracted video feature sequence to obtain a trained online student model;

[0010] Step S4: aligning features of the trained offline teacher model and the trained online student model at different levels of the model level;

[0011] Step S5: After feature alignment, the online student model learns from multiple different offline teacher models through progressive model distillation;

[0012] Step S6: Use the extracted video feature sequence to predict the result of action detection at the current moment according to the online student model after distillation learning.

[0013] Preferably, the video feature extraction network adopts: using two 3D convolutional neural networks to respectively extract RGB and optical flow features of the video clips, and splicing the extracted RGB and optical flow features;

[0014] The 3D convolutional neural network can continuously extract features from each video clip in the video time direction, thereby obtaining a feature sequence of the video.

[0015] Preferably, the offline teacher model adopts:

[0016]

[0017] Among them, based on the action probability prediction at time t0, represents the output target of the offline teacher model; x t represents the feature sequence of the extracted video; t represents time; t0 represents the current moment; the input data observed by the offline teacher model T ranges from t0-m to t0+n; different lengths of time n after the current moment are intercepted to train different offline teacher models respectively;

[0018] The offline teacher model T uses a one-dimensional convolution operation as a basic unit. When the constructed offline teacher model observes video sequence features and the range before and after the current moment is different m≠n, the one-dimensional convolution kernel is an asymmetric structure; by stacking multiple layers of one-dimensional convolution layers, an offline teacher model is obtained.

[0019] Preferably, the online student model adopts:

[0020]

[0021] Among them, based on the action probability prediction at time t0, represents the output target of the online student model; x t represents the extracted video feature sequence; t represents time; t0 represents the current moment; the online student model S observes that the input data range is from t0 - m to t0;

[0022] The online student model S uses a one - dimensional convolution operation as the basic unit. The constructed online student model can only observe the video sequence features at the current moment and before. Then the one - dimensional convolution kernel is an asymmetric structure. By stacking multiple layers of one - dimensional convolution layers, the online student model is obtained.

[0023] Preferably, the step S4 adopts:

[0024] Step S4.1: For the feature expressions of the offline teacher model and the online student model at the same moment, use the divided preset - dimension online student model features to learn from the offline teacher model, so as to ensure that the privilege information differences corresponding to the feature expressions of the offline teacher model and the online student model at the same moment are the same;

[0025] Step S4.2: For the feature expressions of the offline teacher model and the online student model at the same layer, use the divided preset - dimension online student model features to learn from the offline feature model features, so as to ensure that the privilege information differences corresponding to the feature expressions of the offline teacher model and the online student model at the same layer are the same.

[0026] Preferably, the step S5 adopts: The online student model performs distillation learning with multiple different offline teacher models respectively through the way of progressive model distillation according to preset requirements, so as to complete progressive distillation when the privilege information of the offline teacher model is increasing continuously; the specific distillation forms include constraints at the output probability level and constraints at the feature level;

[0027] The constraints at the output probability level adopt:

[0028]

[0029] The constraints at the feature level adopt:

[0030]

[0031] Among them, L and N respectively represent the number of layers of the model and the length of the video feature sequence; represents the function for measuring the KL divergence; τ is the temperature coefficient; σ represents the softmax function; and Respectively represent the action probability output of the online student model and the offline teacher model at time t; and They represent the aligned feature expressions of the online student model and the offline teacher model at time t and the lth layer of the model respectively.

[0032] According to the present invention, an online action detection system for progressive privilege information distillation is provided, comprising:

[0033] Module M1: Use the video feature extraction network to extract features from the uncropped video to obtain a video feature sequence that can express the semantic information of the video;

[0034] Module M2: capture videos of different lengths after the current moment, and use the corresponding video feature sequences to train multiple different offline teacher models;

[0035] Module M3: train the online student model using the extracted video feature sequence to obtain a trained online student model;

[0036] Module M4: Align features of the trained offline teacher model and the trained online student model at different levels of the model level;

[0037] Module M5: After feature alignment, the online student model learns from multiple different offline teacher models through progressive model distillation;

[0038] Module M6: Use the extracted video feature sequence to predict the result of action detection at the current moment based on the online student model after distillation learning.

[0039] Preferably, the video feature extraction network adopts: using two 3D convolutional neural networks to respectively extract RGB and optical flow features of the video clips, and splicing the extracted RGB and optical flow features;

[0040] The 3D convolutional neural network can continuously extract features from each video clip in the video time direction, thereby obtaining a feature sequence of the video.

[0041] Preferably, the offline teacher model adopts:

[0042]

[0043] Among them, based on the action probability prediction at time t0, represents the output target of the offline teacher model; x tIt represents the feature sequence of the extracted video; t represents time; t0 represents the current moment; the input data range observed by the offline teacher model T is from t0 - m to t0 + n; different durations n after the current moment are intercepted, and different offline teacher models are trained respectively;

[0044] The offline teacher model T takes one-dimensional convolution operation as the basic unit. When the constructed offline teacher model observes different ranges before and after the current moment in the video sequence features (m ≠ n), the one-dimensional convolution kernel has an asymmetric structure; by stacking multiple layers of one-dimensional convolutional layers, the offline teacher model is obtained;

[0045] The online student model adopts:

[0046]

[0047] Among them, based on the action probability prediction at the t0 moment, It represents the output target of the online student model; x t It represents the extracted video feature sequence; t represents time; t0 represents the current moment; the input data range observed by the online student model S is from t0 - m to t0;

[0048] The online student model S takes one-dimensional convolution operation as the basic unit. The constructed online student model can only observe the video sequence features at the current moment and before it, so the one-dimensional convolution kernel has an asymmetric structure. By stacking multiple layers of one-dimensional convolutional layers, the online student model is obtained.

[0049] Preferably, the module M4 adopts:

[0050] Module M4.1: For the feature expressions of the offline teacher model and the online student model at the same moment, use the divided preset dimension of the online student model features to learn from the offline teacher model, so as to ensure that the privilege information differences corresponding to the feature expressions of the offline teacher model and the online student model at the same moment are the same;

[0051] Module M4.2: For the feature expressions of the offline teacher model and the online student model at the same layer, use the divided preset dimension of the online student model features to learn from the offline feature model features, so as to ensure that the privilege information differences corresponding to the feature expressions of the offline teacher model and the online student model at the same layer are the same;

[0052] The module M5 adopts: The online student model conducts distillation learning with multiple different offline teacher models respectively according to preset requirements through the method of progressive model distillation, so as to complete progressive distillation when the privilege information of the offline teacher model is continuously increasing; the specific distillation forms include constraints at the output probability level and constraints at the feature level;

[0053] The constraints at the output probability level adopt:

[0054]

[0055] The constraints at the feature level adopt:

[0056]

[0057] Among them, L and N respectively represent the number of layers of the model and the length of the video feature sequence; represents the function for measuring the KL divergence; τ is the temperature coefficient; σ represents the softmax function; and respectively represent the action probability outputs of the online student model and the offline teacher model at time t; and respectively represent the aligned feature expressions of the online student model and the offline teacher model at time t and the l-th layer of the model.

[0058] Compared with the prior art, the present invention has the following beneficial effects: When training the online action detection model, the present invention uses the knowledge learned by the offline action detection model, and transfers this part of the learned knowledge, that is, privileged information, to the online action detection model through the method of progressive model distillation. This enables the online action detection model to utilize the implicit subsequent video content information during testing without explicitly predicting the subsequent video content, thereby improving the performance of the online action detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects, and advantages of the present invention will become more apparent:

[0060] Figure 1 is the flowchart of the method in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that those of ordinary skill in the art can make several changes and improvements without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0062] Example 1

[0063] Aiming at the defects in the prior art, the purpose of the present invention is to provide an online action detection method and system for progressive privileged information distillation, which distills the privileged information in the offline teacher model into the online student model progressively, so that the online model that cannot observe the video content after the current moment approaches the performance of the offline model and completes the action localization in the online video.

[0064] An online action detection method for progressive privileged information distillation provided by the present invention, as Figure 1 shown, includes:

[0065] Step S1: Use a video feature extraction network to extract features from the uncropped video to obtain a video feature sequence that can express the semantic information of the video;

[0066] Step S2: Intercept videos of different durations after the current moment, and use the corresponding video feature sequences to train multiple different offline teacher models respectively;

[0067] Step S3: Use the extracted video feature sequence to train the online student model to obtain the trained online student model;

[0068] Step S4: Align the features between different levels at the model level of the trained offline teacher model and the trained online student model;

[0069] Step S5: Since there is an information difference in the input video content between the offline teacher model and the online student model, based on how much of the subsequent video content that the online student model can observe based on the offline teacher model, after feature alignment, the online student model learns from multiple different offline teacher models from easy to difficult through the way of progressive model distillation;

[0070] Step S6: Use the extracted video feature sequence to predict the result of action detection at the current moment according to the online student model after distillation learning;

[0071] The offline teacher model uses one-dimensional convolution operation as the basic unit, and the offline teacher model can observe the video content after the current moment;

[0072] The online student model uses one-dimensional convolution operation as the basic unit, and the online student model can observe the video content at and before the current moment.

[0073] Specifically, the video feature extraction network adopts: using two 3D convolutional neural networks to extract the RGB and optical flow features of the video segment respectively, and splicing the extracted RGB and optical flow features;

[0074] The 3D convolutional neural network can continuously extract features from each video segment in the time direction of the video, so as to obtain the feature sequence of the video.

[0075] Specifically, the offline teacher model adopts: the offline teacher model can obtain the entire video content, including the video content before and after the current moment. For the online scenario, the video content after the current moment cannot be obtained and can be regarded as privileged information;

[0076] The offline teacher model adopts:

[0077]

[0078] Among them, represents the output target of the offline teacher model; x t represents the feature sequence of the extracted video; t represents time; t0 represents the current moment; for the action probability prediction at the t0 moment, the input data range that the offline teacher model T can observe is from t0 - m to t0 + n; according to the amount of video content after the current moment t0 observed, that is, the size of n, different offline teacher models are trained respectively to be used as the teacher models in the subsequent model distillation;

[0079] The offline teacher model T takes one-dimensional convolution operation as the basic unit. When the constructed offline teacher model may have different front and back ranges in observing the video sequence features and the current moment, that is, the sizes of m and n may be different, the one-dimensional convolution kernel is an asymmetric structure; by stacking multiple layers of one-dimensional convolutional layers, the offline teacher model is obtained. Generally, taking the training of 4 offline teacher models as an example, the trained offline teacher models are denoted as T1, T2, T3, and T4.

[0080] Specifically, the online student model adopts: since the online model cannot obtain the video content after the current moment, the expression of the online student model is:

[0081]

[0082] Among them, represents the output target of the online student model; x t represents the feature sequence of the extracted video; t represents time; t0 represents the current moment; for the action probability prediction at the t0 moment, the input data range that the online student model S observes is from t0 - m to t0;

[0083] The online student model S takes one-dimensional convolution operation as the basic unit. The constructed online student model can only observe the video sequence features at the current moment and before, so the one-dimensional convolution kernel is an asymmetric structure. By stacking multiple layers of one-dimensional convolutional layers, the online student model is obtained.

[0084] Specifically, step S4 adopts the following method: The offline teacher model and the online student model are stacked by one-dimensional convolutional layers. However, due to the differences in input information and the differences in feature receptive fields brought about by different model layers, it is somewhat difficult to directly constrain the feature expressions of the offline teacher model and the online student model through model distillation at the feature level. Therefore, in the distillation process of privileged information, it is necessary to align the features in two aspects. For the differences in input information, since the online student model lacks the observation of video content after the current moment, it is unreasonable to constrain the feature expressions of the offline teacher model and the online student model to be exactly the same at the same moment during the distillation process. For the feature expressions of the offline teacher model and the online student model at the same moment, only part of the dimensions of the online student model's features are used to learn from the offline teacher model. For the differences in feature receptive fields brought about by different model layers, for the feature expressions of the offline teacher model and the online student model at the same layer, only part of the dimensions of the online student model's features are used to learn from the offline feature model's features, so as to ensure that the differences in privileged information corresponding to the feature expressions of the offline teacher model and the online student model at the same layer are the same.

[0085] Specifically, step S5 adopts the following method: The online student model performs distillation learning with multiple different offline teacher models from easy to difficult through progressive model distillation, so as to complete progressive distillation under the condition that the privileged information of the offline teacher model is continuously increasing; for example: The online student model S first performs model distillation with the offline teacher model T1, and then performs model distillation with the offline teacher model T2, and so on until the distillation learning with the offline teacher model T4 is completed.

[0086] The specific distillation forms include constraints at the output probability level and constraints at the feature level;

[0087] The constraint at the output probability level adopts:

[0088]

[0089] The constraint at the feature level adopts:

[0090]

[0091] Among them, L and N respectively represent the number of model layers and the length of the video feature sequence; represents the function for measuring the KL divergence; τ is the temperature coefficient; σ represents the softmax function; and respectively represent the action probability outputs of the online student model and the offline teacher model at time t; and respectively represent the aligned feature representations of the online student model and the offline teacher model at time t and the l-th layer of the model. During the distillation process, by constraining and the online student model can learn the knowledge of the offline teacher model, thereby approaching the performance of the offline teacher model.

[0092] In summary, the present invention extracts a video feature sequence through a deep neural network, and trains a number of offline action detection models as offline teacher networks and an online action detection model as an online student network based on this feature. The knowledge learned by the offline teacher network, that is, privileged information, is transmitted to the online student network through the method of progressive model distillation. This enables the online student network to utilize the implicit subsequent video content information during testing without explicitly predicting the subsequent video content, thereby improving the performance of the online action detection model.

[0093] An online action detection system based on progressive privileged information distillation provided by the present invention includes:

[0094] Module M1: Use a video feature extraction network to extract features from the uncropped video to obtain a video feature sequence that can express the semantic information of the video;

[0095] Module M2: Intercept videos of different durations after the current moment, and use the corresponding video feature sequences to train multiple different offline teacher models respectively;

[0096] Module M3: Use the extracted video feature sequence to train the online student model to obtain the trained online student model;

[0097] Module M4: Align the features between different layers at the model level of the trained offline teacher model and the trained online student model;

[0098] Module M5: Since there is an information difference in the input video content between the offline teacher model and the online student model, the online student model is based on the amount of subsequent video content observable by the offline teacher model. After feature alignment, the online student model learns from multiple different offline teacher models from easy to difficult through the method of progressive model distillation;

[0099] Module M6: Use the extracted video feature sequence to predict the result of action detection at the current moment according to the online student model after distillation learning;

[0100] The offline teacher model uses one-dimensional convolution operations as basic units, and the offline teacher model can observe the video content after the current moment;

[0101] The online student model uses one-dimensional convolutional operations as basic units, and the online student model can observe the video content at the current moment and before.

[0102] Specifically, the video feature extraction network adopts: using two 3D convolutional neural networks to extract the RGB and optical flow features of video clips respectively, and splicing the extracted RGB and optical flow features;

[0103] The 3D convolutional neural network can continuously extract features from each video clip in the video time direction, so as to obtain the feature sequence of the video.

[0104] Specifically, the offline teacher model adopts: the offline teacher model can obtain the entire video content, including the video content before and after the current moment. For the online scenario, the video content after the current moment cannot be obtained and can be regarded as privileged information;

[0105] The offline teacher model adopts:

[0106]

[0107] Among them, represents the output target of the offline teacher model; x t represents the extracted feature sequence of the video; t represents time; t0 represents the current moment; for the action probability prediction at the t0 moment, the input data range that the offline teacher model T can observe is from t0 - m to t0 + n; according to the amount of video content after the current moment t0 observed, that is, the size of n, different offline teacher models are trained respectively to be used as the teacher models in the subsequent model distillation;

[0108] The offline teacher model T uses one-dimensional convolutional operations as basic units. When the constructed offline teacher model may have different front and back ranges in observing the video sequence features, that is, the sizes of m and n may be different, the one-dimensional convolutional kernel is an asymmetric structure; by stacking multiple layers of one-dimensional convolutional layers, the offline teacher model is obtained. Generally, taking the training of 4 offline teacher models as an example, the trained offline teacher models are denoted as T1, T2, T3, and T4.

[0109] Specifically, the online student model adopts: since the online model cannot obtain the video content after the current moment, the expression of the online student model is:

[0110]

[0111] Among them, represents the output target of the online student model; x tdenote the extracted video feature sequence; t represents time; t0 represents the current moment; for the action probability prediction at the moment t0, the online student model S observes the input data range from t0 - m to t0;

[0112] The online student model S uses one-dimensional convolution operations as basic units. The constructed online student model can only observe the video sequence features at the current moment and before. Then the one-dimensional convolution kernel has an asymmetric structure. By stacking multiple layers of one-dimensional convolutional layers, the online student model is obtained.

[0113] Specifically, the module M4 adopts: The offline teacher model and the online student model are stacked by one-dimensional convolutional layers. However, due to the different input information and the difference in feature receptive fields brought about by different model layers, it is difficult to directly constrain the feature expressions of the offline teacher model and the online student model through model distillation at the feature level. Therefore, two aspects of feature alignment are required during the distillation process of privileged information. For the different input information, since the online student model lacks the observation of the video content after the current moment, it is unreasonable to constrain the feature expressions of the offline teacher model and the online student model to be exactly the same at the same moment during the distillation process. For the feature expressions of the offline teacher model and the online student model at the same moment, only part of the dimensions of the online student model features learn from the offline teacher model; for the difference in feature receptive fields brought about by different model layers, for the feature expressions of the offline teacher model and the online student model at the same layer, only part of the dimensions of the online student model features learn from the offline feature model features, so as to ensure that the difference in privileged information corresponding to the feature expressions of the offline teacher model and the online student model at the same layer is the same.

[0114] Specifically, the module M5 adopts: The online student model performs distillation learning with multiple different offline teacher models respectively from easy to difficult through progressive model distillation, so as to complete progressive distillation under the condition that the privileged information of the offline teacher model is continuously increasing; for example: the online student model S first performs model distillation with the offline teacher model T1, and then performs model distillation with the offline teacher model T2, and so on until the distillation learning with the offline teacher model T4 is completed.

[0115] The specific distillation forms include constraints at the output probability level and constraints at the feature level;

[0116] The constraints at the output probability level adopt:

[0117]

[0118] The constraints at the feature level adopt:

[0119]

[0120] Wherein, L and N respectively represent the number of layers of the model and the length of the video feature sequence; represents a function for measuring the KL divergence; τ is the temperature coefficient; σ represents the softmax function; and respectively represent the action probability outputs of the online student model and the offline teacher model at time t; and respectively represent the aligned feature representations of the online student model and the offline teacher model at time t and the l-th layer of the model. During the distillation process, by constraining and the online student model can learn the knowledge of the offline teacher model, thereby approaching the performance of the offline teacher model.

[0121] In summary, the present invention extracts a video feature sequence through a deep neural network, and trains several offline action detection models as an offline teacher network and an online action detection model as an online student network based on this feature. The knowledge learned by the offline teacher network, that is, the privileged information, is transmitted to the online student network through the method of progressive model distillation. This enables the online student network to utilize the implicit subsequent video content information during testing without explicitly predicting the subsequent video content, thereby improving the performance of the online action detection model.

[0122] Those skilled in the art know that in addition to implementing the systems, devices, and their respective modules provided by the present invention in the form of pure computer-readable program code, the method steps can be logically programmed to enable the systems, devices, and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same program. Therefore, the systems, devices, and their respective modules provided by the present invention can be considered as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or the structures within the hardware component.

[0123] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments of the present application and the features in the embodiments can be combined arbitrarily with each other.

Claims

1. An online action detection method for progressive privilege information distillation, characterized in that include: Step S1: extracting features from the uncut video using a video feature extraction network to obtain a video feature sequence that can express the semantic information of the video; Step S2: intercept videos of different lengths after the current moment, and use corresponding video feature sequences to train multiple different offline teacher models respectively; Step S3: training the online student model using the extracted video feature sequence to obtain a trained online student model; Step S4: aligning features of the trained offline teacher model and the trained online student model at different levels of the model level; Step S5: After feature alignment, the online student model learns from multiple different offline teacher models through progressive model distillation; Step S6: using the extracted video feature sequence to predict the result of action detection at the current moment according to the online student model after distillation learning; The step S4 adopts: Step S4.1: For the feature expressions of the offline teacher model and the online student model at the same time, the online student model features of the divided preset dimensions are used to learn from the offline teacher model, thereby ensuring that the privileged information differences corresponding to the feature expressions of the offline teacher model and the online student model at the same time are the same; Step S4.2: For the feature expressions of the same layer of the offline teacher model and the online student model, the online student model features are learned from the offline feature model features using the divided preset dimensions, thereby ensuring that the privileged information differences corresponding to the feature expressions of the same layer of the offline teacher model and the online student model are the same; The step S5 adopts: the online student model performs distillation learning with multiple different offline teacher models respectively according to preset requirements by progressive model distillation, so as to complete progressive distillation when the privileged information of the offline teacher model continues to increase; the specific distillation form includes constraints at the output probability level and constraints at the feature level; The constraints at the output probability level are: The feature-level constraints are: Among them, L and N respectively represent the number of layers of the model and the length of the video feature sequence; The function representing the metric KL divergence; τ is the temperature coefficient; σ represents the softmax function; and respectively represent the action probability outputs of the online student model and the offline teacher model at time t; and respectively represent the aligned feature expressions of the online student model and the offline teacher model at time t and the l-th layer of the model.

2. The online action detection method for progressive privilege information distillation according to claim 1, characterized in that The video feature extraction network adopts: using two 3D convolutional neural networks to extract RGB and optical flow features of video clips respectively, and splicing the extracted RGB and optical flow features; The 3D convolutional neural network can continuously extract features from each video clip in the video time direction, thereby obtaining a feature sequence of the video.

3. The online action detection method for progressive privilege information distillation according to claim 1, characterized in that The offline teacher model adopts: Among them, based on the action probability prediction at time t0, represents the output target of the offline teacher model; x t represents the feature sequence of the extracted video; t represents time; t0 represents the current moment; the input data range observed by the offline teacher model T is from t0 - m to t0 + n; different durations n after the current moment are intercepted, and different offline teacher models are trained respectively; The offline teacher model T uses a one-dimensional convolution operation as a basic unit. When the constructed offline teacher model observes video sequence features and the range before and after the current moment is different m≠n, the one-dimensional convolution kernel is an asymmetric structure; by stacking multiple layers of one-dimensional convolution layers, an offline teacher model is obtained.

4. The online action detection method for progressive privilege information distillation according to claim 1, wherein, The online student model uses: Among them, based on the action probability prediction at time t0, represents the output target of the online student model; x t represents the extracted video feature sequence; t represents time; t0 represents the current moment; the online student model S observes the input data range from t0 - m to t0; The online student model S uses a one-dimensional convolution operation as a basic unit. The constructed online student model intelligently observes the video sequence features of the current moment and before. The one-dimensional convolution kernel is an asymmetric structure. The online student model is obtained by stacking multiple layers of one-dimensional convolution layers.

5. An online action detection system for progressive privilege information distillation, characterized in that, include: Module M1: Use the video feature extraction network to extract features from the uncropped video to obtain a video feature sequence that can express the semantic information of the video; Module M2: capture videos of different lengths after the current moment, and use the corresponding video feature sequences to train multiple different offline teacher models; Module M3: train the online student model using the extracted video feature sequence to obtain a trained online student model; Module M4: Align features of the trained offline teacher model and the trained online student model at different levels of the model level; Module M5: After feature alignment, the online student model learns from multiple different offline teacher models through progressive model distillation; Module M6: Use the extracted video feature sequence to predict the result of action detection at the current moment based on the online student model after distillation learning; The module M4 adopts: Module M4.1: For the feature expressions of the offline teacher model and the online student model at the same time, the online student model features of the divided preset dimensions are used to learn from the offline teacher model, thereby ensuring that the privileged information differences corresponding to the feature expressions of the offline teacher model and the online student model at the same time are the same; Module M4.2: For the feature expressions of the same layer of the offline teacher model and the online student model, the online student model features are learned from the offline feature model features using the preset dimensions of the division, thereby ensuring that the privileged information differences corresponding to the feature expressions of the same layer of the offline teacher model and the online student model are the same; The module M5 adopts: the online student model performs distillation learning with multiple different offline teacher models according to preset requirements by progressive model distillation, so as to complete progressive distillation when the privileged information of the offline teacher model continues to increase; the specific distillation form includes constraints at the output probability level and constraints at the feature level; The constraints at the output probability level are: The feature-level constraints are: Among them, L and N respectively represent the number of layers of the model and the length of the video feature sequence; The function representing the metric KL divergence; τ is the temperature coefficient; σ represents the softmax function; and respectively represent the action probability outputs of the online student model and the offline teacher model at time t; and respectively represent the aligned feature expressions of the online student model and the offline teacher model at time t and the l-th layer of the model.

6. The online action detection system for progressive privilege information distillation according to claim 5, wherein The video feature extraction network adopts: using two 3D convolutional neural networks to extract RGB and optical flow features of video clips respectively, and splicing the extracted RGB and optical flow features; The 3D convolutional neural network can continuously extract features from each video clip in the video time direction, thereby obtaining a feature sequence of the video.

7. The online action detection system for progressive privilege information distillation according to claim 5, characterized in that, The offline teacher model adopts: Among them, based on the action probability prediction at time t0, represents the output target of the offline teacher model; x t represents the feature sequence of the extracted video; t represents time; t0 represents the current moment; the input data range observed by the offline teacher model T is from t0 - m to t0 + n; different durations n after the current moment are intercepted, and different offline teacher models are trained respectively; The offline teacher model T uses a one-dimensional convolution operation as a basic unit. When the constructed offline teacher model has a different range m≠n between the observed video sequence features and the current time, the one-dimensional convolution kernel is an asymmetric structure. The offline teacher model is obtained by stacking multiple one-dimensional convolution layers. The online student model uses: Among them, based on the action probability prediction at time t0, represents the output target of the online student model; x t represents the extracted video feature sequence; t represents time; t0 represents the current moment; the online student model S observes the input data range from t0 - m to t0; The online student model S uses a one-dimensional convolution operation as a basic unit. The constructed online student model intelligently observes the video sequence features of the current moment and before. The one-dimensional convolution kernel is an asymmetric structure. The online student model is obtained by stacking multiple layers of one-dimensional convolution layers.

Images