Efficient long video recognition system and method based on adaptive sampling and inference

Abstract

The present invention provides a system and method for efficient long video recognition based on adaptive sampling and reasoning, which solves the problem of efficient recognition of long uncropped videos. The system of the present invention includes a video preprocessing module, a local feature extraction module, a global semantic reasoning module and a video classification module. The method of the present invention includes: preprocessing the video to be recognized, outputting a video frame sequence with standardized size; using a local feature extraction module with a hierarchical structure to locate different local events, and using a global semantic reasoning module to infer the global semantic features of the entire video based on multiple local events; and using a video classification module to map the output video classification distribution. It has been verified through experiments that the system and method of the present invention can more efficiently and accurately complete the recognition of long uncropped videos, solve the problem of efficient recognition of long videos, and eliminate the disadvantages brought about by the centralized decision network of existing recognition technology.

Description

Technical Field

[0001] The present invention relates to the field of video recognition technology, and in particular to a long video recognition system and method based on adaptive sampling and inference. Background Art

[0002] With the explosive growth of online videos, video recognition has attracted significant attention due to its wide range of applications, such as video recommendation, video retrieval, and network security. However, while classic video recognition models have achieved impressive performance, their excessive complexity and computational cost limit their deployment in many resource-constrained applications, such as edge computing and online vision. Therefore, designing efficient video recognition models that maintain recognition accuracy while minimizing computational cost becomes crucial, especially for long, untrimmed videos.

[0003] However, most current video recognition systems and methods do not take into account the efficiency of recognition and the amount of computation consumed. For example, Reference [1] trained a CNN-based recognition library of different behaviors to identify sampled video clips and obtained the probability of different behaviors appearing; Reference [2] encoded visual features and audio features separately, further concatenated them, and then further encoded features based on the self-attention mechanism, ultimately identifying highlight video clips. Some efficient video recognition methods have improved the efficiency of reasoning and can accurately and effectively realize snapshot or clip recognition, but they are still unable to effectively process uncut videos on a longer time scale. Existing efficient methods mainly focus on designing lightweight structures or adaptive methods, such as key segment selection in Reference [3], key region selection in Reference [4], and adaptive network structure design in Reference [5]. In these adaptive methods, the key to achieving efficiency is to build a centralized policy network with a global observation perspective, which means that the model needs to capture the global information of the video for further decision-making. Existing methods can accurately and efficiently memorize the entire content of snapshots or clips containing only a few actions. However, as the length of the video grows, the size of the decision and observation space of the policy network grows exponentially, making it difficult for existing methods to directly scale to long, untrimmed videos.

[0004] References:

[0005] [1] Video recognition method and system based on deep learning training reconstruction image, Chinese patent, publication number CN108009486A, publication date 2018.5.18;

[0006] [2] Highlight video recognition method and device, electronic device and storage medium, Chinese patent, publication number CN114998797A, publication date September 2, 2022;

[0007] [3]Panda R, Chen CFR, Fan Q, et al.Adamml:Adaptive multi-modallearning for efficient video recognition[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2021:7576-7585;

[0008] [4]Wang Y, Yue Y, Lin Y, et al.Adafocus v2: End-to-end training of spatial dynamic networks for video recognition[C] / / 2022IEEE / CVF Conference onComputer Vision and Pattern Recognition(CVPR).IEEE,2022:20030-20040.;

[0009] [5]Li H, Wu Z, Shrivastava A, et al. 2d or not 2d? adaptive 3d convolutionselection for efficient video recognition[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2021:6155-6164. Summary of the Invention

[0010] The present invention addresses the problem of efficient recognition in long uncropped videos, eliminates the drawbacks of centralized decision networks, and provides a long video efficient recognition system and method based on adaptive sampling and reasoning. The system can adaptively capture multiple local events in the video and infer global semantic features based on this. Compared with existing systems and methods, it can efficiently and accurately complete the recognition of long uncropped videos.

[0011] The present invention provides an efficient long video recognition system based on adaptive sampling and reasoning, which includes: a video preprocessing module, a local feature extraction module, a global semantic reasoning module and a video classification module.

[0012] The video preprocessing module decodes the video into video frames and outputs a sequence of video frames with standardized size.

[0013] The local feature extraction module uniformly initializes N samplers on the video frame sequence. Each sampler uses a local context network to encode the video frame it selects to observe. The local feature extraction module locates different local events through the samplers. N is a positive integer. When all samplers stop, the hidden states of all local context networks are spliced ​​and input into the global semantic reasoning module.

[0014] The global semantic reasoning module infers the global semantic features of the entire video based on multiple local events. The global semantic reasoning module uses a multi-layer Transformer encoder and outputs the hidden features output by the last encoding layer of the encoder as the global semantic features of the video to the video classification module.

[0015] The video classification module uses a single-layer fully connected network as a classifier to identify the category of the video based on the global semantic features of the video.

[0016] The sampler includes a policy network f p and evaluation network f r , the policy network and the evaluation network are both composed of fully connected networks; the local context network includes a feature extractor based on a convolutional neural network and a context network, and the parameters of the local context network are shared among all samplers;

[0017] Assume that the feature extractor is a policy network f p Extract frame-level features o from the selected frame at time step t t , the context network is designed and implemented based on LSTM, which is used to model the temporal causal relationship of local events. The context network is based on the feature o t , and the hidden state h of the previous time step t-1 and cell state c t-1 To calculate the current hidden state h t and cell state c t , which is expressed as follows:

[0018] h t ,c t =LSTM(o t ,h t-1 ,c t-1 θ l,i )

[0019] Among them, θ l,i is the parameter of the context network corresponding to sampler i;

[0020] Assume that at time step t, the hidden state of the context network corresponding to sampler i is The cumulative number of frames observed by sampler i is At time step t, sampler i follows the policy network f p Generated policy distribution, select action To decide the next observation frame or stop observing, the strategy distribution is expressed as Among them, θ p,i is the policy network parameter of sampler i; the action space is {0, δ, 2δ, 3δ}. When 0 is selected, the sampler stops. When nδ is selected, the sampler jumps back nδ frames to observe the next frame. δ represents the minimum number of moving frames.

[0021] Evaluation network f r Used to evaluate the policy network f during the training phase p The value of the selected action helps the policy network optimize.

[0022] Accordingly, the present invention provides an efficient long video recognition method based on adaptive sampling and inference, comprising the steps of:

[0023] (1) Decode the video to be identified and output a video frame sequence with standardized size;

[0024] (2) Using the local feature extraction module to locate different local events, including: uniformly initializing N samplers on the video frame sequence, each sampler uses a local context network to encode the video frame it observes; N is a positive integer; when all samplers stop, the hidden states of all local context networks are spliced ​​and input into the global semantic reasoning module;

[0025] (3) Using the global semantic reasoning module to infer the global semantic features of the entire video based on multiple local events; the global semantic reasoning module uses a multi-layer Transformer encoder and outputs the hidden features output by the last encoding layer of the encoder as the global semantic features of the video to the video classification module;

[0026] (4) The video classification module outputs the video classification distribution based on the global semantic features of the video and identifies the category of the video; the video classification module consists of a single-layer fully connected network.

[0027] The advantages and positive effects of the system and method of the present invention are:

[0028] (1) In order to improve the recognition accuracy and efficiency of long untrimmed videos, the system and method of the present invention propose a novel hierarchical framework, which adopts a hierarchical local feature extraction module to capture key local features, and adopts a global semantic reasoning module to infer the global semantic features of the video based on the local features. This is in sharp contrast to the existing video recognition methods that capture long-term memory, and can achieve effective recognition of long untrimmed videos.

[0029] (2) The system and method of the present invention utilize multi-agent reinforcement learning. The local feature extraction module samples multiple local features in the video. Under the guidance of global semantic features, the agents in the local feature extraction module learn how to model the sampling strategies of the overall environment and other agents based on the local features. During the training process, all samplers are trained jointly. Each sampler has a centralized evaluation network that guides the strategy network to model the overall environment and the strategies of other samplers, resulting in higher recognition efficiency and better recognition accuracy for the system and method of the present invention.

[0030] (3) The system and method of the present invention were extensively tested against existing technologies on three large-scale video datasets (Kinetics-Sounds, ActivityNet, and FCVID). The experimental results confirmed that the system and method of the present invention have advantages over the most advanced existing methods and can more efficiently and accurately complete the recognition of long uncropped videos, thus solving the problem of efficient recognition of long videos and eliminating the drawbacks of the centralized decision network of existing recognition technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the implementation principle of the long video efficient recognition system and method of the present invention;

[0032] Figure 2 This is a framework diagram of the long video efficient recognition system of the present invention;

[0033] Figure 3 This is the overall framework diagram of the long video efficient recognition method of the present invention;

[0034] Figure 4 This is a comparison chart of the recognition effects of the long video efficient recognition method of the present invention and the existing recognition technology. DETAILED DESCRIPTION

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0036] Unlike snapshots or clips of only a few actions, long, untrimmed videos usually depict a complex story over a long timeframe. This story is generally composed of multiple related events with strong temporal dependencies between them. The local features of these events and the temporal causal relationships between them constitute the long-term memory of the entire video. Figure 1The video of "Table Tennis Match" shown in the figure contains multiple events, such as "Equipment", "Athlete Interview", "Warm-up", "Playing Table Tennis", etc. Among these events, only one event has an intuitive connection with "Table Tennis Match", while the rest of the scenes do not appear directly. Without the support of long-term memory, the model cannot capture the dependency between the previous events and "Table Tennis Match". The model can only guess which event the video theme is closest to, rather than inferring semantic information through events. The long video efficient recognition system and method based on adaptive sampling and reasoning of the present invention can adaptively capture multiple local events in the video and infer global semantic features on this basis, thereby realizing efficient and accurate recognition of long uncropped videos.

[0037] like Figure 2 As shown, the present invention provides an efficient video recognition system with adaptive sampling and reasoning, including the following components:

[0038] Video preprocessing module: This module decodes the videos in the video dataset into video frames and performs data preprocessing operations such as sampling and cropping, and outputs a video frame sequence with standardized size;

[0039] Local feature extraction module: This module is responsible for interacting with the video frame sequence. Multiple samplers adaptively sample local feature frames to locate different local events.

[0040] Global semantic reasoning module: This module infers the global semantic features of the entire video based on multiple local events;

[0041] Video classification module: This module identifies the category of the video based on the semantic features generated by the global semantic reasoning module.

[0042] The local feature extraction module, global semantic reasoning module and video classification module of the video recognition system of the present invention are implemented as follows: Figure 3 As shown, the figure also reflects the main implementation framework of the present invention's long video efficient recognition method based on adaptive sampling and reasoning.

[0043] (1) The local feature extraction module locates local events in the video.

[0044] The local feature extraction module is used to extract the spatiotemporal features of multiple local events. The key to feature extraction is how to adaptively select keyframes representing local events, which is crucial for accurately locating local events and improving model efficiency.

[0045] In the present invention, the key frame selection problem for multiple events is expressed as multiple sequential decision problems. Since there are strong temporal dependencies between multiple events, the generation of multiple sequential decisions is not independent of each other, so joint modeling is required, which can be naturally placed in the multi-agent reinforcement learning framework. Specifically, N samplers are uniformly initialized on the frame sequence. N is a positive integer. Over time, multiple joint samplers interact with the frame sequence to decide which frames to view and when to stop. During the interaction, each sampler uses a local context network to encode the frame sequence it observes.

[0046] (1) Local context network: The local context network consists of a CNN (convolutional neural network)-based feature extractor and a context network, which is used to model the temporal causal relationship of local events at different time steps.

[0047] With θ o The feature extractor f is parameterized o Used to extract the frame-level features o of the frame selected by the policy network at time step t t It is worth noting that the parameters of the local context network are shared among all samplers.

[0048] In order to model the temporal causal relationship of local events, a context network f based on LSTM (Long Short-Term Memory Artificial Neural Network) is used. l , with θ l Specifically, at time t, the context network of sampler i passes the current feature o t , the previous hidden state h t-1 and the previous cell state c t-1 To calculate the current hidden state h t and cell state c t .

[0049] h t ,c t =LSTM(o t ,h t-1 ,c t-1 θ l,i )

[0050] Among them, θ l,i are the parameters of the context network corresponding to sampler i.

[0051] (2) Sampler: In order to unite all samplers, the single-agent Actor-Critic algorithm is extended to a multi-agent centralized training and distributed execution framework. Each sampler contains a policy network f for decision making. p and the evaluation network f for learning r , the parameters are θ p and θ r ,Both the policy network and the evaluation network consist of fully connected networks.

[0052] At time t, sampler i selects an action based on the policy distribution generated by the policy network. To decide which frames to observe next and whether to stop. The strategy distribution is as follows:

[0053]

[0054] The action space is {0, δ, 2δ, 3δ}, where δ represents the minimum number of steps. When action 0 is selected, the sampler stops; when nδ is selected, the sampler jumps back nδ frames to observe the next moment, where n = 1, 2, 3. p,i are the policy network parameters of sampler i, is the hidden state of the context network of sampler i at time t, is the cumulative number of observation frames of sampler i at time t.

[0055] The evaluation network is used to evaluate the value of the selected action to guide the policy network to model the overall environment and the policies of other samplers. To avoid the problem of overestimating the value of the action during training, the target network and the dual network technique are used in the evaluation network.

[0056] (2) The global semantic reasoning module infers the global semantic features of the entire video based on multiple local events.

[0057] The global semantic reasoning module infers the global semantic features of the entire video based on the multiple local events extracted by the local feature extraction module. Long, untrimmed videos are typically composed of multiple local events, and the complex temporal causal relationships between these events represent the semantic information of the entire video. However, the logical relationships between events span multiple time periods.

[0058] In order to connect events of multiple time spans, θ gThe parameterized global semantic reasoning module uses a multi-layer Transformer encoder. Through the self-attention mechanism, any sampler can pay attention to the features of all other samplers. As a result, the module can learn long-term dependencies between local events to achieve long-term memory for the entire video.

[0059] When all samplers stop sampling, the hidden states of all local context networks will be concatenated and input into the global semantic reasoning module. Specifically, in the k-th encoding layer of the Transformer encoder, the hidden feature h k and the intermediate state h′ k By h k-1 The forward reasoning process is as follows:

[0060] h′ k =MSA(LN(h k-1 ))+h k-1

[0061] h k =MLP(LN(h′ k ))+h′ k

[0062] Among them, MSA stands for multi-head self-attention mechanism, LN stands for layer normalization, and MLP stands for perceptron.

[0063] (3) The video classification module converts video categories based on global semantic features.

[0064] The video classification module is mainly responsible for the conversion from abstract features to category information. The video classification module uses a single-layer fully connected network as the classifier f c , whose parameter is θ c .

[0065] The present invention performs overall training on the above local feature extraction module, global semantic reasoning module and video classification module. The training process is divided into three stages, namely the backbone network training stage, the policy network training stage and the fine-tuning stage. In the backbone network training stage, the parameters of the policy network are frozen, and a random sampling strategy is used to preheat the backbone network. The backbone network includes the local context network, the global semantic reasoning module and the video recognition module in all samplers. In the policy network training stage, all parameters of the backbone network are fixed, and the policy network parameters in the sampler are unfrozen and initialized, and local video frames are sampled to perform policy network training and learning. Finally, in the fine-tuning stage, the backbone network and the policy network are alternately fine-tuned to fine-tune the overall model parameters.

[0066] During training, standard backpropagation is used to optimize the backbone network, while policy gradient is used to optimize the sampler. The reward function and loss function are shown below.

[0067] (a) During the backbone network training process, the standard cross entropy loss is used to evaluate the prediction effect, namely:

[0068]

[0069] where θ b ={θ o ,θ l ,θ g ,θ c}, p v and y represent the predicted label and true label of video v in dataset V, respectively.

[0070] (b) During the policy network training process, the reward function and loss function used are set as follows.

[0071] The reward function is a feedback mechanism from the environment to the agent, and the reward represents the value of the action. In order to encourage the sampler to find the key frame to achieve efficient and accurate recognition, the present invention adopts a multi-objective reward function based on the video prediction distribution to guide the learning of the policy network. At time t, the sum of the number of observation frames of all samplers is obtained as N t , the rewards for all samplers can be generated as follows:

[0072]

[0073] Among them, P t gt and They represent the probability of the video's true label corresponding to the video's predicted distribution at time t and time t-1, respectively, and λ is the balance factor between recognition accuracy and efficiency.

[0074] The goal of training the policy network is to select actions for each state to maximize future rewards and policy entropy. For sampler i, given a tuple sampled from the memory bank D state represents the action selected by sampler i at time t, r t represents the rewards of all samplers at time t; the strategy network loss function J of the present invention π Expressed as:

[0075]

[0076] in, Indicates that all the Seek hope, represents the policy network, α is the dynamic adjustment factor of entropy, and sampler i adopts a dual evaluation network. The superscript T indicates transpose.

[0077] Evaluation network loss function: For sampler i, given the tuple sampled from the memory bank D The algorithm trains the evaluation network by minimizing the soft Bellman residual:

[0078]

[0079]

[0080] in, for The state value; set.gather(element) means matching element items from the set; The output of can be expressed as a set; pre-train the local evaluation network to obtain the target network parameters Then slide and copy its weight to the target evaluation network, that is γ represents the constant discount factor for past reward losses.

[0081] Alpha loss function: α is used to dynamically adjust the scale of entropy and limit the average entropy to be higher than For sampler i, the α loss can be expressed as:

[0082]

[0083] in is a constant vector of target entropy.

[0084] Correspondingly, an implementation step of the long video efficient recognition method based on adaptive sampling and inference of the present invention is as follows:

[0085] (1) Using the video preprocessing module, the video to be identified is decoded into a frame sequence, and preprocessing operations such as size normalization are performed on each video frame;

[0086] (2) Using the local feature extraction module to interact with the video frame sequence, adaptively select observation frames and locate multiple local events;

[0087] (3) Based on multiple local events, the global semantic reasoning module models the temporal relationship between local events located by the local feature extraction module to capture global semantic features;

[0088] (4) The video classification module realizes the conversion of abstract features into category information and obtains the predicted category of the video based on global semantic features.

[0089] The implementation and training of the specific local feature extraction module, global semantic reasoning module and video classification module are described above and will not be repeated here.

[0090] The video recognition method and system of the present invention are extensively tested against the existing technologies on three video benchmark datasets: ActivityNet, FCVID, and Kinetics-Sounds. Figure 4 FIG. 1 shows a comparison of experimental results. AdaSR represents the system / method of the present invention, and the others are existing video recognition technologies. Figure 4 In the figure, the horizontal axis represents the inference cost used to evaluate the computational consumption, and the vertical axis represents the mAP index used to evaluate the target detection accuracy. The higher the mAP value, the higher the target detection accuracy and precision. Figure 4 It can be seen that the system / method of the present invention outperforms the existing technology in both accuracy and efficiency. With only ImageNet-1K pre-training, AdaSR achieves 82.4% and 86.4% mAP on ActivityNet and FCVID with 38.7 and 36.4 GFLOPs respectively.

[0091] Except for the technical features described in the specification, all other technical features are known to those skilled in the art. The present invention omits descriptions of well-known components and well-known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. Based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the present invention.

Claims

1. An efficient long video recognition system based on adaptive sampling and inference, characterized by: It includes video preprocessing module, local feature extraction module, global semantic reasoning module and video classification module; The video preprocessing module decodes the video into video frames and outputs a video frame sequence with standardized size; The local feature extraction module uniformly initializes N samplers on the video frame sequence. Each sampler uses a local context network to encode the video frame it observes. The local feature extraction module locates different local events through the samplers. N is a positive integer. When all samplers stop, the hidden states of all local context networks are spliced ​​and input into the global semantic reasoning module. The sampler includes a policy network f p and evaluation network f r , the policy network and the evaluation network are composed of fully connected networks; at time step t, the sampler i is based on the policy network f p Generated policy distribution, select action To decide the next observation frame or stop observation; evaluate the network f r Used to evaluate the policy network f during the training phase p The value of the selected action, which helps the policy network optimize; The global semantic reasoning module infers the global semantic features of the entire video based on multiple local events; The global semantic reasoning module uses a multi-layer Transformer encoder and outputs the hidden features output by the last encoding layer of the encoder as the global semantic features of the video to the video classification module; The video classification module uses a single-layer fully connected network as a classifier to identify the category of the video based on the global semantic features of the video.

2. The long video efficient recognition system based on adaptive sampling and inference according to claim 1 is characterized in that: The local context network includes a feature extractor based on a convolutional neural network and a context network, and the parameters of the local context network are shared among all samplers; Assume that the feature extractor is a policy network f p The feature o is extracted from the selected frame at time step t t , the context network is designed and implemented based on LSTM, which is used to model the temporal causal relationship of local events. The context network is based on the feature o t , and the hidden state h at the previous time step t-1 and cell state c t-1 To calculate the current hidden state h t and cell state c t , which is expressed as follows: h t ,c t =LSTM(o t ,h t-1 ,c t-1 ;θ l,i ) Among them, θ l,i is the parameter of the context network corresponding to sampler i; Assume that at time step t, the hidden state of the context network corresponding to sampler i is The cumulative number of frames observed by sampler i is The policy distribution is expressed as Among them, θ p,i is the policy network parameter of sampler i; the action space is {0, δ, 2δ, 3δ}. When 0 is selected, the sampler stops. When nδ is selected, the sampler jumps back nδ frames to observe the next frame. δ represents the minimum number of moving frames.

3. The long video efficient recognition system based on adaptive sampling and inference according to claim 1 or 2, characterized in that: In the global semantic reasoning module, in the kth encoding layer, the hidden feature h k and the intermediate state h′ k The hidden feature h of the previous encoding layer k-1 The forward reasoning is as follows: h′ k =MSA(LN(h k-1 ))+h k-1 h k =MLP(LN(h′ k ))+h′ k Among them, MSA stands for multi-head self-attention mechanism, LN stands for layer normalization, and MLP stands for perceptron.

4. The long video efficient recognition system based on adaptive sampling and inference according to claim 1 or 2, characterized in that: The local feature extraction module, global semantic reasoning module and video classification module are trained as a whole, and the training process is divided into three stages, namely the backbone network training stage, the policy network training stage and the fine-tuning stage; in the backbone network training stage, the parameters of the policy network in the sampler are frozen, and the backbone network is trained using a random sampling strategy; the backbone network includes the local context network in the sampler, the global semantic reasoning module and the video recognition module; in the policy network training stage, all parameters of the backbone network are fixed, local video frames are sampled, and the policy network is trained; in the fine-tuning stage, the backbone network and the policy network are alternately fine-tuned.

5. The long video efficient recognition system based on adaptive sampling and inference according to claim 4 is characterized in that: During training, the backbone network uses a standard cross-entropy loss to evaluate prediction results; during training, the policy network uses a multi-objective reward function to reward the observation frames selected by the sampler; At time step t, the rewards of all samplers are in, and They represent the probability of the video’s true label corresponding to the video’s predicted distribution at time steps t and t-1, respectively. λ is the balance factor between recognition accuracy and efficiency. N t The sum of the number of observation frames of all samplers; The goal of training the policy network is to choose actions for each state that maximize future rewards and policy entropy.

6. The long video efficient recognition system based on adaptive sampling and inference according to claim 4 is characterized in that: When the policy network is trained, the loss function J of the policy network is π It is expressed as follows: Where D represents the memory bank, for sampler i, it samples tuples from the memory bank D Represent the states at time steps t and t+1 respectively. The states include the hidden state of the context network corresponding to sampler i and the cumulative number of observed frames. represents the action selected by sampler i at time step t; Indicates that all the Seek expectations; represents the policy network, θ p,i is the parameter of the policy network of sampler i; α is the dynamic adjustment factor of entropy; sampler i adopts a dual evaluation network The superscript T indicates transposition; For the evaluation network, the loss function J of the evaluation network is evaluated by minimizing the soft Bellman residual training. Q as follows: in, for The state value; set.gather(element) means matching element items from the set. The output of is represented as a set; γ represents the constant discount factor of past reward loss; The target network parameters are obtained in advance through local training; The parameter α dynamically adjusts the scale of entropy and limits the average entropy to be higher than For sampler i, the α loss can be expressed as: in, is a constant vector of target entropy.

7. An efficient long video recognition method based on adaptive sampling and inference, characterized by: The steps include: (1) Decode the video to be identified and output a video frame sequence with standardized size; (2) Using the local feature extraction module to locate different local events, including: uniformly initializing N samplers on the video frame sequence, each sampler uses a local context network to encode the video frame it observes; N is a positive integer; when all samplers stop, the hidden states of all local context networks are spliced ​​and input into the global semantic reasoning module; The sampler includes a policy network f p and evaluation network f r , the policy network and the evaluation network are composed of fully connected networks; at time step t, the sampler i is based on the policy network f p Generated policy distribution, select action To decide the next observation frame or stop observation; evaluate the network f r Used to evaluate the policy network f during the training phase p The value of the selected action, which helps the policy network optimize; (3) Using the global semantic reasoning module to infer the global semantic features of the entire video based on multiple local events; the global semantic reasoning module uses a multi-layer Transformer encoder and outputs the hidden features output by the last encoding layer of the encoder as the global semantic features of the video to the video classification module; (4) The video classification module outputs the video classification distribution based on the global semantic features of the video and identifies the category of the video; the video classification module consists of a single-layer fully connected network.

8. The method for efficient long video recognition based on adaptive sampling and inference according to claim 7, characterized in that: In the step (2), the local context network includes a feature extractor based on a convolutional neural network and a context network, and the parameters of the local context network are shared among all samplers; Assume that the feature extractor is a policy network f p The frame-level features o are extracted from the selected frame at time step t t , the context network is designed and implemented based on LSTM, which is used to model the temporal causal relationship of local events. The context network is based on the feature o t , and the hidden state h at the previous time step t-1 and cell state c t-1 To calculate the current hidden state h t and cell state c t , which is expressed as follows: h t ,c t =LSTM(o t ,h t-1 ,c t-1 ;θ l,i ) Among them, θ l,i is the parameter of the context network corresponding to sampler i; Assume that at time step t, the hidden state of the context network corresponding to sampler i is The cumulative number of frames observed by sampler i is At time step t, the policy distribution is expressed as Among them, θ p,i is the policy network parameter of sampler i; the action space is {0, δ, 2δ, 3δ}. When 0 is selected, the sampler stops. When nδ is selected, the sampler jumps back nδ frames to observe the next frame. δ represents the minimum number of moving frames.

9. The method for efficient long video recognition based on adaptive sampling and inference according to claim 7 or 8, characterized in that: In step (3), in the k-th encoding layer of the multi-layer Transformer encoder, the hidden feature h k and the intermediate state h′ k The hidden feature h of the previous encoding layer k-1 The forward reasoning is as follows: h′ k =MSA(LN(h k-1 ))+h k-1 h k =MLP(LN(h′ k ))+h′ k Among them, MSA stands for multi-head self-attention mechanism, LN stands for layer normalization, and MLP stands for perceptron.

Images