A multi-modal based untrimmed long video recognition system and method

Abstract

The application provides a long untrimmed video recognition system and method based on multi-modal. The system comprises a video acquisition and preprocessing module, a to-be-recognized video database, a recognition and detection module and a result database. In the method, the video recognition model in the recognition and detection module is used to process the frame sequences of multiple modes of the video, multiple samplers are used to interact with the frame sequences of each mode, local events are located, the time causal relationship between the local events is inferred by a single-modal semantic inference module, a single-modal prediction distribution is output by a sub-recognition module, finally, a multi-modal fusion module is used to assign attention weight values to different modes, perform multi-modal prediction distribution fusion, and output a video-level multi-modal class prediction result. The application improves the inference efficiency and recognition accuracy of the long untrimmed video recognition system, and the recognition accuracy is obviously higher than that of the current optimal video recognition method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology and video recognition, and particularly relates to a multi-modal adaptive unpruned long video recognition system and method. BACKGROUND

[0002] Long videos such as movies, documentaries, TV series and self-media videos have always been an important part of people's lives. With the development of mobile Internet in recent years and the rapid improvement of network speed, online videos have become an indispensable part of people's lives. With the popularization and development of the Internet, the number of long videos on the network has increased rapidly, and there are still challenges in efficient recognition and detection of long videos. The accuracy and efficiency of recognition are very important for many downstream tasks such as video transmission, video retrieval, video screening and edge storage.

[0003] Specifically, there are two challenges in the unpruned long video recognition task. On the one hand, unlike snapshots or pruned videos with only a few actions, unpruned videos usually describe complex activities from multiple modalities over a long span, containing multiple local events. Eliminating redundant and irrelevant information through multi-dimensional attention allocation is the key to efficient video action recognition. Most existing video recognition methods and systems do not consider the efficiency of recognition. Some methods and systems optimize and improve the efficiency problem, mainly based on global perspective to allocate attention to single modal short or video segments. For example, document [1] manually samples video frame sequences for each shot in the recognition process to reduce the number of video frames to be processed; document [2] constructs a knowledge graph of violent behavior and proposes an optimization strategy to eliminate redundancy between different labels, obtaining a violent correction matrix modification model; document [3] uses a global strategy network to eliminate spatial redundancy at the frame level; document [4] uses a global strategy network to determine the resolution size of local frames; and document [5] adaptively decides the network architecture at different localities. Although the current methods greatly save the amount of computation while improving the recognition accuracy, they are not efficient enough in handling multi-modalities over a long time scale, requiring a large amount of computing resources and being difficult to capture semantic information over a long time span.

[0004] The references are as follows:

[0005] Distributed ultra-high-definition video recognition method and system based on shot recognition and scene recognition, Chinese patent, publication number CN112261440A, publication date 20210122;

[0006] Violence video classification method and system and storage medium, Chinese patent, publication number CN112069884A, publication date 20201211;

[0007] [3]Wang Y, Yue Y, Lin Y, et al. Adafocus v2: End-to-end training of spatial dynamic networks for video recognition [C] / / 2022IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022:20030-20040.

[0008] [4]Meng Y, Lin C C, Panda R, et al. Ar-net: Adaptive frame resolution for efficient action recognition [C] / / European Conference on Computer Vision. Springer, Cham, 2020:86-104.

[0009] [5]Li H, Wu Z, Shrivastava A, et al. 2d or not 2d?Adaptive 3d convolution selection for efficient video recognition [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021:6155-6164. SUMMARY

[0010] In view of the problems of high consumption of computing resources and low recognition efficiency in the above-mentioned unpruned long video recognition technology, the application provides an unpruned long video recognition system and method based on multi-modal, which can adaptively allocate attention and computing resources between local events and modalities of the video, and realize efficient and accurate recognition of unpruned long video.

[0011] The unpruned long video recognition system based on multi-modal of the application comprises a video acquisition and preprocessing module, a to-be-recognized video database, a recognition and detection module and a result database.

[0012] The video acquisition and preprocessing module receives video from the outside and performs preprocessing, and stores the preprocessed to-be-recognized video into the to-be-recognized video database; the recognition and detection module recognizes the category of the to-be-recognized video, and writes the recognition result into the result database.

[0013] The recognition and detection module identifies the video category, including: for each modality of the frame sequence of the video, respectively adopting N samplers to interact with the frame sequence of the corresponding modality, positioning local events, inferring the temporal causal relationship between the local events by a single-modality semantic inference module to capture the semantic information of the single modality, and outputting the prediction distribution of the single modality by a sub-recognition module, and finally assigning attention weights to different modalities by a multi-modal fusion module, fusing the prediction distributions of all modalities, and outputting the multi-modal category prediction result at the video level; N is an integer greater than 2.

[0014] The video acquisition and preprocessing module pre-processes the video to obtain multi-dimensional frame sequences of multiple modalities of the video that are time-aligned and size-standardized, and stores them in a to-be-identified video database; the to-be-identified video database also stores basic information of the video, including video length, frame number, and modality number.

[0015] The recognition and detection module uniformly initializes N samplers in the frame sequence of each modality of the video, and each sampler includes four parts, namely a local sampler, a context-aware network, a policy network, and an evaluation network; the local sampler extracts local information from the selected frames; the context-aware network includes a feature extractor and an LSTM module, the feature extractor encodes the local information extracted by the local sampler to output a feature vector, and the LSTM module models the temporal causal relationship between the selected frames at different time steps; the policy network is composed of a fully connected network, which is used to select the frames for positioning local events; the evaluation network is composed of a fully connected network, which is used to estimate the q value of the action selected by the policy network in the training stage to help the learning of the policy network; each sampler internally adopts an Actor-Critic structure and uses a centralized evaluation and decentralized execution framework to combine all samplers.

[0016] The single-modality semantic inference module is a multi-layer transformer-based encoder, and after all the samplers of the modality k stop, the hidden states of the LSTM modules of the context-aware networks in all the samplers of the modality k at the current time are spliced and input into the single-modality semantic inference module; the single-modality semantic inference module outputs the hidden features of the last layer of the encoder.

[0017] The sub-recognition module adopts a layer of fully connected network, accepts the output of the single-modality semantic inference module, and maps to output the video category prediction distribution of the single modality.

[0018] The multimodal fusion module transforms the dynamic adjustment of fusion weights into a Markov decision process. First, it initializes the fusion weights of all modes, then uses a policy network to gradually adjust the fusion weights, and finally uses the optimized fusion weights to fuse the single-modal prediction distributions. The multimodal fusion module adopts an Actor-Critic structure and uses an evaluation network to help the policy network learn during training.

[0019] The system of this invention also includes a result display module, used to receive external query requests, retrieve results from the result database, and display them. The result database stores the predicted category, category distribution, selected observation frame number, and fusion ratio of different modalities of the video after processing by the recognition and detection modules.

[0020] Correspondingly, the present invention provides a method for recognizing undressed long videos based on multimodal approaches, comprising the following steps:

[0021] (i) Acquire the video to be identified, decode and preprocess the video, and obtain a multi-dimensional frame sequence of multiple modalities of the video that is time-aligned and size-standardized;

[0022] (ii) For each modality of the video frame sequence, N samplers are used to interact with the corresponding modality frame sequence to locate local events. The temporal causal relationship between local events is inferred through the single-modality semantic reasoning module to capture the semantic information of the single modality. Then, the sub-recognition module outputs the single-modality prediction distribution. Finally, the multimodal fusion module assigns attention weights to different modalities, fuses the prediction distributions of all modalities, and outputs the video-level multimodal category prediction result; N is an integer greater than 2.

[0023] In step (ii), N samplers are uniformly initialized in the frame sequence of each modality of the video. Each sampler consists of four parts: a local sampler, a context-aware network, a policy network, and an evaluation network. The local sampler extracts local information from selected frames. The context-aware network includes a feature extractor and an LSTM module. The feature extractor encodes the local information extracted by the local sampler and outputs a feature vector. The LSTM module models the temporal causal relationship between selected frames at different time steps. The policy network consists of a fully connected network and is used to select frames for locating local events. The evaluation network consists of a fully connected network and is used to estimate the q-value of the action selected by the policy network during the training phase to help the policy network learn. Each sampler adopts an Actor-Critic structure and uses a centralized evaluation and distributed execution framework to unite all samplers.

[0024] The single-modal semantic reasoning module is a multi-layer transformer-based encoder, and after all the samplers of the modal k stop, the hidden states of the LSTM modules of the context-aware network in all the samplers of the modal k at the current moment are spliced to input the single-modal semantic reasoning module; the single-modal semantic reasoning module outputs the hidden features of the last layer of the encoder;

[0025] The sub-recognition module adopts a full connection network, accepts the output of the single-modal semantic reasoning module, and maps the output single-modal video category prediction distribution;

[0026] The multi-modal fusion module converts the dynamic adjustment of the fusion weight into a Markov decision process, first initializes the fusion weight of all modes, then gradually adjusts the fusion weight by using a strategy network, and finally optimizes the fusion weight to fuse the single-modal prediction distribution; the multi-modal fusion module adopts an Actor-Critic structure, and uses an evaluation network to help the strategy network learn during training.

[0027] The advantages and positive effects of the present application are that:

[0028] (1) The system and method of the present application distribute the local attention of the model based on the hierarchical game between local events and modes, realize the adaptive extraction and fusion of multi-modal features, and improve the reasoning efficiency and recognition accuracy of the unpruned long video recognition system.

[0029] (2) The unpruned long video recognition model used by the system and method of the present application adopts a hierarchical detection method based on multi-agent reinforcement learning (referred to as AdaSense), which effectively realizes the attention and computing resource allocation at the frame level and the modal level through the adaptive sampling based on multi-agent at the frame level and the adaptive fusion of the multi-modal fusion module; through attention allocation, the samplers of the multi-modal cooperate with each other to capture the overall structural features, and compete with each other to realize the best allocation of computing resources to improve efficiency.

[0030] (3) Extensive experiments on three commonly used large video datasets (ActivityNet, FCVID and Kinetics-Sounds) show that the system and method of the present application not only greatly reduces the computing consumption, such as GFLOPs (billion floating point operations per second) of all datasets reduced by 76.9% on average compared with the basic framework without the game mechanism, but also significantly exceeds the current optimal video recognition method in recognition accuracy on all datasets, with an average improvement of 6.4%. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 It is a framework diagram of the unpruned long video recognition system based on multi-modal of the present application;

[0032] Figure 2 Implementation schematic diagram of the untrimmed long video adaptive recognition and detection method of the present application;

[0033] Figure 3 Overall framework schematic diagram of the untrimmed long video recognition model of the present application;

[0034] Figure 4 CTDE framework and multi-objective loss function schematic diagram of the strategy network in the training of the present application;

[0035] Figure 5 Training and inference flowchart of the untrimmed long video recognition model of the present application;

[0036] Figure 6 Comparison chart of the method of the present application in ActivityNet with the most advanced single-modal (left) and multi-modal (right) methods. DETAILED DESCRIPTION

[0037] The present application will be further described in detail below with reference to the accompanying drawings and examples.

[0038] As shown in Figure 1 , a multi-modal based untrimmed long video recognition system of the present application comprises the following components:

[0039] Video acquisition and preprocessing module: This module receives external video input and performs data preprocessing, including sampling, cropping, modality construction, etc., decodes the video into a plurality of modality long-aligned and size-standardized multi-dimensional frame sequences; modality refers to different information representation ways such as vision and audio;

[0040] To-be-recognized video database: This database serves as a temporary buffer area, stores the preprocessed to-be-recognized video, and contains the multi-modal original information of the video frames and the basic information of the video such as length, frame number, and modality number;

[0041] Recognition and detection module: This module reads the to-be-recognized video data processed by the preprocessing module, obtains the category of the to-be-recognized video through adaptive recognition and detection algorithm, and writes the result into the result database;

[0042] Result database: This database stores the predicted category, category distribution, selected observation frame number, and fusion ratio of different modalities processed by the recognition and detection module, for use by the result display module.

[0043] Result display module: This module is responsible for accepting external query requests, querying and returning the query results from the result database, and displaying the results to the user.

[0044] The recognition and detection module of the present application is provided with an unpruned long video recognition model, that is, a multi-modal based unpruned long video adaptive recognition and detection procedure, as shown in Figure 2 As shown in the figure, the recognition and detection module distributes the local attention of the model based on the hierarchical game between local events and modalities, realizes the adaptive extraction and fusion of multi-modal features, and improves the reasoning efficiency and recognition accuracy of the recognition system.

[0045] As shown in Figure 3 In the recognition and detection module of the present application, video category recognition is performed through a sampler, a single-modal semantic reasoning module, a sub-recognition module and a multi-modal fusion module. Specifically, given a video with k modalities, multiple joint samplers interact with the frame sequence in different modalities of the video to approximately locate multiple local events. When all samplers stop, the single-modal semantic reasoning module is used to capture the temporal causal relationship between the samplers to reason the semantic information at the modality level. Then, the sub-recognition module outputs the prediction distribution of the single modality. Further, the multi-modal fusion module assigns attention weights to fuse the predictions of all modalities into a video-level prediction. During the training process, under the guidance of a multi-objective reward function based on accuracy and efficiency, the samplers within each modality compete for computing resources to observe more frames and cooperate with each other to extract structured features about the entire modality. At the same time, the multi-modal fusion module estimates the fusion weights to control the fusion and mixing game between modalities. With this mechanism, the samplers of different modalities learn to cooperate to output accurate prediction results based on multi-modal, and compete with each other to attract more fusion weight distribution. The implementation of the recognition and detection module of the specific system is described below in the multi-modal based unpruned long video recognition method.

[0046] The multi-modal based unpruned long video recognition method implemented by the present application generally includes: (1) obtaining a video to be recognized, decoding and preprocessing the video, and obtaining multi-dimensional frame sequences of multiple modalities of the video that are time-aligned and size-standardized; (2) performing video category prediction: for the frame sequence of each modality of the video, N samplers are used to interact with the frame sequence of the corresponding modality to locate local events, the temporal causal relationship between the local events is reasoned by a single-modal semantic reasoning module to capture the semantic information of the single modality, the prediction distribution of the single modality is output by a sub-recognition module, and finally a multi-modal fusion module is used to assign attention weights to different modalities, fuse the prediction distributions of all modalities, and output a video-level multi-modal category prediction result; N is an integer greater than 2. The function implementation of the recognition and detection module in the system of the present application is described below, that is, the implementation of the video category prediction method of the present application.

[0047] As shown in Figure 3As shown, the video in the embodiment of the present application has 2 modalities, which are vision and hearing respectively. In the frame sequence of different modalities of the video, a sampler, a single-modality semantic reasoning module and a sub-recognition module are arranged, and then the single-modality prediction distribution output by the sub-recognition module is input into a multi-modality fusion module for video prediction. The implementation of each module involved in the multi-modality based unpruned long video recognition will be described below.

[0048] (I) N samplers are uniformly arranged in the frame sequence of each modality of the video. Each sampler module includes four parts, which are a local sampler, a context-aware network, a policy network and an evaluation network.

[0049] (1) Local sampler: Given a video, N samplers are uniformly initialized in each modality. Each sampler includes a local sampler. Each local sampler extracts information from the frame to obtain local temporal information. The information extracted from the frame is encoded by a context-aware network to obtain the local temporal information of the context. During the inference process, the sampler models the behavior of other samplers based on local observations to make optimal decisions and effectively capture local events nearby. An Actor-Critic structure is used inside each sampler, and a centralized training and distributed execution (CTDE) framework is used to combine all samplers.

[0050] (2) Context-aware network: includes a feature extractor and an LSTM module. The CNN-based feature extractor observes the selected frame, encodes the extracted local temporal information, and outputs the encoded feature vector O t . For all samplers, the parameters of the CNN-based feature extractor are shared. The LSTM module is responsible for modeling the temporal causal relationship between the selected frames at different time steps. Specifically, at the sampling step t, the LSTM network i takes the current feature vector O t , the previous hidden state h t-1 and the previous cell state c t-1 to calculate the current hidden state h t and the cell state c t of the LSTM network:

[0051] h t ,c t =LSTM(O t ,h t-1 ,c t-1 ) (1)

[0052] (3) Policy network: the policy network f p is parameterized by θ pis composed of fully connected networks, which is responsible for determining which frames should be observed to locate local events. Specifically, at sampling step t, sampler i of modality k selects a frame to observe according to the policy distribution estimated by the policy network from the action space Ω s at time t, where to determine which frame to observe next or whether to stop, where denotes the cumulative number of frames observed by sampler i at time t, denotes the hidden state of the context-aware network corresponding to sampler i at time t. The action space Ω s is composed of four actions, i.e., {0, δ, 2δ, 3δ}, where δ represents the minimum movement step, i.e., the number of frames. At sampling step t, when action 0 is selected, the sampler will stop; when action nδ is selected, the sampler will jump back nh frames for observation at step t, n = 1, 2, 3. denotes the parameters of the policy network of sampler i corresponding to modality k.

[0053] (4) Evaluation network: parameterized by θ r The evaluation network is also composed of fully connected networks, which is used to estimate the q value of the selected action to help the learning of the policy network. During the training phase, the evaluation network globally observes the states and actions of all policy networks, and is not involved in the inference process. During the training process, under the guidance of the multi-objective reward, the evaluation network guides the policy network to learn how to model the overall environment and action policy of other samplers to control the cooperation and competition between samplers.

[0054] The present application adopts a multi-objective reward to measure the accuracy and efficiency of model recognition from the perspectives of single modality and multi-modality. The centralized evaluation network is responsible for guiding the policy network to learn how to model the overall environment and the behavior of other samplers based on local observations. The specific training will be described below.

[0055] (ii) When all samplers stop, the time causal relationship between the local events observed by all samplers under the current modality is inferred by the single-modality semantic inference module to capture the semantic information of the single modality.

[0056] The logical relationship between events has different time spans and great differences for different instances, and capturing this timing feature is good at self-attention mechanism. Through the self-attention mechanism, any sampler can be connected to all other samplers, which means that the mechanism can learn long-term dependencies across multiple events. Considering this, in order to capture the single-modal semantic features, the single-modal semantic reasoning module of the present application adopts a multi-layer transformer-based encoder to describe the logical relationship between local events on multiple local samplers. Specifically, when all samplers stop, the hidden states of the LSTM modules of the context-aware network in all samplers at the current time are spliced as the input of the single-modal semantic reasoning module, and the single-modal semantic reasoning module generates the hidden features h l and the intermediate state h′ l from the h l-1 of the previous layer, as follows:

[0057] h′ l = MSA (LN (h l-1 ))+h l-1 (2)

[0058] h l = MLP (LN (h′ l ))+h′ l (3)

[0059] where MSA represents multi-head self-attention mechanism, LN represents layer normalization, and MLP represents perceptron. The input of the single-modal semantic reasoning module at the first layer encoder is the spliced vector of the hidden states of the LSTM modules of the context-aware network in all samplers of the modal k at the current time. The output of the single-modal semantic reasoning module is the hidden feature of the last layer.

[0060] (Three) The sub-recognition module accepts the output of the single-modal semantic reasoning module, and maps the output single-modal video category prediction distribution.

[0061] The method of the present application sets a sub-recognition module in each modal to output a single-modal video category prediction distribution, and the sub-recognition module adopts a one-layer fully connected network to map the output of the single-modal semantic reasoning module to the prediction distribution. In the embodiment of the present application, the video category refers to the action category, and the category distribution prediction of the video under different modalities is obtained through the sub-recognition module.

[0062] (Four) Different attention weights are assigned to the video category prediction distributions of different modalities through the multi-modal fusion module, the prediction distributions of all modalities are fused, and the multi-modal category prediction result at the video level is output as the classification result of the video.

[0063] The multiple modalities provide heterogeneous information, which means the value of a modality is different in different video scenes and categories. The variability of value cannot be ignored for the training of the computational assignment between modalities and the local sampler within a modality, but the static fusion method with fixed weights cannot describe the variability. To solve this, the module makes specific instance-level adjustments to the attention of different modalities based on the competition and cooperation between modalities. Specifically, the multi-modal fusion module converts the dynamic adjustment of fusion weights into a Markov decision process, and uses a policy network f u to learn the optimal strategy of sequential decision-making. The multi-modal fusion module first initializes the fusion weights of all modalities, and adjusts the fusion weights step by step using the policy network until convergence. At adjustment step r, given a video with K modalities, the policy network f u estimates the adjustment policy distribution of the fusion weights of all modalities as follows:

[0064]

[0065] Where p i and represent the single-modal prediction distribution and the fusion weight of modality i at adjustment step t, respectively. For modality i, the multi-modal fusion module selects the adjustment direction and amplitude from the action space Ω f to adjust the fusion weight according to the policy distribution . The action space Ω f is composed of five actions, which are {-ξ, -0.1ξ, 0, +0.1ξ, +ξ}, where ξ represents the maximum adjustment amplitude, and is a pre-set value less than 1. When the adjustment amplitudes of any two modalities are 0, the adjustment process terminates, and the current fusion weights {ω1, ω2, … ω K} are output.

[0066] Then, the final video-level prediction result is:

[0067]

[0068] The multi-modal fusion module of the present application also adopts the Actor-Critic structure, in which the evaluation network is used to guide the learning of the policy network during training.

[0069] In order to correctly train the multi-modal fusion module, i.e. the entire unpruned long video recognition model framework of the method of the present application, a multi-stage training algorithm is designed, including the following three stages. The CTDE framework and multi-objective loss function involved in the training stage are as shown in Figure 4 .

[0070] Phase 1: Warm-up Phase. In this phase, the model's backbone network learns to extract spatial and temporal features from frame sequences using a randomized strategy. Specifically, the training algorithm freezes the policy networks of the sampler and the multimodal fusion module. After warming up the single-modal backbone network, for video samples v collected from dataset V, the training algorithm randomly selects half of the frames. And uniformly blend all modes into And using standard backpropagation, the cross-entropy loss on the training dataset V is minimized as follows:

[0071]

[0072]

[0073] in, y and y represent the predicted category and the true category label of video v, respectively, and P i This represents the predicted distribution of mode i. AvgFusion represents the average pooling fusion method.

[0074] Phase Two: Policy Learning Phase. In this phase, the training algorithm freezes the backbone network and utilizes policy gradients to optimize the policy networks of the multimodal fusion module and the samplers. First, the training algorithm freezes the backbone network trained in the first phase, freezes the policy networks of the samplers in all modalities, and unfreezes the policy network of the multimodal fusion module for training. During the training of the multimodal fusion module, the reward function encourages the policy network to learn the optimal fusion policy for the recognition task. Based on this, the training algorithm employs an intuitive reward function based on the video-level prediction distribution and the ground truth labels, which can progressively improve the probability of correct predictions. Specifically, at time step t, the prediction distribution is known... and video-level predicted tags t It can generate rewards for the multimodal fusion module. as follows:

[0075]

[0076] Where gt represents the video's true tag. Indicates the reward factor. Let l represent the probabilities of the true labels of the videos in the video-level prediction distributions at time t and t-1, respectively. t-1 This represents the video-level predicted label at time step t-1.

[0077] Secondly, the training algorithm freezes the multi-modal fusion module and unfreezes the policy network of all samplers in the modalities. Among these modalities, the CTDE framework is used to jointly optimize all samplers. Samplers compete with each other for computational resources, cooperate with each other, and extract overall structured features related to the identification task. Between modalities, the multi-modal fusion module is used to estimate the value of the modalities and assign fusion weights. Samplers of different modalities cooperate with each other to produce optimal predictions at the video level, and compete with each other to obtain more attention and computational resources.

[0078] To achieve the above hierarchical hybrid game, the training algorithm uses a multi-objective reward function for the policy network of the sampler. Specifically, the reward includes two parts. The first part allocates global rewards to multiple modalities to avoid lazy modalities. At time step t, given the fusion weights The global reward of modality n is defined as follows:

[0079]

[0080] The second part is the local reward based on the evaluation of the single-modal prediction, which is a multi-objective reward that considers the amount of computation consumed and the prediction effect. At time step t, for modality k, the single-modal prediction distribution and the single-modal prediction class label and the number of frames sampled by sampler i, the local reward received by sampler i is:

[0081]

[0082] where C is a constant proportion factor, is the cumulative number of frames observed by sampler i at time t.

[0083] At sampling step t, the reward value of sampler i is as follows:

[0084]

[0085] Here, λ is a constant scale factor that controls the greediness of the sampling strategy of each modality.

[0086] Third stage: fine-tuning stage. In the fine-tuning stage, in order to adjust the entire framework to the optimal state, the training algorithm alternately continues to train the backbone network trained in the first stage and the policy network obtained in the second stage. Based on the trained policy network, continue to train the backbone network, based on the trained backbone network, continue to train the policy network in the second stage, alternately train, fine-tune the entire network framework. For backbone network training, unlike the first stage, the training algorithm puts the trained fixed parameter policy network back into the framework to replace the random and average strategy of sampling and multi-modal fusion, and realizes the fine-tuning of the entire framework.

[0087] As shown in Figure 5 , the multi-modal based long video recognition method without pruning of the application first performs model prediction and then uses the trained model to perform long video recognition. The training stage and the prediction stage are as follows:

[0088] (1) In the training stage, the video recognition model of the application is first pre-trained based on large public datasets, such as Kinetics, ActivityNet and FCVID datasets. The training is divided into three stages, namely preheating, strategy learning and fine-tuning. The video recognition model will learn how to construct class-related spatial and temporal features, and at the same time learn how to adaptively find high-value video frames.

[0089] (2) In the prediction stage, the video to be detected is input. The video recognition model first preprocesses the video, further extracts the spatial features within the frame and the temporal features between the frames after selecting high-value video frames, and finally fuses the features of multiple modalities to output the corresponding video recognition result.

[0090] As shown in Figure 6 , the multi-modal based long video recognition system and method of the application are tested on three commonly used large video datasets—ActivityNet, FCVID and Kinetics-Sounds. Figure 6 The horizontal coordinate in the figure represents the inference cost for evaluating the computational consumption, and the vertical coordinate represents the mAP index for evaluating the target detection accuracy. The higher the mAP value, the higher the target detection accuracy and accuracy.

[0091] Figure 6 In the left graph, MultiAgent, LiteEval, AdaFrame-5, SCSampler, AR-Net, VideoIQ, FrameExit and AdaFocusV2 are all existing single-modal recognition methods. As can be seen from the figure, the application method (AdaSense) and these methods are used for single-modal class recognition on public datasets. The application has better detection accuracy and accuracy, and lower computational consumption.

[0092] Figure 6 In the right graph, ListenToLook and AdaMML are both existing multi-modal recognition methods. As can be seen from the figure, the application method achieves the most advanced long video recognition accuracy, while greatly reducing the inference cost. Specifically, AdaSense not only greatly reduces the computational consumption, but also significantly exceeds the most advanced method in accuracy on all datasets, compared with the basic framework without the game mechanism. For example,Figure 6 As shown, the recognition accuracy in all data sets is improved by 4.9%-12.2%, and GFLOPs is reduced by 47.0%-49.3%.

[0093] In addition to the technical features described in the specification, they are known to those skilled in the art. The present application omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present application. The embodiments described in the above embodiments do not represent all embodiments consistent with the present application. Various modifications or variations made by those skilled in the art without creative labor on the basis of the technical solutions of the present application are still within the protection scope of the present application.

Claims

1. A multi-modal based untrimmed long video recognition system, characterized in that, The application relates to a video recognition system, comprising a video acquisition and preprocessing module, an identification and detection module, a to-be-identified video database and a result database. The video acquisition and preprocessing module is used for receiving a video, pre-processing the video and storing the pre-processed video into the to-be-identified video database. The identification and detection module is used for identifying the category of the to-be-identified video and writing the identification result into the result database. The identification and detection module identifies the video category, comprising the following steps: for each frame sequence of a video mode, N samplers are respectively used to interact with the frame sequence of the corresponding mode, local events are located, the time causal relationship between the local events is inferred by a single-mode semantic reasoning module to capture the semantic information of the single mode, a single-mode prediction distribution is output by a sub-identification module, and finally a multi-mode fusion module is used to assign attention weights to different modes, fuse the prediction distributions of all modes and output a video-level multi-mode category prediction result; N is an integer greater than 2. The identification and detection module uniformly initializes N samplers in the frame sequence of each mode of a video, each sampler contains four parts, namely a local sampler, a context perception network, a strategy network and an evaluation network; the local sampler extracts local information from selected frames; the context perception network comprises a feature extractor and an LSTM module, the feature extractor encodes the local information extracted by the local sampler to output a feature vector, and the LSTM module models the time causal relationship between the selected frames at different time steps; the strategy network is composed of a full connection network and is used for selecting the frames for locating local events; The evaluation network is composed of a full connection network and is used for estimating the q value of the action selected by the strategy network in the training stage to help the learning of the strategy network; an Actor-Critic structure is used in each sampler, and a centralized evaluation and decentralized execution framework is used to combine all the samplers; The single-mode semantic reasoning module is a multi-layer transformer-based encoder, the hidden states of the LSTM modules of the context perception networks of all the samplers of the mode k at the current time are spliced and input into the single-mode semantic reasoning module after all the samplers of the mode k stop; the single-mode semantic reasoning module outputs the hidden features of the last layer of the encoder; The sub-identification module adopts a full connection network, accepts the output of the single-mode semantic reasoning module and maps the output to a single-mode video category prediction distribution; The multi-mode fusion module converts the dynamic adjustment of the fusion weights into a Markov decision process, initializes the fusion weights of all modes, adjusts the fusion weights step by step by using a strategy network, fuses the single-mode prediction distributions by using the finally optimized fusion weights; the multi-mode fusion module adopts an Actor-Critic structure and uses an evaluation network to help the learning of the strategy network during the training.

2. The system of claim 1, wherein, The video acquisition and preprocessing module obtains a plurality of modal time-aligned and size-standardized multi-dimensional frame sequences of the video after preprocessing the video, and stores the frame sequences in a to-be-identified video database; the to-be-identified video database also stores basic information of the video, including video length, frame number and modal number.

3. The system of claim 1, wherein, The LSTM module obtains the feature vector O output by the feature extractor at time t t , combines the hidden state h t-1 and the cell state c t-1 of the LSTM network at time t-1, calculates the hidden state h t and the cell state c t of the LSTM network at time t, and outputs the hidden state to the policy network.

4. The system of claim 1, wherein, The policy network is implemented as follows: Let the parameters of the policy network for sampler i corresponding to mode k be... The policy distribution estimated by the policy network for sampler i corresponding to mode k at time t is: Sampler i is distributed from the action space Ω according to this strategy. s Select Action To determine which frame to observe next or whether to stop observing; where, the action space Ω s It consists of four actions {0,δ,2δ,3δ}, where δ represents the minimum number of frames to move and 0 represents the sampler stopping; This represents the hidden state of the LSTM module in the context-aware network of sampler i at time t; This represents the cumulative number of frames observed by sampler i at time t.

5. The system of claim 1, wherein, The hidden features h of the lth layer encoder of the single-modal semantic reasoning module l Generated as follows: h′ l = MSA(LN(h l-1 ))+h l-1 h l = MLP(LN(h' + h) + h') + h l l ​ wherein h l-1 is the hidden feature of the l-1 layer encoder of the single-modal semantic reasoning module, h′ l is the intermediate state of the l-1 layer encoder of the single-modal semantic reasoning module, MSA represents a multi-head self-attention mechanism, LN represents layer normalization, and MLP represents a perception machine.

6. The system of claim 1, wherein, In the multi-modal fusion module, a strategy network f is used u The fusion weight is adjusted step by step until convergence; assuming that the video has K modes, the fusion weight of the K modes is adjusted according to the following strategy distribution in the adjustment step r of the strategy network f u of the strategy network f. wherein p i represents the single-modal prediction distribution of the modal i, represents the fusion weight of the modal i at the adjustment step r, and the multi-modal fusion module adjusts the fusion weight of the modal i according to the strategy distribution from the action space Ω f , and the adjustment direction and amplitude of the fusion weight are selected from the action space Ω f ={-ξ,-0.1ξ,0,+0.1ξ,+ξ}, wherein ξ is a pre-set maximum adjustment amplitude value; when the adjustment amplitudes of any two modal are 0, the adjustment process is terminated, and the fusion weight {ω1,ω2,…ω K} is output. Finally, the multimodal fusion module outputs a video-level prediction result 7. The system of claim 1, wherein, The identification and detection module is trained in three stages in the training stage, as follows: (1) The first stage, freeze the strategy network of the sampler and the multi-modal fusion module, the backbone network of the single mode extracts the spatial and temporal features of the frame sequence through a random strategy, obtains a video sample v from the training data set V, and randomly extracts half of the frames of the sample v and uniformly fuses all modalities to obtain a prediction result The cross-entropy loss on the training data set V is minimized by using standard back propagation; (2) The second stage, freeze the backbone network of each single modality in the first stage, and optimize the policy network of the sampler and the multi-modal fusion module by using the policy gradient. In this stage, first, freeze the policy network of the sampler in all modalities, train the policy network of the multi-modal fusion module, and use an intuitive reward function based on the video-level prediction result and the true label to encourage the policy network of the multi-modal fusion module to learn the optimal fusion strategy. At time step t, the intuitive reward function is as follows: as follows: wherein, denotes a reward factor, l t denotes a reward factor, l t-1 denotes a video-level prediction label at time step t, t-1, respectively, gt denotes the true label of the video, denotes the probability of the true label of the video corresponding to the true label of the video in the video-level prediction distribution at t, t-1, respectively; Secondly, freeze the multi-modal fusion module, unfreeze the policy network of the sampler in all modalities, and train the policy network of the sampler using a multi-objective reward function; the multi-objective reward function includes two parts, the first part is to distribute the global reward to each modality, and the fusion weight of K modalities obtained at time step t is set as Then the global reward of modality n is As follows: The second part is a local reward based on the single-modal prediction evaluation, which is received by the sampler i at time step t is: where C is a constant scaling factor, denotes the cumulative number of frames observed by sampler i at time t, and denotes the single-modal prediction label at time step t and time step t-1, respectively, and denotes the single-modal prediction distribution at time step t and time step t-1, respectively, obtaining a reward value for sampler i at time step t where λ is a scale factor; (3) In the third stage, the backbone network is continuously trained based on the trained strategy network, the strategy network is continuously trained based on the trained backbone network in the manner of the second stage, and the training is alternately performed to fine-tune the entire network framework.

8. The system of claim 1, wherein, The system further comprises a result display module for receiving an external query request, querying a result from a result database and displaying the result, the result database storing a predicted category, a category distribution, a selected observation frame number and a fusion ratio of different modalities of the video processed by the identification and detection module.

9. A multi-modal based untrimmed long video recognition method, characterized in that, Comprise: (1) obtaining a to-be-identified video, decoding and preprocessing the video, and obtaining a plurality of modal time-aligned and size-standardized multi-dimensional frame sequences of the video; (2) for each modal frame sequence of the video, N samplers are respectively used to interact with the frame sequence of the corresponding modal to locate local events, the time causal relationship between the local events is inferred by a single-modal semantic reasoning module to capture the semantic information of the single modal, a sub-identification module is used to output the prediction distribution of the single modal, and finally a multi-modal fusion module is used to assign attention weights to different modalities, fuse the prediction distributions of all modalities, and output a multi-modal category prediction result at the video level; N is an integer greater than 2; wherein the N samplers are uniformly initialized in each modal frame sequence of the video, each sampler comprises four parts, namely a local sampler, a context-aware network, a strategy network and an evaluation network; the local sampler extracts local information from the selected frames; the context-aware network comprises a feature extractor and an LSTM module, the feature extractor encodes the local information extracted by the local sampler to output a feature vector, and the LSTM module models the time causal relationship between the selected frames at different time steps; the strategy network is composed of a fully connected network and is used to select the frames for locating local events; the evaluation network is composed of a fully connected network and is used to estimate the q value of the action selected by the strategy network in the training stage to help the learning of the strategy network; an Actor-Critic structure is used in each sampler, and a centralized evaluation and decentralized execution framework is used to combine all samplers; the single-modal semantic reasoning module is a multi-layer transformer-based encoder, after all the samplers of the modal k stop, the hidden states of the LSTM modules of the context-aware networks in all the samplers of the modal k at the current time are spliced and input into the single-modal semantic reasoning module; the single-modal semantic reasoning module outputs the hidden features of the last layer of the encoder; the sub-identification module adopts a fully connected network, accepts the output of the single-modal semantic reasoning module, and maps the output to a single-modal video category prediction distribution; the sub-identification module adopts a fully connected network, accepts the output of the single-modal semantic reasoning module, and maps the output to a single-modal video category prediction distribution; The multi-modal fusion module converts dynamic adjustment of the fusion weight into a Markov decision process, first initializes fusion weights of all modes, then gradually adjusts the fusion weights by using a strategy network, and performs fusion of the single-modal prediction distribution by using the finally optimized fusion weights; the multi-modal fusion module adopts an Actor-Critic structure, and uses an evaluation network to help the strategy network learn during training.

Images