A method and device for first-person video action recognition
Through the multi-scale dual-mode spatiotemporal representation attention mechanism and cross-fusion module, the problem of low generalization performance in first-view video action recognition is solved, and higher classification accuracy and better spatiotemporal semantic representation are achieved. Suitable for complex first-view video data.
Patent Information
- Application Number
- CN202210120923.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-09
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-02-09
AI Technical Summary
The existing deep learning methods have low generalization performance in first-view video action recognition tasks, making it difficult to effectively process inter-frame motion information and complex timing relationships, especially in long-term video data, and the classification effect is not good.
The multi-scale dual-mode spatial characterization attention mechanism and cross-fusion module are adopted to extract the multi-scale spatial semantic features of RGB mode and depth mode through convolutional neural network, and combine the Inter-frame Encoder module to process the inter-frame timing relationship to generate Joint feature embedding vectors, enhance spatiotemporal semantic representation and improve classification accuracy.
It significantly improves the classification accuracy of first-view video action recognition, enhances the generalization ability of the model, and can achieve high recognition effects in different scenarios and movements. It improves the accuracy of action recognition through the fusion of multi-scale multi-modal information. sex.
Smart Images

Figure CN114596520B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the fields of deep learning, computer vision, etc., and relates to feature extraction and action recognition technology, especially a first-person video action recognition method and device. Background Art
[0002] Action recognition based on video data is a very important research direction in the field of computer vision. Its task goal is to complete the video classification task according to the human actions appearing in the video for unprocessed or given segmented video clips. This task has broad application value in video security monitoring, daily behavior recognition, behavior interaction and other fields.
[0003] In recent years, with the help of deep learning methods, research in the field of computer vision has made a lot of progress. Convolutional neural networks (CNNs) have powerful feature extraction capabilities and can better extract feature information from feature maps. Among them, 2D convolutional neural networks have become increasingly mature in image classification tasks. They are transferred to video action classification tasks, and the convolutional neural network is directly applied to each frame of data. Deep features are extracted through multi-layer convolutional networks. The features of each frame are fused in a certain way, and then passed through the feedforward network and input into the fully connected layer FC (Full connection) to complete the classification task. This method can better extract the representation information of video frames, and the final classification result is also determined by the representation information of each frame. However, since this method splits the temporal expression between frames, it can only achieve certain results in some tasks (motion information in the video is not easy to distinguish, and representation information is easy to distinguish). For data that requires better distinction between motion information between frames, it lacks excellent generalization performance.
[0004] The image classification task in the field of computer vision is based on two-dimensional data, focusing on processing the spatial information within the frame. On this basis, the video action recognition task not only needs to process the spatial semantics of each frame, but also introduces the time dimension T to represent the temporal order between video frames. The task is more complex and is currently the focus of research in the field of computer vision. Compared with the image classification task, the video classification task adds temporal information. Commonly used 3D convolutional networks (such as combining two-dimensional convolution in the spatial domain with one-dimensional convolution in the temporal domain) simultaneously process the spatial information within the frame in the video clip and the temporal information between adjacent frames. While processing the representation information of the video frame, the motion information in the video is considered, which has a higher accuracy than the 2D convolutional neural network method. As an optimization of the 3D convolutional network, the Inflated3D ConvNet (I3D) method expands the trained 2D convolution kernel to the 3D category. It can benefit from the pre-trained model of the large ImageNet dataset and can further improve the accuracy of the model, but it will also produce a large computational loss.
[0005] The method of combining time and space is one of the mainstream research directions in the field of action recognition. It mainly focuses on the processing of spatial information and temporal information. The representation data of each frame represents spatial information, such as targets, scenes, etc., and the motion expression between frames represents temporal information, such as the movement of people and objects in the video, the movement of the camera, etc. The method of spatiotemporal network often uses RGB frames as one input to process spatial flow information, and uses multiple stacked optical frames as one input to process temporal information. It adopts two-way joint training, and finally complements and fuses the extracted deep features to complete the classification task. Although the combination of time and space can well integrate spatiotemporal information, the classification effect is not good when processing highly complex and long-term video data, and it cannot be well modeled.
[0006] The Long Short-Term Memory (LSTM) method is often used in sequence modeling problems such as natural language processing and is suitable for processing sequence data. Considering the complex temporal relationship between video frames, migrating it to the video action classification task can solve the gradient vanishing and gradient explosion problems in the long sequence training process. At the same time, the LSTM method can control the transmission state through gating, model long-term sequence information, and obtain higher accuracy than the recurrent neural network (RNN) in longer sequence data. However, in processing long video sequence tasks, the LSTM method has high computational complexity, high training difficulty, and low efficiency in temporal modeling.
[0007] In recent years, the attention mechanism has become an important method in deep learning and has been extensively studied in different application fields. The attention mechanism in deep learning is similar to the attention mechanism of human vision, which is to focus on important points among a lot of information, select key information, and ignore other unimportant information. The attention mechanism breaks through the limitation that the RNN model cannot be calculated in parallel. At the same time, it allows the dependencies of the input and output sequences to be modeled without considering their distance in the sequence. It can well solve the long-term dependency problem and has been widely used in third-person action recognition tasks.
[0008] Algorithms based on deep learning are widely used in human activity recognition. At the same time, first-person perspective visual tasks have also attracted much attention due to their many practical applications. However, due to the existence of problems such as the movement of the shooting lens and the sampling distance of the camera, there are relatively large changes in the movement of the mover, object and background between the frames of the first-person perspective visual data, and there are large differences between the first-person perspective video data and the third-person perspective video. At the same time, traditional video action recognition algorithms do not pay much attention to the correlation of spatial information within the frame, and pay little attention to the distribution of spatial features on the feature map, which leads to certain difficulties in the first-person perspective video action recognition task and low generalization performance. Summary of the invention
[0009] The purpose of the present invention is to overcome the shortcomings of existing deep learning methods and provide a method and device for first-person video action recognition. Combined with the characteristics of first-person video datasets, it can obtain the relevant weights of video data through a multi-scale bimodal spatiotemporal representation attention mechanism, enhance the spatiotemporal semantic representation, and combine with a cross-fusion module to obtain a higher classification accuracy.
[0010] The technical solution to achieve the purpose of the present invention is:
[0011] On the one hand, the present invention provides a first-person video action recognition method, which inputs a first-person video action data set into a multi-scale network based on RGB mode and depth mode to extract spatial semantics, wherein the network adopts a convolutional neural network CNN, and selects two different convolution blocks in the convolutional neural network CNN to respectively output feature maps of two scales, wherein the first type of feature map contains certain spatial information, and the second type of feature map contains rich high-level semantic information, and the first type of feature map is processed by an MCIAMⅠ module, and the second type of feature map is processed by an MCIAMⅡ module, thereby obtaining a feature embedding vector with rich multi-scale bimodal spatial semantics, and the feature embedding vector is processed as the input of an Inter-frame Encoder module, and after multiple Inter-frame The Encoder module is used to extract the temporal relationship between frames and obtain three feature embedding vectors, which are generated by the RGB branch, the depth branch and the multi-scale fusion branch respectively. The data of the RGB branch and the depth branch are fused through the CFAM module, and the fusion of the feature embedding vector of the multi-scale fusion branch is completed to generate a joint feature embedding vector. The joint feature embedding vector is processed through a linear layer to obtain the action classification result of each frame. Then, the video frames of an action clip are averaged along the temporal direction to output the final recognition result.
[0012] Furthermore, the MCIAMⅠ module first segments the first type of feature map by average segmentation or edge cross segmentation, and then maps the feature map into an embedding vector through feature embedding and linear mapping, adds position information, calculates the weight matrix of the embedding vector generated by the same cutting method of the RGB modality and the depth modality, and enhances the spatial correlation between the modalities.
[0013] Furthermore, the MCIAMⅡ module adopts a feature segmentation method of horizontal and vertical segmentation to calculate the spatial correlation between the RGB modality and the depth modality, and fuses the four embedding vectors generated by the MCIAMⅠ module according to the different modalities to complete multi-scale dual-modal spatial semantic enhancement.
[0014] Furthermore, the Inter-frame Encoder module uses trainable linear projection to complete linear mapping to generate a corresponding feature embedding vector, adds position encoding to it, and is used to encode the position information of sequence frames or patches. Then, a multi-head attention operation is performed to obtain an intermediate vector, and finally, the intermediate vector is input into the feedforward network, and residual connection and layer regularization operations are completed.
[0015] The present invention uses the cross entropy loss function to evaluate the difference between the real action label of the video and the prediction result of the current model. The loss function is as follows:
[0016]
[0017] In the formula, n represents the number of behavior categories, i represents the real category number of the video data processed by the current network, and y i represents the true label of the corresponding category, p i Represents the probability value of the corresponding category predicted by the model.
[0018] Moreover, data preprocessing and data enhancement are performed before the first-person video action dataset is input into the network.
[0019] The convolutional neural network CNN uses the multi-layer convolution residual block of ResNet-34 as a basic component and uses a ResNet-34 model pre-trained on the ImageNet dataset.
[0020] Another aspect of the present invention further provides a first-person video action recognition device, comprising:
[0021] Convolutional neural network (CNN) module is used to extract feature information, perform feature extraction of RGB mode and depth mode at the 2D feature map level, and output feature maps of two scales (N1, N1) and (N2, N2);
[0022] The multimodal cross-frame attention module MCIAMⅠ is used to process feature maps of (N1, N1) scale. It uses two different feature segmentation methods: average segmentation and edge cross segmentation. Then, it uses feature embedding and linear mapping to map the feature map into an embedded vector, adds position information, and calculates the weight matrix of the embedded vector generated by the same cutting method of the two modalities to enhance the spatial correlation between the modalities.
[0023] The multimodal cross-frame attention module MCIAMⅡ is used to process the feature map of (N2, N2) scale, and adopts the feature segmentation method of horizontal and vertical segmentation to calculate the spatial correlation between modalities. The calculation process is the same as that of the MCIAMⅠ module, and the four embedding vectors generated by the MCIAMⅠ module are fused according to the different modalities to complete the multi-scale dual-modal spatial semantic enhancement;
[0024] Inter-frame Encoder: It is used to model the time process, use the self-attention mechanism to complete the processing of action timing information, model the long-term relationship of movement, reasonably allocate the weight of inter-frame feature embedding, and can suppress the interference of irrelevant targets and objects in the video, and allocate more attention resources to the focus area;
[0025] The cross-fusion attention module CFAM is used to complete the joint representation of the temporal network RGB modality and the deep modality. It learns the shared structure between different modalities through the mutual attention mechanism. The cross-fusion module is responsible for the final multi-scale spatiotemporal information fusion and generates a joint feature embedding vector.
[0026] Furthermore, the device also includes a preprocessing module for preprocessing and data enhancement of the first-person video action data by random cropping.
[0027] The third aspect of the present invention is to provide a storage medium storing a computer program, wherein the program can be executed by a terminal device or a computer when it is run to execute the method described in the first aspect of the present invention.
[0028] The network based on the modeling of the temporal relationship between frames can well complete the action recognition under the third perspective, but due to the shooting, the first perspective video has irregular jitter and background movement. At the same time, the first perspective action recognition tasks are mostly interactive behaviors (such as the interaction between people and objects), so more fine-grained action recognition is required, which not only needs to recognize the actions of the moving subject, but also the interactive objects. Before the temporal information learning module, the present invention adopts different segmentation interaction methods for feature maps of different scales, which enhances the representation ability of the moving subject operation objects and enhances the network's learning ability of spatial information.
[0029] By analyzing the complementary relationship between modalities, the present invention proposes a feature embedding vector grouping fusion scheme, which is analyzed from three levels: at the spatial level, the potential spatial semantic relationship of different scales between modalities can be well excavated; at the fusion period level, two fusion architectures of different periods are adopted. The first architecture first obtains multi-scale semantic information at the spatial level after inter-modal fusion, and then processes the temporal relationship between frames to complete the multi-scale fusion of spatiotemporal information. The second architecture first processes the temporal information of the RGB modality and the depth modality, and then completes the fusion. The above two architectures each pay attention to different attention information and enrich the fusion semantics; at the multi-stream architecture level, the complementarity of different information is achieved through cross-fusion, which greatly improves the accuracy of action recognition.
[0030] A large number of experiments conducted on the FPHA benchmark in the present invention prove the effectiveness of each module. More importantly, the present invention provides a new method for first-person action recognition that combines multi-scale and multi-modal spatiotemporal information fusion.
[0031] Advantages and beneficial effects of the present invention:
[0032] 1. The present invention uses multimodal cross-frame attention modules MCIAMⅠ and MCIAMⅡ to implement a multi-scale multimodal spatial interactive attention mechanism based on Patch, which can fuse and enhance information of different modalities, learn the best representation effect, and provide rich semantic information for subsequent temporal information processing. At the same time, combined with the characteristics of the first-person video action dataset, a variety of Patch segmentation strategies are adopted to make full use of the edge information of the central area of the crop, which can enhance the generalization ability of the model and help achieve good results in experiments of different scenes and movements.
[0033] 2. The present invention designs a dual-modal feature embedding vector grouping fusion strategy to retain the unique attributes of each modality, and better fuse the low-level features containing rich spatial information with the high-level features containing richer deep semantics through certain embedding mapping transformations and other methods. At the same time, it combines the respective characteristics of RGB and depth data to explore shareable information in a unified deep architecture. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1a This is an example of dual-modal data used in the present invention (clapping action, RGB mode);
[0035] Figure 1b This is an example of dual-modal data used in the present invention (high-five action, depth mode);
[0036] Figure 2 This is a network overall structure diagram of a first-person video action recognition method according to an example of the present invention;
[0037] Figure 3 This is the model diagram of the MCIAMⅠ module;
[0038] Figure 4 This is the model diagram of the MCIAMⅡ module;
[0039] Figure 5 This is the model diagram of the Inter-frame Encoder module;
[0040] Figure 6 This is the model diagram of the CFAM module; DETAILED DESCRIPTION
[0041] The present invention is further described in detail below through specific examples. The following examples are only illustrative and not restrictive, and the protection scope of the present invention cannot be limited thereto.
[0042] The present invention proposes a first-person video action recognition method, which uses an end-to-end network for training.
[0043] like Figure 1a , Figure 1b As shown, the present invention uses dual-modal first-person video action data to carry out experiments.
[0044] like Figure 2 As shown, the present invention adopts a network structure based on RGB modality and depth modality, and multi-stream network is trained together. At the same time, the data of the two modalities correspond one to one in the video frame content in time sequence. The present invention selects the FPHA dataset for experiments. The dataset consists of 1175 videos of 24 different objects participating in 45 action categories.
[0045] like Figure 1a , Figure 1b As shown, the video data clips of the first-person action recognition task are mostly short, and the range of motion of the action is small and the visual space occupied is large, which will lead to a certain context redundancy. In order to reduce overfitting, the present invention performs data enhancement by means of resizing, flipping, random cropping, etc. to improve the randomness of the training data, and then uniformly samples 32 frames in each video segment as the input of the network.
[0046] The present invention constructs a feature extraction network, which uses the multi-layer convolution residual block of ResNet-34 as the basic component, and uses the ResNet-34 model pre-trained on the ImageNet data set. Considering that as the depth of the convolutional neural network continues to increase, the extracted feature space semantics gradually decreases, and the high-level semantics gradually increase, and both semantic information play a relatively important role in the first-person action recognition task, in order to take into account the role of the pre-trained feature extraction model, ensure a certain degree of high-level semantics, and extract the spatial information of the feature map to a greater extent, and improve the intra-frame correlation, the present invention selects the fourth layer convolution block and the fifth layer convolution block in the ResNet-34 network to output feature maps of two scales respectively. The first type of feature map contains certain spatial information, and the second type of feature map contains relatively rich high-level semantic information. The present invention uses a multi-stream and multi-scale network structure to extract rich spatial semantics, and designs MCIAMⅠ and MCIAMⅡ modules at the same time.
[0047] After being processed by the MCIAMⅠ and MCIAMⅡ modules, the network obtains a feature embedding vector with rich multi-scale bimodal spatial semantics, which is processed as the input of the Inter-frame Encoder module.
[0048] After being processed by multiple Inter-frame Encoder modules, the temporal relationship between frames is extracted and three feature embedding vectors are obtained, which are generated by the RGB branch, the depth branch and the multi-scale fusion branch respectively. The data of the RGB branch and the depth branch are fused through the CFAM module, and the feature embedding vector of the multi-scale fusion branch is fused to generate the final feature embedding vector.
[0049] like Figure 3 , Figure 4 As shown, the present invention introduces the specific implementation of the multimodal cross-frame attention module (MCIAMⅠ, MCIAMⅡ). The two modules use similar underlying attention mechanisms. Next, the present invention takes the RGB modality branch in the architecture as an example for example.
[0050] For the input feature map F∈BTxCxHxW (BT: batch size*numbers of frames, C: channel, H: height, W: width), the input feature map of the MCIAMⅠ module is f_R1∈4x256xN1xN1, and the input feature map of the MCIAMⅡ module is f_R2∈128x512xN2xN2. Two segmentation methods are used for the feature map f_R1. The first method is to segment along the central axis, divide the feature map into 4 patches within the frame, and extract the correlation between patches. Specifically, the feature map f_R1 is deformed into f_R1∈BTxNxCP 2 In the form of, P represents the width and height of the newly generated feature map, Represents the number of generated Patch blocks, and CP is converted to 2 Mapped to D to get the feature embedding vector, which is then input into the cross-fusion attention module (which has the same architecture as the CFAM module) to generate the feature embedding vector F_R 1' ∈128x4x512. The second segmentation method is to make full use of the edge information of the segmentation position through flattening operation, enhance the richness of the spatial information of the edge position of the learned feature, and then input the cross-fusion attention module to generate the feature embedding vector F_R 1'‘ ∈128x4x512.
[0051] The input feature map of the MCIAMⅡ module is f_R2∈4x512xN2xN2, which is input into the cross-fusion attention module to generate the feature embedding vector F_R 2'‘ ∈128x4x512, the F_R generated by the MCIAMⅠ module using two segmentation interaction methods 1' and F_R 1'‘ , with F_R 2'‘ Add them together and add them to the F_D generated by the depth modal branch at the same time2'‘ Add them together to generate a feature embedding vector F_Cross that combines the spatial correlations of the two modalities at two scales.
[0052] like Figure 5 As shown in FIG. 1 , the Inter-frameEncoder module of the present invention is responsible for extracting the temporal relationship between frames. Taking the feature map f_R2 input to the module as an example, a trainable linear projection is used to complete the linear mapping to generate the corresponding feature embedding vector f_R 2' , adding position coding to it to encode the position information of sequence frames or patches. Then, multi-head attention operation is performed. Specifically, linear projection is first used to map the embedding vector to a series of query, key, and value vectors of dimension D. Then, the dot product of query and key vectors is calculated by softmax to obtain the corresponding attention weight, which is then weighted and summed with the value vector. For the operation in each head, it can be expressed by the following formula:
[0053]
[0054] Among them, Q represents the query vector, K represents the key vector, and V represents the value vector. The input of each Head is connected, and then dropout, residual connection, layer regularization and other operations are performed. The specific formula can be expressed as:
[0055] f 2” =LN(Dropout(Concat(MSA))+f_R 2' +x pos )
[0056] After that, the intermediate vector f_R 2” Input the feedforward network and complete operations such as residual connection and layer regularization. The specific formula can be expressed as:
[0057] F_R2=LN(Dropout(FFN(f_R 2” ))+f_R 2” )
[0058] Among them, FFM is a feed-forward network consisting of two layers of convolution with a convolution kernel size of 1, and F_R2 is the final output of the Inter-frame Encoder module.
[0059] like Figure 6As shown in the figure, the CFAM module responsible for cross-fusion attention in the present invention. Different modalities have feature diversity, and the fusion operation is performed through the CFAM module to complete the joint representation of the two branches. The feature embedding vector F_R2 of the RGB modal branch and the feature embedding vector F_D2 of the depth branch are processed with the new position information, and the subsequent operation processing is basically the same as Figure 5 The Inter-frameEncoder module in is the same, but the MSA calculation method is different, which can be expressed by the following formula:
[0060] F_D 2_M ,F_R 2_M =CFAM(F_R2,F_D2)
[0061] The F_D generated by the CFAM module 2_M and F_R 2_M Multiplication generates a joint representation F_M. At the same time, F_Cross completes the temporal correlation extraction to generate a new feature embedding vector F_Cross', which is added to F_M and input into the linear layer to complete the category mapping. After that, the results of each frame are averaged to obtain the final recognition result.
[0062] The above network model uses the cross entropy loss function. The cross entropy loss function is used to evaluate the difference between the real action label of the video and the prediction result of the current model. The smaller the value of the loss function, the better the effect of the trained model. The loss function is as follows:
[0063]
[0064] In the formula, n represents the number of behavior categories, i represents the real category number of the video data processed by the current network, and y i represents the true label of the corresponding category, p i Represents the probability value of the corresponding category predicted by the model. The network loss can optimize the network parameters and improve the performance of the model through continuous reverse iterative propagation.
[0065] This paper uses the Pytorch deep learning framework to train and test on the FPHA dataset, and the test standard is recognition accuracy. The experimental results are shown in the following table:
[0066] Methods Modality Accuracy(%) HON4D Depth 70.61 Two stream-color RGB 61.56 H+O RGB 82.43 1-layer LSTM Pose 78.73 2-layer LSTM Pose 80.14 Gram Matrix Pose 85.39 Two stream Flow+RGB 75.3 HOG2-depth+pose Pose+Depth 66.78 JOULE-all Pose+RGB+Depth 78.78 Ours RGB+Depth 96.70
[0067] As can be seen from the above table, the effectiveness of the present invention can be proved through comparative evaluation with other mainstream algorithms. Through the multi-scale bimodal spatiotemporal representation attention mechanism and the reasonable allocation of relevant weights of feature embedding, the spatiotemporal semantic representation can be enhanced. At the same time, combined with the cross-fusion module, a higher recognition accuracy can be obtained, and it has certain practical value.
[0068] The above description is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept, which all belong to the protection scope of the present invention.
Claims
1. A method for first-person video action recognition, characterized in that: The first-person video action dataset is input into a multi-scale network based on RGB modality and depth modality to extract spatial semantics. The network adopts a convolutional neural network CNN. Two different convolution blocks in the convolutional neural network CNN are selected to output feature maps of two scales respectively. The first type of feature map contains certain spatial information, and the second type of feature map contains rich high-level semantic information. The first type of feature map is processed by a multi-modal cross-frame attention module MCIAMⅠ, and the second type of feature map is processed by a multi-modal cross-frame attention module MCIAMⅡ, so as to obtain a feature embedding vector with rich multi-scale bi-modal spatial semantics. The feature embedding vector is processed as the input of the Inter-frameEncoder module. After multiple Inter-frame The encoder module is used to extract the temporal relationship between frames and obtain three feature embedding vectors, which are generated by the RGB branch, the depth branch and the multi-scale fusion branch respectively. The data of the RGB branch and the depth branch are fused through the cross fusion attention module CFAM, and the feature embedding vector of the multi-scale fusion branch is fused to generate a joint feature embedding vector. The joint feature embedding vector is processed through a linear layer to obtain the action classification result of each frame. Then, the video frames of an action clip are averaged along the temporal direction to output the final recognition result. The multimodal cross-frame attention module MCIAMⅠ first segments the first type of feature map by average segmentation or edge cross segmentation, then maps the feature map into an embedding vector by feature embedding and linear mapping, adds position information, calculates the weight matrix of the embedding vector generated by the same cutting method of the RGB mode and the depth mode, and enhances the spatial correlation between the modes; The multimodal cross-frame attention module MCIAMⅡ adopts a feature segmentation method of horizontal and vertical segmentation, calculates the spatial correlation between the RGB modality and the depth modality, and fuses the four embedding vectors generated by the multimodal cross-frame attention module MCIAMⅠ according to the different modalities to complete multi-scale bimodal spatial semantic enhancement.
2. The method for first-person video action recognition according to claim 1, characterized in that: The Inter-frameEncoder module uses trainable linear projection to complete linear mapping to generate the corresponding feature embedding vector, adds position encoding to it, and is used to encode the position information of sequence frames or patches. It then performs multi-head attention operation to obtain the intermediate vector, and finally inputs the intermediate vector into the feedforward network, and completes the residual connection and layer regularization operations.
3. The method for first-person video action recognition according to claim 1, characterized in that: The cross entropy loss function is used to evaluate the difference between the real action label of the video and the prediction result of the current model. The loss function is as follows: In the formula, n represents the number of behavior categories, i represents the real category number of the video data processed by the current network, and y i represents the true label of the corresponding category, p i Represents the probability value of the corresponding category predicted by the model.
4. The method for first-person video action recognition according to claim 1, characterized in that: Data preprocessing and data enhancement are performed before the first-person video action dataset is input into the network.
5. The method for first-person video action recognition according to claim 1, characterized in that: The convolutional neural network CNN uses the multi-layer convolution residual block of ResNet-34 as a basic component and uses a ResNet-34 model pre-trained on the ImageNet dataset.
6. A first-person video action recognition device, characterized in that: include: Convolutional neural network (CNN) module is used to extract feature information, perform feature extraction of RGB mode and depth mode at the 2D feature map level, and output feature maps of two scales (N1, N1) and (N2, N2); The multimodal cross-frame attention module MCIAMⅠ is used to process feature maps of (N1, N1) scale. It uses two different feature segmentation methods: average segmentation and edge cross segmentation. Then, it uses feature embedding and linear mapping to map the feature map into an embedded vector, adds position information, and calculates the weight matrix of the embedded vector generated by the same cutting method of the two modalities to enhance the spatial correlation between the modalities. The multimodal cross-frame attention module MCIAMⅡ is used to process the feature map of (N2, N2) scale, and adopts the feature segmentation method of horizontal and vertical segmentation to calculate the spatial correlation between modalities. The calculation process is the same as that of the multimodal cross-frame attention module MCIAMⅠ, and the four embedding vectors generated by the multimodal cross-frame attention module MCIAMⅠ are fused according to the different modalities to complete the multi-scale dual-modal spatial semantic enhancement; Inter-frame Encoder: It is used to model the time process, use the self-attention mechanism to complete the processing of action timing information, model the long-term relationship of movement, reasonably allocate the weight of inter-frame feature embedding, and can suppress the interference of irrelevant targets and objects in the video, and allocate more attention resources to the focus area; The cross-fusion attention module CFAM is used to complete the joint representation of the temporal network RGB modality and the deep modality. It learns the shared structure between different modalities through the mutual attention mechanism. The cross-fusion module is responsible for the final multi-scale spatiotemporal information fusion and generates a joint feature embedding vector.
7. The first-view video action recognition device according to claim 6, characterized in that: The method also includes a preprocessing module for preprocessing and data enhancement of the first-person video action data by random cropping.
8. A storage medium, characterized in that: The storage medium stores a computer program, wherein the program can be executed by a terminal device or a computer to execute the method described in any one of claims 1 to 5.
