Video identification method and device, electronic equipment, medium and program product

Abstract

The invention discloses a video identification method and device, electronic equipment, a medium and a program product, and relates to the technical field of communication, a plurality of video frames in a video to be identified are acquired, and the respective pixel matrix of each video frame is determined; performing position coding on each pixel matrix to obtain a position enhancement matrix commonly corresponding to each pixel matrix; determining a cross-frame motion feature from the position enhancement matrix through a preset cross attention module; and performing video action classification on the cross-frame motion features through a preset classifier to obtain an action classification result of the to-be-identified video. The technical problem that the calculation cost of video action recognition is high is solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technology, and in particular to a video recognition method, device, electronic device, medium, and program product. Background Art

[0002] Video action recognition plays an important role in human-computer interaction, unmanned driving, health monitoring and other fields. Currently, most video action recognition is implemented based on the 3D CNN (3D Convolutional Neural Network) model. Since videos have two dimensions, time and space, when performing action video recognition, the 3D CNN model needs to perform convolution in the time and space dimensions of the video to perform video action recognition. However, as the resolution of the video increases, the computational complexity of the 3D CNN model in the time dimension will also increase significantly, thereby increasing the computational cost of video action recognition.

[0003] The above contents are only used to assist in understanding the technical solutions of the embodiments of the present application, and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a video recognition method, device, electronic device, medium and program product, aiming to solve the technical problem of high computational cost of video action recognition.

[0005] To achieve the above object, the present application provides a video recognition method, including:

[0006] Acquire multiple video frames in the video to be identified, and determine the pixel matrix of each video frame;

[0007] Performing position encoding on each of the pixel matrices to obtain a position enhancement matrix corresponding to each of the pixel matrices;

[0008] Determining cross-frame motion features from the position enhancement matrix through a preset cross-attention module;

[0009] The cross-frame motion features are subjected to video action classification by a preset classifier to obtain an action classification result of the video to be identified.

[0010] In one embodiment, the step of obtaining multiple video frames in the video to be identified and determining the pixel matrix of each video frame includes:

[0011] Acquire the video to be identified, and extract a preset first number of video frames from the video to be identified;

[0012] Based on a preset flip probability, determining a preset second number of video frames to be flipped from each of the video frames, and horizontally flipping each of the video frames to be flipped to obtain flipped video frames;

[0013] Based on a preset video image size, each flipped video frame and a non-flipped video frame in the video frames are cropped to obtain each pixel matrix.

[0014] In one embodiment, the step of performing position encoding on each of the pixel matrices to obtain a position enhancement matrix corresponding to each of the pixel matrices comprises:

[0015] Based on a preset area division unit, each of the pixel matrices is divided to obtain a pixel area corresponding to each of the pixel matrices;

[0016] According to a preset embedding matrix, embedding operations are performed on each of the pixel regions respectively to obtain a pixel embedding matrix of each of the pixel regions;

[0017] The pixel embedding matrices, the preset pixel global matrix and the preset position encoding matrix are spliced ​​to obtain a position enhancement matrix.

[0018] In one embodiment, the step of determining the cross-frame motion feature from the position enhancement matrix by using a preset cross-attention module comprises:

[0019] Normalizing the position enhancement matrix to obtain a normalized position matrix, and determining a pixel block coding sequence of the normalized position matrix;

[0020] Determine, according to the cross attention module and the pixel block coding sequence, an attention matrix of each attention module of the pixel block coding sequence in the cross attention module;

[0021] Perform cross-frame operations in each of the attention matrices to obtain a cross-frame attention head for each of the attention modules;

[0022] The cross-frame attention heads are concatenated to obtain cross-frame motion features.

[0023] In one embodiment, the attention matrix comprises a block attention matrix of a pixel block; and the step of determining, according to the cross attention module and the pixel block coding sequence, the attention matrix of each attention module of the pixel block coding sequence in the cross attention module comprises:

[0024] The pixel block encoding sequence is input into the cross-attention module. For each of the attention modules in the cross-attention module, a block attention matrix of each pixel block in the pixel block encoding sequence is calculated according to the attention weight of the attention module. The block attention matrix includes a block query matrix, a block key matrix and a block value matrix.

[0025] In one embodiment, the step of performing cross-frame operations in each of the attention matrices to obtain a cross-frame attention head of each of the attention modules includes:

[0026] Determining a target shift matrix in each of the attention matrices, the target shift matrix being any of the attention matrices;

[0027] For each pixel block in the target shift matrix belonging to the same video frame, obtaining, in each attention matrix, respective adjacent block key matrices and adjacent block value matrices of adjacent pixel blocks in adjacent video frames;

[0028] Each block query matrix in the target shift matrix and the adjacent block value matrix and adjacent block key matrix corresponding to the target shift matrix are spliced ​​to obtain a cross-frame attention head of the attention module where the target shift matrix is ​​located.

[0029] In one embodiment, the method comprises:

[0030] Obtaining a training sample set, wherein the training sample set includes a plurality of training samples, each training sample includes a to-be-trained video and a classification label;

[0031] Inputting the video to be trained into a preset model to be trained to obtain an action recognition result, wherein the model to be trained includes a cross-attention module to be trained and a classifier to be trained;

[0032] Calculating the training loss of the model to be trained based on the action recognition result and the classification label;

[0033] The model to be trained is optimized according to the training loss to obtain a trained video action recognition model, wherein the video action recognition model includes a cross-attention module and a classifier.

[0034] In one embodiment, after the step of optimizing the model to be trained according to the training loss to obtain a trained video action recognition model, the method further includes:

[0035] Determine the result type of the action recognition result corresponding to each of the training samples;

[0036] Counting the number of first positive class samples whose sample types are training positive class and whose corresponding result types are positive class results, and counting the number of second positive class samples whose sample types are training positive class and whose corresponding result types are negative class results;

[0037] Count the number of first negative classes whose sample types are training negative classes and whose result types corresponding to the training samples are negative class results, and count the number of second negative classes whose sample types are training negative classes and whose result types corresponding to the training samples are positive class results;

[0038] Calculating the prediction accuracy of the video action recognition model according to the first positive class number and the second negative class number;

[0039] Calculating the recall rate of the video action recognition model according to the number of the first positive classes and the number of the second positive classes;

[0040] The specificity of the video action recognition model is calculated according to the number of the first negative classes and the number of the second negative classes.

[0041] In addition, to achieve the above-mentioned purpose, the embodiment of the present application provides a video recognition device, the device comprising:

[0042] An acquisition module is used to acquire multiple video frames in the video to be identified and determine the pixel matrix of each video frame;

[0043] A position enhancement module, used for performing position encoding on each of the pixel matrices to obtain a position enhancement matrix corresponding to each of the pixel matrices;

[0044] A cross-frame module, used for determining cross-frame motion features from the position enhancement matrix through a preset cross-attention module;

[0045] The classification module is used to classify the cross-frame motion features into video actions through a preset classifier to obtain the action classification result of the video to be identified.

[0046] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides an electronic device, which includes: a memory, a processor, and a program of the video recognition method stored in the memory and executable on the processor. When the program of the video recognition method is executed by the processor, the steps of the video recognition method as described above can be implemented.

[0047] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a computer-readable storage medium, on which is stored a program for implementing the video recognition method. When the program of the video recognition method is executed by a processor, the steps of the video recognition method as described above are implemented.

[0048] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video recognition method when executed by a processor.

[0049] One or more technical solutions proposed in the embodiments of the present application have at least the following technical effects:

[0050] The present application obtains multiple video frames of a video to be identified, determines pixel matrices of the multiple video frames, performs position encoding on each pixel matrix, and obtains a position enhancement matrix corresponding to each pixel matrix, thereby realizing position enhancement of each video frame in the video to be identified. Furthermore, through a preset cross-attention module, cross-frame action features are directly extracted from the position encoding matrix. Since the cross-frame action features can reflect the dynamic image features of the video to be identified, the cross-frame action features can be classified through a preset classifier to obtain the action classification result of the video to be identified, thereby realizing video action recognition of the video to be identified.

[0051] The present application avoids directly performing convolution on the time and space dimensions of the video to be identified. Instead, it constructs a position enhancement matrix corresponding to multiple video frames in the video to be identified, and directly extracts cross-frame action features in the position encoding matrix through a cross-attention module to realize the recognition of the dynamic image features of the video to be identified, thereby reducing the dependence on time and space convolution, so that the computational complexity will not increase with the increase of video resolution, thereby reducing the computational cost of video action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the embodiments of the present application, and together with the description are used to explain the principles of the embodiments of the present application.

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 This is a flow chart of an embodiment of a video recognition method according to an embodiment of the present application;

[0055] Figure 2 This is a flow chart of another embodiment of the video recognition method of the present application;

[0056] Figure 3 This is a flow chart of an example of a video recognition method in an embodiment of the present application;

[0057] Figure 4 This is a schematic diagram of the module structure of the video recognition device according to an embodiment of the present application;

[0058] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the video recognition method in the embodiment of the present application.

[0059] The purpose, features and advantages of the embodiments of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the embodiments of the present application and are not used to limit the embodiments of the present application.

[0061] In order to better understand the technical solutions of the embodiments of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0062] Identifying human actions in videos is an important topic in computer vision. Video action recognition plays an important role in human-computer interaction, intelligent robots, unmanned driving, health monitoring and other fields. Compared with images, videos have an additional time dimension. Videos can be regarded as a collection of images that are continuous in time. Most video action recognition is based on 3D CNN models. 3D CNN models trained with large-scale datasets can be effectively migrated to smaller datasets and have good generalization. However, their computational cost is high and requires large-scale training datasets. At the same time, videos inherently contain spatial and temporal information, and 3D CNN models also find it difficult to capture the complex dependencies between spatial and temporal dimensions. It is also difficult to recognize video actions.

[0063] Based on this, the present application embodiment provides a video recognition method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the video recognition method of the present application. The video recognition method includes steps S10 to S40:

[0064] Step S10, obtaining multiple video frames in the video to be identified, and determining the pixel matrix of each video frame;

[0065] It should be noted that the video to be identified may be a video that needs to perform video action recognition, and the number of the multiple video frames obtained in the video to be identified may be a preset first number. The video frame may be a video image extracted from the video to be identified, and the video image is usually composed of regularly arranged pixels. The pixel matrix is ​​a two-dimensional matrix of pixels in the image, and the size of the pixel matrix of each video frame may be the same. Each video frame has its own pixel matrix.

[0066] Before obtaining multiple video frames in the video to be identified, it can be determined whether the video to be identified needs to be segmented based on the duration of the video to be identified. If the duration of the video to be identified is less than or equal to the preset duration, multiple video frames are directly extracted from the video to be identified. If the duration of the video to be identified is greater than the preset duration, the video to be identified is segmented according to the preset segmentation unit to obtain each segmented video, and each of the multiple video frames is extracted from the segmented video. For example, the preset duration and the preset segmentation unit can both be 10s, so that the accuracy of video action video recognition can be guaranteed, and the problem of inaccurate recognition caused by the length of the video to be identified can be avoided. After the video to be identified is segmented into multiple segmented videos, video action recognition can be performed on each segmented video to determine the action classification results corresponding to each segmented video.

[0067] Exemplarily, a video to be identified is obtained, a preset first number of video frames are obtained from the video to be identified, and a pixel matrix corresponding to each video frame is determined. In other embodiments, each video frame may be cropped to obtain a pixel matrix corresponding to each video frame. For example, each video frame may be cropped directly based on a preset video image size to obtain a pixel matrix.

[0068] In a feasible embodiment, step S10 further includes steps S11 to S13:

[0069] Step S11, obtaining a video to be identified, and extracting a preset first number of video frames from the video to be identified;

[0070] Step S12, based on a preset flip probability, determining a preset second number of video frames to be flipped from each video frame, and horizontally flipping each video frame to be flipped to obtain flipped video frames;

[0071] It should be noted that the preset first number can be configured based on actual conditions, for example, it can be 8 frames, 6 frames or 10 frames, etc., and this embodiment does not make specific limitations on this. The preset flip probability is characterized by the proportion of video frames that are horizontally flipped to all video frames. The preset flip probability can also be set based on actual conditions, for example, it can be 50%, and this embodiment does not make specific limitations on this.

[0072] The preset second number is the product of the preset flip probability and the preset first number. The flipped video frame is an image that has been horizontally flipped. The to-be-flipped video frame is a video frame that needs to be horizontally flipped.

[0073] Step S13 , based on a preset video image size, cropping each flipped video frame and the video frames that are not flipped in each video frame, respectively, to obtain each pixel matrix.

[0074] The preset video image size is the size of the pixel matrix. The size of the video frame can be adjusted within the range of [224, 300] while ensuring that the aspect ratio of the image remains unchanged. The flipped video frame and the non-flipped video frame are cropped based on the preset video image size to obtain a pixel matrix. For example, the preset video image size can be 224×224. This ensures that the size of the cropped video frame is large enough to contain enough information so that effective feature extraction can be performed on the video frame later. When the preset video image size is 224×224, it can ensure that there is enough information in the video frame and minimize the amount of calculation when the video frame is subsequently recognized.

[0075] Exemplarily, a preset first number of video frames can be extracted from the video to be identified, and the product of the preset flip probability and the preset first number is called the preset second number. A preset second number of video frames to be flipped is randomly determined from the preset first number of video frames, and each video frame to be flipped is horizontally flipped to obtain each flipped video frame. Based on the preset video image size, each flipped video frame and the video frames that are not flipped in each video frame are cropped to obtain each pixel matrix. In other embodiments, the preset pixel range can be [224, 300], and the video frame size can be adjusted based on the aspect ratio of each flipped video frame and each non-flipped video frame. Then, based on the preset video image size, each flipped video frame with the adjusted image size and the image that is not flipped in the image are cropped to obtain each pixel matrix. In other embodiments, the preset first number of video frames can also be directly cropped based on the preset video image size to obtain each pixel matrix. This embodiment does not specifically limit this.

[0076] Step S20, position encoding each pixel matrix to obtain a position enhancement matrix corresponding to each pixel matrix;

[0077] In this embodiment, position coding may refer to position enhancement of each pixel matrix, and the position enhancement matrix may describe all pixel matrices that have been position-coded as a whole. Exemplarily, each pixel matrix may be position-coded based on a preset embedding matrix and a preset position coding matrix to obtain a position enhancement matrix corresponding to all pixel matrices.

[0078] In a feasible embodiment, step S20 further includes steps S21 to S23:

[0079] Step S21, dividing each pixel matrix based on a preset area division unit to obtain pixel regions corresponding to each pixel matrix;

[0080] Step S22, performing embedding operations on each pixel region according to a preset embedding matrix to obtain a pixel embedding matrix for each pixel region;

[0081] Step S23, concatenating each pixel embedding matrix, a preset pixel global matrix and a preset position encoding matrix to obtain a position enhancement matrix.

[0082] It should be noted that the preset area division unit can be set based on actual conditions, each pixel matrix can be divided into multiple pixel areas, the pixel area can be a patch (a patch usually refers to a small area in an image), and the preset area division unit can be p×p, where p can be 16 or can be determined based on actual conditions. This embodiment does not make specific limitations on this. When p is 16, a pixel matrix of 224×224 can be divided into 196 pixel areas.

[0083] Embedding a pixel region refers to fusing a pixel region with a preset embedding matrix. For example, the pixel region and the preset embedding matrix may be multiplied to embed the pixel region. The preset embedding matrix is ​​used to convert a pixel region (usually a two-dimensional array) into a one-dimensional feature vector (also called tokens). The preset pixel global matrix is ​​used to capture the global features of the image, thereby facilitating the subsequent improvement of the accuracy of video action recognition, and can solve the problem that only local learning can be achieved in traditional 3DCNN models. The preset position encoding matrix can be used to determine the correlation between different pixel regions.

[0084] Exemplarily, the concatenated position enhancement matrix can be expressed as:

[0085]

[0086] Among them, z0 is the position enhancement matrix, z0∈R T×(N+1)×D , D is the dimension of the position enhancement matrix, T is the preset first number, N is the total number of pixel regions, c0 is the preset pixel global matrix, the preset pixel global matrix can be a class token (image global information), c0∈R T×D , E pos is the preset position encoding matrix, E is the preset embedding matrix, is the i-th pixel area, i is an integer ranging from 0 to N, H is the height of the pixel matrix, W is the width of the pixel matrix, d is the dimension of the pixel area, d = 3P 2 , where 3 refers to the number of RGB (red, green, blue) channels. is the pixel embedding matrix of the i-th pixel region.

[0087] In this embodiment, position encoding is performed on each pixel matrix, and the position encoding matrix can be determined in the ViT (Vision Transformer, a computer vision task architecture) architecture. In the ViT architecture, the image can be preprocessed, that is, each pixel matrix is ​​segmented to obtain each pixel area, and the corresponding position enhancement matrix is ​​determined. After image preprocessing is performed in the ViT architecture and the position enhancement matrix is ​​obtained, the position enhancement matrix can be input into the video recognition model in the ViT architecture. The video recognition model in the ViT architecture is a trained Transformer model (a neural network model based on a self-attention mechanism), and the video recognition model includes a cross attention module and a classifier. In this embodiment, by determining the position enhancement matrix, it is convenient for the video recognition model to process each pixel area based on the position enhancement matrix, and it is also convenient for the video recognition model to distinguish the position of each pixel area in the video frame, and it is also convenient for the video recognition model to capture the global features of the video frame, so that the video recognition model can more accurately perform video action recognition. In this embodiment, when iteratively training the Transformer model to obtain the video recognition model, it is also possible to iterate the preset position encoding matrix at the same time so that each pixel area can be more accurately position encoded.

[0088] Step S30, determining cross-frame motion features from the position enhancement matrix through a preset cross-attention module;

[0089] It should be noted that the cross attention module can be MSCA (Multi-Scale Cross Attention), which can establish connections between pixel areas based on the position enhancement matrix to facilitate determining the cross-frame motion features corresponding to the position enhancement matrix.

[0090] Cross-frame motion features can be used to describe dynamic features in the video to be identified, and can reflect the human motion information in the video to be identified. The dynamic features can reflect the characteristics of the video to be identified in the spatial dimension and the temporal dimension.

[0091] In one example, when using the cross-attention module and the position enhancement matrix to determine the cross-frame motion features, the position enhancement matrix can be normalized first to ensure that the vectors in the position enhancement matrix remain consistent, so as to subsequently improve the accuracy of the cross-frame motion features. Each pixel area can correspond to a vector in the position enhancement matrix, and the vector can be the aggregated pixel embedding matrix and the preset position encoding matrix of the pixel area in the position enhancement matrix. The aggregated pixel embedding matrix and the preset position encoding matrix refer to the splicing of the pixel embedding matrix and the preset position encoding matrix. Then, based on the cross-attention module and the normalized position enhancement matrix, the cross-frame motion features are determined.

[0092] Step S40, using a preset classifier, classify the cross-frame motion features into video actions to obtain an action classification result of the video to be identified.

[0093] It should be noted that the classifier may be an MLP (Multilayer Perceptron, fully connected layer), and the classifier is used to classify the cross-frame motion features into video actions, and video action classification refers to determining the human motion category to which the cross-frame motion features belong, and each human motion category may be interaction between people, body movement, etc., and body movement may be further subdivided into subdivided movements such as running, walking, and playing ball, etc., which is not specifically limited in this embodiment. The action classification result may be used to describe the human motion category existing in the video to be identified by the classifier.

[0094] In one example, the cross-frame motion features can be normalized, and the cross-frame motion features can be normalized in a normalization layer in the video recognition model, and then the normalized cross-frame motion features are used by the classifier to classify the video action to obtain the action classification result of the video to be recognized. In other embodiments, the position enhancement matrix can also be normalized, and the classifier can combine the normalized position enhancement matrix and the normalized cross-frame motion features to determine the action classification result present in the video to be recognized, so as to enhance the transfer between features, so that the position enhancement matrix input into the video recognition model can be input into the classifier in addition to the cross-attention module, so as to ensure the accuracy of the classifier's recognition.

[0095] The embodiment of the present application obtains multiple video frames of the video to be identified, determines the pixel matrices of the multiple video frames, performs position encoding on each pixel matrix, and obtains a position enhancement matrix corresponding to each pixel matrix, thereby realizing position enhancement of each video frame in the video to be identified. Furthermore, through a preset cross-attention module, cross-frame action features are directly extracted from the position encoding matrix. Since the cross-frame action features can reflect the dynamic image features of the video to be identified, the cross-frame action features can be classified through a preset classifier to obtain the action classification result of the video to be identified, thereby realizing video action recognition of the video to be identified.

[0096] The embodiment of the present application avoids directly performing convolution on the time and space dimensions of the video to be identified. Instead, it constructs a position enhancement matrix that corresponds to multiple video frames in the video to be identified, and directly extracts cross-frame action features in the position encoding matrix through a cross-attention module to achieve the recognition of dynamic image features of the video to be identified, thereby reducing the dependence on time and space convolution, so that the computational complexity will not increase with the increase of video resolution, thereby reducing the computational cost of video action recognition.

[0097] Further, see Figure 2, step S30 also includes steps S31 to S34:

[0098] Step S31, normalizing the position enhancement matrix to obtain a normalized position matrix, and determining a pixel block coding sequence of the normalized position matrix;

[0099] Step S32, determining the attention matrix of each attention module of the pixel block coding sequence in the cross attention module according to the cross attention module and the pixel block coding sequence;

[0100] It should be noted that the normalization process of the position enhancement matrix may be performed in a normalization layer in the video recognition model to obtain a normalized position matrix. The normalized position matrix may be input into the video recognition model, and the video recognition model may determine a pixel block coding sequence of the normalized position matrix, wherein the pixel block coding sequence includes pixel image features of multiple pixel blocks, each pixel region may correspond to a pixel block, and the pixel image features are used to describe relevant information of the pixel block, such as the pixel value of the pixel block.

[0101] The cross-attention module includes multiple attention modules, each of which has its own corresponding attention weight. The same pixel block coding sequence can determine an attention matrix in different attention modules. It can be understood that the attention matrices generated based on the pixel block coding sequence in different attention modules are different.

[0102] Exemplarily, the position enhancement matrix can be input into a video recognition model, the video recognition model determines the pixel block coding sequence of the normalized position matrix, and passes the pixel block coding sequence to a cross-attention module in the video recognition model to determine the attention matrix of each attention module in the cross-attention module.

[0103] In a feasible embodiment, the attention matrix includes a block attention matrix of the pixel block, and step S32 also includes step S321: inputting the pixel block encoding sequence into the cross-attention module, for each attention module in the cross-attention module, calculating the block attention matrix of each pixel block in the pixel block encoding sequence according to the attention weight of the attention module, the block attention matrix includes a block key matrix and a block value matrix.

[0104] It should be noted that the cross attention module may include multiple attention modules, for example, the cross attention module may include 8 attention modules, and each attention module has a different attention weight. The attention weight includes query weight, key weight and value weight. The query weight, key weight and value weight refer to the weight matrices for generating the query matrix, key matrix and value matrix in the attention mechanism, respectively.

[0105] The block attention matrix includes the block query matrix, the block key matrix and the block value matrix. The block query matrix is ​​the query matrix of the pixel block, the block key matrix is ​​the key matrix of the pixel block, and the block value matrix is ​​the value matrix of the pixel block. Each pixel block in the pixel block encoding sequence can have its own corresponding block attention matrix. The same pixel block has different fast attention matrices in different attention modules.

[0106] Exemplarily, for each attention module, the block query matrix, block key matrix and block value matrix corresponding to each pixel block in the pixel block coding sequence are calculated according to the query weight, key weight and value weight of the attention module. This embodiment can capture the relationship between different pixel blocks in the image by determining the block attention matrix of each pixel block in the pixel block coding sequence, thereby facilitating the subsequent shifting of the block key matrix and block value matrix of the pixel block, so that the pixel blocks in other video frames can also pay attention to the block key matrix and block value matrix in other video frames, thereby facilitating the improvement of the accuracy of the video recognition model in identifying the video action category.

[0107] For example, the formula for calculating the block attention matrix in the attention matrix can be:

[0108] K (t) ,Q (t) ,V (t) =z (t) [W k ,W q ,W v ]

[0109] Among them, z (t) is the pixel block of frame t in the pixel block coding sequence, W k , W q , W v are query weight, key weight and value weight respectively; K (t) ,Q (t) ,V (t) They are the block query matrix, block key matrix and block value matrix corresponding to the pixel block at time t respectively.

[0110] Step S33, performing cross-frame operations in each attention matrix to obtain a cross-frame attention head for each attention module;

[0111] Step S34, concatenate the cross-frame attention heads to obtain cross-frame motion features.

[0112] It should be noted that each video frame in the cross-frame attention head may contain the block value matrix and block key matrix of the pixel blocks in the adjacent video frame, and the cross-frame motion feature may be obtained by splicing multiple cross-frame attention heads. The cross-frame motion feature may include 8 cross-frame attention heads. Adjacent video frames may include the previous video frame and / or the next video frame. The previous video frame is the video frame of the previous time step of the current video frame, and the next video frame is the video frame of the next time step of the current video frame. The current video frame may be any corresponding video frame in the attention matrix.

[0113] Exemplarily, the attention matrix is ​​operated across frames to obtain the adjacent block key matrix and adjacent block value matrix of the adjacent video frame of the video frame, so that the video frame can also contain the adjacent block key matrix and adjacent block value matrix in the adjacent video frame, so that the video frame contains information in other video frames, thereby facilitating the video recognition model to understand the entire pixel block coding sequence. For example, it can facilitate the classifier in the video recognition model to understand the entire pixel block coding sequence, thereby facilitating more accurate video action classification.

[0114] In a feasible embodiment, step S33 further includes steps S331 to S332:

[0115] Step S331, determining a target shift matrix in each attention matrix, where the target shift matrix is ​​any attention matrix;

[0116] It should be noted that the target shift matrix can be divided into a target forward shift matrix and a target backward shift matrix. The target forward shift matrix refers to the video frame in the target forward shift matrix, and the front block key matrix and the front value key matrix of the front video frame are obtained from each attention matrix. The target backward shift matrix refers to the video frame in the target backward shift matrix, and the rear block key matrix and the rear block value matrix of the rear video frame are obtained from each attention matrix. There may be one or more target forward shift matrices and target backward shift matrices in each attention matrix, and there may also be an attention matrix that is not shifted, and the cross-frame attention head of the attention module where the shifted attention matrix is ​​located is not the attention matrix itself. The target forward shift matrix and the target backward shift matrix can be determined in the attention matrix. For example, any matrix can be determined in the attention matrix as a target forward shift matrix or a target backward shift matrix, and the target forward shift matrix and the target backward shift matrix are not the same. In other embodiments, the target forward shift matrix and the target backward shift matrix can also be determined based on the sorting position of the attention module in the cross attention module. For example, the attention matrix of the attention module ranked first can be determined as the target forward shift matrix, and the attention matrix of the attention module ranked third can be determined as the target backward shift matrix, etc. The above is only one example, and this embodiment does not make specific limitations on this. In other embodiments, the target shift matrix can also be a target front-back matrix, and the target front-back matrix refers to the video frame in the target front-back matrix. The front block key matrix and the front value key matrix of the front video frame are obtained from each attention matrix, and the rear block key matrix and the rear block value key matrix of the rear video frame are obtained. The target front-back matrix can also be determined based on actual conditions.

[0117] Step S332, for each pixel block in the target shift matrix belonging to the same video frame, obtain the adjacent block key matrix and adjacent block value matrix of each adjacent pixel block in the adjacent video frame in each attention matrix;

[0118] Step S333, concatenate each block query matrix in the target shift matrix and the adjacent block value matrix and adjacent block key matrix corresponding to the target shift matrix to obtain the cross-frame attention head of the attention module where the target shift matrix is ​​located.

[0119] It should be noted that each video frame can include multiple pixel blocks, and the pixel image features of the pixel blocks of each video frame can be corresponding in the pixel block coding sequence. The adjacent block value matrix is ​​the value matrix of the pixel blocks in the adjacent video frames, and the adjacent block key matrix is ​​the key matrix of the pixel blocks in the adjacent video frames. The adjacent key matrix can also include the front block key matrix and / or the rear block key matrix, and the adjacent block value matrix can also include the front block value matrix and / or the rear block value matrix. The adjacent pixel blocks refer to the pixel blocks in the adjacent pixel frames. The adjacent pixel blocks include the front pixel blocks and / or the rear pixel blocks. The front block key matrix is ​​the block key matrix of the front pixel blocks of the front video frame, the rear block key matrix is ​​the block key matrix of the rear pixel blocks of the rear video frame, the front block value matrix is ​​the block value matrix of the front pixel blocks of the front video frame, and the rear block value matrix is ​​the block value matrix of the rear pixel blocks of the rear video frame.

[0120] In one example, for any target shift matrix, for each identical video frame corresponding to the target shift matrix, the adjacent block key matrix and adjacent block value matrix of each adjacent pixel block in the adjacent video frame are obtained in each attention matrix, the adjacent block key matrix and adjacent block value matrix of the adjacent pixel block are concatenated with the block query matrix of the corresponding pixel block in the target shift matrix to obtain the adjacent block matrix of the corresponding pixel block in the target shift matrix, and each adjacent block matrix is ​​used as the cross-frame attention head of the attention module where the target shift matrix is ​​located.

[0121] Specifically, when the target shift matrix is ​​a target forward shift matrix, for each identical video frame corresponding to the target forward shift matrix, the forward block key matrix and the forward block value matrix of each forward pixel block in the previous video frame are obtained in each attention matrix, and the forward block key matrix and the forward block value matrix of the forward pixel block are spliced ​​with the block query matrix of the corresponding pixel block in the target shift matrix to obtain the forward block matrix of the corresponding pixel block in the target forward shift matrix, and each forward block matrix is ​​spliced ​​to obtain the cross-frame attention head of the attention module where the target forward shift matrix is ​​located. The target forward shift matrix may include multiple forward block matrices. The forward block matrix includes the block query matrix of the pixel block in the target forward shift matrix and the corresponding forward block key matrix and the forward block value matrix.

[0122] When the target shift matrix is ​​a target back-shift matrix, for each identical video frame corresponding to the target back-shift matrix, the back-block key matrix and the back-block value matrix of each back-pixel block in the back video frame are obtained in each attention matrix, and the back-block key matrix and the back-block value matrix of the back-pixel block are spliced ​​with the block query matrix of the corresponding pixel block in the target shift matrix to obtain the back-shift block matrix of the corresponding pixel block in the target back-shift matrix, and each back-shift block matrix is ​​spliced ​​to obtain the cross-frame attention head of the attention module where the target back-shift matrix is ​​located. The target back-shift matrix may include multiple back-shift block matrices. The back-shift block matrix includes the block query matrix of the pixel block in the target back-shift matrix and the corresponding back-block key matrix and back-block value matrix.

[0123] When the target shift matrix is ​​the target front-back matrix, for each same video frame corresponding to the target front-back matrix, the rear block key matrix and rear block value matrix of each rear pixel block in the rear video frame are obtained in each attention matrix, and the front block key matrix and front block value matrix of each front pixel block in the front video frame are obtained in each attention matrix; the rear block key matrix and rear block value matrix of the rear pixel block are spliced ​​with the block query matrix of the corresponding pixel block in the target shift matrix to obtain the rear block matrix of the corresponding pixel block in the target rear shift matrix, the front block key matrix and front block value matrix of the front pixel block are spliced ​​with the block query matrix of the corresponding pixel block in the target shift matrix to obtain the front block matrix of the corresponding pixel block in the target front shift matrix, and each rear shift block matrix is ​​spliced ​​with each front shift block matrix to obtain the cross-frame attention head of the attention module where the target front-back matrix is ​​located. The target rear shift matrix may include multiple rear shift block matrices. The rear shift block matrix includes the block query matrix of the pixel block in the target rear shift matrix and the corresponding rear block key matrix and rear block value matrix.

[0124] For example, the cross-frame attention head can be expressed as:

[0125]

[0126] in, is the cross-frame attention head of the ith attention module, t can be the tth video frame or the time t, and the video frame corresponding to the time t is the tth video frame. t-1 is the t-1th frame, which can be considered as the previous video frame of the tth video frame, t+1 is the t+1th frame, which can be considered as the next video frame of the tth video frame, and h b is the number of the first attention matrix to be moved forward, for example, h b It can be the smallest number among the target forward matrices, h f is the number of the first attention matrix to be shifted back, for example, h f It can be the largest number in each target backward shift matrix. h is the total number of attention modules. The numbers of the target forward shift matrix and the target backward shift matrix can be the same as the numbers of their respective attention modules. The numbers of each attention module can be customized. The numbers between attention modules are different but continuous. b +h f is the number of the intermediate matrix between the first attention matrix that needs to be moved forward and the first attention matrix that needs to be moved backward. The intermediate matrix is ​​also one of the attention matrices. For example, Denoted as the block query matrix of the pixel block in the t video frame in the i-th cross-frame attention head, is the block key matrix of the pixel block in the t video frame in the i-th cross-frame attention head, is the block value matrix of the pixel block in the t video frame in the i-th cross-frame attention head, is the block key matrix of the pixel block in the t+1 video frame in the i-th cross-frame attention head, It can be the block key matrix of the pixel block in the t+1 video frame obtained from other attention matrices, is the block key matrix of the pixel block in the t-1 video frame in the i-th cross-frame attention head, It can be the block key matrix of the pixel block in the t-1 video frame obtained from other attention matrices; is the block value matrix of the pixel block in the t+1 video frame in the i-th cross-frame attention head, It can be the block value matrix of the pixel block in the t+1 video frame obtained from other attention matrices, is the block value matrix of the pixel block in the t-1 video frame in the i-th cross-frame attention head, which can be the block value matrix of the pixel block in the t-1 video frame obtained from other attention matrices. Expressed as and The attention between Expressed as and The attention between Expressed as and Attention between. can be the representation of all forward block matrices of t video frames in the i-th cross-frame attention head, can be the representation of all the backward block matrices of t video frames in the i-th cross-frame attention head, It can be the attention matrix of the i-th cross-frame attention head itself.

[0127] in, is the calculation formula for attention, Q is the query matrix, K is the key matrix, D is the dimension of the pixel block encoding sequence, and softmax is the activation function. and You can substitute it into the attention calculation formula to calculate and It can also be substituted into the attention calculation formula to get and It can also be substituted into the attention calculation formula to get

[0128] For a better understanding of this embodiment, please refer to Figure 3 , briefly describe the process of this embodiment: wherein, Figure 3There are video frames 1 to 8 in the video, a total of 8 video frames. These 8 video frames can be encoded, that is, the position enhancement matrix of video frames 1 to 8 is determined. The position enhancement matrix can be input into Norm1. Norm1 indicates that the position enhancement matrix is ​​normalized to obtain a normalized position matrix. The pixel block encoding sequence of the normalized position matrix can be input into MSCA. MSCA outputs cross-frame motion features. The position enhancement matrix can also skip the MSCA layer and be directly combined with the output of MSCA and input into Norm2 together. Norm2 indicates that the cross-frame motion features and the position encoding matrix are normalized and then output to MLP. MLP outputs the action classification result. The cross-frame motion features can also be output together with the action classification results. For example, the output result can include cross-frame motion features and action classification results. Figure 3 The symbol C1 in the figure indicates that the residual structure is used in the prediction process of the video recognition model to improve the accuracy of video recognition. Figure 3 It also specifically shows the cross-frame operations in MSCA. For example, Q, K and V corresponding to time t-1, time t and time t+1 are respectively shown. Q, K and V are block query matrix, block key matrix and block value matrix respectively. For example, at time t, the block key matrices corresponding to time t-1 and time t+1 can be obtained, and K' containing the block key matrices at time t, time t-1 and time t+1 can be obtained. At time t, the block value matrices corresponding to time t-1 and time t+1 can be obtained, and V' containing the block key matrices at time t, time t-1 and time t+1 can be obtained. The symbol referred to by C1 can represent the aggregation operation. For example, the aggregation operation between the block query matrices Q and K' at time t can be to calculate the attention between Q and K' at time t, and then aggregate the attention with V'. For example, the attention can be multiplied by V', and then the cross-frame attention head corresponding to time t is obtained. Figure 3 Here, L1 refers to the block key matrix and block value matrix corresponding to time t-1 and time t+1 respectively shifted to time t, and L2 refers to the block key matrix and block value matrix corresponding to time t can be shifted to time t+1 and time t-1. Figure 3 The cross-frame operation shown in is just one example, and this embodiment does not specifically limit this.

[0129] Combination Figure 3 , the formula for determining the action classification result of the video to be identified based on the cross attention module and the classifier can be:

[0130] z′ l =MSCA(LN(z l-1 ))+z l-1

[0131] z l =MLP(LN(z′l ))+z′ l

[0132] Among them, MSCA is the cross attention module, MLP is the classifier, LN is the normalization layer, LN(z l-1 ) is expressed as z l-1 Normalization is performed, LN(z′ l ) is expressed as z′ l Normalize it, z l-1 is the content input to the cross-attention module, for example, it can be a pixel block encoding sequence, etc., z′ l It can be the output of the cross-attention module or the input to the MLP classifier. l It may be the output of the encoding module where the MLP is located, for example, it may be a motion classification result.

[0133] This embodiment achieves effective exchange between adjacent video frames by moving feature channels in the time dimension. This strategy can enhance the video recognition model's ability to capture temporal dynamic features without increasing the amount of additional computation. This time shift operation does not require additional computation because it only involves the rearrangement of features between frames, rather than complex spatiotemporal convolution calculations, avoiding the high computational cost and memory usage problems that traditional convolution may encounter when processing videos, thereby reducing computational costs.

[0134] And in this embodiment, since MSCA can directly interact in the time dimension on the feature map (exchange of block key matrix and block value matrix between adjacent video frames), additional shift operations are avoided, which can reduce the amount of calculation and the number of parameters, thereby reducing the complexity of the video recognition model. And the attention mechanism involved in MSCA allows the model to perform effective information interaction between different time steps (that is, the exchange of block key matrix and block value matrix between adjacent video frames), which helps the model capture long-term dependencies in the pixel block encoding sequence, thereby improving the performance of the video recognition model in processing videos. Traditional shift operations, such as TokenShift, may introduce additional computing and memory overhead. MSCA avoids this overhead through the attention mechanism, making the video recognition model more efficient.

[0135] In a feasible embodiment, the video recognition method further includes steps X10 to X40:

[0136] Step X10, obtaining a training sample set, the training sample set includes multiple training samples, each training sample includes a video to be trained and a classification label;

[0137] Step X20, inputting the video to be trained into a preset model to be trained to obtain an action recognition result, wherein the model to be trained includes a cross-attention module to be trained and a classifier to be trained;

[0138] It should be noted that the model to be trained can be a Transformer model, which includes a cross-attention module to be trained and a classifier to be trained. The training sample set can include multiple training samples, and the training samples include videos to be trained and corresponding classification labels. The classification label is the video action type of the video to be trained. For example, the classification label can be a body movement label, a human interaction label, etc. This embodiment does not make specific restrictions on this, and the classification label can be determined based on actual conditions. The action recognition result is the action recognition result of the video to be trained recognized by the model to be trained.

[0139] The training position enhancement matrix can also be to extract multiple training video frames from the video to be trained, determine the training pixel matrix of each training video frame, and perform position encoding on each training pixel matrix to obtain the position enhancement matrix corresponding to each training pixel. Wherein, in the training stage, multiple training video frames can be horizontally flipped and cropped. For example, the process of determining the training pixel matrix of the training video frame can be the same as the process of determining the pixel matrix of the video frame. For example, the description of step S11 to step S13 can be referred to, and this embodiment will not be repeated here. In the training process, the training video frame is also horizontally flipped, which can increase the diversity of training data, so that the model to be trained can be exposed to more visual patterns, thereby improving the generalization ability of the model to be trained, and in practical applications, the video content may be horizontally flipped due to changes in viewing angles (for example, surveillance cameras are installed in different directions). By horizontally flipping the frame, the model to be trained can learn the representation of objects in different directions and improve the adaptability to such changes. And by horizontally flipping the frame, the size and complexity of the training sample can be increased, which helps to reduce the overfitting of the training sample by the model to be trained.

[0140] At the same time, the short side size of the training video frame can be randomly adjusted while ensuring that the aspect ratio of the training video frame remains unchanged, so as to crop the short side of the training video frame, thereby ensuring that the cropped image size is large enough to contain enough information for effective feature extraction. The size of the training pixel matrix can also be 224x224, which is a balance point that maintains that the image includes enough information while reducing the amount of calculation of the model to be trained, so that the model to be trained can be processed more efficiently.

[0141] Step X30, calculating the training loss of the model to be trained based on the action recognition result and the classification label;

[0142] Step X40, optimizing the model to be trained according to the training loss to obtain a trained video action recognition model, wherein the video action recognition model includes a cross-attention module and a classifier.

[0143] It should be noted that the training loss can be calculated by a preset loss function, which can be a cross entropy function. The training loss can be used to describe the difference between the action recognition result and the corresponding classification label. The larger the training loss, the greater the difference between the action recognition result and the corresponding classification label. The smaller the training loss, the smaller the difference between the action recognition result and the corresponding classification label.

[0144] For example, the preset loss function can be expressed as:

[0145]

[0146] Where loss is the training loss of the model to be trained, n is the number of training samples input to the model to be trained, m is the number of human motion categories, and each human motion category can be multiple, such as body movement, interaction between people, etc.; i represents the number of the training sample, i is the i-th training sample, j is the number of the human motion category, j is the j-th human motion category, p(x ij ) is the classification label, q(x ij ) represents the action recognition result; specifically, the classification label p(x ij ) can be expressed as the probability that the i-th sample actually belongs to the j-th class. The action recognition result q(x ij ) can represent the probability that the i-th sample predicted by the training model belongs to the j-th category.

[0147] Exemplarily, a training sample is obtained from a training sample set, a training position enhancement matrix of a video to be trained in the training sample is determined, the training position enhancement matrix is ​​input into a preset model to be trained, and the action recognition result of the training sample is obtained. The training loss of the model to be trained is calculated according to a preset loss function, the action recognition result and the classification label. If the training loss is less than or equal to a preset loss threshold, the training of the model to be trained is completed, and a video action recognition model is obtained. If the training loss is greater than the preset loss threshold, the training sample is obtained again from the training sample set, and the training position enhancement matrix of the video to be trained in the training sample is determined, and the training position enhancement matrix is ​​input into the preset model to be trained until the training loss is less than or equal to the preset loss threshold. This embodiment calculates the training loss of the model to be trained by a preset loss function, thereby facilitating the accuracy of the video action recognition model.

[0148] In a feasible embodiment, after step X40, steps X41 to X46 are also included:

[0149] Step X41, determining the result type of the action recognition result corresponding to each training sample;

[0150] Step X42, counting the number of first positive class samples whose sample type is the training positive class and whose corresponding result type is the positive class result, and counting the number of second positive class samples whose sample type is the training positive class and whose corresponding result type is the negative class result;

[0151] Step X43, counting the number of first negative classes whose sample types are training negative classes and whose corresponding result types are negative class results, and counting the number of second negative classes whose sample types are training negative classes and whose corresponding result types are positive class results;

[0152] It should be noted that each training sample has its own corresponding sample type, and the sample type can be divided into training positive class and training negative class. The training positive class is characterized by the training sample being a positive sample, and the training negative class is characterized by the training sample being a negative sample. The positive sample represents that there is a specific action recognition in the video to be trained in the training sample, for example, there is human body movement, or interaction between people in the video to be trained. The negative sample represents that there is no specific action recognition in the video to be trained in the training sample, for example, there is no human being shown in the video to be trained, and there is no body movement, interaction between people, etc. The sample type to which each training sample belongs can be set based on actual conditions.

[0153] Each action recognition result may also have its own corresponding result type, which may be divided into a positive result class and a negative result class. The positive result class is characterized by the difference between the action recognition result and the classification label of the corresponding training sample being less than or equal to a preset difference threshold, and the negative result class is characterized by the difference between the action recognition result and the classification label of the corresponding training sample being greater than a preset difference threshold.

[0154] When the sample type of the training sample is a training positive class, and the result type corresponding to the training sample is a positive result, it means that the training sample is a true positive example. For example, the training model predicts that the action recognition result of the training sample is body movement, and the classification label corresponding to the training sample is also body movement. The number of the first positive class is the number of true positive examples in the training process of the training model.

[0155] When the sample type of the training sample is a training positive class, and the result type corresponding to the training sample is a negative class result, it means that the training sample is a false positive example. For example, if the difference between the action recognition result of the training sample of the training positive class predicted by the training model and the classification label of the training sample is greater than the preset difference threshold, it means that the training sample is a false positive example. For example, the training model predicts that the action recognition result of the training sample of the training positive class is empty (predicting that there is no human movement in the training video in the training sample), and the classification label of the training sample is human movement, or, there is human movement a in the training video in the training sample, but the classification label of the training sample is b, so at this time, the training model to be trained has a prediction error for the training sample of the training positive class. The first positive class number is the number of false positive examples during the training process of the model to be trained.

[0156] When the sample type is a training negative class, and the result type corresponding to the training sample is the number of first negative classes of negative class results, it means that the training sample is a true negative example. For example, the training model predicts that the action recognition result of the training sample of the training negative class is empty (it predicts that there is no human motion in the training video in the training sample), and the classification label of the training sample also indicates no human motion. The number of first negative classes is the number of true negative examples during the training process of the training model.

[0157] When the sample type is a training negative class, and the result type corresponding to the training sample is the number of the second negative class of the positive class result, it means that the training sample is a false negative example. For example, the training model predicts that the action recognition result of the training sample of the training negative class indicates that there is human motion in the training video of the training sample, while the classification label of the training sample indicates that there is no human motion. At this time, the training model predicts that there is an error in the training sample of the training negative class. The number of the second negative class is the number of false negative examples in the training process of the training model.

[0158] Step X44, calculating the prediction accuracy of the video action recognition model according to the number of the first positive class and the number of the second negative class;

[0159] In this embodiment, the prediction accuracy can be used to evaluate the accuracy of the video recognition model in predicting positive video data, where the positive video data is a video containing human motion.

[0160] Exemplarily, the formula for calculating the prediction accuracy may be:

[0161]

[0162] Among them, Precision is the prediction accuracy, TP is the number of the first positive class, and FP is the number of the second negative class. The higher the Precision, the higher the accuracy of the video recognition model in predicting positive video data.

[0163] Step X45, calculating the recall rate of the video action recognition model according to the number of the first positive class and the number of the second positive class;

[0164] In this embodiment, the recall rate can be used to evaluate the missed recognition rate of the video recognition model for positive class video data. Exemplarily, the formula for calculating the recall rate can be:

[0165]

[0166] Among them, Recall is the recall rate, FN is the number of second positive classes, the higher the Recall, the lower the missed recognition rate, and the lower the Recall, the higher the missed recognition rate.

[0167] Step X46, calculating the specificity of the video action recognition model according to the number of first negative classes and the number of second negative classes.

[0168] In this embodiment, the specificity can be used to evaluate the misrecognition rate of the video recognition model for negative video data. For example, the formula for calculating the specificity can be:

[0169]

[0170] Among them, Specificity is specificity. The higher the specificity, the lower the false recognition rate, and the lower the specificity, the higher the false recognition rate.

[0171] This embodiment calculates the prediction accuracy, recall rate, and specificity of the video recognition model through the number of first positive classes, the number of second positive classes, the number of first negative classes, and the number of second negative classes, so as to comprehensively evaluate the performance of the video recognition model and facilitate subsequent further optimization of the video recognition model.

[0172] The present application also provides a video recognition device. Figure 4 , the device comprises:

[0173] An acquisition module 10 is used to acquire multiple video frames in the video to be identified and determine the pixel matrix of each video frame;

[0174] A position enhancement module 20 is used to perform position encoding on each of the pixel matrices to obtain a position enhancement matrix corresponding to each of the pixel matrices;

[0175] A cross-frame module 30, configured to determine cross-frame motion features from the position enhancement matrix through a preset cross-attention module;

[0176] The classification module 40 is used to classify the cross-frame motion features into video actions using a preset classifier to obtain an action classification result of the video to be identified.

[0177] The video recognition device provided in the embodiment of the present application adopts the video recognition method in the above embodiment, aiming to solve the technical problem of high computational cost of video action recognition. Compared with the prior art, the beneficial effects of the video recognition method provided in the embodiment of the present application are the same as those of the video recognition method provided in the above embodiment, and other technical features in the video recognition device are the same as those disclosed in the above embodiment method, which will not be described in detail here.

[0178] An embodiment of the present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video recognition method in the above-mentioned embodiment.

[0179] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0180] like Figure 5 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM) 1004. Various programs and data required for the operation of the electronic device are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus.

[0181] Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0182] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1009, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0183] The electronic device provided in the embodiment of the present application adopts the video recognition method in the above embodiment 1 to solve the technical problem of high computational cost of video action recognition. Compared with the prior art, the beneficial effect of product flow data distribution provided in the embodiment of the present application is the same as the beneficial effect of the video recognition method provided in the above embodiment, and the other technical features in the video recognition device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0184] It should be understood that the various parts of the present disclosure can be implemented with hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0185] The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be based on the protection scope of the claims.

[0186] This embodiment provides a computer-readable storage medium having computer-readable program instructions stored thereon, and the computer-readable program instructions are used to execute the video recognition method in the above-mentioned embodiment 1.

[0187] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, equipment or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable EPROM (Electrical Programmable Read Only Memory, read-only memory) or flash memory, an optical fiber, a portable compact disk CD-ROM (compact disc read-only memory, read-only memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution device, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency, radio frequency), etc., or any suitable combination of the above.

[0188] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0189] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: obtains multiple video frames in the video to be identified, and determines the pixel matrix of each video frame; performs position encoding on each pixel matrix to obtain a position enhancement matrix corresponding to each pixel matrix; determines cross-frame motion features from the position enhancement matrix through a preset cross-attention module; and performs video action classification on the cross-frame motion features through a preset classifier to obtain an action classification result of the video to be identified.

[0190] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a LAN (local area network) or WAN (Wide Area Network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0191] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0192] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0193] The computer-readable storage medium provided in the embodiment of the present application stores computer-readable program instructions for executing the above-mentioned video recognition method, aiming to solve the technical problem of high computational cost of video action recognition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the video recognition method provided in the above-mentioned embodiment, and will not be described in detail here.

[0194] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned video recognition method when executed by a processor.

[0195] The computer program product provided in the embodiment of the present application is intended to solve the technical problem of high computational cost of video action recognition. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of the present application are the same as the beneficial effects of the video recognition method provided in the above embodiment, which will not be repeated here.

[0196] The above are only preferred embodiments of the embodiments of the present application, and are not intended to limit the patent scope of the embodiments of the present application. Any equivalent structure or equivalent process transformation made using the description and drawings of the embodiments of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent processing scope of the embodiments of the present application.

Claims

1. A video recognition method, characterized in that: The method includes: Acquire multiple video frames in the video to be identified, and determine the pixel matrix of each video frame; Performing position encoding on each of the pixel matrices to obtain a position enhancement matrix corresponding to each of the pixel matrices; Determining cross-frame motion features from the position enhancement matrix through a preset cross-attention module; The cross-frame motion features are subjected to video action classification by a preset classifier to obtain an action classification result of the video to be identified.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of video frames in the video to be identified and determining a pixel matrix of each video frame comprises: Acquire the video to be identified, and extract a preset first number of video frames from the video to be identified; Based on a preset flip probability, determining a preset second number of video frames to be flipped from each of the video frames, and horizontally flipping each of the video frames to be flipped to obtain flipped video frames; Based on a preset video image size, each flipped video frame and a non-flipped video frame in the video frames are cropped to obtain each pixel matrix.

3. The method according to claim 1, characterized in that The step of performing position encoding on each of the pixel matrices to obtain a position enhancement matrix corresponding to each of the pixel matrices comprises: Based on a preset area division unit, each of the pixel matrices is divided to obtain a pixel area corresponding to each of the pixel matrices; According to a preset embedding matrix, embedding operations are performed on each of the pixel regions respectively to obtain a pixel embedding matrix of each of the pixel regions; The pixel embedding matrices, the preset pixel global matrix and the preset position encoding matrix are spliced ​​to obtain a position enhancement matrix.

4. The method according to claim 1, characterized in that The step of determining the cross-frame motion features from the position enhancement matrix by using a preset cross-attention module comprises: Normalizing the position enhancement matrix to obtain a normalized position matrix, and determining a pixel block coding sequence of the normalized position matrix; Determine, according to the cross attention module and the pixel block coding sequence, an attention matrix of each attention module of the pixel block coding sequence in the cross attention module; Perform cross-frame operations in each of the attention matrices to obtain a cross-frame attention head for each of the attention modules; The cross-frame attention heads are concatenated to obtain cross-frame motion features.

5. The method according to claim 4, characterized in that The attention matrix includes a block attention matrix of a pixel block; and the step of determining, according to the cross attention module and the pixel block coding sequence, the attention matrix of each attention module of the pixel block coding sequence in the cross attention module includes: The pixel block encoding sequence is input into the cross-attention module. For each of the attention modules in the cross-attention module, a block attention matrix of each pixel block in the pixel block encoding sequence is calculated according to the attention weight of the attention module. The block attention matrix includes a block query matrix, a block key matrix and a block value matrix.

6. The method according to claim 4, characterized in that The step of performing cross-frame operations in each of the attention matrices to obtain the cross-frame attention heads of each of the attention modules comprises: Determining a target shift matrix in each of the attention matrices, the target shift matrix being any of the attention matrices; For each pixel block in the target shift matrix belonging to the same video frame, obtaining, in each attention matrix, respective adjacent block key matrices and adjacent block value matrices of adjacent pixel blocks in adjacent video frames; Each block query matrix in the target shift matrix and the adjacent block value matrix and adjacent block key matrix corresponding to the target shift matrix are spliced ​​to obtain a cross-frame attention head of the attention module where the target shift matrix is ​​located.

7. The method according to claim 1, characterized in that The method includes: Obtaining a training sample set, wherein the training sample set includes a plurality of training samples, each training sample includes a to-be-trained video and a classification label; Inputting the video to be trained into a preset model to be trained to obtain an action recognition result, wherein the model to be trained includes a cross-attention module to be trained and a classifier to be trained; Calculating the training loss of the model to be trained based on the action recognition result and the classification label; The model to be trained is optimized according to the training loss to obtain a trained video action recognition model, wherein the video action recognition model includes a cross-attention module and a classifier.

8. The method according to claim 7, characterized in that After the step of optimizing the model to be trained according to the training loss to obtain a trained video action recognition model, the method further includes: Determine the result type of the action recognition result corresponding to each of the training samples; Counting the number of first positive class samples whose sample types are training positive class and whose corresponding result types are positive class results, and counting the number of second positive class samples whose sample types are training positive class and whose corresponding result types are negative class results; Counting the number of first negative classes whose sample types are training negative classes and whose result types corresponding to the training samples are negative class results, and counting the number of second negative classes whose sample types are training negative classes and whose result types corresponding to the training samples are positive class results; Calculating the prediction accuracy of the video action recognition model according to the first positive class number and the second negative class number; Calculating the recall rate of the video action recognition model according to the number of the first positive classes and the number of the second positive classes; The specificity of the video action recognition model is calculated according to the number of the first negative classes and the number of the second negative classes.

9. A video recognition device, characterized in that: The device comprises: An acquisition module is used to acquire multiple video frames in the video to be identified and determine the pixel matrix of each video frame; A position enhancement module, used for performing position encoding on each of the pixel matrices to obtain a position enhancement matrix corresponding to each of the pixel matrices; A cross-frame module, used for determining cross-frame motion features from the position enhancement matrix through a preset cross-attention module; The classification module is used to classify the cross-frame motion features into video actions through a preset classifier to obtain the action classification result of the video to be identified.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the video recognition method described in any one of claims 1 to 8.

11. A medium, characterized in that The medium is a computer-readable storage medium, on which is stored a program for implementing the video recognition method, and the program for implementing the video recognition method is executed by a processor to implement the steps of the video recognition method as claimed in any one of claims 1 to 8.

12. A program product, characterized in that The program product is a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the video recognition method according to any one of claims 1 to 8 are implemented.

Images