A multi-cooking action recognition method based on action collaborative analysis

Abstract

The present invention provides a method for recognizing multiple cooking actions based on action collaborative analysis. The method first pre-processes the input video, and then uses a neural network to recognize the cooking actions in the video; the neural network includes a feature extraction branch and a relationship extraction branch, the feature extraction branch extracts the feature vector V1 of the image, the relationship extraction branch extracts the relationship weight matrix W between each action category, V1 is matrix multiplied with W to obtain a fusion vector V2, and each element of the fusion vector V2 is input into a sigmoid activation function to obtain the final output V3 of the neural network, and the value of each element in V3 is between 0 and 1, indicating the probability of each action occurring. Through the scheme of the present invention, the accuracy of simultaneous detection of multiple actions is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for recognizing multiple cooking actions based on action collaborative analysis. Background Art

[0002] Use deep learning models (including convolutional neural networks (CNN) and self-attention models Transformer) to perform action recognition on video data. Classification schemes are usually used. The last two layers of the model are usually fully connected layers and softmax layers. The model outputs the probability value of the sample belonging to each category. Finally, the category corresponding to the maximum probability is selected as the final classification result of the sample. In order to avoid interference from invalid actions, the validity of the probability value is usually judged. It is usually believed that when the probability value of a certain index position is greater than a preset threshold, the action corresponding to the index position is valid.

[0003] The probability output after the softmax layer processing in the prior art is suitable for video samples containing only one action. When two actions appear in the video, the probability values ​​of the index positions corresponding to the two actions are similar. Since the sum of the probability values ​​corresponding to each category is 1, the probability values ​​corresponding to the two index positions are both low and cannot reach the preset threshold, so no valid results can be output. For actions that occur simultaneously, some technicians have proposed using the sigmoid function instead of softmax so that the indexes corresponding to each category do not affect each other. However, this method still does not use the connections between the actions, such as the temporal sequence and the probability of simultaneous occurrence, to further optimize the action recognition results. Summary of the invention

[0004] In response to the above-mentioned problems existing in video action recognition, the present invention proposes a method for recognizing multiple cooking actions based on action collaborative analysis. The method utilizes prior knowledge of the characteristics between action categories, such as the sequence of actions, the probability of two actions occurring at the same time, etc., to construct a neural network that can detect multiple cooking actions simultaneously, thereby forming a method for recognizing multiple cooking actions based on action collaborative analysis.

[0005] A method for recognizing multiple cooking actions based on action collaborative analysis, the method comprising the following steps:

[0006] Step S1, preprocessing the input video, including center cropping, downsampling, and random brightness processing;

[0007] Step S2: using a neural network to identify cooking actions in the video, the neural network includes a feature extraction branch and a relationship extraction branch, specifically:

[0008] Step S21, the feature extraction branch uses a 3D convolutional neural network or a visual self-attention model VisionTransformer network (ViT network) to extract feature information in the spatial and temporal domains, and outputs a 1*D-dimensional feature vector V1, where D is the implicit data dimension;

[0009] Step S22, the relationship extraction branch uses a graph neural network to extract the relationship between each action, where the relationship includes the temporal sequence or the simultaneity of two actions, and outputs a D*C-dimensional relationship weight matrix W, where C is the number of action categories, and the relationship weight matrix represents the mutual relationship between each action in the action set;

[0010] Step S23, perform matrix multiplication on V1 and W to obtain a vector V2 with a dimension of 1xC, and input each element of the vector into the sigmoid function to obtain a vector V3 with a dimension of 1xC. V3 is the final output. The value of each element in V3 is between 0 and 1, indicating the probability of each action occurring.

[0011] Step S24, compare the value of each element in V3 with the preset threshold T. When the value of an element is greater than T, that is, the probability value of the index position where the element is located is greater than T, it means that the corresponding action is detected;

[0012] Step S25: output all detected actions.

[0013] Furthermore, the action set includes seven operations: lifting the lid, covering the lid, stir-frying, adding ingredients, removing from the pot, turning off the knob, and turning on the knob.

[0014] Furthermore, an adjacency matrix Ma is constructed to represent the relationship between various action categories. The dimension of Ma is 7*7. The element value indexed as [i,j] in Ma represents the probability that the jth action occurs at the same time as the i-th action occurs, that is, the conditional probability P (action[j]=True|action[i]=True), where action represents an action and True represents that the condition is true, that is, the action occurs.

[0015] Furthermore, we use a small ViT network with multi-head pooling attention mechanism MHPA and phased expansion of channel dimension, namely ViT-s network, to extract features. MHPA reduces the number of tokens involved in attention calculation and reduces the computational cost by pooling tokens. Token refers to a series of vector representations obtained after specific processing of input video data. Let the input sequence be X, and the weights be Two-dimensional linear mapping and pooling operation , we get the Query vector Q, Key vector K, and Value vector V. The calculation formula is: , , , these vectors are used to calculate the attention weights in the multi-head pooling attention mechanism and to generate the final output.

[0016] Furthermore, the Vision Transformer network used for feature extraction includes an input layer, a backbone layer, and an output layer. A multi-scale attention mechanism and a phased expansion of the channel dimension are introduced in the input layer. The input layer inputs a sequence of video frames, represented as T frames, each frame consists of H×W pixels, H represents the height of the image, and W represents the width of the image. Backbone includes input embedding, position encoding, multi-head attention mechanism, fully connected layer, residual connection and layer normalization. A multi-scale attention mechanism is introduced to allow the model to focus on different information at different time scales. Phased expansion of the channel dimension refers to gradually increasing the channel dimension in the feature extraction branch of the ViT-s network, which is achieved by gradually increasing the number of convolution kernels or adding convolution layers.

[0017] Furthermore, the input embedding in Backbone embeds the pixel values ​​of each frame into a vector form to form a sequence; the position encoding layer introduces position encoding to help the model capture the temporal information in the sequence; a multi-head self-attention mechanism is introduced in each Transformer block to allow the model to focus on the relationship between different frames; after the self-attention mechanism, a fully connected layer is used to integrate information at each position; residual connections and layer normalization are added between each Transformer block to improve training stability.

[0018] Furthermore, the relationship extraction uses a graph convolutional neural network GCN, which includes an input layer, a GCN layer, an activation function and normalization, a multi-layer GCN, a multi-head GCN, a residual connection and a graph convolution pooling; the input layer is a 1*D-dimensional feature vector V1 and a D*C-dimensional adjacency matrix, where D is the implicit data dimension and C is the number of action categories; the GCN layer uses a graph convolutional network for relationship extraction, with the input layer as input, and generates an updated node feature matrix, wherein the nodes correspond to different action categories, and the edges of the graph convolutional network correspond to the relationships between different action categories; an activation function is added after the GCN layer, the ReLU function is used as the activation function, and the node features are layer-normalized; multiple GCN layers are stacked to form a multi-layer GCN to increase the expressive power of the model and deep-level relationship extraction; a multi-head GCN mechanism is introduced to allow the model to focus on different action relationship subspaces; residual connections are added between GCN layers to promote information flow and gradient propagation; a graph convolution pooling layer is added after the GCN layer to further reduce the dimension and extract more representative features.

[0019] Furthermore, the graph neural network has a total of 5 layers. The input of the first layer of the graph neural network is X and Ma, where X is the initialized relationship feature vector of each action, and the dimension of X is 7*256. The weight of the first layer of the graph neural network is W1, which is the parameter to be trained, with a dimension of 256*256. The output is H1=σ(Ma·X·W1), where σ is a nonlinear activation function, and the dimension of H1 is 7*256. The input of the second layer of the graph neural network is H1, and the output of the second layer of the graph neural network is obtained using the same method as the first layer of the graph neural network. And so on. The final output of the graph neural network is H5, with a dimension of 7*256.

[0020] Furthermore, H5 is transposed to H5T with a dimension of 256*7, and V1*H5T is calculated to obtain the feature vector V2 with a dimension of 1*7. Each element in the feature vector V2 is input into the sigmoid function to obtain the probability value of each action. When a probability value is greater than 0.7, the action is considered to have occurred, and all the actions that have occurred are output.

[0021] The present invention designs a neural network including a feature extraction branch and a relationship extraction branch, and uses the prior knowledge of the characteristics between the cooking actions to detect the cooking actions, thereby achieving the effect of enhancing the accuracy of simultaneous detection of multiple actions. The present invention has the following beneficial technical effects:

[0022] 1. Multi-action collaborative analysis: By introducing prior knowledge of action collaborative analysis, including the order of actions and the probability of two actions occurring at the same time, the model can better understand the relationship between multiple cooking actions. At the same time, attention weight correction is introduced in the relationship extraction branch to adjust the relationship weights between different action categories according to task requirements. This analysis helps to improve the recognition accuracy of multiple actions occurring at the same time in complex scenarios.

[0023] 2. Combination of ViT and GCN: Using ViT for feature extraction can effectively capture the spatial and temporal features in the video, while the relationship extraction branch uses GCN to learn the relationship between action categories. The combination of the two enables the model to fully explore the temporal relationship and simultaneous occurrence probability of actions, thereby improving the recognition accuracy;

[0024] 3. Multi-head attention mechanism: Introducing a multi-head attention mechanism in the feature extraction branch helps the model better focus on different action relationship subspaces and feature subspaces, enhancing the network's ability to model complex relationships;

[0025] 4. Graph Convolutional Neural Network (GCN): GCN is used in the relationship extraction branch, and the relationship between action categories is learned by using the graph structure, which further improves the modeling effect of multiple action relationships in complex scenes. The introduction of GCN increases the model's ability to understand graph structure information, making relationship extraction more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 is a neural network structure diagram provided by an embodiment of the present invention;

[0028] Figure 2 It is a schematic diagram of multiple actions occurring simultaneously provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The present invention proposes a multi-cooking action recognition method which utilizes prior knowledge of characteristics between action categories, such as the order of actions, the probability of two actions occurring simultaneously, etc., to realize simultaneous detection of multiple cooking actions.

[0031] A method for recognizing multiple cooking actions based on action collaborative analysis, the method comprising the following steps:

[0032] Step S1, preprocessing the input video, including center cropping, downsampling, and random brightness processing;

[0033] Step S2, using a neural network to recognize cooking actions in the video, wherein the neural network includes a feature extraction branch and a relationship extraction branch;

[0034] The structure of the neural network is as follows Figure 1 As shown, specifically including:

[0035] Step S21, the feature extraction branch uses a 3D convolutional neural network or a visual self-attention model vision transformer network (ViT network) to extract feature information in the spatial and temporal domains, and outputs a 1*D-dimensional feature vector V1, where D is the implicit data dimension;

[0036] What is extracted here is the spatial and temporal information of the video clip, specifically, the spatial and temporal information of a continuous video frame. Spatial information refers to the information in a single picture, mainly including the scene information described in the picture. For example, the spatial information of a cooking scene mainly includes the operator position information, pot position information, kitchen scene information, etc. in a single picture; temporal information mainly refers to the action information that occurs in a continuous period of time, such as the movement of the operator's arm, the change of the stove position, the change of the kitchen environment, etc. Using spatial and temporal information at the same time can classify actions more accurately.

[0037] Step S22, the relationship extraction branch uses a graph neural network to extract the relationship between each action, where the relationship includes the temporal sequence or the simultaneity of two actions, and outputs a D*C-dimensional relationship weight matrix W, where C is the number of action categories, and the relationship weight matrix represents the mutual relationship between each action in the action set;

[0038] Step S23, perform matrix multiplication on V1 and W to obtain a vector V2 with a dimension of 1xC, and input each element of the vector into the sigmoid function to obtain a vector V3 with a dimension of 1xC. V3 is the final output. The value of each element in V3 is between 0 and 1, indicating the probability of each action occurring.

[0039] Step S24, compare the value of each element in V3 with the preset threshold T. When the value of an element is greater than T, that is, the probability value of the index position where the element is located is greater than T, it means that the corresponding action is detected;

[0040] Step S25: output all detected actions.

[0041] The video is obtained by shooting cooking scenes with a camera. Specific cooking scenes include cooking, washing vegetables, washing dishes, washing pots, etc. The action set that needs to be detected includes seven operations: lifting the lid, covering the lid, stir-frying, adding ingredients, taking out of the pot, closing the knob, and opening the knob. The video sample is a 16-frame segment captured from the video. These 16 frames contain a complete action information, such as stir-frying, adding ingredients, etc. The image size is 224*224 pixels, divided into three RGB channels, and the hidden layer dimension is 256.

[0042] Construct an adjacency matrix Ma to represent the relationship between action categories. The dimension of Ma is 7*7. The element value indexed as [i,j] in Ma represents the probability that the jth action occurs at the same time as the i-th action occurs, that is, the conditional probability P (action[j]=True|action[i]=True), where action represents an action and True represents that the condition is true, that is, the action occurs. Figure 2 The figure shows a schematic diagram of multiple actions occurring simultaneously.

[0043] Furthermore, a small ViT network with multi-head pooling attention mechanism MHPA and phased expansion of channel dimension, namely ViT-s network, is used for feature extraction. MHPA reduces the number of tokens involved in attention calculation and reduces the computational cost by pooling tokens. Token refers to a series of vector representations obtained after specific processing of input video data. Let the input sequence be X, which is the preprocessed video data with weights Two-dimensional linear mapping and pooling operation , we get the Query vector Q, Key vector K, and Value vector V. The calculation formula is: , , , these vectors are used to calculate the attention weights in the multi-head pooling attention mechanism and to generate the final output.

[0044] In the attention mechanism, Query vector (Q), Key vector (K) and Value vector (V) are usually used to calculate attention weights and generate outputs; Query vector is a vector used to compare with the Key vectors of other tokens. In the attention mechanism, each Query vector will perform a dot product operation with all Key vectors to calculate the attention weight of each token. The Query vector determines the shape of the attention distribution, that is, which tokens have a greater impact on the current token; Key vector is used to measure the degree of association between other tokens and the current token. In the attention mechanism, each Key vector performs a dot product operation with the corresponding Query vector to calculate the attention weight. The Key vector determines the similarity between the current token and other tokens; Value vector is a representation of each input token. They are multiplied by the attention weight and summed to obtain the final output. The Value vector determines the contribution of each token in generating the output.

[0045] Vision Transformer is a deep learning model for video understanding based on the Transformer architecture. Unlike traditional 3D convolutional neural networks, Vision Transformer does not usually rely on convolution operations. Instead, it decomposes video sequences into a series of frames or blocks, and then uses these frames or blocks as the input sequence of the Transformer. Each frame or block is encoded as a vector and interacts with the information of other frames through the self-attention mechanism to learn global spatiotemporal dependencies. Vision Transformer is a video feature extractor based on the attention mechanism. It has an encoder-decoder structure, which can extract features from continuous video frames and finally output classification results through the classification layer.

[0046] The Vision Transformer network used for feature extraction includes an input layer, a backbone layer, and an output layer. A multi-scale attention mechanism and a phased expansion of the channel dimension are introduced in the input layer. The input layer inputs a sequence of video frames, represented as T frames, each frame consists of H×W pixels, H represents the height of the image, and W represents the width of the image. The backbone includes input embedding, position encoding, multi-head attention mechanism, fully connected layers, residual connections, and layer normalization. A multi-scale attention mechanism is introduced to allow the model to focus on different information at different time scales. Phased expansion of the channel dimension refers to gradually increasing the channel dimension in the feature extraction branch of the ViT-s network, which is achieved by gradually increasing the number of convolution kernels or adding convolution layers.

[0047] The input embedding in Backbone embeds the pixel values ​​of each frame into a vector form to form a sequence; the position encoding layer introduces position encoding to help the model capture the temporal information in the sequence; a multi-head self-attention mechanism is introduced in each Transformer block to allow the model to focus on the relationship between different frames; after the self-attention mechanism, a fully connected layer is used to integrate information at each position; residual connections and layer normalization are added between each Transformer block to improve training stability.

[0048] The relationship extraction uses a graph convolutional neural network GCN, which includes an input layer, a GCN layer, an activation function and normalization, a multi-layer GCN, a multi-head GCN, a residual connection and a graph convolution pooling; the input layer is a 1*D-dimensional feature vector V1 and a D*C-dimensional adjacency matrix, where D is the implicit data dimension and C is the number of action categories; the GCN layer uses a graph convolutional network for relationship extraction, with the input layer as input, and generates an updated node feature matrix, wherein the nodes correspond to different action categories, and the edges of the graph convolutional network correspond to the relationships between different action categories; an activation function is added after the GCN layer, the ReLU function is used as the activation function, and the node features are layer-normalized; multiple GCN layers are stacked to form a multi-layer GCN to increase the expressive power of the model and deep-level relationship extraction; a multi-head GCN mechanism is introduced to allow the model to focus on different action relationship subspaces; residual connections are added between GCN layers to promote information flow and gradient propagation; a graph convolution pooling layer is added after the GCN layer to further reduce the dimension and extract more representative features.

[0049] Graph neural network is a deep learning model specially designed to process graph structured data. When traditional neural networks cannot directly process non-Euclidean data or unstructured data, graph neural network GNNs came into being. It can effectively process complex graph data containing nodes, edges, and possible attributes. In this invention, action labels are used as nodes, and the relationship between actions is used as edges to form a graph relationship, which is used as the input of the graph neural network.

[0050] The graph neural network in the present invention has a total of 5 layers. The input of the first layer of the graph neural network is X and Ma, where X is the relationship feature vector of each initialized action, and the dimension of X is 7*256. The weight of the first layer of the graph neural network is W1, which is the parameter to be trained, with a dimension of 256*256, and the output H1=σ(Ma·X·W1), where σ is a nonlinear activation function, and the dimension of H1 is 7*256. The input of the second layer of the graph neural network is H1, and the output of the second layer of the graph neural network is obtained using the same method as the first layer of the graph neural network, and so on. The final output of the graph neural network is H5, with a dimension of 7*256.

[0051] Finally, H5 is transposed to H5T with a dimension of 256*7, and V1*H5T is calculated to obtain the feature vector V2 with a dimension of 1*7. Each element in the feature vector V2 is input into the sigmoid function to obtain the probability value of each action. When a probability value is greater than 0.7, the action is considered to have occurred, and all the actions that have occurred are output.

[0052] The present invention is further described below by a specific embodiment:

[0053] 1. The image size of the feature extraction branch input is 16*3*224*224. After feature extraction using 3D convolutional neural network (such as I3D, C3D, etc.) or Video Transformer (such as Video MAE or Video ViT, etc.), the dimension of the feature vector V is 1*256;

[0054] 2. The dataset is obtained by collecting cooking videos and annotating the intervals containing actions. For example, if a video contains three actions at time nodes T1, T2, and T3, then the video can be divided into three action samples, and each sample corresponds to an action classification (action label). For example, stir-frying appears in the 20th second, adding ingredients appears in the 40th second, and lifting the lid appears in the 50th second. Then this video generates 3 video samples, and the categories (labels) are stir-frying, adding ingredients, and lifting the lid. Collecting several videos and extracting samples of the actions contained therein will generate a large number of labeled video samples, thus forming a dataset.

[0055] This data set is composed of a training set and a test set in a certain ratio (for example, 8:2). The training set is used to train the neural network, and the test set is used to test and evaluate the trained network. The existing training set is used to count the correlation between various actions. This embodiment takes the simultaneity in the correlation as an example for explanation. The adjacency matrix here is used to represent the graph relationship between several actions. Representing the relationship between these actions in the form of an adjacency matrix is ​​more convenient for computer calculation. The adjacency matrix Ma is constructed. The dimension of Ma is 7*7. The element value indexed as [i,j] in Ma represents the probability that the jth action occurs at the same time when the i-th action occurs, that is, the conditional probability P (action[j]=True|action[i]=True), which is statistically derived from the data set and contains a video sample data set of seven actions. For example: Ma[0,1]=0 means that in the existing data set, the probability of covering the pot lid while lifting the pot lid is 0; Ma[2,3]=0.12 means that in the existing data, the probability of adding ingredients while stir-frying is 0.12.

[0056] 3. Initialize the relationship feature vector X of each action, and the dimension of X is 7*256.

[0057] 4. The input of the first layer of the graph neural network is X and Ma. The weight of this layer is W1, which is the parameter to be trained, with a dimension of 256*256, and the output is H1=σ(Ma·X·W1), where σ is a nonlinear activation function, which is the ReLU function in this embodiment, and the dimension of H1 is 7*256. The input of the second layer of the graph neural network is H1, and the output of the second layer of the graph neural network is obtained using the same method as the first layer of the graph neural network. The graph neural network has a total of 5 layers, and the final output is H5, with a dimension of 7*256.

[0058] 5. Transpose H5 to H5T with a dimension of 256*7, calculate V*H5T, and obtain the feature vector V2 with a dimension of 1*7. Input each element in the feature vector V2 into the sigmoid function to obtain the probability value of each action. When the probability value of a certain index position is greater than 0.7, it is considered that the action has occurred, and all the actions that have occurred are output.

[0059] The following results were obtained through comparative analysis of specific embodiments:

[0060] After training with the traditional ViT network and classification head, the final average accuracy is 95.1%, and the recall rates of various actions are as follows:

[0061] Table 1 Recall rate of various actions of traditional ViT network

[0062] action Turn the knob Cover the pot Lift the pot lid Pot Remove from heat Adding materials stir fry Recall 0.965 0.939 0.938 0.954 0.911 0.938 0.963

[0063] The improved ViT network was trained in combination with the relationship extraction branch network, and the final average accuracy was 98.54%. The recall rates of various actions are as follows:

[0064] Table 2 Recall rate of various actions of the improved ViT network

[0065] action Turn the knob Cover the pot Lift the pot lid Pot Remove from heat Adding materials stir fry Recall 0.983 0.958 0.966 0.981 0.946 0.953 0.975

[0066] It can be seen that the present invention can more comprehensively and accurately capture the relationship between actions in multiple cooking action recognition tasks, improve the adaptability to complex scenes, and make the recognition results more reliable and accurate. The integration and optimization of these technical solutions enable the model to have good performance in practical applications.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for recognizing multiple cooking actions based on collaborative action analysis, characterized in that: The method comprises the following steps: Step S1, preprocessing the input video, including center cropping, downsampling, and random brightness processing; Step S2: Use a neural network to analyze the cooking actions in the video Identification, the neural network includes a feature extraction branch and a relationship extraction branch, specifically: Step S21, the feature extraction branch uses a 3D convolutional neural network or a visual self-attention model VisionTransformer network, i.e., a ViT network, to extract feature information in the spatial and temporal domains, and outputs a 1*D-dimensional feature vector V1, where D is the implicit data dimension; Step S22, the relationship extraction branch uses a graph neural network to extract the relationship between each action, where the relationship includes the temporal sequence or the simultaneity of two actions, and outputs a D*C-dimensional relationship weight matrix W, where C is the number of action categories, and the relationship weight matrix represents the mutual relationship between each action in the action set; Step S23, perform matrix multiplication on V1 and W to obtain a vector V2 with a dimension of 1xC, and input each element of the vector into the sigmoid function to obtain a vector V3 with a dimension of 1xC. V3 is the final output. The value of each element in V3 is between 0 and 1, indicating the probability of each action occurring. Step S24, compare the value of each element in V3 with the preset threshold T. When the value of an element is greater than T, that is, the probability value of the index position where the element is located is greater than T, it means that the corresponding action is detected; Step S25: output all detected actions.

2. The method according to claim 1, characterized in that The action set includes 7 operations: lifting the lid, covering the lid, stir-frying, adding ingredients, removing from the pot, turning off the knob, and turning on the knob.

3. The method according to claim 1, characterized in that: Construct an adjacency matrix Ma to represent the relationship between action categories. The dimension of Ma is 7*7. The element value indexed as [i,j] in Ma represents the probability that the jth action occurs at the same time as the i-th action occurs, that is, the conditional probability P (action[j]=True|action[i]=True), where action represents an action and True represents that the condition is true, that is, the action occurs.

4. The method according to claim 1, characterized in that The multi-head pooling attention mechanism MHPA and the small ViT network with phased channel dimension expansion, namely the ViT-s network, are used for feature extraction. MHPA reduces the number of tokens involved in attention calculation and reduces the computational cost by pooling tokens. Token refers to a series of vector representations obtained after processing the input video data. Suppose the input sequence is X, and the weight is Two-dimensional linear mapping and pooling operation , we get the Query vector Q, Key vector K, and Value vector V. The calculation formula is: , , , these vectors are used to calculate the attention weights in the multi-head pooling attention mechanism and to generate the final output.

5. The method according to claim 1, characterized in that The Vision Transformer network used for feature extraction includes an input layer, a backbone layer, and an output layer. A multi-scale attention mechanism and a phased expansion channel dimension are introduced in the input layer. The input layer inputs a video frame sequence, represented as T frames, each frame consists of H×W pixels, H represents the height of the image, and W represents the width of the image. The backbone includes input embedding, position encoding, multi-head attention mechanism, fully connected layers, residual connections, and layer normalization. A multi-scale attention mechanism is introduced to allow the model to focus on different information at different time scales. Phased expansion of the channel dimension refers to gradually increasing the channel dimension in the feature extraction branch of the ViT-s network by gradually increasing the number of convolution kernels or adding convolution layers.

6. The method according to claim 5, characterized in that The input embedding in Backbone embeds the pixel values ​​of each frame into a vector form to form a sequence; the position encoding layer introduces position encoding to help the model capture the temporal information in the sequence; a multi-head self-attention mechanism is introduced in each Transformer block to allow the model to focus on the relationship between different frames; after the self-attention mechanism, a fully connected layer is used to integrate information at each position; residual connections and layer normalization are added between each Transformer block to improve training stability.

7. The method according to claim 1, characterized in that The relationship extraction uses a graph convolutional neural network GCN, which includes an input layer, a GCN layer, an activation function and normalization, a multi-layer GCN, a multi-head GCN, a residual connection, and a graph convolution pooling; the input layer is a 1*D-dimensional feature vector V1 and a D*C-dimensional adjacency matrix, where D is the implicit data dimension and C is the number of action categories; the GCN layer uses a graph convolutional network for relationship extraction, with the input layer as input, and generates an updated node feature matrix, where the nodes correspond to different action categories, and the edges of the graph convolutional network correspond to the relationships between different action categories; Add an activation function after the GCN layer, use the ReLU function as the activation function, and perform layer normalization on the node features; Stack multiple GCN layers to form a multi-layer GCN; introduce a multi-head GCN mechanism to allow the model to focus on different action relationship subspaces; Add residual connections between GCN layers; Add a graph convolutional pooling layer after the GCN layer.

8. The method according to claim 3, characterized in that There are 5 layers of graph neural network. The input of the first layer of graph neural network is X and Ma, where X is the initialized relationship feature vector of each action, and the dimension of X is 7*256. The weight of the first layer of graph neural network is W1, which is the parameter to be trained, with a dimension of 256*256. The output is H1=σ(Ma·X·W1), where σ is a nonlinear activation function, and the dimension of H1 is 7*256. The input of the second layer of graph neural network is H1, and the output of the second layer of graph neural network is obtained using the same method as the first layer of graph neural network. And so on. The final output of the graph neural network is H5, with a dimension of 7*256.

9. The method according to claim 8, characterized in that Transpose H5 to H5T with a dimension of 256*7, calculate V1*H5T, and get the feature vector V2 with a dimension of 1*7. Input each element in the feature vector V2 into the sigmoid function to get the probability value of each action. When a probability value is greater than 0.7, the action is considered to have occurred, and all the actions that occurred are output.

Images