Motion recognition method, device and storage medium

Abstract

The present invention provides a method, device, and storage medium for action recognition. The method includes: obtaining multiple feature matrices by processing acquired video data to be tested; the multiple feature matrices are obtained by processing multiple video frames containing a target object; processing the multiple feature matrices using a temporal feature extraction network model to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices; wherein, the network layer in each network block of the temporal feature extraction network model fuses, reuses, and groups the preceding feature matrices; the preceding feature matrix includes: the feature matrix input by the network block corresponding to the network layer, and the feature matrix output by the preceding network layer sorted before the network layer; and determining the action recognition results of the target object in multiple video frames using the multiple temporal action feature matrices. This solution can reduce the occupation of computing resources while also improving recognition accuracy.

Description

Technical Field

[0001] The embodiments of the present invention relate to the fields of computer and Internet technologies, and in particular to a motion recognition method, device, and storage medium. Background Art

[0002] Traditional action recognition methods use the VGG (Visual Geometry Group Network) model, which uses several consecutive convolution kernels instead of larger convolution kernels for action recognition. This model increases the depth of the network while maintaining the same perceptual field of view, proving that increasing network depth can affect network performance to a certain extent and improve network effectiveness. However, this model uses more parameters and consumes more computing resources. Another action recognition method uses the GoogLeNet model to convert full connections into sparse connections, increasing the depth and width of the network while reducing the network parameters to a certain extent. However, as the number of network layers increases, the network degenerates, leading to inaccurate recognition. Therefore, the related technologies have the problem of high computing resource usage and poor recognition accuracy. Summary of the Invention

[0003] The embodiments of the present invention provide a method, device, and storage medium for motion recognition, which can reduce the usage of computing resources while also improving recognition accuracy.

[0004] The technical solution of the present invention is achieved as follows:

[0005] An embodiment of the present invention provides a method for motion recognition, comprising:

[0006] Processing the acquired video data to be tested to obtain a plurality of feature matrices; the plurality of feature matrices are obtained by processing a plurality of video frames containing the target object within a predetermined time period;

[0007] The plurality of feature matrices are processed using a temporal feature extraction network model to obtain a plurality of temporal action feature matrices corresponding to the plurality of feature matrices; wherein the network layer in each network block in the temporal feature extraction network model fuses, reuses, and groups the preceding feature matrices; the preceding feature matrices include: the feature matrix input by the network block corresponding to the network layer, and the feature matrix output by the preceding network layer sorted before the network layer;

[0008] The action recognition results of the target object in the multiple video frames are determined by using the multiple temporal action feature matrices.

[0009] In the above solution, a temporal feature extraction network model is used to process multiple feature matrices to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices, including:

[0010] Each network layer in the first network block is used to fuse, reuse, and group multiple feature matrices and the feature matrices output by the previous network layer to obtain the first group of time series feature matrices output by the first network block for processing by the second network block.

[0011] Until each network layer in the m-th network block is used to fuse, reuse and group the m-1th group of temporal feature matrices and the feature matrices output by the previous network layer to obtain multiple temporal action feature matrices; the temporal feature extraction network model includes: m network blocks; m is a positive integer greater than or equal to 2; the m-1th group of temporal feature matrices is the feature matrix output by the m-1th network block.

[0012] In the above scheme, each network layer in the mth network block is used to fuse, reuse, and group the m-1th group of time series feature matrices and the feature matrices output by the previous network layer to obtain multiple time series action feature matrices, including:

[0013] Use the first network layer to perform group convolution processing on the m-1th group of time series feature matrices to obtain the first group of feature matrices. After reducing the dimension of the first group of feature matrices, fuse them with the m-1th group of time series feature matrices to obtain the first group of intermediate feature matrices for processing by the second network layer.

[0014] The i-th network layer performs group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices. After dimensionality reduction, the i-th group of feature matrices is fused with the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices to obtain the i-th group of intermediate feature matrices for processing by the i+1th network layer. The first i-1 groups of intermediate feature matrices are the feature matrices output by the first i-1 network layers respectively. The preceding network layers include: the first i-1 network layers; i is an integer greater than 1 and less than n.

[0015] Until the n-1th group of intermediate feature matrices is grouped and convolved using the n-1th network layer to obtain multiple temporal action feature matrices; the n-1th group of intermediate feature matrices is the feature matrix output by the n-1th network layer; the m-th network block includes: n network layers; n is a positive integer greater than or equal to 2.

[0016] In the above scheme, the i-th network layer is used to perform group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices. After the i-th group of feature matrices is reduced in dimension, it is fused with the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices to obtain the i-th group of intermediate feature matrices, including:

[0017] Use each convolution layer in the i-th network layer to perform group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices;

[0018] Calculate the matrix of the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices, and multiply the matrix by the predetermined coefficient to obtain the target matrix;

[0019] After reducing the dimension of the i-th group of feature matrices according to half the number of input channels of the i+1-th network layer, it is fused with the target matrix to obtain the i-th group of intermediate feature matrices.

[0020] In the above scheme, each convolution layer includes: the first convolution layer, the second convolution layer and the third convolution layer;

[0021] The convolution layers in the i-th network layer are used to perform group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices, including:

[0022] Use the first convolutional layer to process the i-1th group of intermediate feature matrices to obtain multiple i-th matrices to be grouped;

[0023] Divide the multiple i-th matrices to be grouped into multiple i-th group matrices, and use the second convolutional layer to process each group of matrices in the multiple i-th group matrices respectively to obtain the i-th feature matrix to be fused corresponding to each group of matrices;

[0024] After fusing multiple i-th feature matrices to be fused, they are processed using the third convolutional layer to obtain the i-th group of feature matrices.

[0025] In the above scheme, the action recognition results of the target object in multiple video frames are determined by using multiple temporal action feature matrices, including:

[0026] Pooling and dimensionality reduction are performed on multiple temporal action feature matrices to obtain multiple intermediate temporal feature matrices;

[0027] Using a preset network model to process multiple intermediate time series feature matrices to obtain a final time series feature matrix;

[0028] Using a preset function to process the final time series feature matrix to obtain multiple probability values;

[0029] The action recognition result is determined in the matching database according to the rank of the maximum probability value among the multiple probability values; the matching database pre-stores mapping information between the ranks of the multiple probability values ​​and the corresponding action recognition results.

[0030] In the above solution, the acquired video data to be tested is processed to obtain multiple feature matrices, including:

[0031] Extracting multiple video frames containing the target object from the acquired video data to be tested according to the temporal features;

[0032] Determining a plurality of color value matrices according to a color value of each pixel point in a plurality of video frames corresponding to a predetermined color value space;

[0033] Multiple color value matrices are normalized, convolved, and pooled to obtain multiple feature matrices.

[0034] An embodiment of the present invention further provides a motion recognition device, comprising:

[0035] A processing unit is used to process the acquired video data to be tested to obtain a plurality of feature matrices; the plurality of feature matrices are obtained by processing a plurality of video frames containing a target object within a predetermined time period;

[0036] The processing unit is further configured to process the plurality of feature matrices using a temporal feature extraction network model to obtain a plurality of temporal action feature matrices corresponding to the plurality of feature matrices; wherein the network layer in each network block in the temporal feature extraction network model performs fusion reuse and grouping processing on the preceding feature matrices; the preceding feature matrix includes: a feature matrix input by the network block corresponding to the network layer, and a feature matrix output by the preceding network layer sorted before the network layer;

[0037] A determination unit is configured to determine the action recognition results of the target object in the multiple video frames using the multiple temporal feature matrices.

[0038] An embodiment of the present invention further provides a motion recognition device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps in the above method when executing the program.

[0039] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method are implemented.

[0040] In an embodiment of the present invention, multiple feature matrices are obtained by processing acquired video data to be tested. The multiple feature matrices are obtained by processing multiple video frames containing a target object within a predetermined time period. A temporal feature extraction network model is used to process the multiple feature matrices to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices. The network layer in each network block of the temporal feature extraction network model fuses, reuses, and groups the preceding feature matrices. The preceding feature matrices include: the feature matrix input to the network block corresponding to the network layer, and the feature matrix output by the preceding network layer that precedes the network layer. The multiple temporal action feature matrices are used to determine the action recognition results of the target object in the multiple video frames. Because the network layer fuses and reuses the preceding feature matrices in the corresponding network block, it effectively utilizes the transfer of features, prevents the loss of more detailed information, avoids network degradation, and improves recognition accuracy. Furthermore, because the network layer groups the feature matrices, reducing computing resource usage, this solution can reduce computing resource usage while also improving recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 An optional flowchart of the motion recognition method provided by an embodiment of the present invention;

[0042] Figure 2 An optional flowchart of the motion recognition method provided by an embodiment of the present invention;

[0043] Figure 3 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0044] Figure 4 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0045] Figure 5 An optional flowchart of the motion recognition method provided by an embodiment of the present invention;

[0046] Figure 6 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0047] Figure 7 An optional flowchart of the motion recognition method provided by an embodiment of the present invention;

[0048] Figure 8 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0049] Figure 9 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0050] Figure 10 An optional flowchart of the motion recognition method provided by an embodiment of the present invention;

[0051] Figure 11 An optional flowchart of the motion recognition method provided by an embodiment of the present invention;

[0052] Figure 12 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0053] Figure 13 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0054] Figure 14 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0055] Figure 15 An optional flowchart of the motion recognition method provided by an embodiment of the present invention;

[0056] Figure 16 An optional effect diagram of the action recognition method provided by an embodiment of the present invention;

[0057] Figure 17 A schematic diagram of the structure of a motion recognition device provided by an embodiment of the present invention;

[0058] Figure 18 A schematic diagram of a hardware entity of a motion recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0061] If similar descriptions of "first / second" appear in the invention document, the following explanation is added. In the following description, the terms "first\second\third" involved are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first\second\third" can be interchanged with the specific order or sequence where permitted, so that the embodiments of the invention described herein can be implemented in an order other than that illustrated or described herein.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0063] Figure 1 An optional flow chart of the action recognition method provided by the embodiment of the present invention is combined with Figure 1 The steps shown are explained.

[0064] S101 , processing the acquired video data to be tested to obtain a plurality of feature matrices; the plurality of feature matrices are obtained by processing a plurality of video frames containing a target object within a predetermined time period.

[0065] In an embodiment of the present invention, the terminal processes the acquired video data to be tested to obtain a plurality of feature matrices, wherein the plurality of feature matrices are obtained by processing a plurality of video frames containing a target object within a predetermined time period.

[0066] In an embodiment of the present invention, a terminal first obtains video data to be tested that includes a target object. The target object can be a person, an animal, or other object. The format of the video data to be tested can be any video format. The terminal extracts multiple video frames containing the target object according to temporal features within a predetermined time period from the video data to be tested. The terminal forms a color value matrix corresponding to each video frame based on the color values ​​of the pixels in each video frame. The terminal performs normalization, convolution, and pooling on each color value matrix to obtain a feature matrix corresponding to each video frame, thereby obtaining multiple feature matrices.

[0067] In an embodiment of the present invention, the terminal can mark the target object image in each video frame through a target detection model, such as a real-time fast target detection model (YouOnly Look Once, YOLO) or a single-machine multi-card target detection algorithm (Single Shot MultiBox Detector, SSD). The terminal then converts the target object image into a certain color value space. The terminal divides the target object image into a number of pixel units based on the pixel points on the target object image, and then calculates the color value of the center of each pixel unit. The terminal forms a color value matrix for each video frame through the color values ​​of the centers of several pixel units.

[0068] S102. Use a temporal feature extraction network model to process multiple feature matrices to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices; wherein, the network layer in each network block in the temporal feature extraction network model fuses, reuses, and groups the preceding feature matrices; the preceding feature matrix includes: the feature matrix of the network block input corresponding to the network layer, and the feature matrix of the preceding network layer output that is sorted in front of the network layer.

[0069] In an embodiment of the present invention, a terminal processes multiple feature matrices using a temporal feature extraction network model to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices. The network layer in each network block in the temporal feature extraction network model fuses, reuses, and groups the preceding feature matrices. The preceding feature matrices include the feature matrices input from the network block corresponding to the network layer and the feature matrices output from the preceding network layer that precedes the network layer.

[0070] In an embodiment of the present invention, a temporal feature extraction network model may include multiple network blocks, and each network block may include multiple network layers. The terminal uses the first network layer in the first network block to group and process multiple feature matrices to obtain a first group of feature matrices. The first group of feature matrices is merged with multiple feature matrices to obtain a first group of intermediate feature matrices. The second network layer in the first network block is then used to group and process the first group of intermediate feature matrices to obtain a second group of feature matrices. This process is continued until all network layers of the first network block are processed to obtain the first group of temporal feature matrices corresponding to the first network block. The second network block is then used to process the first group of temporal feature matrices until multiple temporal action feature matrices output by the last network block are obtained, thereby obtaining multiple temporal action feature matrices.

[0071] The preceding network layer is the network layer that comes first in the network block to which any network layer belongs. For example, if the network layer currently processing the feature matrix is ​​the third network layer in the first network block, the preceding network layers of the third network layer are the first and second network layers in the first network block.

[0072] S103 : Determine the action recognition results of the target objects in the multiple video frames through the multiple temporal action feature matrices.

[0073] In the embodiment of the present invention, the terminal determines the action recognition results of the target object in the multiple video frames through multiple temporal action feature matrices.

[0074] In an embodiment of the present invention, a terminal performs pooling and dimensionality reduction processing on multiple temporal action feature matrices to obtain multiple intermediate temporal feature matrices. The multiple intermediate temporal feature matrices are processed using a long short-term memory network model to obtain multiple final temporal feature matrices. The final temporal feature matrices are processed using a preset function to obtain the maximum probability value. The terminal determines the action recognition result corresponding to the maximum probability value using a preset matching database, and further determines the action recognition results corresponding to the target objects in multiple video frames.

[0075] In some embodiments, see Figure 2 , Figure 2 An optional flow chart of the motion recognition method provided by an embodiment of the present invention is provided. Figure 1 The illustrated steps S101 to S103 are followed by steps S201 to S210 , which will be described in conjunction with each step.

[0076] S201. Video data.

[0077] S201: Data preprocessing.

[0078] In the embodiment of the present invention, the terminal pre-processes the video data, converts the image into a matrix form and performs normalization processing.

[0079] S201, 7×7 convolution.

[0080] S201, maximum pooling.

[0081] In the embodiment of the present invention, the terminal performs convolution on the data through a 7×7 filter, performs pooling processing on the obtained result, and then inputs it into the first network block in the temporal feature extraction network model.

[0082] S201, the first network block.

[0083] S201, the second network block.

[0084] S201, the third network block.

[0085] In this embodiment of the present invention, in the temporal feature extraction network model, the data features within each network block are integrated and reused. This maximizes the transfer of features between them, ensuring that more detailed information is not lost. After the temporal feature extraction network model is used, multiple temporal action feature matrices are obtained.

[0086] S201: Pooling dimensionality reduction processing.

[0087] S201, LSTM classification.

[0088] S201, action result.

[0089] After average pooling and vectorization dimensionality reduction, multiple temporal action feature matrices are input into the Long Short-Term Memory (LSTM) network for classification, and the final result of the video action classification is obtained.

[0090] To verify the performance of the present invention, the inventors used the present network model and the classic networks Resnet50, Resnet101, VGG19, and Inception_V3 under the same hyperparameters and hardware conditions to extract temporal action feature matrices. The extracted temporal action feature matrices were then classified using a multi-stage LSTM network with the same settings to obtain action results. Training was performed under the same training conditions, with 35 batches, 36 samples, and an initial learning rate of 0.0001. The accuracy and loss obtained from the training were saved and compared.

[0091] By comparing the changes in the average accuracy of each network during the training process, the network basically reached convergence after 31 iterations. The comparison found that the network model of the present invention had improved accuracy compared with several other networks in the same batch. The inventor used the trained network model to test action recognition and displayed the classification results and the probability of the action category in real time. Some results are shown as follows Figure 3 and Figure 4 As shown. Combined Figure 3 , we can get the corresponding action recognition result: Action: playing the flute. Eigenvalue: 0.988373. Figure 4 , we can get the corresponding action recognition result: Action: Figure Skating. Eigenvalue: 0.988373.

[0092] In an embodiment of the present invention, after determining multiple action recognition results in multiple video frames within a predetermined time period, the terminal can extract multiple video frames of the next time period of the video data to be tested. The terminal processes the next batch of multiple video frames according to the same method as above, and can obtain the action recognition results of the target object in the next batch of multiple video frames.

[0093] In an embodiment of the present invention, multiple feature matrices are obtained by processing the acquired video data to be tested. The multiple feature matrices are obtained by processing multiple video frames containing a target object. A temporal feature extraction network model is used to process the multiple feature matrices to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices. The network layer in each network block of the temporal feature extraction network model fuses, reuses, and groups the preceding feature matrices. The preceding feature matrices include: the feature matrix input to the network block corresponding to the network layer, and the feature matrix output by the preceding network layer that precedes the network layer. The action recognition results of the target object in the video data to be tested are determined using the multiple temporal action feature matrices. Because the network layer fuses and reuses the preceding feature matrices in the corresponding network block, the transfer between features is greatly utilized, so that more detailed information is not lost, network degradation is avoided, and recognition accuracy is improved. Because the network layer groups the feature matrices, thereby reducing computing resource usage, this solution can reduce computing resource usage while also improving recognition accuracy.

[0094] In some embodiments, see Figure 5 , Figure 5 An optional flow chart of the motion recognition method provided by an embodiment of the present invention is provided. Figure 1 The illustrated S102 can be implemented through S104 to S105 , which will be described in conjunction with each step.

[0095] S104. Utilize each network layer in the first network block to fuse, reuse, and group multiple feature matrices and feature matrices output by the preceding network layer to obtain the first group of time series feature matrices output by the first network block for processing by the second network block.

[0096] In an embodiment of the present invention, the terminal utilizes each network layer in the first network block in the timing feature extraction network model to fuse, reuse, and group multiple feature matrices and the feature matrix output by the preceding network layer to obtain the first group of timing feature matrices output by the first network block for processing by the second network block.

[0097] In this embodiment of the present invention, the terminal uses the first network layer in the first network block to group and process multiple feature matrices to obtain a first set of feature matrices. The first set of feature matrices is then fused with multiple feature matrices (i.e., the pre-order feature matrix) to obtain a first set of intermediate feature matrices. The terminal continues processing according to this scheme until the network layer in the first network block completes processing, obtaining a first set of time-series feature matrices. The terminal then uses the network layer in the second network block to process the first set of time-series feature matrices using the same method to obtain a second set of intermediate feature matrices.

[0098] In the embodiment of the present invention, if the current network layer is the i-th network layer, the corresponding previous network layers include: the first i-1 network layers, where i is an integer greater than 1 and less than n, and n is the number of network layers.

[0099] In an embodiment of the present invention, the terminal uses the first network layer in the first network block to perform group convolution processing on multiple feature matrices to obtain a first set of feature matrices. After reducing the dimensionality of the first set of feature matrices, the first set of feature matrices is fused with multiple feature matrices to obtain a first set of intermediate feature matrices for processing by the second network layer. The terminal uses the second network layer in the first network block to perform group convolution processing on the first set of intermediate feature matrices to obtain a second set of feature matrices. After reducing the dimensionality of the second set of feature matrices, the second set of feature matrices is fused with multiple feature matrices and the first set of intermediate feature matrices to obtain a second set of intermediate feature matrices for processing by the third network layer. This continues until the terminal uses the nth network layer in the first network block to perform group convolution processing on the n-1th set of intermediate feature matrices to obtain a first set of time series feature matrices. Each network block includes: n network layers; n is a positive integer greater than or equal to 2.

[0100] S105, until each network layer in the mth network block is used to fuse, reuse and group the m-1th group of time series feature matrices and the feature matrices output by the previous network layer to obtain multiple time series action feature matrices.

[0101] In this embodiment of the present invention, the terminal utilizes each network layer in the mth network block to fuse, reuse, and group the m-1th group of temporal feature matrices and the feature matrices output by the preceding network layer, thereby obtaining multiple temporal action feature matrices. The temporal feature extraction network model includes m network blocks, where m is a positive integer greater than 2. The m-1th group of temporal feature matrices is the matrix obtained by processing the temporal feature matrix input by the preceding network block in the m-1th network block.

[0102] For example, combined Figure 6. The temporal feature extraction network model in the embodiment of the present invention includes: a first network block, a second network block and a third network block. Each network block contains three network layers. And each network layer contains three convolutional layers. The terminal convolves the feature matrix of size 224x224x3 through the Conv7x7 convolutional layer, then performs pooling through BatchNorm, and then inputs it into the first network layer in the first network block in the temporal feature extraction network model after processing through the activation function (Rectified Linear Unit, ReLU). The number of input channels of the first network layer is 64, and the number of output channels is 128. The terminal performs dimensionality reduction processing on the first feature matrix, and then fuses it with multiple feature matrices to obtain the first intermediate feature matrix. The terminal inputs the first intermediate feature matrix into the second network layer. Among them, the convolutional layers in the first network layer are 64, Conv1x1, 128; 128, Conv3x3, 128; 128, Conv1x1, 128 respectively. The number of input and output channels of the convolutional layers of subsequent network layers increases successively. Each time the terminal processes a feature matrix through a convolutional layer, it must perform pooling and activation function processing. This continues until the first network block outputs the first set of temporal feature matrices. The terminal then uses the second network module to process the first set of temporal feature matrices using the same method, obtaining the second set of temporal feature matrices. The terminal then uses the third network module to process the second set of temporal feature matrices using the same method, obtaining multiple temporal action feature matrices. Finally, the terminal further processes these multiple temporal action feature matrices using activation functions.

[0103] In the embodiment of the present invention, the terminal utilizes the network layers in the m network blocks to perform fusion reuse and group processing on the feature matrix, which not only reduces the occupancy rate of computing resources but also improves recognition accuracy.

[0104] In some embodiments, see Figure 7 , Figure 7 An optional flow chart of the motion recognition method provided by an embodiment of the present invention is provided. Figure 5 The illustrated S105 can be implemented through S106 to S108 , which will be described in conjunction with each step.

[0105] S106. Use the first network layer to perform group convolution processing on the m-1th group of time series feature matrices to obtain the first group of feature matrices. After reducing the dimension of the first group of feature matrices, fuse them with the m-1th group of time series feature matrices to obtain the first group of intermediate feature matrices for processing by the second network layer.

[0106] In an embodiment of the present invention, the terminal uses the first network layer to perform group convolution processing on the m-1th group of time series feature matrices to obtain the first group of feature matrices. After reducing the dimension of the first group of feature matrices, they are fused with the m-1th group of time series feature matrices to obtain the first group of intermediate feature matrices for processing by the second network layer.

[0107] In an embodiment of the present invention, the first network layer may include three convolutional layers. The terminal processes multiple feature matrices using the first convolutional layer, groups the obtained multiple feature matrices, and then processes them in sequence using the second convolutional layer. Finally, the third convolutional layer processes the matrix output by the second convolutional layer to obtain the first group of feature matrices. The method for processing the input feature matrix by the convolutional layer in the subsequent network layer can be the same as the processing method in the first network layer, and will not be repeated here.

[0108] In this embodiment of the present invention, the terminal performs group convolution processing on the m-1th group of time series feature matrices using each convolutional layer in the first network layer to obtain the first group of feature matrices. The terminal calculates the target matrix by multiplying the m-1th group of time series feature matrices by the predetermined coefficients corresponding to the current network layer. The terminal performs dimensionality reduction on the first group of feature matrices according to half the number of input channels of the second network layer, and then fuses the result with the target matrix to obtain the first group of intermediate feature matrices.

[0109] For example, combined Figure 6 , the number of input channels of the second network layer is 128. The number of output channels of the first network layer is also 128. The terminal reduces the dimension of the first set of feature matrices to 64. The number of input channels of the first network layer is 64. At this time, the dimensions of multiple feature matrices and the first set of feature matrices after dimensionality reduction are both 64. The terminal multiplies the multiple feature matrices with predetermined coefficients and then combines them with the first set of feature matrices after dimensionality reduction to obtain a first intermediate feature matrix with a dimension of 128.

[0110] S107. Use the i-th network layer to perform group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices. After reducing the dimension of the i-th group of feature matrices, fuse them with the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices to obtain the i-th group of intermediate feature matrices for processing by the i+1th network layer.

[0111] In an embodiment of the present invention, the terminal uses the i-th network layer to perform group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices. After reducing the dimension of the i-th group of feature matrices, the i-th group of feature matrices is fused with the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices to obtain the i-th group of intermediate feature matrices for processing by the i+1th network layer. The first i-1 groups of intermediate feature matrices are the feature matrices output by the first i network layers respectively. The preceding network layers of the i-th network layer include: the first i network layers; i is an integer greater than 1 and less than n.

[0112] In an embodiment of the present invention, the terminal performs group convolution processing on the i-1th group of intermediate feature matrices using each convolutional layer in the i-th network layer to obtain the i-th group of feature matrices. The terminal calculates a matrix formed by combining the m-1th group of temporal feature matrices and the first i-1 groups of intermediate feature matrices, and multiplies the matrix by predetermined coefficients to obtain a target matrix. The terminal performs dimensionality reduction on the i-th group of feature matrices according to half the number of input channels of the i+1th network layer, and then fuses it with the target matrix to obtain the i-th group of intermediate feature matrices.

[0113] In an embodiment of the present invention, the terminal can also use the convolution layer in the second network layer in the nth network block to process the first group of intermediate feature matrices, reduce the dimension of the second group of feature matrices, and fuse them with the first group of intermediate feature matrices and multiple feature matrices to obtain the second group of intermediate feature matrices.

[0114] S108, until the nth network layer is used to perform group convolution processing on the n-1th group of intermediate feature matrices to obtain multiple temporal action feature matrices.

[0115] In an embodiment of the present invention, the terminal performs group convolution processing on the n-1th group of intermediate feature matrices using the nth network layer to obtain multiple time-series action feature matrices. The n-1th group of intermediate feature matrices is the feature matrix output by the n-1th network layer; the mth network block includes: n network layers; n is a positive integer greater than or equal to 2.

[0116] In an embodiment of the present invention, the terminal processes the n-1th group of intermediate feature matrices using the first convolutional layer in the nth network layer to obtain multiple nth matrices to be grouped. The terminal then divides the multiple nth matrices to be grouped into multiple nth groups of matrices and processes each of the multiple nth groups of matrices using the second convolutional layer to obtain the nth feature matrix to be fused corresponding to each group of matrices. After fusing the multiple nth feature matrices to be fused, the terminal processes them using the third convolutional layer in the nth network layer to obtain multiple time-series action feature matrices.

[0117] In the embodiment of the present invention, the temporal feature extraction network model of the present invention can be formed by connecting three network blocks, each of which is composed of three network layers. The structure of a single network block is as follows: Figure 8 As shown in Figure 2, the input feature matrix of each network layer is a fusion matrix of the output matrix of the previous network layers and multiple feature matrices. In other words, the input of the second network layer is the output feature matrix of the first network layer and the fusion matrix of multiple feature matrices, and the input of the third network layer is the output feature matrix of the first network layer, the output feature matrix of the second network layer, and the fusion matrix of multiple feature matrices.

[0118] For example, combined Figure 9 The input of each network layer is a fusion of the inputs of the previous layers and the output of the previous layer, and all network layers in the network block are connected to the network layer. This further strengthens the transmission of features and fuses features of different scales. During the fusion process, dimensionality reduction is used to ensure the consistency of the number of channels, alleviating the problem of vanishing gradients during training. In other words, after the input of the first network layer, the output of the first network layer is obtained. The input of the second network layer is the fusion matrix of the feature matrix of the output of the first network layer after dimensionality reduction and the input of the first network layer. The input of the third network layer is the fusion matrix of the feature matrix of the output of the first network layer after dimensionality reduction, the feature matrix of the output of the second network layer after dimensionality reduction, and the input of the first network layer.

[0119] In an embodiment of the present invention, the terminal groups the input feature matrix through each network layer in the nth network block, and at the same time integrates the feature matrix output by the previous network layer, thereby improving the recognition accuracy while reducing the occupation of computing resources.

[0120] In some embodiments, see Figure 10 , Figure 10 An optional flow chart of the motion recognition method provided by an embodiment of the present invention is provided. Figure 7 The illustrated S107 can be implemented through S109 to S111 , which will be described in conjunction with each step.

[0121] S109. Perform group convolution processing on the i-1th group of intermediate feature matrices using each convolution layer in the i-th network layer to obtain the i-th group of feature matrices.

[0122] In the embodiment of the present invention, the terminal uses each convolution layer in the i-th network layer to perform group convolution processing on the i-1-th group of intermediate feature matrices to obtain the i-th group of feature matrices.

[0123] Each convolution layer may include 3 convolution layers or 4 convolution layers. The specific number of convolution layers is not limited in the embodiment of the present invention.

[0124] S110 , calculating a matrix obtained by combining the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices, and multiplying the matrix by a predetermined coefficient to obtain a target matrix.

[0125] In the embodiment of the present invention, the terminal calculates a matrix obtained by combining the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices, and obtains a target matrix by multiplying the matrix by a predetermined coefficient.

[0126] In this embodiment of the present invention, the terminal combines the m-1th group of time series feature matrices with the first i-1 groups of intermediate feature matrices according to the number of input channels of the i-th network layer to obtain the matrix. The terminal then multiplies the matrix by the predetermined coefficient corresponding to the i-th network layer to obtain the target matrix.

[0127] S111. After reducing the dimension of the i-th group of feature matrices according to half the number of input channels of the i+1-th network layer, the matrix is ​​fused with the target matrix to obtain the i-th group of intermediate feature matrices.

[0128] In the embodiment of the present invention, the terminal performs dimension reduction on the i-th group of feature matrices according to half the number of input channels of the i+1-th network layer, and then fuses the result with the target matrix to obtain the i-th group of intermediate feature matrices.

[0129] In the embodiment of the present invention, when i is equal to 2, the terminal performs dimensionality reduction on the second set of feature matrices according to half the number of input channels of the third network layer, and then fuses the resultant matrix with the corresponding matrix multiplied by a predetermined coefficient to obtain a second set of intermediate feature matrices. The predetermined coefficients are obtained during the training of the temporal feature extraction network model of the present invention.

[0130] In the embodiment of the present invention, the terminal can calculate the intermediate characteristic matrix O of the i-th group by formula (2): m

[0131] O m =Bn(Z concat )+w m a m (2)

[0132] Among them, Bn(Z concat ) indicates that the feature matrix is ​​processed by the third convolutional layer in the i-th network layer, w m represents the predetermined coefficient of the i-th network layer, a m The terminal will w m with a m The corresponding target matrix is ​​obtained by multiplication and then fused with the i-th group of feature matrices. The predetermined coefficients of different network layers can be the same or different.

[0133] In an embodiment of the present invention, the terminal may perform dimensionality reduction on the calculated second set of feature matrices by half the number of input channels of the third network layer through the second network layer, and then fuse the resultant matrix with the target matrix obtained by multiplying the plurality of feature matrices and the first set of intermediate feature matrices by predetermined coefficients, to obtain the second set of intermediate feature matrices. In this case, the predetermined coefficients are predetermined coefficients corresponding to the second network layer.

[0134] In some embodiments, see Figure 11 , Figure 11 An optional flow chart of the motion recognition method provided by an embodiment of the present invention is provided. Figure 10 The illustrated S109 can be implemented through S112 to S114 , which will be described in conjunction with each step.

[0135] S112. Use the first convolutional layer to process the i-1th group of intermediate feature matrices to obtain multiple i-th matrices to be grouped.

[0136] In the embodiment of the present invention, the terminal uses the first convolutional layer in the i-th network layer to process the i-1-th group of intermediate feature matrices to obtain multiple i-th matrices to be grouped.

[0137] For example, combined Figure 6 , the first convolution layer in the second network layer of the third network block is 512, Conv1x1,1024 convolution layer, the number of input channels of the first convolution layer is 512, and the number of output channels is 1024.

[0138] For example, combined Figure 12 , the first convolution layer can also be 128, 1×1 convolution, 256 convolution layer, the number of input channels of the first convolution layer is 128, and the number of output channels is 256.

[0139] In the embodiment of the present invention, when the number of output channels of the first convolutional layer is 128, the first convolutional layer outputs a matrix to be grouped with 128 dimensions.

[0140] S113. Divide the multiple i-th matrices to be grouped into multiple i-th group matrices, and use the second convolutional layer to process each group of matrices in the multiple i-th group matrices respectively to obtain the i-th feature matrix to be fused corresponding to each group of matrices.

[0141] In an embodiment of the present invention, the terminal divides multiple i-th matrices to be grouped into multiple i-th group matrices, and uses the second convolutional layer to process each group of matrices in the multiple i-th group matrices respectively to obtain the i-th feature matrix to be fused corresponding to each group of matrices.

[0142] In the embodiment of the present invention, the terminal may divide the plurality of i-th matrices to be grouped into 24 groups or 32 groups. The specific number of groups is not limited in the embodiment of the present invention.

[0143] For example, combined Figure 12 , the terminal divides multiple i-th matrices to be grouped into 32 groups of matrices. The terminal processes each group of matrices using a convolution layer with a size of 3×3 convolution to obtain the i-th feature matrix to be fused corresponding to each group of matrices.

[0144] In the task of recognition and classification, training a neural network is a very complex process, which is closely related to the structure of the designed neural network itself. Grouped convolution is a very good strategy. Grouped convolution can be regarded as a sparse operation. Grouped convolution can not only reduce the overall number of parameters of the network, but also achieve good results with fewer parameters. The schematic diagram of ordinary convolution and grouped convolution is as follows Figure 13 and Figure 14 shown.

[0145] Combine Figure 13 , when ordinary convolution is performed, if the size of the input feature matrix is ​​C×H×W (where C represents the input dimension or the number of input channels; H represents the feature map height; W represents the feature map width) then Figure 13 The number of channels of the input feature matrix is ​​6, and there are 3 convolution kernels. Then the output feature matrix is ​​the same as the number of convolution kernels, which is 3. The size of each convolution kernel is C×K×K (where K represents the length or width of the convolution kernel). The total number of parameters of the three convolution kernels is 3×C×K×K. Figure 14 , the main purpose of group convolution is to group the input feature matrix and then perform convolution on each group separately. Figure 14 The size of the input feature matrix is ​​C×H×W, and the number of output feature matrices is 3. Assuming that it is divided into 3 groups, the number of input feature matrices in each group is C / 3, and the number of output feature matrices in each group is 3 / 3, that is, 1. The size of each convolution kernel is C×K×K / 3, and the total number of convolution kernels is still 3. The number of convolution kernels in each group is 1. The convolution kernel is only convolved with the input feature map of the same group. The total number of parameters of the convolution kernel is 3×C×K×K / 3. It is easy to obtain that the total number of parameters is reduced to 1 / 3 of the ordinary convolution.

[0146] Combine Figure 14Within the network layer, the input data must first be batch normalized, and then the normalized feature matrix is ​​grouped. The size of the group is usually determined by the number of output channels after the previous convolution, generally 24 or 32. The grouped feature maps are each convolved with a 3×3 filter. The feature matrix obtained after the convolution is aggregated to fuse the extracted data features. The combined data is batch normalized, which can solve the difficulties caused by the irregular distribution of each batch of data during training. The input data of this layer of the network is reused, that is, the input data is fused with the batch normalized data, and the resulting feature data is used as the output of the current network layer. This improves the accuracy of recognition.

[0147] S114: After fusing multiple i-th feature matrices to be fused, process them using the third convolutional layer to obtain the i-th group of feature matrices.

[0148] In the embodiment of the present invention, the terminal fuses multiple i-th feature matrices to be fused, and then processes them using the third convolutional layer to obtain the i-th group of feature matrices.

[0149] In this embodiment of the present invention, the terminal may multiply each i-th matrix to be fused by the fusion coefficient of the corresponding group, and then multiply it by the corresponding vector aggregation parameter to obtain the product of each matrix to be fused. The terminal then adds the multiple products to obtain the i-th group of feature matrices. The fusion coefficient and vector aggregation parameter are obtained during the training of the time series feature extraction network model.

[0150] For example, the terminal can calculate the fusion matrix Z of multiple i-th feature matrices to be fused by formula (1): concat .

[0151]

[0152] Among them, T a represents the vector aggregation parameter of the ath group, C is the number of groups, K a represents the fusion coefficient of the a-th group, X a represents the i-th matrix to be fused corresponding to the a-th group. The terminal will K a 、T a With X a Multiply them to get a product corresponding to each group. The terminal adds multiple products to get the fusion matrix Z of multiple feature matrices to be fused concat .

[0153] In some embodiments, see Figure 15 , Figure 15 An optional flow chart of the motion recognition method provided by an embodiment of the present invention is provided. Figure 1The illustrated S101 can be implemented through S115 to S117 , which will be described in conjunction with each step.

[0154] S115 , extracting multiple video frames containing the target object from the acquired video data to be tested according to the temporal features.

[0155] In the embodiment of the present invention, the terminal extracts a plurality of video frames containing the target object from the acquired video data to be tested according to the time sequence characteristics.

[0156] In the embodiment of the present invention, the terminal can extract 36 video frames containing the target object from 60 seconds of video data to be tested.

[0157] S116 , determining a plurality of color value matrices according to the color value of each pixel point in the plurality of video frames corresponding to the predetermined color space.

[0158] In the embodiment of the present invention, the terminal determines a plurality of color value matrices according to the color value of each pixel in the plurality of video frames corresponding to the predetermined color space.

[0159] In an embodiment of the present invention, the terminal converts each video frame into a color value space. Based on the pixels in each video frame, the terminal divides the video frame into a number of pixel units and calculates the color value at the center of each pixel unit. The terminal forms a color value matrix for each video frame using the color values ​​at the centers of the pixel units, thereby determining multiple color value matrices.

[0160] The predetermined color value space may be any one of the LAB color value space, the HSV color value space, and the RGB color value space.

[0161] S117. Perform normalization, convolution, and pooling processing on the multiple color value matrices to obtain multiple feature matrices.

[0162] In the embodiment of the present invention, the terminal performs normalization, convolution and pooling processing on multiple color value matrices to obtain multiple feature matrices.

[0163] In an embodiment of the present invention, the terminal performs normalization, convolution and pooling on multiple color value matrices to obtain multiple feature matrices, so that the matrices of the input temporal feature extraction network model are highly unified, redundant data information is removed, and the calculation efficiency is higher.

[0164] In some embodiments, see Figure 15 , Figure 15 An optional flow chart of the motion recognition method provided by an embodiment of the present invention is provided. Figure 1 The illustrated S103 can be implemented through S118 to S121 , which will be described in conjunction with each step.

[0165] S118. Pooling and dimensionality reduction processing are performed on the multiple time series action feature matrices to obtain multiple intermediate time series feature matrices.

[0166] In the embodiment of the present invention, the terminal performs pooling and dimensionality reduction processing on multiple time series action feature matrices to obtain multiple intermediate time series feature matrices.

[0167] S119: Processing the multiple intermediate time series feature matrices using a preset network model to obtain a final time series feature matrix.

[0168] In the embodiment of the present invention, the terminal processes a plurality of intermediate time series characteristic matrices using a preset network model to obtain a final time series characteristic matrix.

[0169] The preset network model can be a multi-layer long short-term memory network model (LSTM network). The multi-layer LSTM network makes the model more in-depth and more accurate in expressing time series information. The terminal inputs multiple intermediate time series feature matrices into the multi-layer LSTM network for training and classification, and obtains the final time series feature matrix corresponding to the multiple intermediate time series feature matrices.

[0170] For example, combined Figure 16 , where x <t>Represents the intermediate time series feature matrix corresponding to the video frame at time t. The size is usually expressed in time_steps. Time_steps is unique to LSTM and indicates the number of input moments. Input_dim is usually used to represent the input dimension, that is, the dimension of the input vector at each moment. c1 <t-1>a1 represents the updated value of the memory cell at time t-1 in the first layer of LSTM network, <t>It represents the activation value of the first layer LSTM network transmitted to the second layer LSTM network at time t, y <t>Represents the output of the last state. Each LSTM network contains hidden nodes. The number of hidden nodes determines the dimension of the input to the next layer of LSTM activation value or the output y <t>In the task of training an action recognition network, the parameter values ​​are continuously updated through partial derivatives during backpropagation to obtain the optimal parameters and network.

[0171] S120: Using a preset function to process the final time series feature matrix to obtain multiple probability values.

[0172] In the embodiment of the present invention, the terminal processes the final time series feature matrix using a preset function to obtain multiple probability values.

[0173] The preset function may be a softmax function. In other embodiments, the preset function may also be other functions, which is not limited in the embodiment of the present invention.

[0174] S121 , determining an action recognition result in a matching database according to a ranking of a maximum probability value among a plurality of probability values.

[0175] In the embodiment of the present invention, the terminal determines the action recognition result according to the ranking of the maximum probability value among the multiple probability values ​​in the matching database.

[0176] The matching database pre-stores mapping information between the rankings of multiple probability values ​​and corresponding action recognition results.

[0177] In an embodiment of the present invention, after determining multiple action recognition results in multiple video frames, the terminal can extract the next batch of multiple video frames of the video data to be tested. The terminal processes the next batch of multiple video frames according to the same method as described above, and can obtain the action recognition results of the target object in the next batch of multiple video frames.

[0178] In the embodiment of the present invention, the terminal uses LSTM to classify multiple temporal action feature matrices, thereby being able to more accurately distinguish the actions of the target object in each video frame, thereby also improving the recognition accuracy.

[0179] See Figure 17 , Figure 17 This is a schematic diagram of the structure of a motion recognition device provided by an embodiment of the present invention.

[0180] The embodiment of the present invention also provides a motion recognition device 800, comprising: a processing unit 803 and a determination unit 804.

[0181] The processing unit 803 is configured to process the acquired video data to obtain a plurality of feature matrices; the plurality of feature matrices are obtained by processing a plurality of video frames containing the target object within a predetermined time period;

[0182] The processing unit 803 is further configured to process the plurality of feature matrices using a temporal feature extraction network model to obtain a plurality of temporal action feature matrices corresponding to the plurality of feature matrices; wherein the network layer in each network block in the temporal feature extraction network model performs fusion, reuse, and grouping processing on the preceding feature matrices; the preceding feature matrices include: the feature matrix input by the network block corresponding to the network layer, and the feature matrix output by the preceding network layer sorted before the network layer;

[0183] The determining unit 804 is configured to determine the action recognition results of the target object in the multiple video frames using the multiple temporal action feature matrices.

[0184] In an embodiment of the present invention, the processing unit 803 in the action recognition device 800 is used to use each network layer in the first network block to fuse, reuse and group multiple feature matrices and feature matrices output by the previous network layer to obtain the first group of temporal feature matrices output by the first network block for processing by the second network block; until each network layer in the mth network block is used to fuse, reuse and group the m-1th group of temporal feature matrices and feature matrices output by the previous network layer to obtain multiple temporal action feature matrices; the temporal feature extraction network model includes: m network blocks; m is a positive integer greater than or equal to 2; the m-1th group of temporal feature matrices is the feature matrix output by the m-1th network block.

[0185] In the embodiment of the present invention, the processing unit 803 in the action recognition device 800 is used to perform group convolution processing on the m-1th group of time series feature matrices using the first network layer to obtain the first group of feature matrices, reduce the dimension of the first group of feature matrices, and fuse them with the m-1th group of time series feature matrices to obtain the first group of intermediate feature matrices for processing by the second network layer; perform group convolution processing on the i-1th group of intermediate feature matrices using the i-th network layer to obtain the i-th group of feature matrices, reduce the dimension of the i-th group of feature matrices, and fuse them with the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices. The intermediate feature matrices are fused to obtain the i-th group of intermediate feature matrices for processing by the i+1-th network layer; the first i-1 groups of intermediate feature matrices are the feature matrices output by the first i-1 network layers respectively; the preceding network layers include: the first i-1 network layers; i is an integer greater than 1 and less than n; until the n-1-th group of intermediate feature matrices is grouped and convolved using the n-1-th network layer to obtain multiple temporal action feature matrices; the n-1-th group of intermediate feature matrices is the feature matrix output by the n-1-th network layer; the m-th network block includes: n network layers; n is a positive integer greater than or equal to 2.

[0186] In an embodiment of the present invention, the processing unit 803 in the action recognition device 800 is used to use each convolution layer in the i-th network layer to perform group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices; calculate the matrix after the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices are combined, and multiply the matrix with a predetermined coefficient to obtain a target matrix; after reducing the dimension of the i-th group of feature matrices according to half the number of input channels of the i+1th network layer, it is fused with the target matrix to obtain the i-th group of intermediate feature matrices.

[0187] In the embodiment of the present invention, each convolution layer includes: a first convolution layer, a second convolution layer, and a third convolution layer; the processing unit 803 in the action recognition device 800 is used to process the i-1th group of intermediate feature matrices using the first convolution layer to obtain multiple i-th matrices to be grouped;

[0188] Divide the multiple i-th matrices to be grouped into multiple i-th group matrices, and use the second convolutional layer to process each group of matrices in the multiple i-th group matrices respectively to obtain the i-th feature matrix to be fused corresponding to each group of matrices;

[0189] After fusing multiple i-th feature matrices to be fused, they are processed using the third convolutional layer to obtain the i-th group of feature matrices.

[0190] In an embodiment of the present invention, the determination unit 804 in the action recognition device 800 is used to perform pooling and dimensionality reduction processing on the multiple time-series action feature matrices to obtain multiple intermediate time-series feature matrices; use a preset network model to process the multiple intermediate time-series feature matrices to obtain a final time-series feature matrix; use a preset function to process the final time-series feature matrix to obtain multiple probability values; determine the action recognition result in the matching database based on the ranking of the maximum probability value among the multiple probability values; the matching database pre-stores mapping information between the rankings of multiple probability values ​​and the corresponding action recognition results.

[0191] In an embodiment of the present invention, the processing unit 803 in the action recognition device 800 is used to extract the multiple video frames containing the target object from the acquired video data to be tested according to the temporal features; determine multiple color value matrices based on the color value of each pixel point in the multiple video frames corresponding to the predetermined color value space; and perform normalization, convolution and pooling processing on the multiple color value matrices to obtain the multiple feature matrices.

[0192] In an embodiment of the present invention, processing unit 803 processes acquired video data to be tested to obtain multiple feature matrices. The multiple feature matrices are obtained by processing multiple video frames containing a target object. Processing unit 803 processes the multiple feature matrices using a temporal feature extraction network model to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices. The network layer in each network block of the temporal feature extraction network model fuses, reuses, and groups the preceding feature matrices. The preceding feature matrices include: the feature matrix input to the network block corresponding to the network layer, and the feature matrix output by the preceding network layer sorted before the network layer. Determination unit 804 determines the action recognition results of the target object in the multiple video frames using the multiple temporal action feature matrices. Because the network layer fuses and reuses the preceding feature matrices in the corresponding network block, it greatly utilizes the transmission between features, prevents the loss of more detailed information, avoids network degradation, and thus improves recognition accuracy. Furthermore, because the network layer groups the feature matrices, thereby reducing computing resource usage, this solution can reduce computing resource usage while also improving recognition accuracy.

[0193] It should be noted that, in the embodiment of the present invention, if the above-mentioned motion recognition method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a motion recognition device (which can be a personal computer, etc.) to execute all or part of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present invention is not limited to any specific combination of hardware and software.

[0194] Correspondingly, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0195] Correspondingly, an embodiment of the present invention provides a motion recognition device 800, including a memory 802 and a processor 801, wherein the memory 802 stores a computer program that can be run on the processor 801, and the processor 801 implements the steps in the above method when executing the program.

[0196] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of the present invention, please refer to the description of the method embodiments of the present invention for understanding.

[0197] It should be noted that Figure 18 A hardware entity diagram of a motion recognition device provided by an embodiment of the present invention Figure 1 ,like Figure 18 As shown, the hardware entity of the motion recognition device 800 includes: a processor 801 and a memory 802, wherein;

[0198] The processor 801 generally controls the overall operation of the motion recognition device 800 .

[0199] The memory 802 is configured to store instructions and applications executable by the processor 801, and can also cache data to be processed or processed by the processor 801 and each module in the motion recognition device 800 (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0200] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present invention. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention. The serial numbers of the above-mentioned embodiments of the present invention are for description only and do not represent the advantages and disadvantages of the embodiments.

[0201] It should be noted that, in this document, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0202] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0203] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0204] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0205] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0206] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0207] The above description is merely an embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.< / t> < / t> < / t> < / t>

Claims

1. A method for motion recognition, characterized in that: include: Processing the acquired video data to be tested to obtain multiple feature matrices; The multiple feature matrices are obtained by processing multiple video frames containing the target object within a predetermined time period; The multiple feature matrices are processed using a temporal feature extraction network model to obtain multiple temporal action feature matrices corresponding to the multiple feature matrices; wherein, the network layer in each network block in the temporal feature extraction network model fuses, reuses and groups the preceding feature matrices; the preceding feature matrix includes: the temporal feature matrix input by the network block corresponding to the network layer, and the corresponding group feature matrix output by the preceding network layer sorted in front of the network layer; the action recognition results of the target object in the multiple video frames are determined through the multiple temporal action feature matrices.

2. The motion recognition method according to claim 1, characterized in that The method of processing the plurality of feature matrices using a temporal feature extraction network model to obtain a plurality of temporal action feature matrices corresponding to the plurality of feature matrices includes: Using each network layer in the first network block, the plurality of feature matrices and the feature matrix output by the preceding network layer are fused, reused, and grouped to obtain a first group of time series feature matrices output by the first network block for processing by the second network block; Until each network layer in the mth network block is used to fuse, reuse and group the m-1th group of timing feature matrices and the feature matrix output by the preceding network layer to obtain the multiple timing action feature matrices; the timing feature extraction network model includes: m network blocks; m is a positive integer greater than or equal to 2; the m-1th group of timing feature matrices is the feature matrix output by the m-1th network block.

3. The motion recognition method according to claim 2, characterized in that The method of using each network layer in the mth network block to fuse, reuse, and group the m-1th group of time series feature matrices and the feature matrices output by the preceding network layer to obtain the multiple time series action feature matrices includes: Using the first network layer to perform group convolution processing on the m-1th group of time series feature matrices to obtain a first group of feature matrices, reducing the dimension of the first group of feature matrices and fusing them with the m-1th group of time series feature matrices to obtain a first group of intermediate feature matrices for processing by the second network layer; The i-th network layer performs group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices. After dimensionality reduction, the i-th group of feature matrices is fused with the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices to obtain the i-th group of intermediate feature matrices for processing by the i+1th network layer. The first i-1 groups of intermediate feature matrices are feature matrices outputted by the first i-1 network layers respectively. The preceding network layers include: the first i-1 network layers; i is an integer greater than 1 and less than n. Until the n-1th group of intermediate feature matrices is subjected to group convolution processing using the n-1th network layer to obtain the multiple temporal action feature matrices; the n-1th group of intermediate feature matrices is the feature matrix output by the n-1th network layer; the m-th network block includes: n network layers; n is a positive integer greater than or equal to 2.

4. The motion recognition method according to claim 3, characterized in that The i-th group of intermediate feature matrices is subjected to group convolution processing by the i-th network layer to obtain the i-th group of feature matrices, and the i-th group of feature matrices is subjected to dimensionality reduction and then fused with the m-1-th group of time series feature matrices and the first i-1 groups of intermediate feature matrices to obtain the i-th group of intermediate feature matrices, including: Using each convolution layer in the i-th network layer to perform group convolution processing on the i-1th group of intermediate feature matrices to obtain the i-th group of feature matrices; Calculate a matrix obtained by combining the m-1th group of time series feature matrices and the first i-1 groups of intermediate feature matrices, and multiply the matrix by a predetermined coefficient to obtain a target matrix; After reducing the dimension of the i-th group of feature matrices according to half the number of input channels of the i+1-th network layer, the matrix is ​​fused with the target matrix to obtain the i-th group of intermediate feature matrices.

5. The motion recognition method according to claim 4, characterized in that: The convolutional layers include: a first convolutional layer, a second convolutional layer, and a third convolutional layer; The method of performing group convolution processing on the i-1th group of intermediate feature matrices using each convolution layer in the i-th network layer to obtain the i-th group of feature matrices includes: Using the first convolutional layer to process the i-1th group of intermediate feature matrices to obtain a plurality of i-th matrices to be grouped; Dividing the plurality of i-th matrices to be grouped into a plurality of i-th group matrices, and using the second convolutional layer to process each group of matrices in the plurality of i-th groups of matrices respectively, to obtain an i-th feature matrix to be fused corresponding to each group of matrices; After fusing multiple i-th feature matrices to be fused, the third convolutional layer is used to process them to obtain the i-th group of feature matrices.

6. The motion recognition method according to claim 1, characterized in that: Determining the action recognition results of the target object in the multiple video frames by using the multiple temporal action feature matrices includes: Performing pooling and dimensionality reduction processing on the multiple time series action feature matrices to obtain multiple intermediate time series feature matrices; Processing the plurality of intermediate time series feature matrices using a preset network model to obtain a final time series feature matrix; Processing the final time series feature matrix using a preset function to obtain multiple probability values; The action recognition result is determined in a matching database according to the rank of the maximum probability value among the multiple probability values; the matching database pre-stores mapping information between the ranks of the multiple probability values ​​and the corresponding action recognition results.

7. The motion recognition method according to claim 1, characterized in that: The obtained video data to be tested is processed to obtain multiple feature matrices, including: Extracting the plurality of video frames containing the target object from the acquired video data to be tested according to the time sequence features; Determining a plurality of color value matrices according to the color value of each pixel in the plurality of video frames corresponding to a predetermined color value space; Normalization, convolution, and pooling are performed on the multiple color value matrices to obtain the multiple feature matrices.

8. A motion recognition device, characterized in that: include: A processing unit is used to process the acquired video data to be tested to obtain a plurality of feature matrices; the plurality of feature matrices are obtained by processing a plurality of video frames containing the target object within a predetermined time period; The processing unit is further configured to process the plurality of feature matrices using a temporal feature extraction network model to obtain a plurality of temporal action feature matrices corresponding to the plurality of feature matrices; wherein the network layer in each network block in the temporal feature extraction network model performs fusion, reuse, and grouping processing on the preceding feature matrix; the preceding feature matrix includes: the temporal feature matrix input by the network block corresponding to the network layer, and the corresponding group feature matrix output by the preceding network layer sorted before the network layer; A determination unit is configured to determine the action recognition results of the target object in the multiple video frames using the multiple temporal action feature matrices.

9. A motion recognition device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images