A motion recognition system, method, device and model training method and device

Abstract

The present invention provides an action recognition system, method, device and model training method and device, wherein the action recognition system includes: a bandpass filtering module for extracting a dynamic feature map according to multiple frames of continuous images in a segment; a static feature extraction module for obtaining a static feature map according to multiple frames of continuous images in a segment; a static feature difference and feature displacement network for performing feature displacement operations on the static feature map, and calculating the difference features between the static feature maps corresponding to each segment, to obtain static classification features; a dynamic feature difference and feature displacement network for performing feature displacement operations on the dynamic feature map, and calculating the difference features between the dynamic feature maps corresponding to each segment, to obtain dynamic classification features; a classification network for obtaining action recognition results according to the static classification features and the dynamic classification features. By implementing the present invention, an action recognition model can be obtained by training with less data, and accurate recognition of actions can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an action recognition system, method, device, and model training method and device. Background Art

[0002] Surveillance cameras are very popular nowadays. Whether in companies, factories, shopping malls, roads, or train stations, you can see surveillance cameras everywhere. However, it is difficult to achieve the purpose of real-time monitoring of violations and abnormal behaviors by relying solely on cameras. When abnormal behaviors occur, it is also very time-consuming and labor-intensive to review surveillance videos frame by frame, and it is easy to miss them. If real-time detection of specific abnormal behaviors can be performed through motion recognition technology, it can greatly save manpower and material resources and improve efficiency. Therefore, motion recognition has important practical value.

[0003] The action recognition algorithm of the video needs to extract the time information between the video frames, and the network model needs to have the ability of time modeling. The action recognition technology based on deep learning is mainly divided into: the method based on the two-stream network and the method based on the 3D convolution network. Among them, the method based on the two-stream network uses optical flow as the time information, which needs to be calculated in advance and stored in the local hard disk. For large data sets, it often takes up a lot of memory. At the same time, due to the need to calculate the optical flow in advance, the real-time effect of the method based on the two-stream network is also poor. The method based on the 3D convolution network relies on 3D convolution to achieve the effect of time modeling, which requires more video frames as input, has a large number of parameters, and requires a lot of computing power to train the network, and is also difficult to deploy. In recent years, with the success of Transformer in the fields of natural language processing and computer vision, many researchers have also applied Transformer to the field of video action recognition and achieved good results. However, the number of Transformer parameters is huge, and it often requires a large amount of data to fit, so its real-time performance and practical application are difficult to guarantee. Summary of the invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the defect in the prior art that a large amount of data is required to fit a model for identifying actions, thereby providing an action recognition system, method, device and model training method and device.

[0005] The first aspect of the present invention provides an action recognition system, comprising: an information separation network, a static feature difference and feature displacement network, a dynamic feature difference and feature displacement network, and a classification network. The information separation network comprises a bandpass filtering module and a static feature extraction module. The bandpass filtering module is used to extract a dynamic feature map based on multiple frames of continuous images in a segment; the static feature extraction module is used to perform time average pooling on multiple frames of continuous images in a segment to obtain a feature map, and to subtract the feature map from the dynamic feature map to obtain a static feature map; the static feature difference and feature displacement network is used to perform feature displacement operations on static feature maps corresponding to multiple segments, and calculate the difference features between the static feature maps corresponding to each segment to obtain static classification features; the dynamic feature difference and feature displacement network is used to perform feature displacement operations on dynamic feature maps corresponding to multiple segments, and calculate the difference features between the dynamic feature maps corresponding to each segment to obtain dynamic classification features; the classification network is used to obtain action recognition results based on static classification features and dynamic classification features.

[0006] Optionally, in the action recognition system provided by the present invention, the bandpass filtering module includes a spatial convolution layer and a temporal convolution layer.

[0007] Optionally, in the action recognition system provided by the present invention, the static feature difference and feature displacement network, and / or the dynamic feature difference and feature displacement network includes: an image segmentation module, an initial feature extraction module, and at least one intermediate feature extraction module, the image segmentation module is used to segment the input feature map according to a first preset size to obtain a first feature vector; the initial feature extraction module includes a linear embedding submodule and at least one feature difference and feature displacement submodule, the linear embedding submodule is used to transform the first feature vector according to a preset number of channels to obtain a second feature vector; the feature difference and feature displacement submodule is used to perform a feature displacement operation on the second feature vector, and calculate the difference features between the second feature vectors corresponding to each segment to obtain an initial classification feature; the intermediate feature extraction module includes a feature merging submodule and at least one feature difference and feature displacement submodule, the feature merging submodule is used to merge the initial classification features according to the second preset size to obtain a third feature vector; the feature difference and feature displacement submodule is used to perform a feature displacement operation on the third feature vector, and calculate the difference features between the third feature vectors corresponding to each segment to obtain a classification feature.

[0008] Optionally, in the action recognition system provided by the present invention, the static feature difference and feature displacement network, and / or the dynamic feature difference and feature displacement network includes three intermediate feature extraction modules: a first intermediate feature extraction module, a second intermediate feature extraction module, and a third intermediate feature extraction module, and the image segmentation module, the initial feature extraction module, the first intermediate feature extraction module, the second intermediate feature extraction module, and the third intermediate feature extraction module are connected in sequence; the number of feature difference and feature displacement sub-modules in the initial feature extraction module, the first intermediate feature extraction module, and the third intermediate feature extraction module is the same; the number of feature difference and feature displacement sub-modules in the second intermediate feature extraction module is greater than the number of feature difference and feature displacement sub-modules in the initial feature extraction module, the first intermediate feature extraction module, and the third intermediate feature extraction module.

[0009] Optionally, in the action recognition system provided by the present invention, the feature difference and feature displacement submodule includes a first normalization layer, a feature displacement unit, a feature difference unit, a second normalization layer, a first fully connected layer, a first GELU function layer, and a second fully connected layer, wherein the first normalization layer, the feature displacement unit, the feature difference unit, the second normalization layer, the first fully connected layer, the first GELU function layer, and the second fully connected layer are connected in sequence; the input data of the second normalization layer is the residual calculation result of the input data of the first normalization layer and the output data of the feature difference unit; the output data of the feature difference and feature displacement submodule is the residual calculation result of the input data of the second normalization layer and the output data of the second fully connected layer.

[0010] Optionally, in the action recognition system provided by the present invention, the feature displacement unit includes a first channel fully connected layer, a horizontal feature displacement layer, a second channel fully connected layer, a vertical feature displacement layer, a third channel fully connected layer, and a fourth channel fully connected layer, wherein the first channel fully connected layer is used to fully connect the channels of the input data to obtain a fully connected result, and input the fully connected result into the horizontal feature displacement layer and the vertical feature displacement layer respectively; the horizontal feature displacement layer is used to horizontally shift the fully connected result to obtain a horizontal displacement result, and input the horizontal displacement result into the second channel fully connected layer; the vertical feature displacement layer is used to vertically shift the fully connected result to obtain a vertical displacement result, and input the vertical displacement result into the third channel fully connected layer; the fourth channel fully connected layer is used to process the sum of the output results of the second channel fully connected layer and the third channel fully connected layer to obtain the output result of the feature displacement unit.

[0011] Optionally, in the action recognition system provided by the present invention, the feature difference unit includes: an input layer, a maximum pooling layer, a third fully connected layer, a second GELU function layer, a fourth fully connected layer, an upsampling layer, a fifth fully connected layer, a third GELU function layer, a sixth fully connected layer, and a feature difference output layer, wherein the input layer, the maximum pooling layer, the third fully connected layer, the second GELU function layer, the fourth fully connected layer, the upsampling layer, and the feature difference output layer are connected in sequence; the input layer, the fifth fully connected layer, the third GELU function layer, the sixth fully connected layer, and the feature difference output layer are connected in sequence; the input layer is used to subtract the input feature corresponding to the current moment segment from the input feature corresponding to the previous moment segment, and input the difference feature into the maximum pooling layer and the fifth fully connected layer respectively; the feature difference output layer is used to sum the output results of the upsampling layer and the sixth fully connected layer to obtain a sum result; the sum result is point-by-point multiplied with the input feature corresponding to the previous moment segment to obtain a multiplication result; the multiplication result is added to the input feature corresponding to the previous moment segment to obtain the output result of the feature difference unit.

[0012] Optionally, in the action recognition system provided by the present invention, the classification network includes a first time average pooling layer, a second time average pooling layer, a static feature classifier, a dynamic feature classifier, and an output layer. The first time average pooling layer is used to perform time average pooling on the static classification features corresponding to multiple segments, and input the pooling results into the static feature classifier; the second time average pooling layer is used to perform time average pooling on the dynamic classification features corresponding to multiple segments, and input the pooling results into the dynamic feature classifier; the static feature classifier is used to obtain a first classification result based on the static classification features; the dynamic feature classifier is used to obtain a second classification result based on the dynamic classification features; the recognition result output layer is used to output the weighted average result of the first classification result and the second classification result.

[0013] The second aspect of the present invention provides a method for training an action recognition model, comprising: obtaining multiple image sequences, in which the pedestrian action types are marked; dividing each image sequence into multiple sub-sequences to obtain a training data set; inputting the training data set into a neural network system, training the neural network system until the loss value of the loss function meets the loss condition, and obtaining an action recognition model, wherein the neural network system is the action recognition system provided by the first aspect of the present invention.

[0014] Optionally, in the action recognition model training method provided by the present invention, the loss function is obtained by combining an orthogonal projection loss function and a cross entropy loss function.

[0015] Optionally, in the action recognition model training method provided by the present invention, the loss function is: L = L ce +β×L opl , where Lce represents the cross entropy loss function, L opl represents the orthogonal projection loss function, and β represents the hyperparameter that controls the weight of the orthogonal projection loss.

[0016] The third aspect of the present invention provides an action recognition method, comprising: obtaining an image sequence of a target object, dividing the image sequence into a plurality of subsequences; inputting the subsequences into an action recognition model to generate an action recognition result, wherein the action recognition model is trained by the action recognition model training method provided by the second aspect of the present invention.

[0017] Optionally, in the action recognition method provided by the present invention, after the step of dividing the image sequence into a plurality of sub-sequences and before the step of inputting the sub-sequences into the action recognition model, the method further includes: proportionally scaling the images in the image sequence to obtain scaled images, wherein the size of the short side of the scaled images is within a preset interval; randomly cropping the scaled images to obtain cropped images, wherein the size of the cropped images satisfies preset conditions; and using the cropped images as the images in the sub-sequences to execute the step of inputting the sub-sequences into the action recognition model.

[0018] The fourth aspect of the present invention provides an action recognition model training device, including: an image acquisition module, used to acquire multiple image sequences, the image sequences are marked with pedestrian action types; a training data acquisition module, used to divide each image sequence into multiple sub-sequences to obtain a training data set; a model training module, used to input the training data set into a neural network system, train the neural network system, and obtain an action recognition model, the neural network system is the action recognition system provided by the first aspect of the present invention.

[0019] The fifth aspect of the present invention provides an action recognition device, comprising: an image acquisition module, used to obtain an image sequence of a target object and divide the image sequence into multiple sub-sequences; an action recognition module, used to input the sub-sequences into an action recognition model to generate an action recognition result, and the action recognition model is trained by the action recognition model training method provided by the second aspect of the present invention.

[0020] The sixth aspect of the present invention provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, thereby executing the action recognition system provided in the first aspect of the present invention, or, executing the action recognition model training method provided in the second aspect of the present invention, or, executing the action recognition method provided in the third aspect of the present invention.

[0021] The seventh aspect of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the action recognition system provided by the first aspect of the present invention, or to execute the action recognition model training method provided by the second aspect of the present invention, or to execute the action recognition method provided by the third aspect of the present invention.

[0022] The technical solution of the present invention has the following advantages:

[0023] 1. The motion recognition system provided by the present invention includes an information separation network, a static feature difference and feature displacement network, a dynamic feature difference and feature displacement network, and a classification network. First, the dynamic feature map and the static feature map in the image are separated by the information separation network, and then the dynamic feature map and the static feature map are respectively input into the dynamic feature difference and feature displacement network and the static feature difference and feature displacement network. By analyzing the dynamic feature map separately, the short-term time information of the video can be captured. By analyzing the static feature map separately, the static scene in the video can be identified. The static feature difference and feature displacement network and the dynamic feature difference and feature displacement network perform displacement operations on the feature map, and calculate the difference features between the feature maps corresponding to each segment to obtain classification features. By performing displacement operations on the feature map, the network can obtain spatial local information with less computational effort, thereby ensuring the running speed of the network. By calculating the difference features between the feature maps, the long-term time relationship in the video can be captured, so that the network has the ability of time modeling, thereby ensuring the accuracy of network motion recognition. It can be seen that the motion recognition system provided by the present invention can use less data training to obtain the motion recognition model, and the motion recognition model obtained by training the motion recognition system can realize accurate recognition of the action.

[0024] 2. The motion recognition model training method and device provided by the present invention, after obtaining the training data set, input the training data set into the motion recognition system provided by the first aspect of the present invention, train the motion recognition system to obtain the motion recognition model, the motion recognition system provided by the first aspect of the present invention includes an information separation network, a static feature difference and feature displacement network, a dynamic feature difference and feature displacement network, and a classification network, the motion recognition system first separates the dynamic feature map and the static feature map in the image through the information separation network, and then inputs the dynamic feature map and the static feature map into the dynamic feature difference and feature displacement network and the static feature difference and feature displacement network respectively, by separately analyzing the dynamic feature map, it is possible to capture the short-term time information of the video, and by separately analyzing the static feature map, it is possible to The static scenes in the video can be recognized. The static feature difference and feature displacement network and the dynamic feature difference and feature displacement network perform displacement operations on the feature graphs, and calculate the difference features between the feature graphs corresponding to each segment to obtain classification features. By performing displacement operations on the feature graphs, the network can obtain spatial local information with less computational effort, thereby ensuring the network's operating speed. By calculating the difference features between the feature graphs, the long-term time relationship in the video can be captured, so that the network has the ability of time modeling, thereby ensuring the accuracy of network action recognition. It can be seen that the action recognition model training method provided by the present invention can use less data training to obtain the action recognition model, and the action recognition model obtained by training with the action recognition training method and device provided by the present invention can realize accurate recognition of actions.

[0025] 3. The action recognition method and device provided by the present invention divide the image sequence into multiple subsequences after acquiring the image sequence of the target object, and input the subsequences into the action recognition model trained by the action recognition model training method provided by the second aspect of the present invention. The action recognition model training method provided by the second aspect of the present invention obtains the action recognition model by training the action recognition system. The action recognition system first separates the dynamic feature map and the static feature map in the image through the information separation network, and then inputs the dynamic feature map and the static feature map into the dynamic feature difference and feature displacement network and the static feature difference and feature displacement network respectively. By analyzing the dynamic feature map separately, the short-term time information of the video can be captured. By analyzing the static feature map separately, the short-term time information of the video can be captured. Analysis shows that static scenes in videos can be identified. The static feature difference and feature displacement network and the dynamic feature difference and feature displacement network perform displacement operations on feature maps, and calculate the difference features between feature maps corresponding to each segment to obtain classification features. By performing displacement operations on feature maps, the network can obtain spatial local information with less computational effort, thereby ensuring the network's operating speed. By calculating the difference features between feature maps, the long-term time relationship in the video can be captured, so that the network has the ability of time modeling, thereby ensuring the accuracy of network action recognition. It can be seen that the action recognition model trained by the action recognition training method provided by the second aspect of the present invention can realize accurate recognition of actions. Therefore, accurate recognition of actions can be achieved by implementing the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 is a principle block diagram of a specific example of the action recognition system in an embodiment of the present invention;

[0028] Figure 2 A principle block diagram of a specific example of a static feature difference and feature displacement network and / or a dynamic feature difference and feature displacement network in an embodiment of the present invention;

[0029] Figure 3 is a principle block diagram of a specific example of a feature difference and feature shift submodule in an embodiment of the present invention;

[0030] Figure 4 is a principle block diagram of a specific example of a characteristic displacement unit in an embodiment of the present invention;

[0031] Figure 5 is a principle block diagram of a specific example of a characteristic difference unit in an embodiment of the present invention;

[0032] Figure 6 is a flowchart of a specific example of the method for training an action recognition model in an embodiment of the present invention;

[0033] Figure 7 is a flowchart of a specific example of the action recognition method in an embodiment of the present invention;

[0034] Figure 8 is a principle block diagram of a specific example of the action recognition model training device in an embodiment of the present invention;

[0035] Fig. 9 is a flowchart of a specific example of the action recognition method in an embodiment of the present invention;

[0036] Fig.10 It is a principle block diagram of a specific example of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] In the description of the present invention, it should be noted that the terms “first”, “second” and “third” are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0039] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0040] The embodiment of the present invention provides an action recognition system, such as Figure 1 As shown, it includes: an information separation network 11, a static feature difference and feature displacement network 12, a dynamic feature difference and feature displacement network 13, and a classification network 14. The information separation network 11 includes a bandpass filtering module 111 and a static feature extraction module 112.

[0041] The bandpass filtering module 111 is used to extract a dynamic feature map according to multiple frames of continuous images in a segment.

[0042] In an optional embodiment, before performing action recognition by the action recognition system provided by the embodiment of the present invention, it is necessary to first convert the collected video data containing the target object into an image sequence, and then divide the image sequence into N segments {P1, P2, ..., P N Finally, N segments of images are input into the action recognition system, and K consecutive frames of images are randomly sampled in each segment, and a total of T = N × K frames of images are obtained, where the number of frames of images in a segment can be set according to actual needs. For example, the value of N is 8 or 16, and the value of K is 3.

[0043] In an optional embodiment, when performing action recognition through the action recognition system provided by the embodiment of the present invention, the bandpass filtering module 111 extracts the dynamic feature map corresponding to each segment based on the continuous images in each segment in turn.

[0044] The static feature extraction module 112 is used to perform temporal average pooling on multiple frames of continuous images in a segment to obtain a feature map, and to subtract the feature map from the dynamic feature map to obtain a static feature map.

[0045] In an embodiment of the present invention, the static feature extraction module 112 first performs temporal average pooling on multiple frames of continuous images in a segment through a temporal average pooling layer to obtain a feature map with a time dimension of 1, and subtracts the feature map from the dynamic feature map corresponding to the segment to obtain a static feature map of the segment.

[0046] In an optional embodiment, when performing action recognition through the action recognition system provided by the embodiment of the present invention, the static feature extraction module 112 extracts the static feature graph corresponding to each segment based on the continuous images in each segment in turn.

[0047] The static feature difference and feature shift network 12 is used to perform feature shift operations on static feature maps corresponding to multiple segments, and calculate the difference features between the static feature maps corresponding to each segment to obtain static classification features.

[0048] In an optional embodiment, when the action recognition system provided by the embodiment of the present invention performs action recognition, the static feature difference and feature displacement network 12 performs feature displacement operations on the static feature graphs corresponding to each segment, that is, when the feature displacement operation is performed on the static feature graph corresponding to any segment, it will not be affected by the static feature graphs corresponding to other segments. However, when the static feature difference and feature displacement network 12 calculates the difference features between the static feature graphs corresponding to each segment, it is necessary to combine the static feature graphs corresponding to two adjacent segments to calculate the difference features.

[0049] The dynamic feature difference and feature displacement network 13 is used to perform feature displacement operations on the dynamic feature maps corresponding to multiple segments, and calculate the difference features between the dynamic feature map values ​​corresponding to each segment to obtain dynamic classification features.

[0050] In an optional embodiment, the dynamic feature difference and feature displacement network 13 performs feature displacement operations on the dynamic feature graphs corresponding to each segment, similar to the static feature difference and feature displacement network 12. That is, when the feature displacement operation is performed on the dynamic feature graph corresponding to any segment, it will not be affected by the static feature graphs corresponding to other segments. However, when the dynamic feature difference and feature displacement network 13 calculates the difference features between the dynamic feature graphs corresponding to each segment, it is necessary to combine the dynamic feature graphs corresponding to two adjacent segments to calculate the difference features.

[0051] The classification network 14 is used to obtain action recognition results based on static classification features and dynamic classification features.

[0052] In an optional embodiment, the classification network includes a classifier, and the classifier is used to analyze the static classification features and the dynamic classification features to obtain the action recognition result.

[0053] The action recognition system provided by the embodiment of the present invention includes an information separation network 11, a static feature difference and feature displacement network 12, a dynamic feature difference and feature displacement network 13, and a classification network 14. First, the dynamic feature map and the static feature map in the image are separated by the information separation network 11, and then the dynamic feature map and the static feature map are respectively input into the dynamic feature difference and feature displacement network 13 and the static feature difference and feature displacement network 12. By analyzing the dynamic feature map separately, the short-term time information of the video can be captured. By analyzing the static feature map separately, the static scene in the video can be identified. The static feature difference and feature displacement network 12 and the dynamic feature map are respectively input into the dynamic feature difference and feature displacement network 13 and the static feature difference and feature displacement network 12. The difference and feature displacement network 13 performs displacement operations on the feature maps, and calculates the difference features between the feature maps corresponding to each segment to obtain classification features. By performing displacement operations on the feature maps, the network can obtain spatial local information with less computational effort, thereby ensuring the network's operating speed. By calculating the difference features between the feature maps, the long-term time relationship in the video can be captured, so that the network has the ability of time modeling, thereby ensuring the accuracy of network action recognition. It can be seen that the action recognition system provided by the embodiment of the present invention can obtain an action recognition model using less data training, and the action recognition model obtained by training the action recognition system can realize accurate recognition of actions.

[0054] In an optional embodiment, the bandpass filtering module 111 includes a spatial convolution layer and a temporal convolution layer.

[0055] For each K consecutive frames in a segment, define P(t,x,y) as the pixel value, where x and y represent the spatial dimension and t represents the time dimension. P(t,x,y) corresponds to the pixel value at (x,y) of the tth frame. Then the output F(t,x,y) of the bandpass filter is:

[0056]

[0057] Among them, * represents the convolution operation, LoG μ (x,y) represents the Laplacian of Gaussian with parameter μ:

[0058]

[0059] The second-order derivative with respect to t is approximated by the finite difference numerical h(i) as follows:

[0060]

[0061] Then formula (1) can be expressed as:

[0062]

[0063] Where K represents the number of image frames in a segment, and “·” represents multiplication.

[0064] It can be seen from formula (4) that the bandpass filter function is completely differentiable. In order to improve robustness, two consecutive convolutional layers are used in the embodiment of the present invention to implement the bandpass filter module 111. Therefore, the bandpass filter module 111 in the embodiment of the present invention is trainable. After the convolutional layer is used, formula (4) is restated as follows:

[0065]

[0066] in A spatial convolution layer with a convolution kernel size of k×k is used, and the Laplacian operator with parameter μ is used to initialize the parameters. The sum of the parameter values ​​of the convolution kernel is normalized to 1. A temporal convolution layer with a time step of s is used, and the convolution kernel value is initialized to

[0067] In an optional embodiment, the static feature difference and feature displacement network 12 and the dynamic feature difference and feature displacement network 13 may have the same network structure or different network structures. When the static feature difference and feature displacement network 12 and the dynamic feature difference and feature displacement network 13 have the same network structure, their network parameter values ​​are different.

[0068] In an alternative embodiment, if Figure 2As shown, the static feature difference and feature displacement network 12 and / or the dynamic feature difference and feature displacement network 13 include: an image segmentation module 121 , an initial feature extraction module 122 , and at least one intermediate feature extraction module 123 .

[0069] The image segmentation module 121 is used to segment the input feature map according to a first preset size to obtain a first feature vector.

[0070] In an optional embodiment, the input data of the static feature difference and feature displacement network 12 is a static feature map, and the image segmentation module 121 in the static feature difference and feature displacement network 12 segments the static feature map; the input data of the dynamic feature difference and feature displacement network 13 is a dynamic feature map, and the image segmentation module 121 in the dynamic feature difference and feature displacement network 13 segments the dynamic feature map.

[0071] In an alternative embodiment, for an input RGB image Where H and W represent the size of the image, and 3 represents the number of channels of the image. The image segmentation module 121 segments the image into blocks of size 4×4, and synthesizes each 4×4 block obtained by segmentation into a vector, and obtains a feature size of in It represents the number of blocks and 48 is the number of channels.

[0072] The initial feature extraction module 122 includes a linear embedding submodule 1221 and at least one feature difference and feature displacement submodule 1222. The linear embedding submodule 1221 is used to transform the first feature vector according to a preset number of channels to obtain a second feature vector; the feature difference and feature displacement submodule 1222 is used to perform a feature displacement operation on the second feature vector, and calculate the difference features between the second feature vectors corresponding to each segment to obtain the initial classification feature.

[0073] In an optional embodiment, when the image segmentation module 121 segments the input feature map into blocks of size 4×4 and obtains blocks of size When the first eigenvector of the linear embedding submodule 1221 is projected to Where C represents the number of channels.

[0074] In an optional embodiment, the initial feature extraction module 122 includes two continuous feature difference and feature displacement submodules 1222. After the first feature vector is processed by the linear embedding submodule 1221 to obtain the second feature vector, the second feature vector is processed by the two continuous feature difference and feature displacement submodules 1222 to obtain the initial classification feature.

[0075] The intermediate feature extraction module 123 includes a feature merging submodule 1231 and at least one feature difference and feature displacement submodule 1222. The feature merging submodule 1231 is used to merge the initial classification features according to a second preset size to obtain a third feature vector; the feature difference and feature displacement submodule 1222 is used to perform a feature displacement operation on the third feature vector, and calculate the difference features between the third feature vectors corresponding to each segment to obtain the classification feature.

[0076] In an optional embodiment, the feature merging submodule 1231 in the intermediate feature extraction module 123 merges the initial classification features obtained in the previous stage into blocks of 2×2 size to synthesize a vector, and obtains a feature size of Then it passes through at least one feature difference and feature displacement submodule 1222 and is output.

[0077] In an optional embodiment, when the static feature difference and feature displacement network 12, and / or the dynamic feature difference and feature displacement network 13 include multiple intermediate feature extraction modules 123, the number of feature difference and feature displacement sub-modules 1222 in different intermediate feature extraction modules 123 may be the same or different. For example, the number of feature difference and feature displacement sub-modules 1222 in the intermediate feature extraction module 123 may be 2, 6, etc.

[0078] In an optional embodiment, the static feature difference and feature displacement network 12, and / or the dynamic feature difference and feature displacement network 13 includes three intermediate feature extraction modules 123: a first intermediate feature extraction module 123, a second intermediate feature extraction module 123, and a third intermediate feature extraction module 123.

[0079] The image segmentation module 121, the initial feature extraction module 122, the first intermediate feature extraction module 123, the second intermediate feature extraction module 123, and the third intermediate feature extraction module 123 are connected in sequence. That is, in the embodiment of the present invention, the output data of the image segmentation module 121 is the input data of the initial feature extraction module 122, the output data of the initial feature extraction module 122 is the input data of the first intermediate feature extraction module 123, the output data of the first intermediate feature extraction module 123 is the input data of the second intermediate feature extraction module 123, and the output data of the second intermediate feature extraction module 123 is the input data of the third intermediate feature extraction module 123.

[0080] The number of feature differences in the initial feature extraction module 122 , the first intermediate feature extraction module 123 , and the third intermediate feature extraction module 123 is the same as the number of feature displacement submodules 1222 .

[0081] The number of feature difference and feature displacement submodules 1222 in the second intermediate feature extraction module 123 is greater than the number of feature difference and feature displacement submodules 1222 in the initial feature extraction module 122 , the first intermediate feature extraction module 123 , and the third intermediate feature extraction module 123 .

[0082] In an optional embodiment, the number of feature difference and feature displacement submodules 1222 in the initial feature extraction module 122, the first intermediate feature extraction module 123, and the third intermediate feature extraction module 123 is 2, and the number of feature difference and feature displacement submodules 1222 in the second intermediate feature extraction module 123 is 6.

[0083] In an optional embodiment, when the static feature difference and feature displacement network 12 and the dynamic feature difference and feature displacement network 13 include three intermediate feature extraction modules 123, the static feature difference and feature displacement network 12 and the dynamic feature difference and feature displacement network 13 finally output feature sizes of

[0084] In an alternative embodiment, if Figure 3 As shown, the feature difference and feature displacement submodule 1222 includes a first normalization layer, a feature displacement unit, a feature difference unit, a second normalization layer, a first fully connected layer, a first GELU function layer, and a second fully connected layer, wherein the first normalization layer, the feature displacement unit, the feature difference unit, the second normalization layer, the first fully connected layer, the first GELU function layer, and the second fully connected layer are connected in sequence.

[0085] The input data of the second normalization layer is the residual calculation result of the input data of the first normalization layer and the output data of the feature difference unit.

[0086] The output data of the feature difference and feature displacement submodule 1222 is the residual calculation result of the input data of the second normalization layer and the output data of the second fully connected layer.

[0087] In an alternative embodiment, if Figure 4 As shown, the feature displacement unit includes a first channel fully connected layer, a horizontal feature displacement layer, a second channel fully connected layer, a vertical feature displacement layer, a third channel fully connected layer, and a fourth channel fully connected layer. The horizontal feature displacement layer is connected to the second channel fully connected layer, and the vertical feature displacement layer is connected to the fourth channel fully connected layer. The structure formed by connecting the horizontal feature displacement layer to the second channel fully connected layer and the structure formed by connecting the vertical feature displacement layer to the fourth channel fully connected layer are parallel structures.

[0088] The first channel fully connected layer is used to fully connect the channels of the input data to obtain a fully connected result, and input the fully connected result into the horizontal feature displacement layer and the vertical feature displacement layer respectively.

[0089] The horizontal feature displacement layer is used to horizontally shift the fully connected result to obtain a horizontal displacement result, and input the horizontal displacement result into the second channel fully connected layer.

[0090] In an optional embodiment, the fully connected result has three dimensions: height, width, and channel. When the fully connected result is horizontally shifted, if the number of shift groups is 3 and the shift size is 1, then each group of channel feature graphs in the channel dimension is horizontally shifted in the manner of [+1, 0, -1], and the empty part is filled with zeros. Specifically, the fully connected result is divided into 3 groups of data in the channel dimension, the first group of data is horizontally shifted to the left by a unit length, and after the first group of data is moved, the empty part is filled with zeros, the third group of data remains unchanged, the second group of data is horizontally shifted to the right by a unit length, and after the third group of data is moved, the empty part is filled with zeros. If the number of shift groups is 5 and the shift size is 2, then each group of channel feature graphs in the channel dimension is horizontally shifted in the manner of [+4, +2, 0, -2, -4], and the empty part is filled with zeros.

[0091] The vertical feature displacement layer is used to vertically displace the fully connected result to obtain a vertical displacement result, and input the vertical displacement result into the third channel fully connected layer.

[0092] The difference between performing a vertical displacement on the fully connected result and performing a horizontal displacement on the fully connected result is that the moving direction of the vertical displacement is the vertical direction, and the moving direction of the horizontal displacement is the horizontal direction.

[0093] The fourth channel fully connected layer is used to process the sum of the output results of the second channel fully connected layer and the third channel fully connected layer to obtain the output result of the feature shift unit.

[0094] In an alternative embodiment, if Figure 5 As shown, the feature difference unit includes: an input layer, a maximum pooling layer, a third fully connected layer, a second GELU function layer, a fourth fully connected layer, an upsampling layer, a fifth fully connected layer, a third GELU function layer, a sixth fully connected layer, and a feature difference output layer.

[0095] The input layer, the maximum pooling layer, the third fully connected layer, the second GELU function layer, the fourth fully connected layer, the upsampling layer, and the feature difference output layer are connected in sequence.

[0096] The input layer, the fifth fully connected layer, the third GELU function layer, the sixth fully connected layer, and the feature difference output layer are connected in sequence.

[0097] The input layer is used to subtract the input features corresponding to the current segment from the input features corresponding to the previous segment, and input the difference features into the maximum pooling layer and the fifth fully connected layer respectively.

[0098] The feature difference output layer is used to sum the output results of the upsampling layer and the sixth fully connected layer to obtain the sum result; multiply the sum result point by point with the input feature corresponding to the segment at the previous moment to obtain the multiplication result; add the multiplication result to the input feature corresponding to the segment at the previous moment to obtain the output result of the feature difference unit.

[0099] Specifically, for the input features [F1, F2, ..., F N ],in The feature at time t is the difference between the feature at time t and the feature at time t+1, and then it is divided into two paths. One path is downsampled through the maximum pooling layer, then passes through two fully connected layers with a GELU function in the middle, and then upsampled through the upsampling layer; the other path directly passes through two fully connected layers with a GELU function in the middle. Finally, the feature maps output by the two paths are added, multiplied point by point with the original input features, and then added to get the final output.

[0100] In an alternative embodiment, if Figure 1 As shown, in the action recognition system provided by the embodiment of the present invention, the classification network 14 includes a first time average pooling layer 141, a second time average pooling layer 143, a static feature classifier 142, a dynamic feature classifier 144, and an output layer 145.

[0101] The first temporal average pooling layer 141 is used to perform temporal average pooling on the static classification features corresponding to the multiple segments, and input the pooling results into the static feature classifier 142 .

[0102] The second time average pooling layer 143 is used to perform time average pooling on the dynamic classification features corresponding to the multiple segments, and input the pooling results into the dynamic feature classifier 144.

[0103] In an optional embodiment, when performing action recognition through the action recognition system provided by the embodiment of the present invention, if the image sequence corresponding to the video is divided into N segments, the N segments are processed in sequence by the information separation network 11, the static feature difference and feature displacement network 12, and the dynamic feature difference and feature displacement network 13 to obtain N static classification features and N dynamic classification features. The first time average pooling layer 141 is used to perform time average pooling on the N static classification features to obtain a static classification feature with a time attribute. The second time average pooling layer 143 is used to perform time average pooling on the N dynamic classification features to obtain a dynamic classification feature with a time attribute. The static classification features and dynamic classification features with time attributes can more accurately realize the recognition of actions.

[0104] The static feature classifier 142 is used to obtain a first classification result according to the static classification feature.

[0105] The dynamic feature classifier 144 is used to obtain a second classification result according to the dynamic classification feature.

[0106] The recognition result output layer 145 is used to output the weighted average result of the first classification result and the second classification result as the output result.

[0107] In an optional embodiment, the static feature classifier 142 and the dynamic feature classifier 144 are Softmax classifiers.

[0108] The embodiment of the present invention provides a method for training an action recognition model. Figure 6 As shown, including:

[0109] Step S21: Acquire multiple image sequences, where the image sequences are marked with pedestrian action types.

[0110] Step S22: Divide each image sequence into multiple sub-sequences to obtain a training data set.

[0111] Step S23: Input the training data set into the neural network system, train the neural network system until the loss value of the loss function meets the loss condition, and obtain the action recognition model. The neural network system is the action recognition system provided in any of the above embodiments. For details about the action recognition system, please refer to the above embodiments.

[0112] In an optional embodiment, the initialized action recognition system is trained to obtain an action recognition model. In the action recognition system, the bandpass filter module 111 is initialized using the Gaussian Laplacian operator, and the feature difference and feature displacement network are initialized using a pre-trained model. The pre-trained model is obtained by pre-training using ImageNet or other large data sets.

[0113] In an optional embodiment, the action recognition system can be fully trained using a large-scale dataset, or it can be fine-tuned based on a pre-trained model.

[0114] The action recognition model training method provided by the embodiment of the present invention, after obtaining the training data set, inputs the training data set into the action recognition system provided by the above embodiment, trains the action recognition system to obtain the action recognition model, the action recognition system provided by the above embodiment includes an information separation network, a static feature difference and feature displacement network, a dynamic feature difference and feature displacement network, and a classification network, the action recognition system first separates the dynamic feature map and the static feature map in the image through the information separation network, and then inputs the dynamic feature map and the static feature map into the dynamic feature difference and feature displacement network and the static feature difference and feature displacement network respectively, by separately analyzing the dynamic feature map, it is possible to capture the short-term time information of the video, and by separately analyzing the static feature map, it is possible to recognize the video In the static scene in the video, the static feature difference and feature displacement network and the dynamic feature difference and feature displacement network perform displacement operations on the feature maps, and calculate the difference features between the feature maps corresponding to each segment to obtain classification features. By performing displacement operations on the feature maps, the network can obtain spatial local information with less computational effort, thereby ensuring the network's operating speed. By calculating the difference features between the feature maps, the long-term temporal relationship in the video can be captured, so that the network has the ability of temporal modeling, thereby ensuring the accuracy of network action recognition. It can be seen that the action recognition model training method provided by the embodiment of the present invention can use less data training to obtain the action recognition model, and the action recognition model trained by the action recognition training method provided by the embodiment of the present invention can realize accurate recognition of actions.

[0115] In an optional embodiment, the loss function is obtained by combining an orthogonal projection loss function and a cross entropy loss function.

[0116] In an optional embodiment, the following formula is defined for a batch B:

[0117]

[0118]

[0119] where y i ,y j F represents the i-th and j-th labeled true values, and i,j∈B. i ,F j represents the features after temporal average pooling of the network output features of the N segments of the i-th and j-th videos, ||·||2 represents l2 regularization, and “·” represents the vector dot product. s and d represent the results of the cosine similarity operation on the intermediate features when the true value categories are consistent and inconsistent, respectively. The final orthogonal projection loss function L opl as follows:

[0120] L opl=(1-s)+α×|d|, (4)

[0121] Among them, α is a hyperparameter that controls the weight, and |·| represents the absolute value operation.

[0122] The above orthogonal projection loss function is combined with the cross entropy loss L ce The final loss function L is obtained by joint use:

[0123] L=L ce +β×L opl , (5)

[0124] Among them, β is a hyperparameter that controls the weight of the orthogonal projection loss.

[0125] In an embodiment of the present invention, the orthogonal projection loss function is used in conjunction with the cross entropy loss to obtain the final loss function. The orthogonal projection loss function is introduced to orthogonalize the intermediate layer features to achieve the effects of inter-class separation and intra-class clustering, so that the trained action recognition model can more accurately recognize actions.

[0126] In an optional embodiment, the above step S21 specifically includes:

[0127] First, the images in the image sequence are scaled in equal proportion to obtain scaled images, and the size of the short side of the scaled images is within a preset interval. For example, the preset interval may be [256, 320].

[0128] Then, the scaled image is randomly cropped to obtain a cropped image, and the size of the cropped image meets a preset condition. In an optional embodiment, the size of the cropped image is 224×224.

[0129] Finally, the image sequence formed by the cropped images is divided into multiple sub-sequences to obtain a training data set.

[0130] The embodiment of the present invention provides a method for action recognition, such as Figure 7 As shown, including:

[0131] Step S31: Acquire an image sequence of the target object, and divide the image sequence into multiple sub-sequences.

[0132] Step S32: input the subsequence into the action recognition model to generate an action recognition result. The action recognition model is trained by the action recognition model training method provided in the above embodiment. For details about the action recognition model, please refer to the description in the above embodiment, which will not be repeated here.

[0133] The action recognition method provided by the embodiment of the present invention divides the image sequence into multiple subsequences after acquiring the image sequence of the target object, and inputs the subsequences into the action recognition model trained by the action recognition model training method provided by the above embodiment. The action recognition model training method provided by the above embodiment obtains the action recognition model by training the action recognition system. The action recognition system first separates the dynamic feature map and the static feature map in the image through the information separation network, and then inputs the dynamic feature map and the static feature map into the dynamic feature difference and feature displacement network and the static feature difference and feature displacement network respectively. By analyzing the dynamic feature map separately, the short-term time information of the video can be captured, and by analyzing the static feature map separately, the It is capable of identifying static scenes in videos, the static feature difference and feature displacement network and the dynamic feature difference and feature displacement network perform displacement operations on feature maps, and calculate the difference features between feature maps corresponding to each segment to obtain classification features, and by performing displacement operations on feature maps, the network can obtain spatial local information with less computational effort, thereby ensuring the network's operating speed, and by calculating the difference features between feature maps, it is capable of capturing long-term temporal relationships in videos, so that the network has the ability of temporal modeling, thereby ensuring the accuracy of network action recognition, it can be seen that the action recognition model trained by the action recognition training method provided in the above embodiment can realize accurate recognition of actions, and therefore, accurate recognition of actions can be achieved by implementing the embodiments of the present invention.

[0134] In an optional embodiment, in the action recognition method provided in the embodiment of the present invention, after the above step S31 and before step S32, the method further includes:

[0135] First, the images in the image sequence are scaled proportionally to obtain scaled images, and the short side size of the scaled images is within a preset range. For details, please refer to the description in the above embodiment, which will not be repeated here.

[0136] Then, the scaled image is randomly cropped to obtain a cropped image, and the size of the cropped image meets a preset condition. For details, please refer to the description in the above embodiment, which will not be repeated here.

[0137] Finally, the cropped image is used as the image in the subsequence, and the step of inputting the subsequence into the action recognition model is performed. For details, please refer to the description in the above embodiment, which will not be repeated here.

[0138] In an embodiment of the present invention, the image is scaled proportionally so that the size of the short side of the image is within a preset range, and then randomly cropped to an input size accepted by the network, thereby achieving the purpose of data enhancement. The scaled and cropped data is analyzed, and the action recognition model can analyze the effective information in the image, thereby improving the analysis efficiency and the accuracy of the analysis results.

[0139] The embodiment of the present invention provides a motion recognition model training device, such as Figure 8 As shown, including:

[0140] The image acquisition module 21 is used to acquire multiple image sequences, in which the pedestrian action types are marked. For details, please refer to the description of step S21 in the above embodiment, which will not be repeated here.

[0141] The training data acquisition module 22 is used to divide each image sequence into multiple sub-sequences to obtain a training data set. For details, please refer to the description of step S22 in the above embodiment, which will not be repeated here.

[0142] The model training module 23 is used to input the training data set into the neural network system, train the neural network system, and obtain the action recognition model. The neural network system is the action recognition system provided in the above embodiment. For details, please refer to the description of step S23 in the above embodiment, which will not be repeated here.

[0143] The motion recognition model training device provided by the embodiment of the present invention, after acquiring the training data set, inputs the training data set into the motion recognition system provided by the above embodiment, trains the motion recognition system to obtain the motion recognition model, the motion recognition system provided by the above embodiment includes an information separation network, a static feature difference and feature displacement network, a dynamic feature difference and feature displacement network, and a classification network, the motion recognition system first separates the dynamic feature map and the static feature map in the image through the information separation network, and then inputs the dynamic feature map and the static feature map into the dynamic feature difference and feature displacement network and the static feature difference and feature displacement network respectively, by separately analyzing the dynamic feature map, it is possible to capture the short-term time information of the video, and by separately analyzing the static feature map, it is possible to recognize the video In the static scene in the video, the static feature difference and feature displacement network and the dynamic feature difference and feature displacement network perform displacement operations on the feature maps, and calculate the difference features between the feature maps corresponding to each segment to obtain classification features. By performing displacement operations on the feature maps, the network can obtain spatial local information with less computational effort, thereby ensuring the network's operating speed. By calculating the difference features between the feature maps, the long-term temporal relationship in the video can be captured, so that the network has the ability of temporal modeling, thereby ensuring the accuracy of network action recognition. It can be seen that the action recognition model training device provided by the embodiment of the present invention can use less data training to obtain the action recognition model, and the action recognition model trained by the action recognition training device provided by the embodiment of the present invention can realize accurate recognition of actions.

[0144] The embodiment of the present invention provides a motion recognition device, such as Fig. 9 As shown, including:

[0145] The image acquisition module 31 is used to obtain an image sequence of the target object and divide the image sequence into multiple sub-sequences. For details, please refer to the description of step S31 in the above embodiment, which will not be repeated here.

[0146] The action recognition module 32 is used to input the subsequence into the action recognition model to generate an action recognition result. The action recognition model is trained by the action recognition model training method provided in the above embodiment. For details, please refer to the description of step S32 in the above embodiment, which will not be repeated here.

[0147] The action recognition device provided by the embodiment of the present invention divides the image sequence into multiple sub-sequences after acquiring the image sequence of the target object, and inputs the sub-sequences into the action recognition model trained by the action recognition model training method provided by the above embodiment. The action recognition model training method provided by the above embodiment obtains the action recognition model by training the action recognition system. The action recognition system first separates the dynamic feature map and the static feature map in the image through the information separation network, and then inputs the dynamic feature map and the static feature map into the dynamic feature difference and feature displacement network and the static feature difference and feature displacement network respectively. By analyzing the dynamic feature map separately, the short-term time information of the video can be captured, and by analyzing the static feature map separately, the short-term time information of the video can be captured. It is capable of identifying static scenes in videos, the static feature difference and feature displacement network and the dynamic feature difference and feature displacement network perform displacement operations on feature maps, and calculate the difference features between feature maps corresponding to each segment to obtain classification features, and by performing displacement operations on feature maps, the network can obtain spatial local information with less computational effort, thereby ensuring the network's operating speed, and by calculating the difference features between feature maps, it is capable of capturing long-term temporal relationships in videos, so that the network has the ability of temporal modeling, thereby ensuring the accuracy of network action recognition, it can be seen that the action recognition model trained by the action recognition training method provided in the above embodiment can realize accurate recognition of actions, and therefore, accurate recognition of actions can be achieved by implementing the embodiments of the present invention.

[0148] An embodiment of the present invention provides a computer device, such as Fig.10 As shown, the computer device mainly includes one or more processors 41 and a memory 42. Fig.10 A processor 41 is taken as an example.

[0149] The computer device may further include: an input device 43 and an output device 44 .

[0150] The processor 41, the memory 42, the input device 43 and the output device 44 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0151] Processor 41 can be a central processing unit (CPU). Processor 41 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The memory 42 may include a storage program area and a storage data area, wherein the storage program area can store an operating system, an application required for at least one function; the storage data area can store data created according to the use of the action recognition system, or, the action recognition model training device, or, the action recognition device, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage devices. In some embodiments, the memory 42 optionally includes a memory remotely arranged relative to the processor 41, and these remote memories can be connected to the action recognition system, or, the action recognition model training device, or, the action recognition device through a network. The input device 43 can receive a calculation request (or other digital or character information) input by a user, and generate a key signal input related to the motion recognition system, or the motion recognition model training device, or the motion recognition device. The output device 44 can include a display device such as a display screen to output the calculation result.

[0152] The embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions, and the computer storage medium stores computer executable instructions, which can execute the action recognition model training method or the action recognition method in any of the above method embodiments. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memories.

[0153] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A motion recognition system, characterized in that: include: Information separation network, static feature difference and feature displacement network, dynamic feature difference and feature displacement network, classification network, the information separation network includes a bandpass filtering module, a static feature extraction module, The bandpass filtering module is used to extract a dynamic feature map according to multiple frames of continuous images in a segment, and the bandpass filtering module includes a spatial convolution layer and a temporal convolution layer; The static feature extraction module is used to perform time average pooling on multiple frames of continuous images in the segment to obtain a feature map, and to subtract the feature map from the dynamic feature map to obtain a static feature map; The static feature difference and feature displacement network is used to perform feature displacement operations on static feature maps corresponding to multiple segments, and calculate the difference features between the static feature maps corresponding to each segment to obtain static classification features; The dynamic feature difference and feature displacement network is used to perform feature displacement operations on the dynamic feature maps corresponding to the multiple segments, and calculate the difference features between the dynamic feature maps corresponding to the segments to obtain dynamic classification features; The classification network is used to obtain action recognition results according to the static classification features and the dynamic classification features.

2. The motion recognition system according to claim 1, characterized in that: The static feature difference and feature displacement network, and / or the dynamic feature difference and feature displacement network comprises: an image segmentation module, an initial feature extraction module, and at least one intermediate feature extraction module. The image segmentation module is used to segment the input feature map according to a first preset size to obtain a first feature vector; The initial feature extraction module includes a linear embedding submodule and at least one feature difference and feature displacement submodule, wherein the linear embedding submodule is used to transform the first feature vector according to a preset number of channels to obtain a second feature vector; the feature difference and feature displacement submodule is used to perform a feature displacement operation on the second feature vector, and calculate the difference features between the second feature vectors corresponding to each segment to obtain an initial classification feature; The intermediate feature extraction module includes a feature merging submodule and at least one feature difference and feature displacement submodule. The feature merging submodule is used to merge the initial classification features according to a second preset size to obtain a third feature vector; the feature difference and feature displacement submodule is used to perform a feature displacement operation on the third feature vector, and calculate the difference features between the third feature vectors corresponding to each segment to obtain a classification feature.

3. The motion recognition system according to claim 2, characterized in that: The static feature difference and feature displacement network, and / or the dynamic feature difference and feature displacement network, includes three intermediate feature extraction modules: a first intermediate feature extraction module, a second intermediate feature extraction module, and a third intermediate feature extraction module. The image segmentation module, the initial feature extraction module, the first intermediate feature extraction module, the second intermediate feature extraction module, and the third intermediate feature extraction module are connected in sequence; The number of feature differences and feature displacement submodules in the initial feature extraction module, the first intermediate feature extraction module, and the third intermediate feature extraction module is the same; The number of feature difference and feature displacement submodules in the second intermediate feature extraction module is greater than the number of feature difference and feature displacement submodules in the initial feature extraction module, the first intermediate feature extraction module, and the third intermediate feature extraction module.

4. The motion recognition system according to claim 2 or 3, characterized in that: The feature difference and feature shift submodule includes a first normalization layer, a feature shift unit, a feature difference unit, a second normalization layer, a first fully connected layer, a first GELU function layer, and a second fully connected layer, wherein: The first normalization layer, the feature displacement unit, the feature difference unit, the second normalization layer, the first fully connected layer, the first GELU function layer, and the second fully connected layer are connected in sequence; The input data of the second normalization layer is a residual calculation result of the input data of the first normalization layer and the output data of the feature difference unit; The output data of the feature difference and feature shift submodule is a residual calculation result of the input data of the second normalization layer and the output data of the second fully connected layer.

5. The motion recognition system according to claim 4, characterized in that: The feature displacement unit includes a first channel fully connected layer, a horizontal feature displacement layer, a second channel fully connected layer, a vertical feature displacement layer, a third channel fully connected layer, and a fourth channel fully connected layer, wherein: The first channel fully connected layer is used to fully connect the channels of the input data to obtain a fully connected result, and input the fully connected result into the horizontal feature displacement layer and the vertical feature displacement layer respectively; The horizontal feature displacement layer is used to horizontally shift the fully connected result to obtain a horizontal displacement result, and input the horizontal displacement result into the second channel fully connected layer; The vertical feature displacement layer is used to vertically displace the fully connected result to obtain a vertical displacement result, and input the vertical displacement result into the third channel fully connected layer; The fourth channel fully connected layer is used to process the sum of the output results of the second channel fully connected layer and the third channel fully connected layer to obtain the output result of the feature shift unit.

6. The motion recognition system according to claim 4, characterized in that: The feature difference unit includes: an input layer, a maximum pooling layer, a third fully connected layer, a second GELU function layer, a fourth fully connected layer, an upsampling layer, a fifth fully connected layer, a third GELU function layer, a sixth fully connected layer, and a feature difference output layer, wherein: The input layer, the maximum pooling layer, the third fully connected layer, the second GELU function layer, the fourth fully connected layer, the upsampling layer, and the feature difference output layer are connected in sequence; The input layer, the fifth fully connected layer, the third GELU function layer, the sixth fully connected layer, and the feature difference output layer are connected in sequence; The input layer is used to make a difference between the input feature corresponding to the segment at the current moment and the input feature corresponding to the segment at the previous moment, and input the difference feature into the maximum pooling layer and the fifth fully connected layer respectively; The feature difference output layer is used to sum the output results of the upsampling layer and the sixth fully connected layer to obtain a summed result; multiply the summed result point by point with the input feature corresponding to the segment at the previous moment to obtain a multiplication result; add the multiplication result to the input feature corresponding to the segment at the previous moment to obtain the output result of the feature difference unit.

7. The motion recognition system according to claim 1, characterized in that: The classification network includes a first time average pooling layer, a second time average pooling layer, a static feature classifier, a dynamic feature classifier, and an output layer. The first time average pooling layer is used to perform time average pooling on the static classification features corresponding to the multiple segments, and input the pooling results into the static feature classifier; The second time average pooling layer is used to perform time average pooling on the dynamic classification features corresponding to the multiple segments, and input the pooling results into the dynamic feature classifier; The static feature classifier is used to obtain a first classification result according to the static classification feature; The dynamic feature classifier is used to obtain a second classification result according to the dynamic classification feature; The recognition result output layer is used to output a weighted average result of the first classification result and the second classification result as an output result.

8. A method for training an action recognition model, characterized in that: include: Acquire multiple image sequences, wherein the image sequences are labeled with pedestrian action types; Dividing each of the image sequences into a plurality of subsequences to obtain a training data set; The training data set is input into a neural network system, and the neural network system is trained until the loss value of the loss function satisfies the loss condition, thereby obtaining the action recognition model. The neural network system is the action recognition system according to any one of claims 1 to 7.

9. The method for training an action recognition model according to claim 8, characterized in that: The loss function is obtained by combining an orthogonal projection loss function and a cross entropy loss function.

10. The method for training an action recognition model according to claim 9, characterized in that: The loss function is: L=L ce +β×L opl , Among them, L ce represents the cross entropy loss function, L opl represents the orthogonal projection loss function, and β represents the hyperparameter that controls the weight of the orthogonal projection loss.

11. A method for motion recognition, characterized in that: include: Acquire an image sequence of a target object, and divide the image sequence into a plurality of subsequences; The subsequence is input into an action recognition model to generate an action recognition result, wherein the action recognition model is trained by the action recognition model training method according to any one of claims 8 to 10.

12. The motion recognition method according to claim 11, characterized in that: After the step of dividing the image sequence into a plurality of subsequences and before the step of inputting the subsequences into the action recognition model, the method further comprises: Proportionally scaling the images in the image sequence to obtain scaled images, wherein the size of the short side of the scaled images is within a preset range; Randomly cropping the scaled image to obtain a cropped image, wherein the size of the cropped image meets a preset condition; The cropped image is used as an image in a subsequence, and the step of inputting the subsequence into an action recognition model is performed.

13. A motion recognition model training device, characterized in that: include: An image acquisition module, used to acquire a plurality of image sequences, wherein the image sequences are marked with pedestrian action types; A training data acquisition module, used for dividing each of the image sequences into a plurality of sub-sequences to obtain a training data set; A model training module is used to input the training data set into a neural network system, train the neural network system, and obtain the action recognition model. The neural network system is the action recognition system described in any one of claims 1 to 7.

14. A motion recognition device, characterized in that: include: An image acquisition module, used to acquire an image sequence of a target object and divide the image sequence into multiple subsequences; The action recognition module is used to input the subsequence into an action recognition model to generate an action recognition result, and the action recognition model is trained by the action recognition model training method according to any one of claims 8 to 10.

15. A computer device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor, thereby executing the action recognition system as described in any one of claims 1-7, or, executing the action recognition model training method as described in any one of claims 8-10, or, executing the action recognition method as described in claim 11 or 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the action recognition system as described in any one of claims 1-7, or, to execute the action recognition model training method as described in any one of claims 8-10, or, to execute the action recognition method as described in claim 11 or 12.

Images