Action recognition method and device based on data expansion, equipment and storage medium

Abstract

The application relates to the technical field of video analysis, and specifically discloses a motion recognition method and device based on data expansion, equipment and a storage medium. The training data is constructed by means of multiple sparse frame extraction of each complete motion image segment and feature superposition, the training data has the advantages of removing redundant information and covering more motion features, the ability of a neural network model to recognize motions in a video is trained, the video motion feature can be learned faster and more accurately, a video action recognition network is used to recognize a to-be-recognized video, and a motion recognition result can be obtained with higher quality. On this basis, the information quantity is improved by performing sparse frame extraction according to the motion change rate distribution of the complete motion image segment. The complete motion image segment can be encoded and the encoding community center can be calculated by using the video motion recognition network, and the detection efficiency is improved by comparing the encoding obtained by recognizing the to-be-recognized video with the encoding community center.

Description

Technical Field

[0001] This application relates to the field of video analytics technology, and in particular to a method, apparatus, device, and storage medium for motion recognition based on data expansion. Background Technology

[0002] With the development of artificial intelligence technology and the increase in video business data, video intelligent analysis technology has gradually become a research hotspot. It aims to automatically analyze and identify target behaviors in videos through video analysis. It is an important part of industrial monitoring, security analysis, medical assistance, and sports motion analysis. Motion recognition is the core area of ​​video intelligent analysis, and the accuracy and reaction speed in motion recognition are important measures of this research.

[0003] Action recognition is a natural extension of image recognition by introducing the time dimension. Video images are data structures composed of several frames of images arranged in time sequence. Action recognition not only needs to analyze the spatial content of redundant images in the video, but also needs to mine the temporal information between images. However, existing deep learning algorithms generally perform poorly in terms of accuracy and reaction speed in action recognition.

[0004] Video data is massive in scale, with information redundancy between frames. The number of parameters in action recognition network models is positively correlated with the amount of data, and training these models requires substantial computational resources. Action recognition models are generally complex in structure, making convergence difficult during training. Currently, single-channel and dual-channel optical flow recognition of video data shows good results, but extracting optical flow is extremely tedious, making end-to-end network training challenging, especially as the accuracy of the temporal information captured by optical flow is difficult to guarantee. Conversely, RGB (red, green, and blue) image information can cover optical flow information, representing another promising direction for solving video action recognition problems.

[0005] Because video action recognition involves a huge amount of data, existing technologies that use image information recognition for video action recognition employ sparse frame extraction to process training data or video data to be recognized. This involves arranging the frames in a video in chronological order and then extracting a portion of the frames as training data for the model or for video action recognition.

[0006] While frame extraction can solve the problem of information redundancy and reduce computation, it can affect the input quality of video data and reduce recognition performance. Summary of the Invention

[0007] The purpose of this application is to provide a method, apparatus, device, and storage medium for action recognition based on data expansion, which can improve the ability of neural network models to recognize actions contained in videos while solving the problem of information redundancy in action recognition during training.

[0008] To address the aforementioned technical problems, this application provides an action recognition method based on data expansion, comprising:

[0009] A complete motion dataset is obtained, and multiple sparse frame extractions are performed on each complete motion image segment in the complete motion dataset. The features of each frame extraction result are superimposed to obtain the first three-dimensional image module corresponding to each complete motion image segment.

[0010] Using the first three-dimensional image module corresponding to each complete action image segment as training data, a neural network model is trained to recognize the corresponding action of each complete action image segment, thus obtaining a video action recognition network.

[0011] The video action recognition network is used to identify the video to be identified, and the action recognition result of the video to be identified is obtained.

[0012] Optionally, the step of performing multiple sparse frame extractions on each complete motion image segment in the complete motion dataset and then superimposing features on the results of each frame extraction to obtain a first three-dimensional image module corresponding to each complete motion image segment specifically includes:

[0013] The complete motion image segment is subjected to multiple sparse frame extractions in chronological order to obtain multiple sets of corresponding first image data.

[0014] Each group of the first image data is input into the convolution kernel, and the output is a first three-dimensional feature set containing the first feature map of the corresponding frame number;

[0015] The first three-dimensional feature sets corresponding to the same complete action video segment are merged to obtain the first three-dimensional image module corresponding to the complete action video segment.

[0016] Optionally, the step of inputting each group of the first image data into the convolution kernel and outputting a first three-dimensional feature set containing the first feature map of the corresponding frame number specifically involves:

[0017] Each group of the first image data is input into the corresponding convolution kernel, and the output is a first three-dimensional feature set containing the first feature map of the corresponding frame number;

[0018] The more frames the first image data in the same group has, the larger the corresponding convolution kernel becomes.

[0019] Optionally, the step of performing multiple sparse frame extractions on each complete motion image segment in the complete motion dataset specifically includes:

[0020] Calculate the difference value between adjacent frames of the complete motion video segment, and determine the motion change rate distribution of the complete motion video segment based on each difference value;

[0021] Based on the distribution of the rate of change of the action, the complete action video segment is subjected to multiple frame extractions.

[0022] Optionally, the neural network model specifically adds a temporal dimension convolutional channel to the densely connected convolutional network DenseNet, and adds a network in each layer to randomly reduce the input relationships of each layer by the inter-layer sparsity rate.

[0023] Optionally, the step of using the first three-dimensional image module corresponding to each of the complete action image segments as training data to train the neural network model to recognize the corresponding actions of each of the complete action image segments, thereby obtaining a video action recognition network, specifically involves:

[0024] Using the first three-dimensional image module corresponding to each complete action image segment as training data, the neural network model is trained to assimilate the encoding generated by the first three-dimensional image module corresponding to the same complete action image segment and to differentiate the encoding generated by the first three-dimensional image module corresponding to different complete action image segments, thereby obtaining the video action recognition network.

[0025] Optionally, the training data is triplet data consisting of two first three-dimensional image modules corresponding to the same complete motion image segment and one first three-dimensional image module corresponding to another complete motion image segment.

[0026] The loss function used to train the neural network model is:

[0027] L(θ)=max(||φ(M1)-φ(M2)|| 2 -||φ(M1)-φ(N)|| 2 +δ,0);

[0028] The cost function used to train the neural network model is:

[0029]

[0030] Where θ represents the parameters of the neural network model, L(θ) is the loss function value, M1 and M2 are two first 3D image modules corresponding to the same complete motion image segment, N is a first 3D image module corresponding to another complete motion image segment, φ(M1) and φ(M2) are the codes generated by the neural network model for the two first 3D image modules corresponding to the same complete motion image segment, φ(N) is the code generated by the neural network model for the first 3D image module corresponding to another complete motion image segment, δ is the interval parameter, T is the total number of motion types included in the complete motion dataset, and M...i1 M i2 For the two first three-dimensional image modules corresponding to the same complete motion image segment in the i-th triplet data, N i For the first three-dimensional image module corresponding to another complete motion image segment in the i-th triplet data, φ(M i1 ), φ(M i2 ) represents the encoding generated by the neural network model for the two first three-dimensional image modules corresponding to the same complete motion image segment in the i-th triplet data, φ(N) i ) is the encoding generated by the neural network model for the first three-dimensional image module corresponding to another complete motion image segment in the i-th triplet data.

[0031] Optionally, the step of using the video action recognition network to identify the video to be recognized and obtaining the action recognition result specifically includes:

[0032] The first three-dimensional image module corresponding to each complete motion image segment is input into the video motion recognition network to obtain the motion feature code of each first three-dimensional image module.

[0033] The video to be identified is processed according to the generation method of the first three-dimensional image module to obtain a second three-dimensional image module corresponding to the video to be identified;

[0034] The second 3D image module is input into the video action recognition network to obtain the action feature encoding of the second 3D image module;

[0035] The motion recognition result is obtained by comparing the motion feature encoding of the second three-dimensional image module with the motion feature encoding of the first three-dimensional image module.

[0036] Optionally, the step of using the video action recognition network to identify the video to be recognized and obtaining the action recognition result specifically includes:

[0037] The first three-dimensional image module corresponding to each complete motion image segment is input into the video motion recognition network to obtain the motion feature code of each first three-dimensional image module.

[0038] Calculate the coding cluster center of the motion feature codes of all the first three-dimensional image modules corresponding to the complete motion image segments of the same type of motion;

[0039] The video to be identified is processed according to the generation method of the first three-dimensional image module to obtain a second three-dimensional image module corresponding to the video to be identified;

[0040] The second 3D image module is input into the video action recognition network to obtain the action feature encoding of the second 3D image module;

[0041] Calculate the distance from the action feature encoding of the second three-dimensional image module to the center of each encoding community. When the obtained distance is less than the distance threshold, determine that the action corresponding to the second three-dimensional image module is the action corresponding to the center of the encoding community.

[0042] To address the aforementioned technical problems, this application also provides an action recognition device based on data expansion, comprising:

[0043] The generation unit is used to acquire a complete motion dataset, perform multiple sparse frame extractions on each complete motion image segment in the complete motion dataset, and perform feature overlay on each frame extraction result to obtain a first three-dimensional image module corresponding to each complete motion image segment.

[0044] The training unit is used to train the neural network model to recognize the corresponding actions of each complete action image segment using the first three-dimensional image module corresponding to each complete action image segment as training data, thereby obtaining a video action recognition network.

[0045] The recognition unit is used to recognize the video to be recognized using the video action recognition network, and to obtain the action recognition result of the video to be recognized.

[0046] To address the aforementioned technical problems, this application also provides an action recognition device based on data expansion, comprising:

[0047] Memory, used to store computer programs;

[0048] A processor for executing the computer program, which, when executed by the processor, implements the steps of the action recognition method based on data expansion as described above.

[0049] To address the aforementioned technical problems, this application also provides a storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the action recognition method based on data expansion as described in any of the preceding claims.

[0050] The action recognition method based on data expansion provided in this application constructs training data by performing multiple sparse frame extractions and feature overlays on each complete action video segment. This training data removes redundant information while encompassing more action features, thereby training a neural network model to recognize actions in videos. This allows for faster and more accurate learning of video action features. The resulting video action recognition network then identifies the video to be recognized, yielding higher-quality action recognition results. Furthermore, sparse frame extraction is performed according to the action change rate distribution of the complete action video segments, increasing the amount of information acquired. The video action recognition network encodes the complete action video segments and calculates the coding cluster centers. Comparing the codes obtained from recognizing the video to be recognized with these coding cluster centers improves detection efficiency.

[0051] This application also provides an action recognition device, equipment, and storage medium based on data expansion, which has the above-mentioned beneficial effects, and will not be elaborated here. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating an action recognition method based on data expansion provided in this application embodiment;

[0054] Figure 2 A schematic diagram of the structure of an action recognition device based on data expansion provided in this application embodiment;

[0055] Figure 3 This is a schematic diagram of the structure of an action recognition device based on data expansion, provided in an embodiment of this application. Detailed Implementation

[0056] The core of this application is to provide a method, apparatus, device, and storage medium for action recognition based on data expansion, which can improve the ability of neural network models to recognize actions contained in videos while solving the problem of information redundancy in action recognition during training.

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] Example 1

[0059] Figure 1 This is a flowchart of an action recognition method based on data expansion provided in an embodiment of this application.

[0060] like Figure 1 As shown, the action recognition method based on data expansion provided in this application includes:

[0061] S101: Obtain a complete motion dataset, perform multiple sparse frame extractions on each complete motion image segment in the complete motion dataset, and superimpose features on each extraction result to obtain the first three-dimensional image module corresponding to each complete motion image segment.

[0062] S102: Using the first three-dimensional image module corresponding to each complete action image segment as training data, train the neural network model to recognize the corresponding action of each complete action image segment, and obtain the video action recognition network.

[0063] S103: Use a video action recognition network to identify the video to be identified and obtain the action recognition results of the video to be identified.

[0064] When recognizing actions (such as running, jumping, or certain dynamic gestures) from videos, there is a large amount of redundant information between video frames, such as background information. Furthermore, when the amplitude of action changes little between adjacent frames, the action information in adjacent frames is also redundant for action recognition. Sparse frame extraction, which extracts a portion of frames from a video segment for model training and recognition, can remove a large amount of redundant information between frames and reduce computational load. However, this obviously affects video data quality and reduces recognition performance. Therefore, the action recognition method based on data expansion provided in this application, based on sparse frame extraction, extracts feature information through multiple frame extractions and superimposes the extracted information. This not only removes redundant information but also improves the quality of video data during model training and recognition, thereby enhancing recognition performance.

[0065] In specific implementation, for S101, the complete motion dataset includes multiple complete motion image segments with known motion types. Let the complete motion dataset be denoted as v. Assume v is a three-channel high-definition RGB image, containing T types of motion, each type corresponding to several complete motion image segments, and each complete motion image segment has been temporally segmented. Then, for each complete motion image segment, multiple sparse frame extractions are performed. For example, three extractions can be performed for each complete motion image segment, extracting frames q1, q2, and q3 respectively, where q1 > q2 > q3, and q1, q2, and q3 are all less than the total number of frames in the complete motion image segment. These can be defined as fast frame channels, medium frame channels, and slow frame channels, arranged from largest to smallest frame number.

[0066] Specifically, convolutional kernels can be used to extract image features, and S101 can specifically include:

[0067] Multiple sparse frame extractions are performed on complete motion image segments in chronological order to obtain multiple sets of corresponding first image data.

[0068] Each group of first image data is input into the convolution kernel, and the output is a first three-dimensional feature set containing the first feature map of the corresponding frame number;

[0069] The first three-dimensional feature sets corresponding to the same complete action video segment are merged to obtain the first three-dimensional image module corresponding to the complete action video segment.

[0070] The significance of convolution lies in channel information compression. That is, for the first image data of frame q1, the first feature map of layer q1 is obtained after processing by the convolution kernel. The first feature maps of layer q1 are stacked to obtain the first three-dimensional feature set A1 corresponding to the fast frame channel.

[0071] By independently repeating the above steps, we can obtain the first three-dimensional feature set A2 corresponding to the medium frame channel and the first three-dimensional feature set A3 corresponding to the slow frame channel.

[0072] The first three-dimensional feature sets A1, A2, and A3 are physically merged to obtain the first three-dimensional image module M corresponding to the complete motion image segment.

[0073] To improve extraction efficiency, a larger convolutional kernel is used for the first image data with a large number of frames, and a smaller convolutional kernel is used for the first image data with a small number of frames. That is, each group of first image data is input into the convolutional kernel, and the output is a first three-dimensional feature set containing the first feature map of the corresponding number of frames. Specifically, each group of first image data is input into the corresponding convolutional kernel, and the output is a first three-dimensional feature set containing the first feature map of the corresponding number of frames. The more frames in the same group of first image data, the larger the corresponding convolutional kernel.

[0074] For example, for the fast frame channel, medium frame channel, and slow frame channel mentioned above, the corresponding convolutional kernels can be set to s1×s1×3, s2×s2×3, and s3×s3×3 respectively, with s1 > s2 > s3. Specifically, s1 = 5, s2 = 3, and s3 = 1 can be defined. The convolutional kernel parameters can be pre-set by technicians based on experience and used as fixed parameters, without needing to participate in the subsequent network training process.

[0075] Understandably, based on experience, more or fewer channels can be set for each complete motion picture segment, as well as other convolution kernel parameters can be adjusted.

[0076] Building upon this, to further enhance the action features contained in the training data, S101 performs multiple sparse frame extractions on each complete action image segment in the complete action dataset. Specifically, this may include:

[0077] Calculate the difference values ​​between adjacent frames of a complete motion video segment, and determine the distribution of motion change rate of the complete motion video segment based on each difference value;

[0078] Based on the distribution of motion change rates, complete motion video clips are subjected to multiple frame extractions.

[0079] Specifically, the difference values ​​between adjacent frames of a complete motion video segment are calculated, and the distribution of motion change rate of the complete motion video segment is determined based on each difference value. This can be calculated using the following formula:

[0080]

[0081] Where p(z) is the probability corresponding to the z-th frame of the complete motion picture segment, F z+1 (x,y) represents the pixel value at coordinates (x,y) on the (z+1)th frame of the complete motion picture segment, F z (x,y) represents the pixel value at coordinates (x,y) on the z-th frame of the complete motion picture segment, F i+1 (x,y) represents the pixel value at coordinates (x,y) on the (i+1)th frame of the complete motion picture segment, F i (x,y) represents the pixel value at coordinates (x,y) on the i-th frame of the complete motion picture clip, and n represents the total number of frames in the complete motion picture clip.

[0082] Since many actions are not uniformly varied—that is, actions change slowly in some consecutive frames and quickly in others—uniform frame extraction would result in acquiring more redundant information in slower-changing segments while failing to extract sufficient information from segments with changing actions. Therefore, random frame extraction based on the distribution of action change rates can better remove redundant information and extract more action features. Based on the above formula, the action change rate distribution of a complete action video segment can be calculated. Then, following the principle of extracting more frames from segments with high action change rates and fewer frames from segments with low action change rates, the frame extraction distribution is determined according to the action change rate distribution.

[0083] For S102, the first three-dimensional image modules corresponding to each complete action image segment obtained through the above steps are used as training data to train the neural network model to recognize the corresponding actions of each complete action image segment. Specifically, the neural network model can be assisted in learning by adding action type labels to each first three-dimensional image module until the convergence condition is reached, thus obtaining the video action recognition network.

[0084] For S103, the video to be recognized is processed according to the method of generating the first 3D image module to obtain a second 3D image module. The second 3D image module is then input into the video action recognition network to obtain the action recognition result of the video to be recognized. The action recognition result can be either that a certain action type has been identified or that no action type has been identified.

[0085] The action recognition method based on data expansion provided in this application, when constructing training data, performs multiple sparse frame extractions on each complete action image segment in the complete action dataset, based on sparse frame extraction, and then superimposes the features of each extraction result to obtain a first three-dimensional image module corresponding to each complete action image segment. This results in training data that removes redundancy and contains more action features. Based on this, the ability of the neural network model to recognize actions in videos can be trained, enabling faster and more accurate learning of video action features. Using the obtained video action recognition network to identify the video to be recognized yields higher-quality action recognition results.

[0086] Example 2

[0087] The number of parameters in an action recognition network model is positively correlated with the amount of data, and training the model requires a large amount of computational resources. To address this, the action recognition method based on data expansion provided in this application offers a preferred neural network architecture.

[0088] In the action recognition method based on data expansion provided in the embodiments of this application, the neural network model specifically adds a temporal dimension convolutional channel to the densely connected convolutional network DenseNet, and adds a network in each layer to randomly reduce the input relationship of each layer by the inter-layer sparsity rate.

[0089] This application improves upon the densely connected convolutional network DenseNet by introducing convolutional channels in the temporal dimension, transforming 2D convolutions into 3D convolutions to capture the temporal relationships between frames. Furthermore, by introducing an inter-layer sparsity rate, specifically a dropping rate of 0.7, the input relationships of each layer are randomly reduced, thus sparsifying the model architecture. Therefore, the operation of the l-th layer is as follows:

[0090]

[0091] Where, x l V is the output data for the l-th layer. l Let x be the concept function of the l-th layer. u This is the output data of the uth layer preceding the lth layer.

[0092] The existing DenseNet uses the output data of all layers before layer l as the input data of layer l. However, in this embodiment, to reduce the amount of computation, layers before layer l are randomly deleted from the input data of layer l. For example, 0.7 layers are left as the input data of layer l (it should be noted that u does not represent the layer number).

[0093] In addition, based on probability dropping out, a portion of the data from a layer preceding the l-th layer in the input data of the l-th layer can be randomly removed.

[0094] In addition, training can be accelerated by appropriately reducing the number of convolutional layers in the first half of the DenseNet convolutional network, based on the training results, or by loading publicly available weight coefficients.

[0095] It can be understood that the goal of training a neural network model is to enable the model to recognize the same complete action image segment as the same result, while recognizing different complete action image segments as different results. For ease of recording, complete action image segments can be identified using encoding. Therefore, in the action recognition method based on data expansion provided in this application embodiment, S102: using the first three-dimensional image module corresponding to each complete action image segment as training data, the neural network model is trained to recognize the corresponding actions of each complete action image segment, resulting in a video action recognition network. Specifically, this can be achieved by using the first three-dimensional image module corresponding to each complete action image segment as training data, training the neural network model to assimilate the encoding generated by the first three-dimensional image module corresponding to the same complete action image segment and to differentiate the encoding generated by the first three-dimensional image module corresponding to different complete action image segments, thus obtaining a video action recognition network.

[0096] In practical implementation, the fully connected layers of the improved densely connected convolutional network DenseNet can be configured as follows:

[0097] fc layer: nn.Linear(fc.infeatures,fc)+Batch Norm+Sigmoid;

[0098] Output layer: nn.Linear(fc,256).

[0099] That is, the fully connected layers of the improved densely connected convolutional network DenseNet are set to encode the first three-dimensional image module corresponding to the complete action image segment and output the encoding in 256-dimensional vector format.

[0100] To train a neural network model to assimilate the encoding of the first 3D image module corresponding to the same complete motion image segment and to differentiate the variations of the first 3D image modules corresponding to different complete motion image segments, the training data can be designed as triplets consisting of two first 3D image modules corresponding to the same complete motion image segment and one first 3D image module corresponding to another complete motion image segment. For example, M1 and M2 are two first 3D image modules corresponding to the same complete motion image segment, and N is a first 3D image module corresponding to another complete motion image segment. The function of training the neural network model φ is:

[0101] ||φ(M1)-φ(M2)|| 2 +δ≤||φ(M1)-φ(N)|| 2 .

[0102] The loss function used to train the neural network model is:

[0103] L(θ)=max(||φ(M1)-φ(M2)||2 -||φ(M1)-φ(N)|| 2 +δ,0);

[0104] The cost function used to train the neural network model is:

[0105]

[0106] Where θ represents the parameters of the neural network model, L(θ) is the loss function value, M1 and M2 are two first 3D image modules corresponding to the same complete motion image segment, N is the first 3D image module corresponding to another complete motion image segment, φ(M1) and φ(M2) are the codes generated by the neural network model for the two first 3D image modules corresponding to the same complete motion image segment, φ(N) is the code generated by the neural network model for the first 3D image module corresponding to another complete motion image segment, δ is the interval parameter, T is the total number of motion types included in the complete motion dataset, and M... i1 M i2 For the two first 3D image modules corresponding to the same complete motion image segment in the i-th triplet data, N i For the first 3D image module corresponding to another complete motion image segment in the i-th triplet data, φ(M i1 ), φ(M i2 φ(N) represents the encoding generated by the neural network model for the two first 3D image modules corresponding to the same complete motion image segment in the i-th triplet data. i ) is the encoding generated by the neural network model for the first three-dimensional image module corresponding to another complete motion image segment in the i-th triplet data.

[0107] Based on the aforementioned neural network architecture and loss function, the parameters θ of the neural network model φ can be trained using the Adam optimization algorithm. Specifically, parameters such as batch size (64), dropping rate (0.7), maximum number of epochs (100), and Batch Norm (deep learning algorithm) can be preset and executed as needed.

[0108] Example 3

[0109] Based on the above embodiments, in the action recognition method based on data expansion provided in this application embodiment, S103: using a video action recognition network to identify the video to be identified and obtaining the action recognition result, specifically may include:

[0110] Each complete action video segment is input into the first three-dimensional image module corresponding to it to obtain the action feature code of each first three-dimensional image module.

[0111] The video to be identified is processed according to the generation method of the first three-dimensional image module to obtain the second three-dimensional image module corresponding to the video to be identified.

[0112] The second 3D image module is input into the video action recognition network to obtain the action feature encoding of the second 3D image module.

[0113] By comparing the motion feature codes of the second 3D image module with those of the first 3D image module, the motion recognition result is obtained.

[0114] Based on the scheme proposed in the above embodiments of this application for training a neural network model to encode the corresponding first three-dimensional image module of a complete action image segment, the trained video action recognition network is used to encode all the pre-generated first three-dimensional image modules to construct an action coding library corresponding to the complete action dataset.

[0115] When it is necessary to identify actions contained in a video, the video to be identified is first processed in the same way as the first 3D image module is generated to obtain a second 3D image module. The second 3D image module is then input into the video action recognition network to obtain the action feature code of the second 3D image module. The action feature code of the second 3D image module is then compared with the action feature codes of all first 3D image modules in the action coding library. If there is the same or similar code (determined by setting a threshold), the action type corresponding to the first 3D image module is taken as the action type contained in the video to be identified; if there is no same or similar code, the recognition result can be output as empty.

[0116] Example 4

[0117] In addition to the methods described above for constructing and recognizing action codes, this application provides another scheme for constructing and recognizing action codes. In the action recognition method based on data expansion provided in this application, S103: using a video action recognition network to recognize the video to be recognized and obtaining the action recognition result, specifically including:

[0118] Each complete action video segment is input into the first three-dimensional image module corresponding to it to obtain the action feature code of each first three-dimensional image module.

[0119] Calculate the coding cluster center of the motion feature codes of all first three-dimensional image modules corresponding to complete motion image segments of the same type of motion;

[0120] The video to be identified is processed according to the generation method of the first three-dimensional image module to obtain the second three-dimensional image module corresponding to the video to be identified.

[0121] The second 3D image module is input into the video action recognition network to obtain the action feature encoding of the second 3D image module.

[0122] Calculate the distance from the action feature encoding of the second 3D image module to the center of each encoding community. When the obtained distance is less than the distance threshold, determine that the action corresponding to the second 3D image module is the action corresponding to the center of the encoding community.

[0123] In specific implementation, if the complete action dataset contains T types of actions, and each type of action corresponds to w complete action image segments, it can be further processed according to the process provided in the above embodiments of this application to obtain p (p≥1) first three-dimensional image modules corresponding to each complete action image segment, and encoded using a video action recognition network to obtain i th Each action feature is encoded as Z. i11 Z i12 …Z i1p Z i21 Z i22 …Z i2p Z iw1 Z iw2 …Z iwp Stored in the action coding library:

[0124] Z i11 =φ(M i11 Z i12 =φ(M i12 ...Z i1p =φ(M i1p ),

[0125] Z i21 =φ(M i21 Z i22 =φ(M i22 ...Z i2p =φ(M i2p ),

[0126] ...

[0127] Z iw1 =φ(M iw1 Z iw2 =φ(M iw2 ...Z iwp =φ(M iwp ),

[0128] (i = 1, 2, ..., T)

[0129] The resulting action coding library stores T×w×p action feature codes, which can be denoted as... Among them, y iLet be the action identifier for the i-th action category. If we understand it as T coding clusters, then the center of each coding cluster can be calculated using the following formula:

[0130]

[0131] in, This is the center of the encoded community corresponding to the i-th action category. As shown in the neural network architecture provided in the above embodiment, if the action feature encoding Z... imn If it is a 256-dimensional vector, then It is also a 256-dimensional vector.

[0132] When performing video action recognition, the video c to be recognized is first processed in the same way as the first 3D image module to obtain the second 3D image module. The second 3D image module is then input into the video action recognition network, φ(c) = Q. c The motion feature encoding Q of the second 3D image module is obtained. c .

[0133] Define distance Set a threshold λ, when At that time, it is confirmed that the video to be identified, c, contains an action recorded in the action coding library, and that the action is identified by the action identifier y. i If the action is not recorded in the action coding library, the video c to be identified is considered to contain no action recorded in the action coding library, and the recognition result is empty (0), thus completing the judgment.

[0134] By processing the action feature codes corresponding to similar actions into coding cluster centers and comparing them with the action feature codes of the video to be identified, the number of action feature codes that need to be compared can be further reduced, thereby improving recognition efficiency.

[0135] The above details various embodiments of the action recognition method based on data expansion. Based on this, this application also discloses an action recognition device, equipment, and storage medium based on data expansion corresponding to the above method.

[0136] Example 5

[0137] Figure 2 This is a schematic diagram of the structure of an action recognition device based on data expansion, provided in an embodiment of this application.

[0138] like Figure 2 As shown, the action recognition device based on data expansion provided in this application embodiment includes:

[0139] The generation unit 201 is used to obtain a complete motion dataset, perform multiple sparse frame extractions on each complete motion image segment in the complete motion dataset, and perform feature overlay on each frame extraction result to obtain the first three-dimensional image module corresponding to each complete motion image segment.

[0140] Training unit 202 is used to train the neural network model to recognize the corresponding actions of each complete action video segment using the first three-dimensional image module corresponding to each complete action video segment as training data, so as to obtain a video action recognition network.

[0141] The recognition unit 203 is used to recognize the video to be recognized using a video action recognition network and obtain the action recognition result of the video to be recognized.

[0142] Furthermore, the generation unit 201 performs multiple sparse frame extractions on each complete motion image segment in the complete motion dataset, and performs feature overlay on the results of each frame extraction to obtain the first three-dimensional image module corresponding to each complete motion image segment, specifically including:

[0143] Multiple sparse frame extractions are performed on complete motion image segments in chronological order to obtain multiple sets of corresponding first image data.

[0144] Each group of first image data is input into the convolution kernel, and the output is a first three-dimensional feature set containing the first feature map of the corresponding frame number;

[0145] The first three-dimensional feature sets corresponding to the same complete action video segment are merged to obtain the first three-dimensional image module corresponding to the complete action video segment.

[0146] Furthermore, the generation unit 201 inputs each group of first image data into the convolution kernel and outputs a first three-dimensional feature set containing the first feature map of the corresponding frame number, specifically:

[0147] Each group of first image data is input into the corresponding convolution kernel, and the output is a first three-dimensional feature set containing the first feature map of the corresponding frame number;

[0148] The more frames in the first image data of the same group, the larger the corresponding convolution kernel.

[0149] Furthermore, the generation unit 201 performs multiple sparse frame extractions on each complete motion image segment in the complete motion dataset, specifically including:

[0150] Calculate the difference values ​​between adjacent frames of a complete motion video segment, and determine the distribution of motion change rate of the complete motion video segment based on each difference value;

[0151] Based on the distribution of motion change rates, complete motion video clips are subjected to multiple frame extractions.

[0152] Furthermore, the neural network model specifically adds temporal convolutional channels to the densely connected convolutional network DenseNet, and adds a network in each layer to randomly reduce the input relationships of each layer through inter-layer sparsity.

[0153] Furthermore, training unit 202 uses the first three-dimensional image module corresponding to each complete action image segment as training data to train the neural network model to recognize the corresponding actions of each complete action image segment, thereby obtaining a video action recognition network, specifically:

[0154] Using the first three-dimensional image modules corresponding to each complete action video segment as training data, a neural network model is trained to assimilate the encoding generated by the first three-dimensional image module corresponding to the same complete action video segment and to differentiate the encoding generated by the first three-dimensional image module corresponding to different complete action video segments, thus obtaining a video action recognition network.

[0155] Furthermore, the training data consists of triplets consisting of two first three-dimensional image modules corresponding to the same complete motion image segment and a first three-dimensional image module corresponding to another complete motion image segment.

[0156] The loss function used to train the neural network model is:

[0157] L(θ)=max(||φ(M1)-φ(M2)|| 2 -||φ(M1)-φ(N)|| 2 +δ,0);

[0158] The cost function used to train the neural network model is:

[0159]

[0160] Where θ represents the parameters of the neural network model, L(θ) is the loss function value, M1 and M2 are two first 3D image modules corresponding to the same complete motion image segment, N is the first 3D image module corresponding to another complete motion image segment, φ(M1) and φ(M2) are the codes generated by the neural network model for the two first 3D image modules corresponding to the same complete motion image segment, φ(N) is the code generated by the neural network model for the first 3D image module corresponding to another complete motion image segment, δ is the interval parameter, T is the total number of motion types included in the complete motion dataset, and M... i1 M i2 For the two first 3D image modules corresponding to the same complete motion image segment in the i-th triplet data, N i For the first 3D image module corresponding to another complete motion image segment in the i-th triplet data, φ(M i1 ), φ(M i2 φ(N) represents the encoding generated by the neural network model for the two first 3D image modules corresponding to the same complete motion image segment in the i-th triplet data. i) is the encoding generated by the neural network model for the first three-dimensional image module corresponding to another complete motion image segment in the i-th triplet data.

[0161] Furthermore, the recognition unit 203 uses a video action recognition network to recognize the video to be recognized and obtains action recognition results, specifically including:

[0162] Each complete action video segment is input into the first three-dimensional image module corresponding to it to obtain the action feature code of each first three-dimensional image module.

[0163] The video to be identified is processed according to the generation method of the first three-dimensional image module to obtain the second three-dimensional image module corresponding to the video to be identified.

[0164] The second 3D image module is input into the video action recognition network to obtain the action feature encoding of the second 3D image module.

[0165] By comparing the motion feature codes of the second 3D image module with those of the first 3D image module, the motion recognition result is obtained.

[0166] Furthermore, the recognition unit 203 uses a video action recognition network to recognize the video to be recognized and obtains action recognition results, specifically including:

[0167] Each complete action video segment is input into the first three-dimensional image module corresponding to it to obtain the action feature code of each first three-dimensional image module.

[0168] Calculate the coding cluster center of the motion feature codes of all first three-dimensional image modules corresponding to complete motion image segments of the same type of motion;

[0169] The video to be identified is processed according to the generation method of the first three-dimensional image module to obtain the second three-dimensional image module corresponding to the video to be identified.

[0170] The second 3D image module is input into the video action recognition network to obtain the action feature encoding of the second 3D image module.

[0171] Calculate the distance from the action feature encoding of the second 3D image module to the center of each encoding community. When the obtained distance is less than the distance threshold, determine that the action corresponding to the second 3D image module is the action corresponding to the center of the encoding community.

[0172] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0173] Example 6

[0174] Figure 3This is a schematic diagram of the structure of an action recognition device based on data expansion, provided in an embodiment of this application.

[0175] like Figure 3 As shown, the action recognition device based on data expansion provided in this application embodiment includes:

[0176] Memory 310 is used to store computer program 311;

[0177] The processor 320 is configured to execute a computer program 311, which, when executed by the processor 320, implements the steps of the action recognition method based on data expansion as described in any of the above embodiments.

[0178] The processor 320 may include one or more processing cores, such as a 3-core processor or an 8-core processor. The processor 320 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 320 may also include a main processor and a coprocessor. The main processor, also known as a Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 320 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 320 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0179] The memory 310 may include one or more storage media, which may be non-transitory. The memory 310 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 310 is used to store at least the following computer program 311, wherein, after being loaded and executed by the processor 320, the computer program 311 is able to implement the relevant steps in the action recognition method based on data expansion disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 310 may also include an operating system 312 and data 313, and the storage method may be temporary storage or permanent storage. The operating system 312 may be Windows. The data 313 may include, but is not limited to, the data involved in the above methods.

[0180] In some embodiments, the motion recognition device based on data expansion may further include a display screen 330, a power supply 340, a communication interface 350, an input / output interface 360, a sensor 370, and a communication bus 380.

[0181] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on motion recognition devices based on data expansion and may include more or fewer components than illustrated.

[0182] The action recognition device based on data expansion provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the action recognition method based on data expansion as described above, with the same effect.

[0183] Example 7

[0184] It should be noted that the device and equipment embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0186] If the integrated modules are implemented as software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application.

[0187] Therefore, embodiments of this application also provide a storage medium storing a computer program, which, when executed by a processor, implements steps such as the action recognition method based on data expansion.

[0188] The storage medium can include various media that can store program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] The computer program contained in the storage medium provided in this embodiment can implement the steps of the action recognition method based on data expansion as described above when executed by the processor, and the effect is the same as above.

[0190] The foregoing provides a detailed description of an action recognition method, apparatus, device, and storage medium based on data expansion provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus, device, and storage medium disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0191] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for action recognition based on data augmentation, characterized in that, The method comprises the following steps: acquiring a complete action data set, performing sparse frame extraction on each complete action video segment in the complete action data set multiple times, and superimposing features of each frame extraction result to obtain a first three-dimensional video module corresponding to each complete action video segment; training a neural network model to recognize the corresponding action of each complete action video segment by taking the first three-dimensional video module corresponding to each complete action video segment as training data, to obtain a video action recognition network; recognizing a to-be-recognized video by using the video action recognition network to obtain an action recognition result of the to-be-recognized video; wherein the complete action data set is subjected to multiple sparse frame extractions, and the features of each frame extraction result are superimposed to obtain a first three-dimensional video module corresponding to each complete action video segment, which comprises the following steps: performing sparse frame extraction on the complete action video segment multiple times in a time sequence to obtain a plurality of groups of first image data; inputting each group of first image data into a convolution kernel to output a first three-dimensional feature set containing a first feature map of a corresponding frame number; merging each first three-dimensional feature set corresponding to the same complete action video segment to obtain the first three-dimensional video module corresponding to the complete action video segment; wherein the complete action data set is subjected to multiple sparse frame extractions, which specifically comprises the following steps: calculating the difference value between adjacent frame images of the complete action video segment, and determining the action change rate distribution of the complete action video segment according to each difference value; performing multiple frame extractions on the complete action video segment according to the action change rate distribution.

2. The data expansion based action recognition method of claim 1, wherein, The step of inputting each group of first image data into a convolution kernel to output a first three-dimensional feature set containing a first feature map of a corresponding frame number specifically comprises the following steps: inputting each group of first image data into a corresponding convolution kernel to output a first three-dimensional feature set containing a first feature map of a corresponding frame number; wherein the more the frame number of each group of first image data, the larger the corresponding convolution kernel.

3. The data expansion based action recognition method of claim 1, wherein, The neural network model specifically comprises a convolution channel with a time sequence dimension added on the basis of a densely connected convolution network DenseNet, and a network with an input relationship of each layer randomly pruned by an interlayer sparsity rate added in each layer network.

4. The data expansion based action recognition method of claim 1, wherein, The step of training a neural network model to recognize the corresponding action of each complete action video segment by taking the first three-dimensional video module corresponding to each complete action video segment as training data to obtain a video action recognition network specifically comprises the following steps: training the neural network model to have the ability to assimilate the encoding of the first three-dimensional video module corresponding to the same complete action video segment and to differentiate the encoding of the first three-dimensional video module corresponding to different complete action video segments by taking the first three-dimensional video module corresponding to each complete action video segment as training data, to obtain the video action recognition network.

5. The data expansion based action recognition method of claim 4, wherein, The training data is a triplet data composed of two first three-dimensional image modules corresponding to the same complete action image segment and the first three-dimensional image module corresponding to another complete action image segment; The loss function used in training the neural network model is: ； The cost function used in training the neural network model is: ； wherein, is a parameter of the neural network model, is a loss function value, , is a first three-dimensional image module corresponding to the same complete action video segment, is the first three-dimensional image module corresponding to another complete action video segment, , is an encoding generated by the neural network model for the two first three-dimensional image modules corresponding to the same complete action video segment, is an encoding generated by the neural network model for the first three-dimensional image module corresponding to another complete action video segment, is an interval parameter, T is the total number of action types contained in the complete action dataset, , is the first three-dimensional image module corresponding to the same complete action video segment in the i-th triplet data, is the first three-dimensional image module corresponding to another complete action video segment in the i-th triplet data, , is an encoding generated by the neural network model for the two first three-dimensional image modules corresponding to the same complete action video segment in the i-th triplet data, is an encoding generated by the neural network model for the first three-dimensional image module corresponding to another complete action video segment in the i-th triplet data.

6. The data expansion based action recognition method of claim 4, wherein, The method for identifying the to-be-identified video by using the video action recognition network to obtain an action recognition result specifically includes: The first three-dimensional image modules corresponding to each complete action image segment are input into the video action recognition network respectively to obtain action feature codes of the first three-dimensional image modules; The to-be-identified video is processed according to the generation method of the first three-dimensional image module to obtain a second three-dimensional image module corresponding to the to-be-identified video; The second three-dimensional image module is input into the video action recognition network to obtain an action feature code of the second three-dimensional image module; The action feature code of the second three-dimensional image module is compared with the action feature codes of the first three-dimensional image modules to obtain the action recognition result.

7. The data expansion based action recognition method of claim 4, wherein, The method for identifying the to-be-identified video by using the video action recognition network to obtain an action recognition result specifically includes: The first three-dimensional image modules corresponding to each complete action image segment are input into the video action recognition network respectively to obtain action feature codes of the first three-dimensional image modules; A coding community center of the action feature codes of all the first three-dimensional image modules corresponding to the complete action image segments of the same type of action is calculated; The to-be-identified video is processed according to the generation method of the first three-dimensional image module to obtain a second three-dimensional image module corresponding to the to-be-identified video; The second three-dimensional image module is input into the video action recognition network to obtain an action feature code of the second three-dimensional image module; A distance between the action feature code of the second three-dimensional image module and each coding community center is calculated, and when the obtained distance is less than a distance threshold, it is determined that the action corresponding to the second three-dimensional image module is the action corresponding to the coding community center.

8. A data expansion-based action recognition apparatus, comprising: It includes: A generation unit is configured to obtain a complete action dataset, perform sparse frame extraction on each complete action image segment in the complete action dataset multiple times, and perform feature superposition on the frame extraction results of each time to obtain a first three-dimensional image module corresponding to each complete action image segment; A training unit is configured to use the first three-dimensional image modules corresponding to each complete action image segment as training data, train a neural network model to identify the corresponding action of each complete action image segment, and obtain a video action recognition network; An identification unit is configured to identify a to-be-identified video by using the video action recognition network to obtain an action recognition result of the to-be-identified video; The method for performing sparse frame extraction on each complete action image segment in the complete action dataset multiple times and performing feature superposition on the frame extraction results of each time to obtain a first three-dimensional image module corresponding to each complete action image segment includes: The complete action image segment is subjected to sparse frame extraction multiple times in a time sequence order to obtain a plurality of groups of first image data corresponding to the complete action image segment; The first image data of each group is input into a convolution kernel to output a first three-dimensional feature set including a first feature map corresponding to a frame number; Each of the first three-dimensional feature sets corresponding to the same complete action image segment is merged to obtain the first three-dimensional image module corresponding to the complete action image segment; The method further comprises the following steps: The difference values between adjacent frame images of the complete action image segment are calculated, and the action change rate distribution of the complete action image segment is determined according to the difference values; The complete action image segment is frame-sampled multiple times according to the action change rate distribution.

9. A data expansion-based motion recognition device, comprising: The method comprises the following steps: a memory for storing a computer program; a processor for executing the computer program, wherein the computer program, when executed by the processor, implements the steps of the action recognition method based on data expansion according to any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program, when executed by the processor, implements the steps of the action recognition method based on data expansion according to any one of claims 1 to 7.

Images