Motion recognition method, device and electronic equipment

Abstract

The embodiments of the present invention provide a method, device and electronic device for action recognition, which relate to the field of action recognition technology. The method includes: obtaining a first feature of K picture frames in a video to be recognized; performing action recognition on the video to be recognized based on the first feature to obtain a first action recognition result; obtaining the target positions of M target picture frames in the video to be recognized based on the first action recognition result; obtaining a second feature of N picture frames before and after each of the target positions; combining the first feature with the second feature to obtain a second action recognition result; wherein M, N, and K are all positive integers, and K is greater than M. The above scheme can not only save manpower, but also make the action recognition results more accurate.

Description

Technical Field

[0001] The present invention relates to the field of motion recognition technology, and in particular to a motion recognition method, device and electronic equipment. Background Art

[0002] At present, short video products are all the rage on the Internet, and the content of short videos is growing rapidly. Video review is becoming increasingly important, and action recognition in video content has become an important part of content review. There are two main steps in short video action recognition: decoding the video and performing action recognition on the decoded image frames according to the time sequence.

[0003] Video decoding can generate a large number of picture frames. The decoded picture frames generally have serious spatial redundancy. Therefore, in existing action recognition scenarios, there are three methods to reduce redundancy: 1) Based on sparsely sampled picture frames, use the Long Short-Term Memory (LSTM) network to predict the start and end time of the action that may occur in the pictures input in chronological order; 2) Starting from the first frame of the decoded picture, use LSTM to gradually predict the position of the next frame of the picture input to the network, so as to achieve the purpose of sparse sampling; 3) Cut the video into segments, and sample each segment sparsely or densely at equal intervals.

[0004] After obtaining the sparsely sampled image frames according to the above method, they are input into the action recognition model. The action recognition model in business scenarios needs to take into account both efficiency and diversity. The existing mainstream technology uses deep learning convolution models, which are mainly the following two types: 1) recognition based on optical flow and LSTM methods; 2) recognition based on three-dimensional (3D) convolution of time and space, spatial 2D convolution + temporal 1D convolution combination on the image. The more efficient second method is widely used in actual business scenarios.

[0005] The above short video action recognition technology has the following problems in the application of audit business:

[0006] First, when LSTM predicts the start and end time of an action, it assumes that the action in the entire short video is continuous and uninterrupted in time. However, in real scenarios, the action may not be within the video screen for a period of time due to factors such as the shaking of the mobile phone lens, resulting in the action occurring only in two time periods. Therefore, LSTM only predicts one start and end time period, resulting in an incomplete number of predicted time periods.

[0007] Second: Existing technical solutions for recommending the start and end time of an action based on LSTM all require marking the start and end time of the action in the video, which requires a lot of labeling manpower.

[0008] Third: In order to reduce computing power consumption, the existing 2D convolution based on space + 3D convolution based on space-time action recognition usually uses sparse sampling data to form a sequence of picture frames and then input them into 3D action recognition. Since the action usually occurs in a local part of the video, this method will lead to inaccurate action recognition. If segmented dense sampling is used, greater computing power is required. Summary of the invention

[0009] The present invention provides a method, device and electronic device for action recognition, so as to solve the problems that global sparse sampling and local dense sampling cannot be taken into account at the same time and large manpower is required in the existing action recognition process to a certain extent.

[0010] In a first aspect of the present invention, a method for action recognition is provided, the method comprising:

[0011] Obtain the first features of K picture frames in the video to be identified;

[0012] Performing action recognition on the video to be recognized according to the first feature to obtain a first action recognition result;

[0013] According to the first action recognition result, obtaining target positions of M target picture frames in the K picture frames in the video to be recognized;

[0014] Obtaining second features of N picture frames before and after each of the target positions;

[0015] Combining the first feature with the second feature to obtain a second action recognition result;

[0016] Among them, M, N, and K are all positive integers, and K is greater than M.

[0017] In a second aspect of the present invention, there is provided a motion recognition device, the device comprising:

[0018] A first acquisition module, used to acquire first features of K picture frames in the video to be identified;

[0019] A first recognition module, configured to perform action recognition on the video to be recognized according to the first feature, to obtain a first action recognition result;

[0020] A second acquisition module is used to acquire target positions of M target picture frames in the K picture frames in the video to be recognized according to the first action recognition result;

[0021] A third acquisition module is used to acquire the second features of N picture frames before and after each of the target positions;

[0022] A second recognition module, used for combining the first feature and the second feature to obtain a second action recognition result;

[0023] Among them, M, N, and K are all positive integers, and K is greater than M.

[0024] In a third aspect of the present invention, there is also provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0025] Memory, used to store computer programs;

[0026] The processor is used to implement the steps in the above-mentioned action recognition method when executing the program stored in the memory.

[0027] In a fourth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored, and when the program is executed by a processor, the action recognition method as described above is implemented.

[0028] In a fifth aspect of the embodiments of the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the motion recognition method as described above.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] In an embodiment of the present invention, by obtaining the first features of K picture frames in a video to be identified, action recognition is performed on the video to be identified based on the first features to obtain a first action recognition result, and based on the first action recognition result, the target positions of M target picture frames in the video to be identified among the K picture frames are obtained, without the need for manual annotation of the start and end time of the action in the video; and, obtaining the second features of N picture frames before and after each of the target positions, and combining the first features with the second features to obtain a second action recognition result. The combination of two action recognitions and the sparse first features and the dense second features can make the action recognition result more accurate.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for describing the embodiments are briefly introduced below.

[0033] Figure 1 One of the flow charts of the action recognition method provided by the embodiment of the present invention;

[0034] Figure 2 The second flowchart of the action recognition method provided by the embodiment of the present invention;

[0035] Figure 3 A schematic diagram of the structure of the ECO model provided by an embodiment of the present invention;

[0036] Figure 4 A schematic diagram of the structure of a 3D convolutional network provided in an embodiment of the present invention;

[0037] Figure 5 One of the structural schematic diagrams of the residual block provided in the embodiment of the present invention;

[0038] Figure 6 A second structural diagram of a residual block provided in an embodiment of the present invention;

[0039] Figure 7 The third flowchart of the action recognition method provided by the embodiment of the present invention;

[0040] Figure 8 A structural block diagram of a motion recognition device provided by an embodiment of the present invention;

[0041] Fig. 9 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0044] When describing the embodiments of the present invention, some concepts used in the following description are first explained.

[0045] Convolutional Neural Network: An end-to-end implicit model that is commonly used to extract image features or video features and use these features for various visual tasks. The input of the model is often an image or video.

[0046] Convolutional layer: A network layer that performs weighted summation of local pixel values ​​and non-linear activation.

[0047] 2D convolution: A convolution operation that performs convolution in two dimensions: height (H) and width (W).

[0048] 3D convolution: A convolution operation that performs convolution in three dimensions: time (T), H, and W.

[0049] Pooling: An operation that aggregates structured data within a certain range.

[0050] Residual block: A convolutional neural network module built using strided connections, typically consisting of two or three convolutional layers.

[0051] Grad-CAM: Using reverse gradient, the heat map is calculated by multiplying the feature map with its corresponding gradient.

[0052] Reverse gradient: The process of finding the gradient in reverse by calculating the cross entropy between the predicted result and the true result.

[0053] Cross Entropy: A loss function for classification problems.

[0054] Specifically, Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a method for action recognition, and the method specifically includes:

[0055] Step 101, obtaining first features of K picture frames in a video to be identified.

[0056] Specifically, in the above step 101, some picture frames included in the video to be identified are obtained, that is, K picture frames are sparsely sampled, and the first feature of each picture frame in the K picture frames is identified and extracted.

[0057] It should be noted that K picture frames can be input into the first deep convolutional neural network (EfficientConvolutional Network, ECO) model, and the first ECO model can perform feature extraction on each of the K picture frames to obtain the first feature. The first ECO model of lightweight video action recognition is used for feature extraction, which can improve the operating efficiency; and according to business needs, feature extraction can be performed through other action recognition network models with fewer parameters or stronger expression capabilities.

[0058] Step 102: Perform action recognition on the video to be recognized according to the first feature to obtain a first action recognition result.

[0059] Specifically, in the above step 102, action recognition is performed on the video to be identified according to the first feature of each of the K picture frames, and a first action recognition result can be obtained; for example, the first action recognition result can include the identified action category and the probability P1 of the action category, etc., and one or more action categories can be identified, which is not specifically limited here.

[0060] It should be noted that the process of obtaining the first action recognition result by action recognition can also be obtained through the first ECO model, that is, K picture frames are input into the first ECO model, and feature extraction and action recognition are performed through the first ECO model to obtain the first feature and the first action recognition result. Using the first ECO model for feature extraction and action recognition can improve operating efficiency.

[0061] Step 103: Obtain target positions of M target picture frames in the to-be-identified video among K picture frames according to the first action recognition result; wherein K is greater than M.

[0062] Specifically, in the above step 103, according to the first action recognition result, the target positions of M target picture frames in the original video (i.e., the video to be recognized) in the K picture frames can be obtained, that is, which target picture frames in the K picture frames have the greatest impact on the correct action classification result in the second action recognition result, and the video positions (i.e., target positions) of these target picture frames in the video to be recognized are obtained. This process saves resources for human annotation of the start and end time of the action in the video. Among them, since the M target picture frames belong to the K picture frames, the value of K is greater than the value of M.

[0063] Step 104: Obtain second features of N picture frames before and after each of the target positions.

[0064] Specifically, in the above step 104, by obtaining each target position, dense sampling can be performed before and after each target position, and each target position can be densely sampled in N picture frames; then the M target position densely sampled picture frames can be spliced ​​into a densely sampled data in time sequence, and the second feature of the densely sampled data can be obtained, that is, the second feature of the N picture frames before and after each target position is obtained.

[0065] It should be noted that N picture frames (i.e., M*N picture frames) before and after each of the target positions can be input into the second ECO model, and the second ECO model can perform feature extraction on each of the M*N picture frames to obtain the second feature. The second ECO model for lightweight video action recognition is used for feature extraction to improve operating efficiency. Moreover, according to business needs, feature extraction can be performed through other action recognition network models with fewer parameters or stronger expression capabilities.

[0066] Step 105, combining the first feature and the second feature to obtain a second action recognition result; wherein M, N, and K are all positive integers.

[0067] Specifically, in the above step 105, the first features of the sparsely sampled K picture frames and the second features of the densely sampled M*N picture frames are combined to obtain a second action recognition result; for example, the second action recognition result may include the recognized action category and the probability P2 of the action category, etc., and one or more action categories may be recognized, which is not specifically limited here.

[0068] It should be noted that the process of obtaining the second action recognition result by action recognition can also be obtained through the second ECO model, that is, M*N picture frames are input into the second ECO model, feature extraction is performed through the second ECO model to obtain the second feature, and the first feature obtained by the first ECO model is combined with the second feature to obtain the second action recognition result. Using the second ECO model for feature extraction and feature combination can improve operating efficiency and improve action recognition accuracy.

[0069] In the above embodiment of the present invention, by obtaining the first features of K picture frames in the video to be identified, action recognition is performed on the video to be identified based on the first features to obtain a first action recognition result, and based on the first action recognition result, the target positions of M target picture frames in the video to be identified among the K picture frames are obtained, without the need for manual annotation of the start and end time of the action in the video; and, obtaining the second features of the N picture frames before and after each of the target positions, and combining the first feature and the second feature to obtain a second action recognition result. The combination of the two action recognitions and the sparsely sampled first feature and the densely sampled second feature can make the obtained second action recognition result more accurate.

[0070] Optionally, the action recognition method can be applied to an action recognition model, such as Figure 2 As shown, specifically, the step 101 of obtaining the first features of K picture frames in the video to be identified may specifically include:

[0071] Step A1: Obtain the video to be identified.

[0072] Specifically, the video to be identified is a video that needs to perform action recognition. It is necessary to identify the action category and the probability of the action category in the video to be identified, thereby knowing the action category to which the video to be identified belongs.

[0073] Step A2: Decode the video to be identified to obtain S picture frames of the video to be identified.

[0074] Specifically, after obtaining the video to be identified, it is necessary to decode the video to be identified and decompose the video to be identified into S picture frames, that is, to obtain a video sequence frame containing S picture frames, that is, the video to be identified is composed of S picture frames.

[0075] Step A3: extract K picture frames from the S picture frames and combine them to obtain a first picture frame sequence.

[0076] Specifically, the S picture frames may be sparsely sampled at equal intervals to obtain K picture frames, which are combined into a first picture frame sequence. Other sampling methods may also be used for sampling according to business needs, and are not limited to equal interval sampling, to ensure the accuracy of the action recognition result.

[0077] For example, the process of sparse sampling is as follows:

[0078] From the decoded first picture frame sequence, the specified K picture frames are obtained. Combined with the average length of the video to be identified, 7S, K can be taken as 15; there will be a certain interval between two adjacent frames of the picture in the video to be identified, such as: for the video to be identified with a frame rate of 30fps, when the interval between adjacent frames A and B in the video is 1S, there are 29 picture frames between frames A and B; the training and prediction of sparse sampling on the first ECO model can be sampled by taking K frames at equal intervals from the first picture frame sequence; and in order to generate training data for the second ECO model, the video frame interval also needs to be kept fixed during training, and the number of frame intervals SG also needs to be obtained.

[0079] Step A4: Process the first picture frame sequence to obtain first features of K picture frames; wherein S is a positive integer and S is greater than K.

[0080] Specifically, the first features of K picture frames can be extracted by subjecting the first picture frame sequence to image processing in the first ECO model, wherein the value of S is greater than the value of K because the K picture frames are part of the S picture frames.

[0081] Optional, such as Figure 3As shown, the step A4 processes the first picture frame sequence through image processing to obtain first features of K picture frames, including:

[0082] Performing a first image convolution process on the first picture frame sequence in image processing to obtain an intermediate feature map during the processing and a first feature map after the processing;

[0083] Performing a second image convolution process on the intermediate feature map in image processing to obtain a processed second feature map;

[0084] The first feature map and the second feature map are concatenated to obtain first features of K picture frames.

[0085] Specifically, the first picture frame sequence 31 is subjected to a first image convolution process, and an intermediate feature map 34 is obtained during the first image convolution process. After the first image convolution process is completed, a first feature map 33 is obtained; and, the intermediate feature map 34 is subjected to a second image convolution process, and after the second image convolution process is completed, a second feature map 36 can be obtained; then, the first feature map 33 and the second feature map 36 are spliced, and then the first features 38 of K picture frames, i.e., the first features of the first picture frame sequence, are obtained through a fully connected layer 37.

[0086] Specifically, in the process of performing the first image convolution processing on the first picture frame sequence 31 to obtain the intermediate feature map 34, the first picture frame sequence 31 can be firstly subjected to vector convolution operation and maximum pooling in sequence, and then subjected to the first image convolution processing to obtain the intermediate result, i.e., the intermediate feature map 34. Among them, the first image convolution processing is to model the spatial information of each picture frame respectively through the 2D convolution structure 32; the 2D convolution uses the Res50 residual network structure; and in practical applications, the number of channels of the convolution layer operation can be halved to improve the operation efficiency.

[0087] Specifically, after the first picture frame sequence 31 is subjected to vector convolution operation and maximum pooling in sequence, it is subjected to the first image convolution processing and global average pooling to obtain a 1024-dimensional global spatial feature, namely the first feature map 33. Similarly, the intermediate feature map 34 is subjected to vector convolution and second image convolution processing and global average pooling to obtain a 512-dimensional global temporal feature, namely the second feature map 36. The first feature map 33 and the second feature map 36 are spliced ​​to obtain a 1536-dimensional global feature, and global action recognition is performed based on this global feature through a fully connected layer 37 to obtain a first feature 38. Among them, the second image convolution processing is to model the temporal information of adjacent picture frames through a 3D convolution structure 35. Since the 3D convolution parameters are large and the calculation is time-consuming, the network structure needs to be small while meeting the network fitting capability: compared with the residual block in the prior art, the network structure can delete the 1*1 convolution layer and only retain the 3*3 convolution; and the total number of residual blocks is controlled to be 5, such as Figure 4 As shown, it includes a first residual block 41, a first projection residual block 42, a second residual block 43, a second projection residual block 44 and a third residual block 45, and the vector convolution operation Conv convolution layer is controlled at 10 layers. Among them, the structural schematic diagrams of the first residual block 41, the second residual block 43 and the third residual block 45 are all as shown Figure 5 As shown, they include two layers of Conv convolutional layers respectively; the structural schematic diagrams of the first projection residual block 42 and the second projection residual block 44 are both as shown Figure 6 As shown, it includes two Conv convolution layers or one Conv convolution layer respectively.

[0088] It should be noted that 3D convolution can be replaced by other lighter-weight convolution methods according to business needs, such as channel-wise 3D convolution.

[0089] Optionally, the step 103 of obtaining target positions of M target picture frames in the to-be-recognized video among the K picture frames according to the first action recognition result may specifically include:

[0090] Step B1, obtaining the thermal value corresponding to each of the K picture frames according to the first action recognition result.

[0091] Specifically, according to the first action recognition result, the thermal value corresponding to each picture frame in the sparsely sampled K picture frames can be obtained, so that the thermal value can be used as the basis for time period recommendation to save manpower for marking the start and end time of the action.

[0092] Step B2, determining M target picture frames among the K picture frames according to the M target thermal values ​​among the thermal values.

[0093] Specifically, M target thermal values ​​are obtained from the thermal values, and M target picture frames from the K picture frames can be determined based on the M target thermal values, that is, the picture frame corresponding to each target thermal value is the target picture frame, and the target thermal value is used as the basis for recommending time periods. Compared with LSTM, not only can the number of recommended time periods be freely controlled by setting the recommended number M in the target thermal value; and only weakly supervised annotations such as category labels are used in the entire ECO model training process, which can save more annotation manpower compared to the strong supervision in LSTM.

[0094] Furthermore, when LSTM pushes a continuous interval segment, the required annotation data are the category label and the start and end time of the action. Annotating the category label and the start and end time of the action at the same time is about 8 times more labor-intensive than annotating only the category label. Therefore, using the target thermal value as the time period recommendation can save about 7 times the annotation manpower.

[0095] Step B3, obtaining the target position of each target picture frame in the video to be identified.

[0096] Specifically, the target positions (L1-LM) corresponding to the M target picture frames in the to-be-recognized video may be obtained according to the inter-frame interval number SG stored in the sparse sampling.

[0097] Optional, such as Figure 7 As shown, the step B1 obtains the thermal value corresponding to each of the K picture frames according to the first action recognition result, which may specifically include:

[0098] Step C1, obtaining a heat map corresponding to each of the K picture frames according to the first action recognition result and the intermediate feature map.

[0099] Specifically, the first picture frame sequence 71 obtains the intermediate feature map 73 through the first branch 72 of the 2D convolution, that is, the first picture frame sequence 71 obtains the intermediate feature map 73 through a part of the 2D convolution, the intermediate feature map 73 obtains the first feature map through the second branch 74 of the 2D convolution, the intermediate feature map 73 obtains the second feature map through the 3D convolution structure 75, and the first feature map and the second feature map are processed by global pooling, splicing, etc. to obtain the first action recognition result 76; wherein, the heat map 77 corresponding to each picture frame in the K picture frames can be obtained through the first action recognition result 76 and the intermediate feature map 73.

[0100] For example: based on the first action recognition result and the intermediate feature map, the Grad-CAM algorithm can be used to perform reverse gradient calculation to obtain a heat map (HM1~HMK) of each frame image input into the 2D convolutional network; among them, other gradient calculation methods besides Grad-CAM can be used to calculate the heat map according to business needs.

[0101] Furthermore, the Grad-CAM algorithm can be used to obtain the heat map of each feature corresponding to each frame image through reverse gradient calculation of 3D convolution and 2D convolution, and then all feature maps under a frame image are globally pooled, so that a frame image generates a corresponding heat map (denoted as: HM), so K frame images can obtain K heat maps (HM1~HM K ).

[0102] Step C2, performing pooling processing on each of the thermal maps to obtain a thermal value corresponding to each of the K picture frames.

[0103] Specifically, the K heat maps 77 are pooled again so that each frame image corresponds to a pooled heat value (H1~HK), and the calculated K heat values ​​78 can be applied to the positions of dense sampling for training the second ECO model.

[0104] Optionally, the step C1 obtains a heat map corresponding to each of the K picture frames according to the first action recognition result and the intermediate feature map, which may specifically include:

[0105] According to the first action recognition result and the preset result, a loss value between the first action recognition result and the preset result is obtained;

[0106] Perform gradient calculation according to the loss value to obtain a gradient value;

[0107] The gradient value and the intermediate feature map are combined to obtain a heat map corresponding to each of the K picture frames.

[0108] Specifically, the first action recognition result is the predicted value P1. According to P1 and the preset result (that is, the real action label), the loss value loss can be obtained. The gradient value is reversely calculated through the loss value. According to the gradient value and the intermediate feature map, the heat map obtained by multiplying the intermediate feature map and its corresponding gradient value is calculated.

[0109] Optionally, the step B2 determines M target picture frames among the K picture frames according to the M target thermal values ​​among the thermal values, and specifically may include:

[0110] Obtain M thermal values ​​with larger thermal values ​​among the thermal values ​​corresponding to the K picture frames;

[0111] Determine M target thermal values ​​according to M thermal values ​​with larger thermal values;

[0112] According to the M target thermal values, M target picture frames are determined.

[0113] Specifically, among the K heat values corresponding to the K picture frames, the M heat values with the highest heat values can be taken as the target heat values, denoted as Top-M (M < K). Each picture frame corresponding to a target heat value is a target picture frame. The target positions of the corresponding M target picture frames in the video to be recognized can be obtained through the M target heat values.

[0114] Optionally, as Figure 2 shown, step 104 of obtaining the second features of N picture frames before and after each of the target positions may specifically include:

[0115] Step D1, obtaining N picture frames before and after each of the target positions.

[0116] Step D2, combining the N picture frames before and after each of the target positions to obtain a second picture frame sequence.

[0117] Step D3, performing image processing on the second picture frame sequence to obtain the second features of N picture frames before and after each of the target positions.

[0118] Specifically, sampling is performed before and after each target position. Each target position corresponds to N picture frames collected. The picture frames before and after the M target positions collected are combined to obtain a second picture frame sequence, that is, the second picture frame sequence contains N * M picture frames, that is, N * M picture frames are densely sampled. The second picture frame sequence is input into the second ECO model for image processing, that is, similar to the processing process in the first ECO model, which will not be specifically described here, and the second features of each picture frame can be obtained.

[0119] It should be noted that the step of feature combination of the first feature and the second feature may be a feature combination without passing through the second ECO model to obtain the second action recognition result, that is, as Figure 2 shown. Or, the step of feature combination of the first feature and the second feature may be performed in the second ECO model, that is, the first feature obtained by the first ECO model is input into the second ECO model, and the first feature is combined with the second feature, so as to output the second action recognition result through the second ECO model.

[0120] For example, the process of dense sampling is as follows:

[0121] The K sparsely sampled picture frames are obtained and predicted by the first ECO model to obtain the first action recognition result; then the Grad-CAM algorithm is used to calculate the K thermal values ​​corresponding to the K picture frames; then the target positions corresponding to the M target thermal values ​​with the highest thermal values ​​are obtained, where K can be 15 and M can be 3), recorded as Top-M. According to the number of inter-frame intervals SG saved in the sparse sampling, M corresponding target positions can be obtained, and N frames (such as K / M frames) are taken forward and backward with each target position as the center position, and the sampling interval can be set to the value of SG / M. Among them, the N frames of pictures taken forward and backward with each target position as the center position can include the frame pictures of the target position.

[0122] Among them, there are M target positions, and K / M picture frames are sampled at each target position. The picture frames of these M target positions are spliced ​​and combined together in time sequence and still still contain a sequence frame combination of K picture frames. The data format input to the second ECO model is consistent with the data format input to the first ECO model.

[0123] The training process of the first ECO model and the second ECO model is described below:

[0124] First, ECO L1 is trained end-to-end with weak supervision using category labels. The input is a first picture frame sequence consisting of K picture frames sampled sparsely and equally (or in other sampling methods). The output is the first action recognition result, i.e., the predicted value P1. The first ECO model is classified and learned based on the loss value loss between P1 and the true action classification label T-label. Compared with the existing strong supervision method of video category + action start and end time, the weak supervision method using video category annotation training can save a lot of labeling manpower and effectively improve the efficiency of model training iteration.

[0125] Secondly, after completing the training of the first ECO model, use the trained first ECO model to generate the thermal values ​​corresponding to the K sparsely sampled picture frames on each video to be identified and the first features corresponding to the K picture frames. According to the thermal value top-M, select the target position for dense sampling and densely sample the training data. Use the densely sampled data and the first ECO model in combination with the second ECO model for end-to-end training, and output the comprehensive judgment result of the entire video to be identified, that is, the second action recognition result; here, the parameters of the first ECO model can be set not to be learned, and the calculation of thermal values ​​is not involved. Only the parameters of the second ECO model are learned. The purpose is to use the first ECO model to assist the second ECO model in training. Using the first ECO model to assist the second ECO model in training during the training process can improve the training effect of the second ECO model compared to training the second ECO model alone.

[0126] The prediction process of the first ECO model and the second ECO model is described below:

[0127] Use sparsely sampled test sequence frames (such as K frames) and pass them through the first ECO model. Save the first feature of the fully connected layer output of the first ECO model, which is recorded as: sparse feature, and calculate the thermal value. According to the target thermal value of Top-M, the recommended target position is densely sampled to obtain a densely sampled sequence frame (second picture frame sequence), which is input into the second ECO model to obtain the second feature of the fully connected layer output, which is recorded as: dense feature. Use the splicing of sparse features and dense features as input, pass through the last fully connected layer of the second ECO model, and output the final second action recognition result. Use sparse sampling and dense sampling in combination to ensure the sparse density of the input data. Combine the first ECO model and the second ECO model to effectively improve the accuracy of action recognition.

[0128] Optional, such as Figure 2 As shown, after the step 105 combines the first feature and the second feature to obtain a second action recognition result, the method may further include:

[0129] Determining whether the second action recognition result includes a target action that meets a preset condition;

[0130] If the second action recognition result includes a target action, the target action is sent to an action approval office.

[0131] Specifically, after obtaining the second action recognition result, it is determined whether the probability corresponding to the action classification in the second action recognition result meets the preset conditions. If the target action that meets the preset conditions is included, the target action is pushed to the action approval department for approval, and it is determined whether it is an illegal action, etc. The preset condition is a condition for determining whether the second action recognition result contains an illegal action category. Specifically, satisfying the preset condition can be determining that the probability is greater than a preset value. If the probability is less than or equal to the preset value, it is determined that the preset condition is not satisfied.

[0132] In summary, the above embodiment of the present invention controls the number of dense sample segments that the sparse samples pay attention to through M target thermal values, and the number of recommended time periods can be freely selected. After splicing, a dense sampling data containing M time periods can be obtained, which solves the problem that when the action occurs in a discontinuous time due to lens shaking when shooting a short video with a mobile phone, the LSTM only predicts a continuous time period; and, only the category label is used to recommend the start and end time periods of the action, and there is no need to additionally mark the start and end time of the action in the existing technical solution (such as LSTM), which can save labeling manpower; and, through global sparse sampling and local dense sampling, the global information of the entire video to be identified and the local information of the action are guaranteed from the data source, thereby improving the accuracy of action recognition.

[0133] like Figure 8 As shown, an embodiment of the present invention provides a motion recognition device 800, the device comprising:

[0134] A first acquisition module 801 is used to acquire first features of K picture frames in a video to be identified;

[0135] A first recognition module 802, configured to perform action recognition on the video to be recognized according to the first feature to obtain a first action recognition result;

[0136] A second acquisition module 803 is used to acquire target positions of M target picture frames in the K picture frames in the video to be recognized according to the first action recognition result;

[0137] A third acquisition module 804 is used to acquire the second features of N picture frames before and after each of the target positions;

[0138] A second recognition module 805, used to combine the first feature and the second feature to obtain a second action recognition result;

[0139] Among them, M, N, and K are all positive integers, and K is greater than M.

[0140] In the above embodiment of the present invention, by obtaining the first features of K picture frames in the video to be identified, action recognition is performed on the video to be identified based on the first features to obtain a first action recognition result, and based on the first action recognition result, the target positions of M target picture frames in the video to be identified among the K picture frames are obtained, without the need for manual annotation of the start and end time of the action in the video; and, obtaining the second features of the N picture frames before and after each of the target positions, and combining the first feature with the second feature to obtain a second action recognition result. The combination of two action recognitions and the sparse first feature and the dense second feature can make the action recognition result more accurate.

[0141] Optionally, the first acquisition module 801 includes:

[0142] A first acquisition unit, used to acquire a video to be identified;

[0143] A decoding unit, configured to decode the video to be identified to obtain S picture frames of the video to be identified;

[0144] A first combining unit, configured to extract K picture frames from the S picture frames and combine them to obtain a first picture frame sequence;

[0145] A first processing unit, configured to process the first picture frame sequence to obtain first features of K picture frames;

[0146] Wherein, S is a positive integer, and S is greater than K.

[0147] Optionally, the first processing unit includes:

[0148] A first processing subunit, configured to perform a first image convolution process on the first picture frame sequence in image processing to obtain an intermediate feature map during the processing and a first feature map after the processing;

[0149] A second processing subunit is used to perform a second image convolution process in image processing on the intermediate feature map to obtain a processed second feature map;

[0150] The third processing subunit is used to splice the first feature map and the second feature map to obtain first features of K picture frames.

[0151] Optionally, the second acquisition module 803 includes:

[0152] A second acquisition unit, configured to acquire a thermal value corresponding to each of the K picture frames according to the first action recognition result;

[0153] A first determining unit, configured to determine M target picture frames among the K picture frames according to the M target thermal values ​​among the thermal values;

[0154] The third acquisition unit is used to acquire the target position of each target picture frame in the video to be identified.

[0155] Optionally, the second acquiring unit includes:

[0156] A first acquisition subunit, configured to acquire a heat map corresponding to each of the K picture frames according to the first action recognition result and the intermediate feature map;

[0157] The fourth processing subunit is used to perform pooling processing on each of the thermal maps to obtain a thermal value corresponding to each of the K picture frames.

[0158] Optionally, the first acquiring subunit includes:

[0159] According to the first action recognition result and the preset result, a loss value between the first action recognition result and the preset result is obtained;

[0160] Perform gradient calculation according to the loss value to obtain a gradient value;

[0161] The gradient value and the intermediate feature map are combined to obtain a heat map corresponding to each of the K picture frames.

[0162] Optionally, the first determining unit includes:

[0163] The second acquisition subunit is used to acquire M thermal values ​​with larger thermal values ​​among the thermal values ​​corresponding to the K picture frames;

[0164] A first determination subunit is used to determine M target thermal values ​​according to M thermal values ​​with larger thermal values;

[0165] The second determination subunit is used to determine M target picture frames according to the M target thermal values.

[0166] Optionally, the third acquisition module 804 includes:

[0167] A fourth acquisition unit, used to acquire N picture frames before and after each of the target positions;

[0168] A second combining unit, used for combining the N picture frames before and after each of the target positions to obtain a second picture frame sequence;

[0169] The second processing unit is used to process the second picture frame sequence through image processing to obtain second features of N picture frames before and after each of the target positions.

[0170] Optionally, the device further comprises:

[0171] A judging module, used for judging whether the second action recognition result includes a target action that meets a preset condition;

[0172] The sending module is used to send the target action to the action approval department if the second action recognition result includes the target action.

[0173] It should be noted that the action recognition device embodiment is a device corresponding to the above-mentioned action recognition method. All implementation methods of the above-mentioned method embodiment are applicable to the device embodiment and can achieve the same technical effects, which will not be repeated here.

[0174] In summary, the above embodiment of the present invention controls the number of dense sample segments that the sparse samples pay attention to through M target thermal values, and the number of recommended time periods can be freely selected. After splicing, a dense sampling data containing M time periods can be obtained, which solves the problem that when the action occurs in a discontinuous time due to lens shaking when shooting a short video with a mobile phone, the LSTM only predicts a continuous time period; and, only the category label is used to recommend the start and end time periods of the action, and there is no need to additionally mark the start and end time of the action in the existing technical solution (such as LSTM), which can save labeling manpower; and, through global sparse sampling and local dense sampling, the global information of the entire video to be identified and the local information of the action are guaranteed from the data source, thereby improving the accuracy of action recognition.

[0175] The embodiment of the present invention also provides an electronic device. Fig. 9 As shown, it includes a processor 901 , a communication interface 902 , a memory 903 and a communication bus 904 , wherein the processor 901 , the communication interface 902 , and the memory 903 communicate with each other via the communication bus 904 .

[0176] The memory 903 is used to store computer programs.

[0177] When the processor 901 is used to execute the program stored in the memory 903, part or all of the steps in the action recognition method provided by the embodiment of the present invention are implemented.

[0178] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0179] The communication interface is used for communication between the above terminal and other devices.

[0180] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0181] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0182] In another embodiment of the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the action recognition method described in the above embodiment.

[0183] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the action recognition method described in the above embodiment.

[0184] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for motion recognition, characterized in that: The method comprises: Obtaining first features of K picture frames in a to-be-recognized video, where the K picture frames are obtained by sparsely sampling some picture frames contained in the to-be-recognized video; Performing action recognition on the video to be recognized according to the first feature to obtain a first action recognition result; According to the first action recognition result, thermal values ​​corresponding to K picture frames are obtained, M target thermal values ​​are determined according to larger M thermal values ​​among the thermal values, M target picture frames among the K picture frames are determined according to the M target thermal values, and a target position of each of the target picture frames in the video to be recognized is obtained; Obtaining second features of N picture frames before and after each of the target positions, wherein the N picture frames are obtained by densely sampling before and after the target position; Combining the first feature with the second feature to obtain a second action recognition result; Among them, M, N, and K are all positive integers, and K is greater than M.

2. The method according to claim 1, characterized in that The step of obtaining first features of K picture frames in the video to be identified includes: Get the video to be identified; Decoding the video to be identified to obtain S picture frames of the video to be identified; Extract K picture frames from the S picture frames and combine them to obtain a first picture frame sequence; Processing the first picture frame sequence to obtain first features of K picture frames; Wherein, S is a positive integer, and S is greater than K.

3. The method according to claim 2, characterized in that The step of processing the first picture frame sequence to obtain first features of K picture frames includes: The first picture frame sequence is subjected to vector convolution operation and maximum pooling in sequence, and then subjected to first image convolution processing to obtain an intermediate feature map; The first picture frame sequence is subjected to vector convolution operation and maximum pooling in sequence, and then subjected to first image convolution processing, and then subjected to global average pooling to obtain a first feature map; Performing a second image convolution process on the intermediate feature map in image processing to obtain a processed second feature map; The first feature map and the second feature map are concatenated to obtain first features of K picture frames.

4. The method according to claim 3, characterized in that The step of obtaining a thermal value corresponding to each of the K picture frames according to the first action recognition result includes: Obtaining a heat map corresponding to each of the K picture frames according to the first action recognition result and the intermediate feature map; Pooling is performed on each of the thermal maps to obtain a thermal value corresponding to each of the K picture frames.

5. The method according to claim 4, characterized in that The step of obtaining a heat map corresponding to each of the K picture frames according to the first action recognition result and the intermediate feature map includes: According to the first action recognition result and the preset result, a loss value between the first action recognition result and the preset result is obtained; Perform gradient calculation according to the loss value to obtain a gradient value; The gradient value and the intermediate feature map are combined to obtain a heat map corresponding to each of the K picture frames.

6. The method according to claim 1, characterized in that The obtaining of the second features of the N picture frames before and after each of the target positions includes: Obtain N picture frames before and after each of the target positions; Combining N picture frames before and after each of the target positions to obtain a second picture frame sequence; The second picture frame sequence is subjected to image processing to obtain second features of N picture frames before and after each of the target positions.

7. The method according to claim 1, characterized in that After combining the first feature and the second feature to obtain a second action recognition result, the method further includes: Determining whether the second action recognition result includes a target action that meets a preset condition; If the second action recognition result includes a target action, the target action is sent to an action approval office.

8. A motion recognition device, characterized in that: The device comprises: A first acquisition module is used to acquire first features of K picture frames in the video to be identified, where the K picture frames are obtained by sparsely sampling some picture frames contained in the video to be identified; A first recognition module, configured to perform action recognition on the video to be recognized according to the first feature, to obtain a first action recognition result; A second acquisition module is used to acquire target positions of M target picture frames in the K picture frames in the video to be recognized according to the first action recognition result; Wherein, the second acquisition module includes: A second acquisition unit, configured to acquire a thermal value corresponding to each of the K picture frames according to the first action recognition result; A first determining unit, configured to determine M target picture frames among the K picture frames according to the M target thermal values ​​among the thermal values; A third acquisition unit, used to acquire a target position of each target picture frame in the video to be identified; Wherein, the first determining unit includes: The second acquisition subunit is used to acquire M thermal values ​​with larger thermal values ​​among the thermal values ​​corresponding to the K picture frames; A first determination subunit is used to determine M target thermal values ​​according to M thermal values ​​with larger thermal values; A second determining subunit is used to determine M target picture frames according to the M target thermal values; A third acquisition module is used to acquire the second features of N picture frames before and after each of the target positions, wherein the N picture frames are obtained by densely sampling before and after the target position; A second recognition module, used for combining the first feature and the second feature to obtain a second action recognition result; Among them, M, N, and K are all positive integers, and K is greater than M.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus; wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps in the action recognition method as described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the action recognition method according to any one of claims 1 to 7 is implemented.

Images