Action recognition method, apparatus, device, and storage medium

Abstract

The application discloses a motion recognition method, device and equipment and a storage medium, and belongs to the image recognition field. The method comprises the following steps: performing feature extraction on a plurality of video frames in a target video according to a plurality of dimensions to obtain a plurality of multi-channel feature maps. Based on the similarity between each two multi-channel feature maps in the plurality of multi-channel feature maps, a first attention weight of each multi-channel feature map is determined, and based on the plurality of multi-channel feature maps and the plurality of first attention weights, the type of the motion performed by a target object in the target video is determined. Through the above steps, in the process of performing motion recognition on the target video, the similarity between the plurality of video frames in the target video is combined to obtain the first attention weight, the first attention weight can represent the correlation degree between the multi-channel feature map and the motion, and the motion recognition based on the attention weight and the multi-channel feature map of the video frame can improve the accuracy of the motion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to a motion recognition method and device, equipment and a storage medium. BACKGROUND

[0002] With the development of computer technology, more and more scenarios need to perform motion recognition, such as recognizing the motion performed by a person in a monitoring video, or recognizing the motion performed by an actor in a movie video, etc.

[0003] In related technologies, a video is often split into multiple video frames, and the multiple video frames are respectively input into an image recognition model, and the image recognition model performs motion recognition based on each video frame, and finally the motion recognition results of the multiple video frames are fused as the motion recognition result of the video.

[0004] However, since the motion in the video often has continuity, when predicting the motion in the video based on the individual video frames, the continuity of the motion is broken, resulting in low accuracy of motion recognition. SUMMARY

[0005] The embodiments of the present application provide a motion recognition method, device, equipment and storage medium, which can improve the motion recognition effect. The technical solution is as follows:

[0006] On the one hand, a motion recognition method is provided, which comprises:

[0007] Obtaining multiple video frames in a target video;

[0008] Performing feature extraction on the multiple video frames according to multiple dimensions respectively, to obtain multiple multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension;

[0009] Determining a first attention weight of each multi-channel feature map based on the similarity between every two multi-channel feature maps in the multiple multi-channel feature maps, the first attention weight being used to represent the correlation degree of the corresponding multi-channel feature map and the motion performed by an object in the target video in the target video;

[0010] Determining the type of the motion based on the multiple multi-channel feature maps and the determined multiple first attention weights.

[0011] On the one hand, a motion recognition method is provided, which comprises:

[0012] Obtaining multiple video frames in a target video;

[0013] The plurality of video frames are respectively subjected to feature extraction according to multiple dimensions, to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension;

[0014] Based on the similarity between a plurality of feature points on the multi-channel feature maps of each two adjacent video frames, a third attention weight of each feature point is determined, the third attention weight being used to represent the correlation degree of the corresponding feature point and an action performed by an object in the target video in the target video;

[0015] Based on the plurality of multi-channel feature maps and the determined plurality of third attention weights, a type of the action is determined.

[0016] In a possible implementation, the determining of the type of the action based on the plurality of third attention feature maps comprises:

[0017] Based on the similarity between each two third attention feature maps, a fourth attention weight of each third attention feature map is determined, the fourth attention weight being used to represent the correlation degree of the corresponding third attention feature map and the action;

[0018] Based on the plurality of third attention feature maps and the determined plurality of fourth attention weights, the type of the action is determined.

[0019] In an aspect, an action recognition apparatus is provided, the apparatus comprising:

[0020] A first acquisition unit is configured to acquire a plurality of video frames in a target video;

[0021] A first feature extraction unit is configured to subject the plurality of video frames to feature extraction according to multiple dimensions, to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension;

[0022] A first attention weight determination unit is configured to determine a first attention weight of each multi-channel feature map based on the similarity between each two multi-channel feature maps in the plurality of multi-channel feature maps, the first attention weight being used to represent the correlation degree of the corresponding multi-channel feature map and an action performed by an object in the target video in the target video;

[0023] A first type determination unit is configured to determine a type of the action based on the plurality of multi-channel feature maps and the determined plurality of first attention weights.

[0024] In a possible implementation, the first feature extraction unit is configured to, for any video frame, perform feature extraction on the any video frame by using a plurality of convolution kernels, to obtain a plurality of single-channel feature maps corresponding to the plurality of convolution kernels respectively, each convolution kernel being configured to extract a single dimension of feature map; and splice the plurality of single-channel feature maps to obtain a multi-channel feature map of the any video frame.

[0025] In a possible implementation, the first attention weight determination unit is configured to perform dimension reduction processing on the plurality of multi-channel feature maps to obtain a plurality of first feature vectors, each multi-channel feature map corresponding to a first feature vector.

[0026] The first attention weight determination unit is configured to determine a first attention weight of each first feature vector based on a similarity between each two first feature vectors in the plurality of first feature vectors.

[0027] In a possible implementation, the first attention weight determination unit is configured to, for any multi-channel feature map, perform dimension reduction processing on a plurality of single-channel feature maps in the any multi-channel feature map respectively, to obtain a plurality of feature values corresponding to the plurality of single-channel feature maps respectively; and combine the plurality of feature values to obtain a first feature vector of the any multi-channel feature map.

[0028] In a possible implementation, the first attention weight determination unit is configured to, for any first feature vector, obtain a plurality of similarities between the any first feature vector and the plurality of first feature vectors respectively; and determine a first attention weight of the any first feature vector based on a sum of products of the plurality of first feature vectors and corresponding similarities.

[0029] In a possible implementation, the first attention weight determination unit is configured to perform normalization processing on the sum of products of the plurality of first feature vectors and corresponding similarities to obtain the first attention weight of the any first feature vector.

[0030] In a possible implementation, the first type determination unit is configured to multiply the plurality of first attention weights and the plurality of multi-channel feature maps to obtain a plurality of first attention feature maps, each video frame corresponding to a first attention feature map; and perform identification based on the plurality of first attention feature maps to obtain the type of the action.

[0031] In a possible implementation, the first type determining unit is configured to, for any first attention feature map, acquire a plurality of second feature vectors corresponding to a plurality of feature points on the any first attention feature map and a plurality of third feature vectors corresponding to the plurality of feature points on a reference attention feature map corresponding to a video frame adjacent to a video frame corresponding to the any first attention feature map; determine a plurality of second attention weights respectively corresponding to the plurality of feature points on the any first attention feature map based on similarities between the plurality of second feature vectors and the plurality of third feature vectors, the second attention weight being used to represent a degree of relevance of a corresponding feature point to the action; multiply the plurality of second attention weights and the plurality of feature points on the any first attention feature map to obtain a second attention feature map of the any first attention feature map; and perform recognition based on a plurality of the second attention feature maps to obtain the type of the action.

[0032] In an aspect, an action recognition apparatus is provided, the apparatus comprising:

[0033] A second acquiring unit is configured to acquire a plurality of video frames in a target video.

[0034] A second feature extracting unit is configured to perform feature extraction on the plurality of video frames respectively according to a plurality of dimensions to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension.

[0035] A third attention weight determining unit is configured to determine a third attention weight of each feature point based on similarities between the plurality of feature points on multi-channel feature maps of each two adjacent video frames, the third attention weight being used to represent a degree of relevance of a corresponding feature point to an action performed by an object in the target video.

[0036] A second type determining unit is configured to determine the type of the action based on the plurality of multi-channel feature maps and the plurality of determined third attention weights.

[0037] In a possible implementation, the third attention weight determining unit is configured to, for a multi-channel feature map of any video frame, acquire a plurality of fourth feature vectors corresponding to a plurality of feature points on the multi-channel feature map of the any video frame and a plurality of fifth feature vectors corresponding to the plurality of feature points on a multi-channel feature map of a reference video frame adjacent to the any video frame; and determine a plurality of third attention weights of the any video frame based on the plurality of fourth feature vectors and the plurality of fifth feature vectors.

[0038] In a possible implementation, the second type determining unit is configured to multiply the plurality of third attention weights with the plurality of multi-channel feature maps to obtain a plurality of third attention feature maps, each video frame corresponding to a third attention feature map; and perform recognition based on the plurality of third attention feature maps to obtain the type of the action.

[0039] In a possible implementation, the second type determining unit is configured to determine a fourth attention weight of each third attention feature map based on a similarity between every two third attention feature maps, the fourth attention weight being used to represent a degree of relevance of the corresponding third attention feature map to the action; and determine the type of the action based on the plurality of third attention feature maps and the plurality of determined fourth attention weights.

[0040] In an aspect, a computer device is provided, which includes one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the action recognition method.

[0041] In an aspect, a computer readable storage medium is provided, which stores at least one computer program, the computer program being loaded and executed by a processor to implement the action recognition method.

[0042] In an aspect, a computer program product or computer program is provided, which includes program code stored in a computer readable storage medium, the program code being read by a processor of a computer device from the computer readable storage medium, and the processor executes the program code to enable the computer device to perform the action recognition method.

[0043] By means of the technical solutions provided in the embodiments of the present application, in the process of recognizing the action of the target video, the similarity between the plurality of video frames in the target video is combined, and since the action change causes the change of the similarity between the video frames, the similarity can reflect the continuity of the action. The attention weight obtained based on the similarity can represent the degree of relevance of the feature map to the action, and the action recognition based on the attention weight and the multi-channel feature map can improve the accuracy of the action recognition. BRIEF DESCRIPTION OF DRAWINGS

[0044] To make the technical solutions in the embodiments of the present application clearer, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0045] Figure 1 This is a schematic diagram of the implementation environment of an action recognition method provided in an embodiment of this application;

[0046] Figure 2 This is a flowchart of an action recognition method provided in an embodiment of this application;

[0047] Figure 3 This is a flowchart of an action recognition method provided in an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of a multi-channel feature map provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of the structure of an action recognition model provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram illustrating the principle of a channel motion enhancement module provided in an embodiment of this application;

[0051] Figure 7 This is a schematic diagram illustrating the principle of a spatial motion enhancement module provided in an embodiment of this application;

[0052] Figure 8 This is a schematic diagram illustrating a module alternation provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram illustrating a feature map visualization provided in an embodiment of this application;

[0054] Figure 10 This is a flowchart of an action recognition method provided in an embodiment of this application;

[0055] Figure 11 This is a flowchart of an action recognition method provided in an embodiment of this application;

[0056] Figure 12 This is a schematic diagram of the structure of an action recognition device provided in an embodiment of this application;

[0057] Figure 13 This is a schematic diagram of the structure of an action recognition device provided in an embodiment of this application;

[0058] Figure 14 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0059] Figure 15 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0060] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0061] The terms "first", "second", and the like are used in the present application to distinguish between the same or similar items with substantially the same function, and it should be understood that there is no logical or chronological dependency between "first", "second", and "nth", and the number and execution order are not limited.

[0062] The term "at least one" in the present application means one or more, and the meaning of "a plurality of" is two or more, for example, a plurality of reference face images means two or more reference face images.

[0063] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0064] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0065] Machine learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning technologies.

[0066] Normalization processing: mapping number series with different value ranges to the (0, 1) interval for data processing. In some cases, the normalized value can be directly implemented as a probability.

[0067] Attention Mechanism: In the hidden layer of a neural network, the attention mechanism adjusts the direction of attention and the weighting model according to the specific task target. By increasing the weighting of the attention mechanism, the content that does not conform to the attention model is weakened or forgotten. If the direction of attention is based on itself, it is called self-attention mechanism.

[0068] Attention weight: can represent the importance of certain data in the training or prediction process, the importance represents the size of the influence of the input data on the output data. The data with high importance has a higher value of the corresponding attention weight, and the data with low importance has a lower value of the corresponding attention weight. In different scenarios, the importance of data is not the same, and the process of training the attention weight of the model is also the process of determining the importance of the data.

[0069] Convolutional Neural Network (CNN): a kind of feedforward neural network, whose neurons can respond to units within the receptive field. CNN is usually composed of multiple convolutional layers and top fully connected layers, which reduces the parameter quantity of the model by sharing parameters, and is widely used in image and speech recognition.

[0070] Figure 1 is an implementation environment schematic diagram of a motion recognition method provided by an embodiment of the present application, referring to Figure 1 The implementation environment can include a terminal 110 and a server 140.

[0071] The terminal 110 is connected to the server 140 through a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto. The terminal 110 is installed and runs an application program supporting motion recognition.

[0072] Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0073] Optionally, the terminal 110 generally refers to one of multiple terminals, and the embodiments of the present application only take the terminal 110 as an example for illustration.

[0074] The skilled in the art can know that the number of the terminal can be more or less. For example, the terminal is only one, or the terminal is dozens or hundreds, or more, and other terminals are included in the implementation environment.

[0075] After introducing the implementation environment of the technical solution provided by the embodiments of the application, the application scenarios of the application are introduced below.

[0076] The technical solution provided by the embodiments of the application can be applied in the action analysis of objects in a video, video classification, and security video anomaly analysis scenarios, and the embodiments of the application do not limit this. The following will take the above three application scenarios as examples for description.

[0077] 1. In the scenario of action analysis of objects in a video, the computer device can execute the technical solution provided by the embodiments of the application on the video, so as to identify the type of the action performed by the object in the video. For example, the video is a video of a dancer dancing, and then the computer device can identify the type of the dance performed by the dancer by executing the technical solution provided by the embodiments of the application on the video. In some embodiments, if the video is a live video, the computer device can execute the technical solution provided by the embodiments of the application on the live video, so as to obtain the type of the action performed by the host in the live video. The computer device can add a label to the live video based on the type of the action, so that the audience of the live video can know the content of the live video in advance.

[0078] 2. In the scenario of video classification, the computer device can execute the technical solution provided by the embodiments of the application on a plurality of videos, so as to identify the type of the action performed by the object in the videos, and classify the plurality of videos based on the type of the action performed by the object in the videos, that is, classify the videos in which the object performs the same type of action into one category. For example, the computer device can add a video label to each of the plurality of videos based on the type of the action performed by the object in the videos, classify the plurality of videos based on the video label, so as to classify the plurality of videos into a plurality of types, facilitating subsequent query and storage.

[0079] 3. In the scenario of security video anomaly analysis, the computer device can execute the technical solution provided by the embodiments of the application on a security video, and identify the type of the action performed by the object in the security video. Based on the identification of the type of the action, the computer device can mark the object performing the type of action as a target action type, so as to facilitate the security personnel to lock the object in time, wherein the target action type includes actions such as climbing over a wall, breaking a door, waving a knife, and entering a vehicle, and the embodiments of the application do not limit this.

[0080] It should be noted that in the above description process, the above three application scenarios are taken as examples for description, and in other possible cases, the technical solutions provided by the embodiments of the present application can also be applied to the action recognition process of other types of videos, and the embodiments of the present application do not limit this.

[0081] In the following description of the technical solutions provided by the embodiments of the present application, a computer device is taken as an example. In other possible embodiments, a server can also be used as an execution subject to perform, or the cooperation between the terminal and the server can be used to perform, and the embodiments of the present application do not limit the type of execution subject.

[0082] Figure 2 is a flowchart of an action recognition method provided by an embodiment of the present application, referring to Figure 2 , the method comprises:

[0083] 201. The terminal obtains a plurality of video frames in a target video.

[0084] Among them, the target video is any video in the above application scenarios, and the embodiments of the present application do not limit this.

[0085] 202. The terminal extracts features of the plurality of video frames according to a plurality of dimensions respectively, obtains a plurality of multi-channel feature maps, each video frame corresponds to a multi-channel feature map, and each channel represents a dimension.

[0086] Among them, the dimension is also the different aspects of the video frame, for example, one dimension is the background of the video frame, then the features extracted in this dimension can represent the background of the video frame; another dimension is the color of the video frame, then the features extracted in this dimension can represent the color of the video frame. The terminal extracts features of the video frame in multiple dimensions, that is, extracts features of the video frame from multiple aspects, and the obtained multi-channel feature map can comprehensively express the features of the video frame from multiple aspects.

[0087] 203. The terminal determines a first attention weight of each multi-channel feature map based on the similarity between every two multi-channel feature maps in the plurality of video frames, and the first attention weight is used to represent the correlation degree between the corresponding multi-channel feature map and the action performed by the object in the target video.

[0088] If there is a moving object in the target video, the position of the object can be different in different video frames of the target video. If the similarity of two multi-channel feature maps is high, the object can not move or the amplitude of the movement is small in the video frames corresponding to the two multi-channel feature maps. If the similarity of two multi-channel feature maps is low, the object can move in the video frames corresponding to the two multi-channel feature maps. The terminal can use the first attention weight to represent the degree of correlation between the multi-channel feature map and the movement, so as to focus on the multi-channel feature map with a high degree of correlation in the subsequent action recognition process.

[0089] 204. The terminal determines the type of action based on the plurality of multi-channel feature maps and the plurality of determined first attention weights.

[0090] The terminal can strengthen the multi-channel feature map related to the movement based on the first attention weight, thereby improving the accuracy of action recognition.

[0091] In the process of action recognition of the target video, the similarity between the plurality of video frames in the target video is combined. Since the change of the action causes the change of the similarity between the video frames, the similarity can reflect the continuity of the action. The attention weight obtained based on the similarity can represent the degree of correlation between the feature map and the action. Action recognition based on the attention weight and the multi-channel feature map of the video frame can improve the accuracy of action recognition.

[0092] The above steps 201-204 are a simple description of the embodiments of the present application. The technical solutions provided by the embodiments of the present application will be described in detail in combination with some examples. Figure 3 is a flowchart of an action recognition method provided by an embodiment of the present application, referring to Figure 3 , the method comprises:

[0093] 301. The terminal acquires a plurality of video frames in the target video.

[0094] In a possible implementation, the terminal can acquire all the video frames in the target video. In this implementation, the terminal can perform action recognition based on all the video frames of the target video. Since all the video frames carry all the information of the target video, the result of subsequent action recognition based on all the video frames is more accurate.

[0095] In a possible implementation, the terminal can perform interval sampling on the target video to obtain a plurality of video frames, that is, interval N video frames are sampled from the target video to obtain a plurality of video frames, and N is a positive integer greater than or equal to 1.

[0096] In this implementation, the terminal does not need to obtain all the video frames in the target video, but obtains the video frames from the target video in an interval sampling manner, which can reduce the number of video frames and improve the efficiency of action recognition.

[0097] For example, if there are 100 video frames in the target video, the terminal can sample every 2 video frames, i.e., the terminal obtains the first video frame, the fourth video frame, the seventh video frame, and so on, to obtain a plurality of video frames.

[0098] 302. The terminal extracts features of the plurality of video frames according to a plurality of dimensions respectively to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension.

[0099] In a possible implementation, for any video frame, the terminal extracts features of the video frame by using a plurality of convolution kernels to obtain a plurality of single-channel feature maps corresponding to the plurality of convolution kernels, each convolution kernel being used to extract a feature map of a dimension. The terminal splices the plurality of single-channel feature maps to obtain a multi-channel feature map of the video frame.

[0100] In this implementation, the terminal can use different convolution kernels to extract features of different dimensions of the video frame, and the plurality of convolution kernels can operate in parallel, thereby improving the efficiency of feature extraction.

[0101] For example, if there is a video frame The numbers in the matrix are the gray values of the video frame, and the terminal can use a convolution kernel and to perform convolution processing on the video frame to obtain three single-channel feature maps and of the video frame In the process of feature extraction, taking a convolution kernel as an example, the terminal can control the convolution kernel to slide on the video frame . In the sliding process, the terminal obtains the sum of products of the values in the convolution kernel and the corresponding values on the video frame , and takes the sum of products as a value in the single-channel feature map. Taking the case where the convolution kernel covers on the video frame as an example, the terminal can obtain the sum of products of the values in the convolution kernel and on the corresponding positions, i.e., 1x2+2x1+1x1+1x3=8, and 8 is also a value in the single-channel feature map. The terminal controls the convolution kernel to slide on the video frame The user slides up to collect multiple values. The terminal then combines these values ​​to obtain the video frame. A single-channel feature map The terminal can process three single-channel feature maps as well as By overlaying the features, a multi-channel feature map of the video frame is obtained.

[0102] In the example above, during the process of using a convolution kernel to perform convolution processing on a video frame and extracting the single-channel feature map of the video frame, the convolution kernel covers the edges of the video frame less frequently. This results in the size of the single-channel feature map being smaller than the size of the video frame, and the reduction in size may lead to the loss of edge information of the video frame. To ensure the integrity of the information, the terminal can also extract multi-channel feature maps of multiple video frames using the following method.

[0103] In one possible implementation, for any video frame, the terminal fills in the edges of the video frame and extracts features from the filled video frame using multiple convolutional kernels, obtaining multiple single-channel feature maps corresponding to each convolutional kernel. The terminal then concatenates these multiple single-channel feature maps to obtain a multi-channel feature map of the video frame.

[0104] In this implementation, before the terminal uses a convolutional kernel to extract the multi-channel feature map of the video frame, edge padding of the video frame can be performed, thereby ensuring that the edge information of the video frame is completely acquired by the convolutional kernel and improving the integrity of the information carried by the multi-channel feature map.

[0105] Still based on video frames Taking a three-kernel convolution as an example, if the terminal directly uses a 3×3 convolution kernel to process the video frame, only one value will be obtained after processing, which may result in the loss of video frames. This involves partial edge information within the video frame. In this case, the terminal can fill the edges of the video frame with 0s to obtain... The terminal uses three convolutional kernels as well as The padded video frames are convolved to obtain three single-channel feature maps. as well as In the feature extraction process, a convolutional kernel is used. For example, the terminal can control the convolution kernel. The video frame after filling The convolution kernel is slid upwards. During the sliding process, the terminal acquires the convolution kernel. The values ​​in the video frame after padding The sum of the products of the corresponding values ​​is used as a single value in the single-channel feature map. This convolution kernel... Overlay on the video frame For example, the terminal can obtain the convolution kernel. The values ​​in The sum of the products at the corresponding positions is 0×1+0×1+0×1+0×1+2×0+1×1+0×0+1×0+1×3=4, where 4 is a value in the single-channel feature map. The terminal controls this convolutional kernel. The video frame after filling The user slides the slider to acquire multiple values. The terminal then combines these values ​​to obtain a single-channel feature map. The terminal superimposes the three one-dimensional feature values ​​to obtain the multi-channel feature map of the video frame. In this implementation, by filling the video frames, the terminal can enable the convolution kernel to collect edge information of the video frames multiple times during the sliding process. This allows the single-channel feature map to retain the information of the video frames to the greatest extent. At the same time, the size of the final single-channel feature map is the same as the original size of the video frame, which facilitates the subsequent extraction of deeper features.

[0106] Furthermore, the above example is based on the premise that the target video is a grayscale video and the video frames in the target video are grayscale video frames. In other possible implementations, if the target video is a color video and the video frames in the target video are color video frames, the terminal can extract multi-channel feature maps of multiple video frames in the following manner.

[0107] In one possible implementation, for any given video frame, the terminal uses multiple convolutional kernel groups to extract features from the video frame, obtaining multiple single-channel feature maps corresponding to each convolutional kernel group. Each convolutional kernel group includes a target number of convolutional kernels, the number of which is the same as the number of color channels in the video frame. The convolutional kernels in one kernel group are used to extract features in one dimension. The terminal then concatenates these multiple single-channel feature maps to obtain a multi-channel feature map of the video frame.

[0108] In this implementation, the terminal can use different sets of convolutional kernels to extract features from different color channels of video frames, thereby achieving feature extraction of color video frames.

[0109] For example, if a video frame includes three color channels (RGB), then the terminal can use three color matrices. as well as This is used to represent the video frame. The terminal can use multiple convolutional kernel groups to perform convolution processing on the video frame, obtaining multiple single-channel feature maps of the video frame. Taking one convolutional kernel group as an example, since the video frame includes three color channels, a convolutional kernel group will also include three convolutional kernels. as well as Three convolutional kernels are used to extract feature maps for the three color channels, respectively. The terminal uses three convolutional kernels. as well as The three color matrices of this video frame as well as Three color feature maps were obtained by performing convolution processing separately. as well as The terminal will display three color feature maps as well as The fusion process yields a single-channel feature map of the video frame, where the terminal combines three color feature maps. as well as The process of fusion is to combine the three color feature maps. as well as The addition process yields a single-channel feature map of the video frame. The terminal can superimpose single-channel feature maps obtained through multiple convolutional kernel groups to obtain multi-channel feature maps of the video frame. The superposition method is the same as the previous example and belongs to the same inventive concept, so it will not be described again here.

[0110] Optionally, based on the above embodiments, the terminal can also perform the following steps:

[0111] In one possible implementation, for any video frame, the terminal fills the edges of the color matrix of that video frame, and uses multiple convolutional kernel groups to extract features from the filled video frame, obtaining multiple single-channel feature maps corresponding to each convolutional kernel group. The terminal then concatenates these multiple single-channel feature maps to obtain a multi-channel feature map of the video frame.

[0112] In this implementation, before the terminal uses a group of convolutional kernels to extract the multi-channel feature map of the video frame, the color matrix of the video frame can be edge-filled, thereby ensuring that the edge information of the video frame is completely acquired by the group of convolutional kernels and improving the integrity of the information carried by the multi-channel feature map.

[0113] For example, if a video frame includes three color channels (RGB), then the terminal can use three color matrices. as well as This is used to represent the video frame. The terminal can process three color matrices. as well as Perform edge filling to obtain three color matrices after filling. as well as The terminal uses multiple convolutional kernel groups to convolve the padded three color matrices, obtaining multiple single-channel feature maps for the video frame. Taking one convolutional kernel group as an example, since the video frame includes three color channels, a convolutional kernel group will contain three convolutional kernels. as well as Three convolution kernels are respectively used to extract feature maps of three color channels. and The terminal respectively performs convolution processing on the three padded color matrices of the video frame to obtain three color feature maps and The terminal fuses the three color feature maps to obtain a single-channel feature map of the video frame, wherein the process of fusing the three color feature maps, that is, the process of adding the three color feature maps, obtains a single-channel feature map of the video frame and The terminal fuses the three color feature maps to obtain a single-channel feature map of the video frame, wherein the process of fusing the three color feature maps, that is, the process of adding the three color feature maps, obtains a single-channel feature map of the video frame and The terminal fuses the three color feature maps to obtain a single-channel feature map of the video frame, wherein the process of fusing the three color feature maps, that is, the process of adding the three color feature maps, obtains a single-channel feature map of the video frame and The terminal fuses the three color feature maps to obtain a single-channel feature map of the video frame, wherein the process of fusing the three color feature maps, that is, the process of adding the three color feature maps, obtains a single-channel feature map of the video frame and The terminal fuses the three color feature maps to obtain a single-channel feature map of the video frame, wherein the process of fusing the three color feature maps, that is, the process of adding the three color feature maps, obtains a single-channel feature map of the video frame The terminal can superimpose the single-channel feature maps obtained through the multiple convolution kernel groups to obtain a multi-channel feature map of the video frame. The superimposition manner belongs to the same inventive concept as the previous example, and thus will not be described herein.

[0114] 303. The terminal determines a first attention weight of each multi-channel feature map based on a similarity between each two multi-channel feature maps in the multiple multi-channel feature maps, wherein the first attention weight is used to represent a correlation degree between the corresponding multi-channel feature map and an action performed by an object in the target video.

[0115] Optionally, the first attention weight is inversely proportional to the similarity between the two multi-channel feature maps, that is, for any multi-channel feature map, if there is another multi-channel feature map with a high similarity with the multi-channel feature map, it means that the two video frames corresponding to the two multi-channel feature maps have little change in time sequence and contain limited information related to the action, and accordingly, the attention weight between the two multi-channel feature maps is small when the action is recognized. If there is another multi-channel feature map with a low similarity with the multi-channel feature map, it means that the video frames corresponding to the two multi-channel feature maps have great change in time sequence and contain more information related to the action, and accordingly, the attention weight between the two multi-channel feature maps is large when the action is recognized.

[0116] In a possible implementation, the terminal performs dimension reduction processing on the multiple multi-channel feature maps to obtain multiple first feature vectors, and each multi-channel feature map corresponds to a first feature vector. The terminal determines a first attention weight of each first feature vector based on a similarity between each two first feature vectors in the multiple first feature vectors.

[0117] To make the above-mentioned embodiments more clear, the following will be divided into two parts to explain the above-mentioned embodiments, wherein the first part explains the method for the terminal to obtain the first feature vector, and the second part explains the method for the terminal to determine the first attention weight.

[0118] In part 1, in a possible implementation, for any multi-channel feature map, the terminal respectively performs dimension reduction processing on the plurality of single-channel feature maps in the multi-channel feature map to obtain a plurality of feature values corresponding to the plurality of single-channel feature maps respectively. The terminal combines the plurality of feature values to obtain the first feature vector of the multi-channel feature map.

[0119] In this implementation, the terminal can reduce the single-channel feature map to the feature value, and splice the feature values of the plurality of single-channel feature maps into the feature vector of the multi-channel feature map. Compared with the multi-channel feature map, the feature vector contains less amount of data, which improves the efficiency of subsequent obtaining of the attention weight, thereby improving the efficiency of action recognition.

[0120] For example, the terminal can obtain the first feature vector of the multi-channel feature map through the following formula (1).

[0121] z t =Conv(x t ,W1),1≤t≤T (1)

[0122] Wherein, t represents the serial number of the first feature vector, z t represents the first feature vector with serial number t, x t represents the multi-channel feature map with serial number t, t is a positive integer, Conv() represents a convolution function, W1 represents a parameter matrix of a 1x1 convolution kernel, and T represents the number of multi-channel feature maps. In some embodiments, x t represents the above-mentioned multi-channel feature map, and z t represents the first feature vector of the above-mentioned multi-channel feature map.

[0123] For example, taking a multi-channel feature map including three single-channel feature maps as an example, if the multi-channel feature map is The three matrices in the brackets are the three single-channel feature maps included in the multi-channel feature map. The terminal can perform average pooling on the three single-channel feature maps to convert the three single-channel feature maps into three feature values, that is, the terminal respectively obtains three single-channel feature maps The average values of 7.9, 5.2 and 8.5 are obtained as three feature values. The terminal combines the three feature values 7.9, 5.2 and 8.5 to obtain the first feature vector (7.9, 5.2, 8.5) of the multi-channel feature map. In this way, the terminal can represent the single-channel feature map by using the average value of the numerical values in the single-channel feature map, thereby reducing the amount of calculation and improving the processing efficiency.

[0124] It should be noted that in the above examples, the terminal is taken as an example to reduce the dimension of the single-channel feature map by using mean pooling. In other possible embodiments, the terminal can also reduce the dimension of the single-channel feature map by using maximum pooling or minimum pooling to obtain the feature values corresponding to the single-channel feature map. The method of reducing the dimension of the single-channel feature map by using maximum pooling by the terminal will be described below.

[0125] For example, taking a multi-channel feature map including three single-channel feature maps as an example, if the multi-channel feature map is The three matrices in the brackets are the three single-channel feature maps included in the multi-channel feature map. The terminal can perform maximum pooling on the three single-channel feature maps to convert the three single-channel feature maps into three feature values, that is, the terminal obtains three feature values 11, 10 and 14 from the three single-channel feature maps The maximum values of 11, 10 and 14 are obtained as three feature values. The terminal combines the three feature values 11, 10 and 14 to obtain the first feature vector (11, 10, 14) of the multi-channel feature map. In this way, the terminal can represent the single-channel feature map by using the maximum value of the numerical values in the single-channel feature map, thereby reducing the amount of calculation and improving the processing efficiency.

[0126] Part 2: For any first feature vector, the terminal respectively obtains a plurality of similarities between the first feature vector and a plurality of first feature vectors. The terminal determines a first attention weight of the first feature vector based on a sum of products of the plurality of first feature vectors and corresponding similarities.

[0127] In this embodiment, the terminal can represent the corresponding multi-channel feature map by using the first feature vector, and represent the similarity between the multi-channel feature maps by using the similarity between the first feature vectors, thereby reducing the number of parameters and improving the operation efficiency.

[0128] For example, the terminal can respectively obtain the similarities between the first feature vector and a plurality of first feature vectors by using the following formula (2), obtain the sum of products of the plurality of first feature vectors and corresponding similarities by using the following formula (3), and obtain the first attention weight of the first feature vector by using the following formula (4).

[0129] d tj = -z t ·z j , 1≤t≤T, 1≤j≤T (2)

[0130] wherein d tj denotes a similarity between a first feature vector z t and a first feature vector z j , z t and z j denote a first feature vector with a serial number t and a first feature vector with a serial number j respectively, T denotes a number of channel feature maps, i.e. a number of first feature vectors corresponding to the channel feature maps, t and j each denote a serial number of a first feature vector, and t and j are each a positive integer. In some embodiments, denotes that a sum of the plurality of similarities is 1.

[0131]

[0132] wherein t denotes a serial number of a product sum, denotes a product sum with a serial number t, and the product sum corresponds to a multi-channel feature map x t .

[0133]

[0134] wherein t denotes a serial number of a first attention weight, a t denotes a first attention weight with a serial number t, σ() denotes a normalization function, W2 denotes a parameter matrix, and the first attention weight a t corresponds to a multi-channel feature map x t . In some embodiments, a t denotes a first attention weight of the first feature vector.

[0135] Taking any first feature vector as an example, the terminal obtains a plurality of similarities between the first feature vector and a plurality of first feature vectors respectively. The terminal normalizes a sum of products of the plurality of first feature vectors and the corresponding similarities to obtain a first attention weight of the first feature vector. For example, if there are three video frames in the target video, the three video frames are marked as video frame A, video frame B and video frame C in the order of playing time. After the processing of part 1, the terminal can obtain three first feature vectors corresponding to the three video frames respectively. In some embodiments, the first feature vector of the video frame A is (1, 2, 3), the first feature vector of the video frame B is (1, 4, 3), and the first feature vector of the video frame C is (2, 1, 2). For the video frame A, the terminal can obtain the cosine similarity 0.94 between the first feature vector (1, 2, 3) of the video frame A and the first feature vector (1, 4, 3) of the video frame B, and the cosine similarity 0.89 between the first feature vector (1, 2, 3) of the video frame A and the first feature vector (2, 1, 2) of the video frame C. The terminal obtains the product (0.94, 3.76, 2.82) of the cosine similarity 0.94 between the first feature vector (1, 2, 3) of the video frame A and the first feature vector (1, 4, 3) of the video frame B and the first feature vector (1, 4, 3) of the video frame B. The terminal obtains the product (1.78, 0.89, 1.78) of the cosine similarity 0.89 between the first feature vector (1, 2, 3) of the video frame A and the first feature vector (2, 1, 2) of the video frame C and the first feature vector (2, 1, 2) of the video frame C. The terminal fuses the first feature vector (1, 2, 3) of the video frame A and the product (0.94, 3.76, 2.82) and the product (1.78, 0.89, 1.78) to obtain the product sum (3.72, 6.65, 7.6). The terminal normalizes the product sum (3.72, 6.65, 7.6) by the Sigmoid (S-shaped growth curve) function to obtain the first attention weight (0.97, 0.99, 0.99), wherein the numbers respectively represent the weights of the three single-channel feature maps in the multi-channel feature map of the video frame A. For the first feature vectors of the video frame B and the video frame C, the terminal can determine the first attention weight of the first feature vector of the video frame B based on the cosine similarity between the first feature vector of the video frame B and the first feature vector of the video frame A and the first feature vector of the video frame C respectively. The terminal can determine the first attention weight of the first feature vector of the video frame C based on the cosine similarity between the first feature vector of the video frame C and the first feature vector of the video frame A and the first feature vector of the video frame B respectively.

[0136] It should be noted that the above is an example of using a sigmoid function to obtain the first attention weight, and using cosine similarity to represent the similarity between vectors. In other possible implementations, the terminal can also use other activation functions to obtain the first attention weight, such as using a Softmax (Softmax) function to normalize the product sum to obtain the first attention weight, and can also represent the similarity between vectors through the dot product of vectors. The embodiments of the present application do not limit this.

[0137] 304、The terminal multiplies the plurality of first attention weights and the plurality of multi-channel feature maps to obtain a plurality of first attention feature maps, each video frame corresponding to a first attention feature map.

[0138] In a possible implementation, the terminal can process each first attention weight and the corresponding multi-channel feature map through the following formula (5) to obtain a plurality of first attention feature maps.

[0139] u t =x t ⊙a t ,1≤t≤T (5)

[0140] Wherein, u t represents the first attention feature map numbered t, and the first attention feature map u t corresponds to the multi-channel feature map x t .

[0141] Taking a multi-channel feature map , the terminal can multiply the multi-channel feature map with the corresponding first attention weight, such as (0.8, 0.75, 0.9), to obtain the first attention feature map of the multi-channel feature map The terminal can obtain a plurality of first attention feature maps by multiplying a plurality of multi-channel feature maps with a plurality of corresponding first attention weights in the above manner. Since the first attention weight represents the correlation degree of the multi-channel feature map and the action, after multiplying the multi-channel feature map with the corresponding first attention weight, the single-channel feature map related to the action in the multi-channel feature map can be highlighted, and the single-channel feature map unrelated to the action can be suppressed, thereby improving the accuracy of subsequent action recognition.

[0142] 305、The terminal performs recognition based on the plurality of first attention feature maps to obtain the type of the action.

[0143] In a possible implementation, for any first attention feature map, the terminal performs full connection processing on the first attention feature map to obtain a first attention feature vector of the first attention feature map. The terminal performs normalization processing on the first attention feature vector to obtain a first prediction vector of the first attention feature vector. The terminal determines the type of the action based on the first prediction vector.

[0144] In this implementation, since the first attention feature map is a feature map determined based on the first attention weight, the first attention weight highlights information related to the action, and the terminal can obtain more accurate recognition effect based on the first attention feature map for action recognition.

[0145] For example, for a first attention feature map , the terminal can multiply the first attention feature map by a weight matrix [2, 2, 3] T , and add the result to a bias matrix [1, 1, 1] T , to obtain three attention feature vectors [17.8, 44.8, 48.8] T , [18, 32.25, 30.75] T , and [39.6, 63.9, 57.6] T of the first attention feature map. The terminal fuses the three attention feature vectors to obtain a first attention feature vector [150.8, 281.9, 137.15] T of the first attention feature map. The terminal performs normalization processing on the first attention feature vector by using a Softmax function to obtain a first prediction vector [0.26, 0.50, 0.24] T of the first attention feature vector. If the value 0.5 in the first prediction vector [0.26, 0.50, 0.24] T corresponds to the action type of running, the terminal can determine running as the action type recognized based on the first attention feature map. Of course, one first attention feature map corresponds to one video frame, and multiple first attention feature maps correspond to multiple video frames. The terminal can perform the above full connection processing and normalization processing on the multiple first attention feature maps to obtain action recognition results of the multiple first attention feature maps. The terminal obtains the type of the action performed by the object in the target video in the target video based on the action recognition results of the multiple first attention feature maps. For example, the terminal can determine the action type with the highest occurrence frequency in the action recognition results of the multiple first attention feature maps as the type of the action performed by the object in the target video in the target video.

[0146] It should be noted that in addition to obtaining the type of action performed by the object in the target video through the action recognition result of the plurality of first attention feature maps, the terminal can also fuse the first prediction vectors corresponding to the plurality of first attention feature maps to obtain a fused prediction vector. The terminal normalizes the fused prediction vector to obtain the type of action performed by the object in the target video.

[0147] In a possible implementation, the step 305 can also be implemented by the following steps 3051-3054.

[0148] 3051. For any first attention feature map, the terminal obtains a plurality of second feature vectors corresponding to a plurality of feature points on the first attention feature map, and a plurality of third feature vectors corresponding to a plurality of feature points on a reference attention feature map of the first attention feature map, wherein the video frame corresponding to the reference attention feature map is adjacent to the video frame corresponding to any first attention feature map.

[0149] In some embodiments, the video frame D is a previous video frame of the video frame E, the video frame corresponding to the first attention feature map is the video frame D, and the video frame corresponding to the reference attention feature map is the video frame E.

[0150] In a possible implementation, since the first attention feature map is a feature map obtained by multiplying the multi-channel feature map by the first attention weight, for each first attention feature map, a plurality of sub-attention feature maps are included, and the number of sub-attention feature maps is the same as the number of single-channel feature maps in the corresponding multi-channel feature map. The feature points at the same positions on the plurality of sub-attention feature maps of the first attention feature map correspond to the same pixel point on the video frame. The terminal can splice the values of the feature points at the same positions on the plurality of sub-attention feature maps of the first attention feature map to obtain a plurality of second feature vectors, and one second feature vector corresponds to one pixel point on the video frame, that is, one second feature vector corresponds to one pixel point on the video frame D. The terminal splices the values of the feature points at the same positions on the plurality of sub-attention feature maps of the reference attention feature map to obtain a plurality of third feature vectors, and one third feature vector corresponds to one pixel point on the video frame, that is, one third feature vector corresponds to one pixel point on the video frame E.

[0151] In this implementation, the terminal obtains the feature vector of each feature point from the plurality of sub-attention feature maps of a first attention feature map, which facilitates subsequent comparison of the differences between the pixel points at the same positions in different video frames, thereby obtaining the action-related information and improving the accuracy of subsequent action recognition.

[0152] For example, refer to Figure 4 , Figure 4 The first attention feature map 401 includes a plurality of sub-attention feature maps 4011, and the reference feature map 402 includes a plurality of sub-attention feature maps 4021. Figure 4 In the first attention feature map, the plurality of sub-attention feature maps 4011 are arranged in a top-to-bottom order, and the positions of the plurality of sub-attention feature maps 4011 are aligned with each other, that is, the projections of the plurality of sub-attention feature maps 4011 from top to bottom are the same. Figure 4 The values of the intersection points of the cube 403 and the plurality of sub-attention feature maps 4011 form a second feature vector. Figure 4 The values of the intersection points of the cube 404 and the plurality of sub-attention feature maps 4021 form a third feature vector.

[0153] 3052、The terminal determines a plurality of second attention weights corresponding to a plurality of feature points on the first attention feature map based on the similarity between the plurality of second feature vectors and the plurality of third feature vectors, and the second attention weight is used to represent the correlation degree of the corresponding feature point and the action.

[0154] Optionally, the second attention weight is inversely proportional to the similarity between the second feature vector and the third feature vector, that is, for any second feature vector, if the similarity between the corresponding third feature vector and the second feature vector is high, it means that the two pixel points corresponding to the second feature vector and the third feature vector change little in time sequence and contain limited information related to the action, and accordingly, the attention weight between the second feature vector and the third feature vector is small when the action is recognized; if the similarity between the corresponding third feature vector and the second feature vector is low, it means that the pixel points corresponding to the second feature vector and the third feature vector change greatly in time sequence and contain more information related to the action, and accordingly, the attention weight between the second feature vector and the third feature vector is large when the action is recognized.

[0155] In a possible implementation, the terminal can obtain the cosine similarity between the plurality of second feature vectors and the corresponding plurality of third feature vectors, and determine a plurality of second attention weights corresponding to a plurality of feature points on the first attention feature map.

[0156] In this implementation, the terminal can determine the difference between the pixel points at the same position in different video frames through the similarity between the second feature vector and the third feature vector, and determine the action information carried by the pixel points based on the difference, thereby improving the accuracy of subsequent action recognition.

[0157] For example, the terminal can obtain the similarity between the second feature vector and the third feature vector through the following formulas (6) and (7).

[0158]

[0159]

[0160] Where t represents the index of the cosine similarity, and s t Represents the second eigenvector and the third eigenvector The cosine similarity between them, where η() represents the function for calculating the cosine similarity. Represents the first attention feature map xz t The second eigenvector of the feature point (i, j) For reference feature map xz t+1 The third eigenvector of the feature point (i, j), where i and j are the x and y coordinates of the feature point, respectively, and ||||2 is the L2 norm.

[0161] Taking a second feature vector of [1, 2, 5, 4, 2] and its corresponding third feature vector of [2, 2, 1, 1, 1] as an example, the terminal can obtain a cosine similarity of 0.512 between the second feature vector [1, 2, 5, 4, 2] and the third feature vector [2, 2, 1, 1, 1]. The terminal can determine 1 - 0.512 = 0.488 as the second attention weight of the feature point corresponding to the second feature vector.

[0162] 3053. The terminal multiplies multiple second attention weights with multiple feature points on the first attention feature map to obtain the second attention feature map of the first attention feature map.

[0163] In one possible implementation, the terminal can obtain the second attention feature map using the following formula (8).

[0164] v t =BN(Conv(xz) t ·(1-s t )))+x t ,1≤t≤T-1 (8)

[0165] Where t represents the index of the second attention feature map, v t This represents the second attention feature map with ID t, where BN is the activation function, and xz t This represents the first attention feature map with the number t.

[0166] For example, for the first attention feature map For example, the terminal can multiply the second attention weight corresponding to the first attention feature map and the first attention feature map to obtain a second attention feature map of the first attention feature map.

[0167] 3054、The terminal performs recognition based on the plurality of second attention feature maps to obtain the type of the action.

[0168] In a possible implementation, for any second attention feature map, the terminal performs full connection processing on the second attention feature map to obtain a first attention feature vector of the second attention feature map. The terminal performs normalization processing on the first attention feature vector to obtain a first prediction vector of the first attention feature vector. The terminal determines the type of the action based on the first prediction vector.

[0169] In this implementation, since the second attention feature map is a feature map determined based on the first attention weight, the first attention weight highlights information related to the action, and the terminal performs action recognition based on the second attention feature map, which can obtain more accurate recognition effect.

[0170] For example, for the second attention feature map The terminal can multiply the second attention feature map and the weight matrix [2, 2, 3] T and add the bias matrix [1, 1, 1] T to obtain three attention feature vectors [8.96, 35.84, 14.4] T , [9.6, 25.2, 10.35] T and [24.48, 45.36, 16.38] T of the second attention feature map. The terminal fuses the three attention feature vectors to obtain a second attention feature vector [43.04, 106.4, 41.13] T of the second attention feature map. The terminal performs normalization processing on the second attention feature vector by using a Softmax function to obtain a second prediction vector [0.23, 0.56, 0.21] T of the second attention feature vector. If the second prediction vector [0.23, 0.56, 0.21] T ​If the value 0.56 in the second attention feature map corresponds to the action type of running, the terminal can determine running as the action type identified based on the second attention feature map. Of course, one second attention feature map corresponds to one video frame, and multiple second attention feature map correspond to multiple video frames. The terminal can perform the full connection processing and the normalization processing on the multiple second attention feature maps, so as to obtain the action recognition results of the multiple second attention feature maps. The terminal obtains the type of the action performed by the object in the target video in the target video based on the action recognition results of the multiple second attention feature maps. For example, the terminal can determine the action type with the highest occurrence frequency in the action recognition results of the multiple second attention feature maps as the type of the action performed by the object in the target video in the target video.

[0171] Through the steps 3051-3054, the terminal can further determine the second attention weight based on the determination of the first attention weight. The first attention weight and the second attention weight are both used to highlight the features related to the action. The action recognition based on the first attention weight and the second attention weight can improve the accuracy of the action recognition.

[0172] All the optional technical solutions described above can be combined to form optional embodiments of the present application, which will not be described one by one here.

[0173] In order to more clearly illustrate the steps 301-305 described above, the following will be described in combination with the various optional embodiments described above, Figure 5 , Figure 6 and Figure 7 .

[0174] In some embodiments, the steps 302-305 described above can be performed by different modules of an action recognition model. For example, referring to Figure 5 , the step 302 is performed by a feature extraction module 5011 of the action recognition model 501, the steps 303 and 304 are performed by a channel-wise motion enhancement module (CME) 5012 of the action recognition model 501, and the first implementation of the step 305 is performed by a prediction module 5013 of the action recognition model 501. In the second implementation of the step 305, the steps 3051-3053 are performed by a spatial-wise motion enhancement module (SME) 5014 of the action recognition model 501, and the step 3054 is performed by the prediction module 5013.

[0175] Figure 6 is a schematic diagram of the channel-wise motion enhancement module 5012. The following will be described in combination with Figure 6The processing procedure of the channel motion increasing module 5012 is described.

[0176] In the channel motion increasing module 5012, the “channel” is the “dimension” in the steps 301-305, and one channel corresponds to one dimension. See Figure 6 , which includes three multi-channel feature maps x1, x i and x T . The three multi-channel feature maps x1, x i and x T correspond to three video frames in the target video. The terminal can perform global average pooling on the three multi-channel feature maps x1, x i and x T through the channel motion increasing module 5012 to obtain three first feature vectors z1, z i and z T corresponding to the three multi-channel feature maps. In some embodiments, for the three first feature vectors z1, z i and z T , z1 is the first feature vector with the serial number 1, which is the first first feature vector in the above formula (1), z i is the first feature vector with the serial number i, i is a positive integer with the same value range as t, z T is the first feature vector with the serial number T, which is the last first feature vector in the above formula (1). The terminal obtains the three first feature vectors z1, z i and z T through the channel motion increasing module 5012, that is, the three first feature vectors z1, z i and z T are obtained by processing the three multi-channel feature maps x1, x i and x T using the formula (1) respectively.

[0177] The terminal obtains d 1i , d 1T and d iT between the three first feature vectors z1, z i and z T through the channel motion increasing module 5012, where d 1i is the distance between the first feature vectors z1 and z i , d 1T is the distance between the first feature vectors z1 and z T , and d iT is the distance between the first feature vectors z i and z TThe similarity between them can optionally be the cosine similarity between the two first feature vectors, or the dot product of the two first feature vectors. In some embodiments, the terminal obtains three first feature vectors z1, z2, z3, z4, z5, z6, z7, z8, z9, z1 ...1, z2, z3, z4, z5, z6, i and z T between d 1i d 1T and d iT The process is to respectively set z1 and z i and z T Substituting into formula (2) above, we get d 1i d 1T and d iT The process.

[0178] The terminal obtains three first feature vectors z1, z2, z3, z4, z5, z6, z7, z8, z9, z1 ... i and z T With the corresponding d 1i d 1T and d iT The product between as well as In some embodiments, for obtaining three first feature vectors z1, z2, z3, z4, z5, z6, z7, z8, z9, z1 ... i and z T With the corresponding d 1i d 1T and d iT The product between as well as In other words, it means z1, z i and z T With the corresponding d 1i d 1T and d iT Substituting each into formula (3) above, we obtain the product and sum. as well as

[0179] The terminal can normalize (σ) the sum of the products of the three multi-channel feature maps by the channel motion enhancement module 5012, thereby obtaining three multi-channel feature maps x1, x2, x3, x4, x5, x6, x7, x8, x9, x1, x1, x9, x1, x1, x2 ... i and x T The corresponding three first attention weights a1, a2, a3, a4, a5, a6, a7, a8, i and a T In some embodiments, the terminal acquires three multi-channel feature maps x1, x2, x3, x4, x5, x6, x7, x8, x9, x1 ...1, x9, x1, x1, x2, x1, x3, x1, x1, x2, x1, x1, x2, x1, x1 i and x T The corresponding three first attention weights a1, a2, a3, a4, a5, a6, a7, a8, i and a T That is, combining the three multi-channel feature maps x1, x2, x3, x4, x5, x6, x7i and x T respectively and respectively i and a T .

[0180] The terminal multiplies the three multi-channel feature maps x1, x i and x T respectively with the three first attention weights a1, a i and a T respectively, to obtain three first attention feature maps u1, u i and u T . In some embodiments, the terminal obtains the three first attention feature maps u1, u i and u T through the channel motion enhancement module 5012, that is, the terminal multiplies the three multi-channel feature maps x1, x i and x T respectively with the three first attention weights a1, a i and a T respectively, to obtain three first attention feature maps u1, u i and u T .

[0181] Figure 7 is a schematic diagram of the spatial motion enhancement module 5014, and the processing process of the spatial motion enhancement module 5014 will be described below in combination with Figure 7 .

[0182] Referring to Figure 7 , the first attention feature map and the reference feature map of the first attention feature map are included, the first attention feature map is the attention feature map corresponding to the video frame F, and the reference feature map is the attention feature map corresponding to the video frame G. Taking a feature point as an example, the terminal can obtain, through the spatial motion enhancement module 5014, a second feature vector corresponding to the feature point on the first attention feature map and a third feature vector corresponding to the feature point at the same position on the reference feature map. The terminal can obtain, through the spatial motion enhancement module 5014, the cosine similarity between the second feature vector and the third feature vector. In some embodiments, the process of obtaining the cosine similarity between the second feature vector and the third feature vector through the spatial motion enhancement module 5014, that is, the process of bringing the second feature vector and the third feature vector into the above formulas (6) and (7), to obtain the corresponding cosine similarity.

[0183] The terminal uses the spatial motion enhancement module 5014 to determine the second attention weight of the feature point based on the cosine similarity, and multiplies the second attention weight with the corresponding feature point to obtain the second attention feature map. In some embodiments, the terminal uses the spatial motion enhancement module 5014 to obtain the second attention feature map based on the above formula (8).

[0184] It should be noted that the action recognition model can include multiple channel motion enhancement modules 5012 and multiple spatial motion enhancement modules 5014. Channel motion enhancement modules 5012 and spatial motion enhancement modules 5014 can be used interchangeably within the action recognition model. In some embodiments, if a module includes only channel motion enhancement modules 5012, it is referred to as CMR-Block A; if a module includes both motion enhancement modules 5012 and spatial motion enhancement modules 5014, it is referred to as CMR-Block B. An alternating example of CMR-Block A and CMR-Block B can be found in [reference needed]. Figure 8 .

[0185] During the experiment, a key observation for the CME module is as follows: After the multi-channel feature maps from the previous layer are processed by the CME module to extract action-related attention information for strengthening certain multi-channel feature maps, which multi-channel feature maps are strengthened (assigned greater weight) and which are weakened (assigned less weight), what information these channels carry, whether they are action-related, and whether they reflect the action attention mechanism. Therefore, the multi-channel feature maps strengthened and weakened by CME are visualized here. First, a multi-channel feature map from a CMR-BlockB module is extracted. Then, the multi-channel feature maps of each frame are rearranged in descending order according to the first attention weights generated by the CME module. The first and last ten multi-channel feature maps of each frame are selected and divided into two groups. The mean of the ten multi-channel feature maps in each group is calculated and compressed into one channel. This single feature map is then used to generate the corresponding heatmap. See [link to relevant documentation]. Figure 9 The first four rows of images show heatmaps generated according to the above rules for the top-10 and bottom-10 scores, respectively. The first and second rows are the average values ​​of all single-channel feature maps of the original video frame and the multi-channel feature map, respectively. The visualization results show that in the top 10 multi-channel feature maps selected by CME for enhancement, the areas containing moving objects show significantly higher attention, while the 10 multi-channel feature maps to be suppressed are more sensitive to static background areas. In other words, the CME module provided in this application can effectively distinguish attention information, enhancing multi-channel feature maps sensitive to moving targets and suppressing multi-channel feature maps that tend to describe static background information.

[0186] For the SME module, visualize the second attention weight map generated by CMR-BlockB. For example... Figure 9 The last two lines show the effect after the heatmap of the second attention weight is superimposed on the original video frame. It can be observed that a highlighted area appears only at the location of an object when its position or shape changes between video frames. The feature values ​​within this area are assigned higher weights to reflect its importance.

[0187] During the experiment, datasets something-something v1 and v2 were used to test models with different structures. Dataset something-something v1 contains 108,499 test videos, and dataset something-something v2 contains 220,847 test videos. Both datasets something-something v1 and v2 include 174 action categories. When testing with test videos from datasets something-something v1 and v2, the video data was preprocessed. Specifically, the shorter side of each video frame was adjusted to 256, then center-cropped and scaled, and finally each video frame was resized to 224×224. For datasets something-something v1 and v2, 8 or 16 frames were uniformly sampled from each test video as a test set, and these test sets were then input into different models.

[0188] Table 1

[0189]

[0190] Table 1 shows the method proposed in the present application and other advanced algorithms in which, I3D is the full name of (Inflated 3D Conv Net, Inflated 3D Conv Net), NL-I3D is a branch of I3D network, NL-I3D+GCN is the full name of (Inflated 3D Conv Net+Graph Convolutional Network, Inflated 3D Conv Net+Graph Convolutional Network), ECO is the full name of (Efficient Convolution Operators for Tracking, Efficient Convolution Operators for Tracking), TSM is the full name of (Temporal Shift Module, video action classification network), TEI is the full name of (Towards an Efficient Architecture for Video Recognition, Efficient Video Recognition Model), STM is the full name of (Structural Topic Model, Propositional Structure Model), TEA is the full name of (Temporal Excitation and Aggregation for Action Recognition, Action Recognition Model), GSM is the full name of (Gate-Shift Networks for Video Recognition, Gate-Shift Networks for Video Recognition Model), TDRL is the full name of (Temporal Distinct Representation Learning For Action Recognition, Temporal Distinct Representation Learning For Action Recognition Model), CMR is the full name of (Comprehensive Motion Representation for Action Recognition, Comprehensive Motion Representation for Action Recognition Model), CMR is the model provided by the application embodiment, and its structure is based on ResNet-50. In addition, for ResNet, the full name is (Residual Network, Residual Network), and the number after ResNet is the model depth. Inception V3 is the third version of the original network.

[0191] As can be seen from Table 1, compared with the ResNet2D method, that is, TDRL, the technical solution (CMR) provided in the embodiments of the present application improves the accuracy by 1.5% in something-something v1 and 1.1% in something-something v2 under the condition of 8 frames. Under the condition of 16 frames, the accuracy is improved by 2.3% in something-something v1 and 1.9% in something-something v2. Compared with other methods, it can also be seen that the method (CMR) provided in the present application achieves higher prediction accuracy under the premise of keeping similar or even less calculation amount than other algorithms.

[0192] In addition, in order to verify the effectiveness of the model, the CME module and the SME module provided in the embodiments of the present application are inserted in different ways on the basis of the ResNet model for testing, and the results are shown in Tables 2-4.

[0193] Table 2

[0194] Method Top-1 ResNet-50 18.1 ResNet-50+CME and SME 43.5 ResNet-50+TIM 46.1 ResNet-50+TIM+CME and SME 51.3

[0195] Table 3

[0196] Method Top-1 Top-5 Latency Baseline + CME all +SME all ]]> 51.0 79.3 30.4ms Baseline + CME all +SME part ]]> 51.3 79.8 21.4ms Baseline + CME part +SME all ]]> 51.2 79.2 26.5ms Baseline + CME part +SME part ]]> 50.4 78.6 17.4ms

[0197] Table 4

[0198] Method Top-1 Top-5 ResNet-50+TIM (baseline) 46.1 74.7 Baseline+MEM 47.4 76.6 Baseline+PEM 48.7 77.8 Baseline+CME 50.6 79.4 Baseline+SME 50.6 78.9 Baseline+CME and SME 51.3 79.8

[0199] Among them, the full name of TIM is Temporal Interaction Module, that is, time interaction module. The full name of MEM is Motion Enchanced Module, that is, motion enhancement module. The full name of PEM is Progressive Enhancement Module, that is, progressive enhancement module. TIM, MEM and PEM are all enhancement modules in related technologies.

[0200] Top-1 represents the probability that the highest probability result is the correct result for a video frame, Top-5 represents the probability that the correct result is included in the top 5 results for a video frame, and delay represents the time of action recognition. The smaller the delay is, the better the performance is.

[0201] In the action recognition process of the target video, the similarity between the plurality of video frames in the target video is combined. Since the action change causes the change of the video frame similarity, the similarity can reflect the continuity of the action. The attention weight obtained based on the similarity can represent the correlation degree of the feature map and the action. The action recognition based on the attention weight and the multi-channel feature map of the video frame can improve the accuracy of the action recognition.

[0202] Figure 10 is a flowchart of an action recognition method provided by an embodiment of the present application, referring to Figure 10 , the method comprises:

[0203] 1001, the terminal obtains a plurality of video frames in a target video.

[0204] 1002, the terminal extracts features of the plurality of video frames according to a plurality of dimensions respectively, obtains a plurality of multi-channel feature maps, each video frame corresponds to a multi-channel feature map, and each channel represents a dimension.

[0205] 1003, the terminal determines a third attention weight of each feature point based on the similarity between a plurality of feature points on the multi-channel feature maps of each two adjacent video frames, and the third attention weight is used to represent the correlation degree of the corresponding feature point and the action performed by the object in the target video.

[0206] If there is a moving object in the target video, the position of the object in different video frames of the target video may be different. In the multi-channel feature map of the video frame, if the similarity of the feature points at the same position on the two multi-channel feature maps is high, the object may not move or the amplitude of the movement is small in the video frames corresponding to the two multi-channel feature maps. If the similarity of the feature points at the same position on the two multi-channel feature maps is low, the object may move in the video frames corresponding to the two multi-channel feature maps. The computer device can use the third attention weight to represent the correlation degree of the feature points in the multi-channel feature map and the movement, so as to focus on the feature points with high correlation degree in the subsequent action recognition process.

[0207] 1004, the terminal determines the type of the action based on the plurality of multi-channel feature maps and the plurality of determined third attention weights.

[0208] The terminal can strengthen the multi-channel feature map related to the movement based on the third attention weight, thereby improving the accuracy of the action recognition.

[0209] In the process of action recognition on the target video, the similarity between the feature points on the plurality of video frames in the target video is combined. Since the action change will cause the change of the similarity between the feature points on the video frames, the similarity can reflect the continuity of the action. The attention weight obtained based on the similarity can represent the correlation degree between the feature map and the action. The action recognition based on the attention weight and the multi-channel feature map of the video frame can improve the accuracy of the action recognition.

[0210] Figure 11 is a flowchart of an action recognition method provided by an embodiment of the present application, referring to Figure 11 , the method comprises:

[0211] 1101. The terminal obtains a plurality of video frames in a target video.

[0212] Step 1101 and step 301 described above belong to the same inventive concept. For the implementation process, refer to the related description of step 301, which will not be repeated here.

[0213] 1102. The terminal extracts features from the plurality of video frames according to a plurality of dimensions respectively, to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension.

[0214] Step 1102 and step 302 described above belong to the same inventive concept. For the implementation process, refer to the related description of step 301, which will not be repeated here.

[0215] 1103. For the multi-channel feature map of any video frame, the terminal obtains a plurality of fourth feature vectors corresponding to a plurality of feature points on the multi-channel feature map of the video frame, and a plurality of fifth feature vectors corresponding to a plurality of feature points on the multi-channel feature map of a reference video frame adjacent to the video frame.

[0216] In a possible implementation, each multi-channel feature map has a plurality of single-channel feature maps. The feature points at the same position on the plurality of single-channel feature maps correspond to the same pixel point on the video frame. The terminal can splice the values of the feature points at the same position on the plurality of single-channel feature maps of the multi-channel feature map to obtain a plurality of fourth feature vectors, one fourth feature vector corresponding to one pixel point. The terminal splices the values of the feature points at the same position on the plurality of single-channel feature maps of the multi-channel feature map of the reference video frame to obtain a plurality of fifth feature vectors, one fifth feature vector corresponding to one pixel point.

[0217] In this implementation, the terminal obtains the feature vector of each feature point from the plurality of single-channel feature maps of one multi-channel feature map, which facilitates the subsequent comparison of the differences of the pixel points at the same position in different video frames, thereby obtaining the information related to the action and improving the accuracy of the subsequent action recognition.

[0218] 1104、The terminal determines, based on the plurality of fourth feature vectors and the plurality of fifth feature vectors, a plurality of third attention weights of any video frame, the third attention weights being used to represent a degree of correlation between the corresponding feature points and an action performed by the object in the target video.

[0219] Optionally, the third attention weight is inversely proportional to a similarity between the fourth feature vector and the fifth feature vector, that is, for any fourth feature vector, if the similarity between the corresponding fifth feature vector and the fourth feature vector is high, it means that the two pixel points corresponding to the fourth feature vector and the fifth feature vector respectively have little change in time sequence and contain limited action-related information, and accordingly, the attention weight between the fourth feature vector and the fifth feature vector is small when action recognition is performed; if the similarity between the corresponding fifth feature vector and the fourth feature vector is low, it means that the pixel points corresponding to the fourth feature vector and the fifth feature vector respectively have large change in time sequence and contain more action-related information, and accordingly, the attention weight between the fourth feature vector and the fifth feature vector is large when action recognition is performed.

[0220] In a possible implementation, the terminal can obtain a cosine similarity between the plurality of fourth feature vectors and the corresponding plurality of fifth feature vectors, and determine a plurality of third attention weights corresponding to the plurality of feature points on the first attention feature map respectively.

[0221] In this implementation, the terminal can determine the difference between the pixel points at the same position on different video frames through the similarity between the fourth feature vector and the fifth feature vector, and determine the action information carried by the pixel points based on the difference, thereby improving the accuracy of subsequent action recognition.

[0222] Taking a fourth feature vector [1, 1, 2, 3, 2] and a fifth feature vector corresponding to the fourth feature vector [1, 2, 3, 1, 1] as an example, the terminal can obtain a cosine similarity of 0.80 between the fourth feature vector [1, 1, 2, 3, 2] and the fifth feature vector [1, 2, 3, 1, 1]. The terminal can determine 1-0.80=0.20 as the third attention weight of the feature point corresponding to the fourth feature vector.

[0223] 1105、The terminal determines, based on the plurality of fourth feature vectors and the plurality of fifth feature vectors, a plurality of third attention weights of any video frame, the third attention weights being used to represent a degree of correlation between the corresponding feature points and an action performed by the object in the target video.

[0224] In a possible implementation, the terminal multiplies the plurality of third attention weights and the plurality of multi-channel feature maps to obtain a plurality of third attention feature maps, each video frame corresponding to a third attention feature map. The type of the action is determined based on the plurality of third attention feature maps.

[0225] For example, the terminal determines a fourth attention weight of each third attention feature map based on the similarity between every two third attention feature maps, the fourth attention weight being used to represent the degree of relevance of the corresponding third attention feature map to the action. The terminal determines the type of the action based on the plurality of third attention feature maps and the plurality of determined fourth attention weights.

[0226] By means of the technical solutions provided in the embodiments of the present application, in the process of action recognition on the target video, the similarity between the feature points on the plurality of video frames in the target video is combined. Since the action changes will cause the similarity between the feature points on the video frames to change, the similarity can reflect the continuity of the action. The attention weight obtained based on the similarity can represent the degree of relevance of the feature map to the action. The action recognition based on the attention weight and the multi-channel feature map of the video frame can improve the accuracy of the action recognition.

[0227] Figure 12 is a structural schematic diagram of an action recognition device provided by an embodiment of the present application, referring to Figure 12 The device comprises a first acquisition unit 1201, a first feature extraction unit 1202, a first attention weight determination unit 1203, and a first type determination unit 1204.

[0228] The first acquisition unit 1201 is configured to acquire a plurality of video frames in a target video.

[0229] The first feature extraction unit 1202 is configured to perform feature extraction on the plurality of video frames respectively according to a plurality of dimensions to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension.

[0230] The first attention weight determination unit 1203 is configured to determine a first attention weight of each multi-channel feature map based on the similarity between every two multi-channel feature maps, the first attention weight being used to represent the degree of relevance of the corresponding multi-channel feature map to an action performed by an object in the target video.

[0231] The first type determination unit 1204 is configured to determine the type of the action based on the plurality of multi-channel feature maps and the plurality of determined first attention weights.

[0232] In a possible implementation, the first feature extraction unit is configured to, for any video frame, perform feature extraction on the video frame by using a plurality of convolution kernels, to obtain a plurality of single-channel feature maps corresponding to the plurality of convolution kernels respectively, each convolution kernel being used to extract a single dimension of feature map.

[0233] In a possible implementation, the first attention weight determination unit is configured to perform dimension reduction processing on the plurality of multi-channel feature maps, to obtain a plurality of first feature vectors, each multi-channel feature map corresponding to a first feature vector.

[0234] The plurality of first attention weights are determined based on similarities between each two first feature vectors in the plurality of first feature vectors.

[0235] In a possible implementation, the first attention weight determination unit is configured to, for any multi-channel feature map, perform dimension reduction processing on a plurality of single-channel feature maps in the multi-channel feature map respectively, to obtain a plurality of feature values corresponding to the plurality of single-channel feature maps respectively. The plurality of feature values are combined to obtain a first feature vector of the multi-channel feature map.

[0236] In a possible implementation, the first attention weight determination unit is configured to, for any first feature vector, obtain a plurality of similarities between the first feature vector and a plurality of first feature vectors respectively. The first attention weight of the first feature vector is determined based on a sum of products of the plurality of first feature vectors and the corresponding similarities.

[0237] In a possible implementation, the first attention weight determination unit is configured to perform normalization processing on the sum of products of the plurality of first feature vectors and the corresponding similarities, to obtain the first attention weight of the first feature vector.

[0238] In a possible implementation, the first type determination unit is configured to multiply the plurality of first attention weights and the plurality of multi-channel feature maps, to obtain a plurality of first attention feature maps, each video frame corresponding to a first attention feature map. The plurality of first attention feature maps are used for recognition, to obtain a type of the action.

[0239] In a possible implementation, the first type determining unit is configured to, for any first attention feature map, acquire a plurality of second feature vectors corresponding to a plurality of feature points on the any first attention feature map, and a plurality of third feature vectors corresponding to the plurality of feature points on a reference attention feature map of the any first attention feature map, the video frame corresponding to the reference attention feature map being adjacent to the video frame corresponding to the any first attention feature map; determine a plurality of second attention weights respectively corresponding to the plurality of feature points on the any first attention feature map based on similarity between the plurality of second feature vectors and the plurality of third feature vectors, the second attention weight being used to represent a degree of relevance of the corresponding feature point to the action; and multiply the plurality of second attention weights with the plurality of feature points on the any first attention feature map to obtain a second attention feature map of the any first attention feature map. The action type is obtained by performing recognition based on the plurality of second attention feature maps.

[0240] It should be noted that the action recognition apparatus provided in the above embodiments is only used as an example for the division of the above functional modules, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above described functions. In addition, the action recognition apparatus and the action recognition method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be described here.

[0241] Through the technical solutions provided in the embodiments of the present application, in the process of recognizing the action of the target video, the similarity between the plurality of video frames in the target video is combined, and since the action change will cause the change of the video frame similarity, the similarity can reflect the continuity of the action. The attention weight obtained based on the similarity can represent the degree of relevance of the feature map to the action, and the action recognition based on the attention weight and the multi-channel feature map of the video frame can improve the accuracy of the action recognition.

[0242] Figure 13 FIG. 1 is a structural schematic diagram of an action recognition apparatus provided in an embodiment of the present application, referring to FIG. 1, Figure 13 The apparatus includes a second acquisition unit 1301, a second feature extraction unit 1302, a third attention weight determining unit 1303, and a second type determining unit 1304.

[0243] The second acquisition unit 1301 is configured to acquire a plurality of video frames in a target video.

[0244] The second feature extraction unit 1302 is configured to perform feature extraction on the plurality of video frames respectively according to a plurality of dimensions to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension.

[0245] The third attention weight determination unit 1303 is configured to determine a third attention weight of each feature point based on a similarity between the feature points on the multi-channel feature map of each two adjacent video frames, and the third attention weight is used to represent a correlation degree of the corresponding feature point and an action performed by the object in the target video.

[0246] The second type determination unit 1304 is configured to determine the type of the action based on the plurality of multi-channel feature maps and the plurality of determined third attention weights.

[0247] In a possible implementation, the third attention weight determination unit is configured to, for the multi-channel feature map of any video frame, obtain a plurality of fourth feature vectors corresponding to the plurality of feature points on the multi-channel feature map of any video frame, and a plurality of fifth feature vectors corresponding to the plurality of feature points on the multi-channel feature map of a reference video frame adjacent to any video frame. The plurality of third attention weights of any video frame are determined based on the plurality of fourth feature vectors and the plurality of fifth feature vectors.

[0248] In a possible implementation, the second type determination unit is configured to multiply the plurality of third attention weights with the plurality of multi-channel feature maps to obtain a plurality of third attention feature maps, each video frame corresponding to a third attention feature map. The type of the action is determined based on the plurality of third attention feature maps.

[0249] In a possible implementation, the second type determination unit is configured to determine a fourth attention weight of each third attention feature map based on a similarity between each two third attention feature maps, and the fourth attention weight is used to represent a correlation degree of the corresponding third attention feature map and the action. The type of the action is determined based on the plurality of third attention feature maps and the plurality of determined fourth attention weights.

[0250] It should be noted that the action recognition apparatus provided in the above embodiments is only used as an example for dividing the above functional modules when identifying the action, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the above described functions. In addition, the action recognition apparatus and the action recognition method provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0251] In the process of action recognition on the target video, the similarity between the feature points on the plurality of video frames in the target video is combined. Since action changes can cause the similarity between the feature points on the video frames to change, the similarity can reflect the continuity of the action. The attention weight obtained based on the similarity can represent the correlation degree between the feature map and the action. Action recognition based on the attention weight and the multi-channel feature map of the video frame can improve the accuracy of action recognition.

[0252] The embodiments of the present application provide a computer device for executing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal is introduced as follows.

[0253] Figure 14 Fig. 1 is a structural schematic diagram of a terminal provided by the embodiments of the present application. The terminal 1400 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but is not limited thereto.

[0254] Generally, the terminal 1400 includes one or more processors 1401 and one or more memories 1402.

[0255] The processor 1401 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1401 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 1401 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1401 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1401 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0256] The memory 1402 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 1402 can also include high-speed random access memory and can include nonvolatile memory, such as one or more magnetic disk storage devices, optical storage devices, flash memory devices, or other nonvolatile solid-state storage devices.

[0257] In some embodiments, the terminal 1400 can further include a peripheral device interface 1403 and at least one peripheral device. The processor 1401, the memory 1402, and the peripheral device interface 1403 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1403 through a bus, a signal line, or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, an audio circuit 1407, a positioning assembly 1408, and a power supply 1409.

[0258] The peripheral device interface 1403 can be used to connect at least one peripheral device related to input / output to the processor 1401 and the memory 1402. In some embodiments, the processor 1401, the memory 1402, and the peripheral device interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1401, the memory 1402, and the peripheral device interface 1403 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0259] The radio frequency circuit 1404 is used to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1404 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1404 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1404 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.

[0260] The display 1405 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display 1405 is a touch display, the display 1405 is further capable of acquiring a touch signal on or above the surface of the display 1405. The touch signal can be input to the processor 1401 as a control signal for processing. At this time, the display 1405 can also be configured to provide a virtual button and / or a virtual keyboard, also known as a soft button and / or a soft keyboard.

[0261] The camera component 1406 is configured to acquire an image or a video. Optionally, the camera component 1406 includes a front-facing camera and a rear-facing camera. Generally, the front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal.

[0262] The audio circuit 1407 can include a microphone and a speaker. The microphone is configured to acquire sound waves of a user and an environment, and convert the sound waves into an electrical signal input to the processor 1401 for processing, or input to the radio frequency circuit 1404 to implement voice communication.

[0263] The positioning component 1408 is configured to locate the current geographical position of the terminal 1400 to implement navigation or LBS (Location Based Service).

[0264] The power supply 1409 is configured to supply power to each component in the terminal 1400. The power supply 1409 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery.

[0265] In some embodiments, the terminal 1400 further includes one or more sensors 1410. The one or more sensors 1410 include, but are not limited to, an acceleration sensor 1411, a gyroscope sensor 1412, a pressure sensor 1413, a fingerprint sensor 1414, an optical sensor 1415, and a proximity sensor 1416.

[0266] The acceleration sensor 1411 can detect the acceleration magnitude in three coordinate axes of a coordinate system established by the terminal 1400.

[0267] The gyroscope sensor 1412 can detect the body direction and rotation angle of the terminal 1400. The gyroscope sensor 1412 can cooperate with the acceleration sensor 1411 to acquire the 3D action of the user on the terminal 1400.

[0268] The pressure sensor 1413 can be arranged at the side frame of the terminal 1400 and / or the lower layer of the display screen 1405. When the pressure sensor 1413 is arranged at the side frame of the terminal 1400, the holding signal of the user to the terminal 1400 can be detected, and the left-hand or right-hand recognition or the shortcut operation is performed by the processor 1401 according to the holding signal collected by the pressure sensor 1413. When the pressure sensor 1413 is arranged at the lower layer of the display screen 1405, the controllable control on the UI interface is controlled by the processor 1401 according to the pressure operation of the user to the display screen 1405.

[0269] The fingerprint sensor 1414 is used to collect the fingerprint of the user, and the identity of the user is recognized by the processor 1401 according to the fingerprint collected by the fingerprint sensor 1414, or the identity of the user is recognized by the fingerprint sensor 1414 according to the collected fingerprint.

[0270] The optical sensor 1415 is used to collect the ambient light intensity. In an embodiment, the processor 1401 can control the display brightness of the display screen 1405 according to the ambient light intensity collected by the optical sensor 1415.

[0271] The proximity sensor 1416 is used to collect the distance between the user and the front of the terminal 1400.

[0272] Those skilled in the art can understand that the structure shown in the above embodiment is not a limitation on the terminal 1400, and the terminal 1400 can include more or less components than the structure shown in the figure, or combine certain components, or adopt a different component arrangement. Figure 14

[0273] The above computer device can also be implemented as a server, and the structure of the server will be introduced as follows:

[0274] Figure 15 FIG. 15 is a structural schematic diagram of a server provided by an embodiment of the present application. The server 1500 can have great differences due to different configurations or performances, and can include one or more processors (Central Processing Units, CPUs) 1501 and one or more memories 1502. The one or more memories 1502 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1501 to implement the method provided by each method embodiment. Of course, the server 1500 can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for realizing the functions of the device, and details are not described herein.

[0275] ​In the example embodiment, a computer readable storage medium, such as a memory including a computer program executable by a processor to perform the action recognition method in the above embodiment, is also provided. For example, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a Compact Disc Read-Only Memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0276] In the example embodiment, a computer program product or computer program including program code stored in a computer readable storage medium is also provided, and a processor of a computer device reads the program code from the computer readable storage medium, and the processor executes the program code to cause the computer device to perform the action recognition method.

[0277] Those of ordinary skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware, or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, such as a Read-Only Memory, a magnetic disk or an optical disk.

[0278] The above is only an optional embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of action recognition, characterized by, The method comprises: obtaining a plurality of video frames in a target video; performing feature extraction on the plurality of video frames respectively according to a plurality of dimensions to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension; determining a first attention weight of each multi-channel feature map based on a similarity between each two multi-channel feature maps in the plurality of multi-channel feature maps, the first attention weight being used to represent a degree of relevance of the corresponding multi-channel feature map to an action performed by an object in the target video; determining a type of the action based on the plurality of multi-channel feature maps and the determined plurality of first attention weights.

2. The method of claim 1, wherein, The performing feature extraction on the plurality of video frames respectively according to a plurality of dimensions to obtain a plurality of multi-channel feature maps comprises: for any video frame, performing feature extraction on the any video frame by using a plurality of convolution kernels to obtain a plurality of single-channel feature maps corresponding to the plurality of convolution kernels respectively, each convolution kernel being used to extract a feature map of a dimension; splicing the plurality of single-channel feature maps to obtain a multi-channel feature map of the any video frame.

3. The method of claim 1, wherein, The determining a first attention weight of each multi-channel feature map based on a similarity between each two multi-channel feature maps in the plurality of multi-channel feature maps comprises: performing dimension reduction processing on the plurality of multi-channel feature maps to obtain a plurality of first feature vectors, each multi-channel feature map corresponding to a first feature vector; determining a first attention weight of each first feature vector based on a similarity between each two first feature vectors in the plurality of first feature vectors.

4. The method of claim 3, wherein, The performing dimension reduction processing on the plurality of multi-channel feature maps to obtain a plurality of first feature vectors comprises: for any multi-channel feature map, performing dimension reduction processing on a plurality of single-channel feature maps in the any multi-channel feature map respectively to obtain a plurality of feature values corresponding to the plurality of single-channel feature maps respectively; combining the plurality of feature values to obtain a first feature vector of the any multi-channel feature map.

5. The method of claim 3, wherein, The determining a first attention weight of each first feature vector based on a similarity between each two first feature vectors in the plurality of first feature vectors comprises: for any first feature vector, obtaining a plurality of similarities between the any first feature vector and the plurality of first feature vectors respectively; determining a first attention weight of the any first feature vector based on a sum of products of the plurality of first feature vectors and corresponding similarities.

6. The method of claim 5, wherein, The determining a first attention weight of the any first feature vector based on a sum of products of the plurality of first feature vectors and corresponding similarities comprises: performing normalization processing on the sum of products of the plurality of first feature vectors and corresponding similarities to obtain the first attention weight of the any first feature vector.

7. The method of claim 1, wherein, The determining a type of the action based on the plurality of multi-channel feature maps and the determined plurality of first attention weights comprises: multiplying the plurality of first attention weights and the plurality of multi-channel feature maps to obtain a plurality of first attention feature maps, each video frame corresponding to a first attention feature map; Perform recognition based on the plurality of first attention feature maps to obtain a type of the action.

8. The method of claim 7, wherein, The performing recognition based on the plurality of first attention feature maps to obtain a type of the action includes: For any first attention feature map, a plurality of second feature vectors corresponding to a plurality of feature points on the any first attention feature map and a plurality of third feature vectors corresponding to a plurality of feature points on a reference attention feature map of the any first attention feature map are obtained, the video frame corresponding to the reference attention feature map being a video frame adjacent to the video frame corresponding to the any first attention feature map; Based on a similarity between the plurality of second feature vectors and the plurality of third feature vectors, a plurality of second attention weights respectively corresponding to the plurality of feature points on the any first attention feature map are determined, the second attention weight being used to represent a degree of correlation of the corresponding feature point with the action; The plurality of second attention weights are multiplied with the plurality of feature points on the any first attention feature map to obtain a second attention feature map of the any first attention feature map; The performing recognition based on the plurality of second attention feature maps to obtain a type of the action.

9. An action recognition apparatus, characterized by comprising: The apparatus includes: A first obtaining unit configured to obtain a plurality of video frames in a target video; A first feature extraction unit configured to perform feature extraction on the plurality of video frames respectively according to a plurality of dimensions to obtain a plurality of multi-channel feature maps, each video frame corresponding to a multi-channel feature map, and each channel representing a dimension; A first attention weight determination unit configured to determine a first attention weight of each multi-channel feature map based on a similarity between every two multi-channel feature maps in the plurality of multi-channel feature maps, the first attention weight being used to represent a degree of correlation of the corresponding multi-channel feature map with an action performed by an object in the target video in the target video; A first type determination unit configured to determine a type of the action based on the plurality of multi-channel feature maps and the plurality of determined first attention weights.

10. A computer device, comprising: The computer device includes one or more processors and one or more memories, the one or more memories storing at least one computer program, the computer program being loaded and executed by the one or more processors to implement the action recognition method according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores at least one computer program, the computer program being loaded and executed by the processor to implement the action recognition method according to any one of claims 1 to 8.

Images