Action recognition method, device, computer equipment and storage medium

Abstract

The present application relates to an action recognition method, device, computer equipment and storage medium. The method includes: obtaining a video to be recognized, extracting a video frame sequence and a current video frame in the video to be recognized; performing a first convolution operation on the video frame sequence to obtain time series features in the video to be recognized, performing a second convolution operation on the current video frame to obtain spatial features in the video to be recognized; performing an attention mechanism adjustment on the time series features and the spatial features to obtain mixed action features; performing classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame. The action category recognition and action position recognition in the action recognition architecture in the present application do not need to be processed separately, which can effectively improve the processing efficiency of the action recognition process.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a motion recognition method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development of computer technology and machine learning technology, computer vision technology has also grown rapidly. Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. Compared with images, the video content and background of the action recognition video are more complex and changeable. Different action categories have similarities, while the same category has different characteristics in different environments. In addition, the occlusion, jitter, and perspective changes caused by shooting also bring further difficulties to action recognition. In practical applications, accurate action recognition is helpful for advertising, security monitoring, etc.

[0003] The commonly used technologies for video action recognition currently include 3D convolution-based neural networks, which are based on a two-stage system architecture, training subsequent box selection networks and classification networks for action recognition. However, two-stage training requires more training time. Summary of the invention

[0004] Based on this, it is necessary to provide a motion recognition method, device, computer equipment and storage medium that can effectively improve the efficiency of motion recognition in response to the above technical problems.

[0005] A method for motion recognition, the method comprising:

[0006] Acquire a video to be identified, and extract a video frame sequence and a current video frame from the video to be identified;

[0007] Performing a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and performing a second convolution operation on the current video frame to obtain spatial features in the video to be identified;

[0008] Performing an attention mechanism adjustment on the time series features and the spatial features to obtain mixed action features;

[0009] Classification and regression processing is performed on the mixed action features to obtain the action category and the action position in the current video frame.

[0010] In one embodiment, obtaining a video to be identified and extracting a video frame sequence and a current video frame in the video to be identified includes:

[0011] Get the video to be identified;

[0012] Determine the number of video frames in the video frame sequence based on the video duration of the video to be identified;

[0013] Determine a video frame interval according to the number of video frames and the video duration;

[0014] Each video frame in the video frame sequence is intercepted from the video to be identified based on the video frame interval, and the last video frame in the video frame sequence is used as the current video frame.

[0015] In one embodiment, the performing of attention mechanism adjustment on the time series feature and the spatial feature to obtain the mixed action feature includes:

[0016] Connecting the time series features and the spatial features in series;

[0017] Performing convolution processing on the concatenated time series features and the spatial features to obtain video fusion features;

[0018] Obtaining a mixed attention feature map according to the video fusion features;

[0019] A convolution operation is performed on the mixed attention feature map to obtain mixed action features.

[0020] In one embodiment, obtaining a mixed attention feature map according to the video fusion feature includes:

[0021] Performing deformation operation and deformation displacement operation on the video fusion feature;

[0022] Obtain a channel feature matrix according to the video fusion features after the deformation operation and the video fusion features after the deformation displacement operation;

[0023] Performing softmax processing on the channel feature matrix to obtain a channel attention map;

[0024] A mixed attention feature map is obtained according to the video fusion features and the channel attention map.

[0025] In one embodiment, performing classification regression processing on the mixed action features to obtain the action category and the action position in the current video frame includes:

[0026] The action category corresponding to the video to be identified is obtained through a classification loss function, and the action position corresponding to the video to be identified is obtained through a regression loss function.

[0027] In one embodiment, after performing classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame, the method further includes:

[0028] When the action in the to-be-identified video is detected as an illegal action according to the action category, illegal detection information is generated according to the action category and the action position.

[0029] A motion recognition device, comprising:

[0030] A video frame extraction module is used to obtain a video to be identified, and extract a video frame sequence and a current video frame in the video to be identified;

[0031] A convolution processing module, configured to perform a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and perform a second convolution operation on the current video frame to obtain spatial features in the video to be identified;

[0032] An attention mechanism module, used to perform attention mechanism adjustment on the time series features and the spatial features to obtain mixed action features;

[0033] The classification and regression module is used to perform classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame.

[0034] In one embodiment, the video frame extraction module is specifically used for:

[0035] Get the video to be identified;

[0036] Determine the number of video frames in the video frame sequence based on the video duration of the video to be identified;

[0037] Determine a video frame interval according to the number of video frames and the video duration;

[0038] Each video frame in the video frame sequence is intercepted from the video to be identified based on the video frame interval, and the last video frame in the video frame sequence is used as the current video frame.

[0039] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0040] Acquire a video to be identified, and extract a video frame sequence and a current video frame from the video to be identified;

[0041] Performing a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and performing a second convolution operation on the current video frame to obtain spatial features in the video to be identified;

[0042] Performing an attention mechanism adjustment on the time series features and the spatial features to obtain mixed action features;

[0043] Classification and regression processing is performed on the mixed action features to obtain the action category and the action position in the current video frame.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0045] Acquire a video to be identified, and extract a video frame sequence and a current video frame from the video to be identified;

[0046] Performing a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and performing a second convolution operation on the current video frame to obtain spatial features in the video to be identified;

[0047] Performing an attention mechanism adjustment on the time series features and the spatial features to obtain mixed action features;

[0048] Classification and regression processing is performed on the mixed action features to obtain the action category and the action position in the current video frame.

[0049] The above-mentioned action recognition method, device, computer equipment and storage medium obtain the video to be recognized, extract the video frame sequence and the current video frame in the video to be recognized; perform a first convolution operation on the video frame sequence to obtain the time series features in the video to be recognized, and perform a second convolution operation on the current video frame to obtain the spatial features in the video to be recognized; perform an attention mechanism adjustment on the time series features and the spatial features to obtain the mixed action features; perform classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame. The present application performs 2D convolution on the current video frame and 3D convolution on the video frame sequence to respectively obtain the spatial features and time series features contained in the video to be recognized, and then adjusts the attention mechanism and performs classification and regression operations based on the spatial features and the time series to recognize the action category and action position in the video to be recognized. The action category recognition and action position recognition in the action recognition architecture do not need to be processed separately, which can effectively improve the processing efficiency of the action recognition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A diagram of an application environment of an action recognition method in an embodiment;

[0051] Figure 2 is a flow chart of an action recognition method in one embodiment;

[0052] Figure 3 In one embodiment Figure 2 Schematic diagram of sub-process of step 201;

[0053] Figure 4 In one embodiment Figure 2 Schematic diagram of sub-process of step 306;

[0054] Figure 5 In one embodiment Figure 2 Schematic diagram of sub-process of step 405;

[0055] Figure 6 is a structural block diagram of a motion recognition device in one embodiment;

[0056] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] The action recognition method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The terminal 102 can send the video to be identified to the server 104 to perform action recognition through the server 104. The server 104 obtains the video to be identified, extracts the video frame sequence and the current video frame in the video to be identified; performs a first convolution operation on the video frame sequence to obtain the time series features in the video to be identified, and performs a second convolution operation on the current video frame to obtain the spatial features in the video to be identified; adjusts the attention mechanism for the time series features and the spatial features to obtain the mixed action features; performs classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented with an independent server or a server cluster composed of multiple servers. In another embodiment, the action recognition method of the present application is specifically used in the field of logistics and transportation to identify illegal actions of logistics personnel. Terminal 102 can be specifically a DVR (Digital Video Recorder) or NVR (Network Video Recorder). Terminal 102 can be connected to a camera in an actual application scenario to collect video through the camera. The camera can be an analog surveillance camera or a network camera. Server 104 can be specifically a CPU server, a GPU server or other forms of computing units. It is specifically used for edge computing. In addition, multiple servers 104 can also be connected to a cloud management platform, which manages the motion recognition of these servers.

[0059] In one embodiment, Figure 2 As shown, a method for action recognition is provided, which is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0060] Step 201: obtain a video to be identified, and extract a video frame sequence and a current video frame in the video to be identified.

[0061] The video to be identified refers to the identification object of the action recognition method of this application. The video frame is the basic component element of the video stream of the video to be identified, specifically the video images that constitute the video. The current video frame specifically refers to the last frame in the video to be identified, and the video frame sequence is composed of multiple video frames extracted from the video stream.

[0062] First, when the terminal needs to perform action recognition, it can upload the collected video data to the server 104 through the network, and the server 104 recognizes the action in the video. The server 104 first obtains the video to be recognized from the terminal through the network, and then decodes the obtained video to be recognized, and extracts the corresponding current video frame and video frame sequence from it.

[0063] Step 203, performing a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and performing a second convolution operation on the current video frame to obtain spatial features in the video to be identified.

[0064] The first convolution operation may be specifically a 3D convolution operation, and the second convolution operation may be specifically a 2D convolution operation. The 2D convolution operation slides a 2D convolution kernel on the image of the video frame. Multiply the grayscale value on the image with the value on the corresponding convolution kernel. Then add all the multiplied values ​​to get the grayscale value of the pixel corresponding to the center of the convolution kernel to get a new image, which is the spatial feature. The 3D convolution operates on an image sequence consisting of multiple video frames arranged in chronological order. The convolution kernel used is also a three-dimensional convolution kernel, and the resulting image also contains motion information between consecutive frames, that is, time series features.

[0065] Specifically, the convolution operation can be performed on the extracted current video frame and the video frame sequence through the pre-trained convolutional neural network, and the corresponding image features can be obtained. The current video frame is used to extract the spatial features in the video to be identified, that is, the action position corresponding features in action recognition. The video frame sequence is used to extract the time series features in the video to be identified, that is, the action category corresponding features in action recognition.

[0066] Step 205: Adjust the attention mechanism for the time series features and the spatial features to obtain mixed action features.

[0067] The attention mechanism originates from the study of human vision. In cognitive science, due to the bottleneck of information processing, humans will selectively focus on part of all information while ignoring other visible information. The above mechanism is usually referred to as the attention mechanism. In this application, the attention mechanism is formed by focusing on the use of masks. The principle of masking is to identify the key features in the image data of the feature image through another layer of new weights, and through learning and training, the deep neural network learns the areas that need to be paid attention to in each new image, thus forming attention. The attention mechanism in computer vision is mainly three attention domains, spatial domain, channel domain, and mixed domain. This application mainly adopts the mechanism of mixed domain, that is, it takes into account the attention mechanism of space and time at the same time.

[0068] Specifically, after extracting the images corresponding to the time series features and the spatial features, the feature images can be further processed through the attention mechanism to adjust the weights of the feature images to form feature images corresponding to the mixed action features.

[0069] Step 207: perform classification regression processing on the mixed action features to obtain the action category and the action position in the current video frame.

[0070] Among them, classification processing refers to a problem with a finite number of possibilities, and what is predicted is a discrete, clear variable. For example, given a picture, to determine whether it is a T-shirt, pants or other types; this category is limited. In the target detection algorithm, generally a certain amount of candidate areas are found first, and then these candidate areas are classified. In this application, classification processing is used to identify the types of actions contained in the video to be identified. Regression refers to a problem with an infinite number of possibilities, and what is predicted is a continuous, approximate variable. In this application, regression processing is mainly used to identify the position of the action in the video to be identified, that is, to identify the position of the action subject in the video to be identified in the video screen of the current video frame. The single-stage method not only takes into account the category information of the video action, but also its position information.

[0071] Specifically, after obtaining the mixed action features, the mixed action features can be classified and processed by the classification loss function completed in advance, and the action categories contained in the video to be identified are identified. At the same time, the mixed action features are regressed by the regression loss function completed in advance, and the action categories contained in the video to be identified are identified. In a specific embodiment, the action recognition method of the present application is mainly used to determine whether there are any illegal handling by the staff in the logistics process, such as illegal throwing of objects. At this time, it can be identified by the action recognition method of the present application, and the actions of the staff can be divided into violations and non-violations, and the judgment of violations and non-violations also needs to be analyzed in combination with the location of the action. After the work video of the staff is collected at the work site, action recognition can be performed through the edge computing unit on site, and the video containing illegal operations can be identified, and uploaded to the cloud server of cloud management for further processing.

[0072] Specifically, in one of the embodiments, a corresponding action recognition model can be constructed according to the action recognition method of the present application, which includes a 3D convolutional neural network model and a 2D convolutional neural network model, and the attention mechanism adjustment and the loss function of classification regression can also be loaded into the action recognition model in the form of a convolutional layer, so that the selection box network and the classification network in the action recognition process can be integrated. When training the classification model, only the action recognition model needs to be trained, that is, only a single-stage video action recognition architecture is required. The present application combines spatiotemporal video features and attention mechanisms, and an end-to-end training method effectively improves the robustness of video actions. At the same time, for different tasks, different 2D and 3D convolutional networks are embedded in the architecture to achieve real-time processing efficiency.

[0073] The above-mentioned action recognition method, device, computer equipment and storage medium obtain the video to be recognized, extract the video frame sequence and the current video frame in the video to be recognized; perform a first convolution operation on the video frame sequence to obtain the time series features in the video to be recognized, and perform a second convolution operation on the current video frame to obtain the spatial features in the video to be recognized; perform an attention mechanism adjustment on the time series features and the spatial features to obtain the mixed action features; perform classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame. The present application performs 2D convolution on the current video frame and 3D convolution on the video frame sequence to respectively obtain the spatial features and time series features contained in the video to be recognized, and then adjusts the attention mechanism and performs classification and regression operations based on the spatial features and the time series to recognize the action category and action position in the video to be recognized. The action category recognition and action position recognition in the action recognition architecture do not need to be processed separately, which can effectively improve the processing efficiency of the action recognition process.

[0074] In one embodiment, Figure 3 As shown, step 201 includes:

[0075] Step 302: Obtain the video to be identified.

[0076] Step 304: determine the number of video frames in the video frame sequence based on the video duration of the video to be identified.

[0077] Step 306: Determine the video frame interval according to the number of video frames and the video duration.

[0078] Step 308 , extract each video frame in a video frame sequence from the video to be identified based on the video frame interval, and use the last video frame in the video frame sequence as the current video frame.

[0079] Among them, according to experimental data, the specific value of the number of video frames can be 8, 10 or 16. For videos with longer duration, the corresponding number of video frames will be more. Specifically, multiple duration segments can be preset, and videos with different duration segments correspond to different numbers of video frames. Then, based on the duration segment in which the duration of the video to be identified is located, the number of video frames that should be included in the video frame sequence is determined. At the same time, since 3D convolution requires video frames with the same time interval for further identification, at this time, the time interval for intercepting video frames can be determined based on the number of video frames in the video frame sequence and the duration of the video, so that the video frames are intercepted from the video to be identified in sequence, and the last video frame is used as the current video frame. In this embodiment, video frames are intercepted at equal intervals based on the duration of the video frame and the number of video frames, which can effectively ensure the accuracy of the 3D convolution process of extracting time series features, thereby improving the accuracy of action recognition.

[0080] like Figure 4 As shown, in one embodiment, step 306 includes:

[0081] Step 401, concatenate the time series features and the spatial features.

[0082] Step 403, performing convolution processing on the concatenated time series features and spatial features to obtain video fusion features.

[0083] Step 405, obtaining a mixed attention feature map based on the video fusion features.

[0084] Step 407, performing a convolution operation on the mixed attention feature map to obtain mixed action features.

[0085] The time series features and the spatial features are connected in series, specifically referring to splicing the feature graphs corresponding to the time series features and the feature graphs corresponding to the spatial features.

[0086] Specifically, in the present application, the attention mechanism is adjusted mainly through the mechanism of the mixed domain. The feature graph obtained in the previous convolution process can be first processed in series. After the series connection, the time series features and the spatial features can be fused by convolution processing on the series graph to obtain the feature graph of the video fusion feature. Then, the channel attention graph for attention mechanism adjustment is extracted from the feature graph of the video fusion feature, and then a mixed attention feature graph is constructed based on the channel attention graph and the feature graph of the video fusion feature, and then the constructed mixed attention feature graph is convoluted once to obtain the final mixed action feature feature graph. In this embodiment, the mixed attention feature graph is obtained by processing the time series features and the spatial features in series and then adjusting the attention mechanism accordingly, which can effectively pave the way for subsequent classification and regression operations and improve the recognition accuracy of the subsequent action recognition process.

[0087] like Figure 5 As shown, in one embodiment, step 405 includes:

[0088] Step 502: Perform deformation operation and deformation replacement operation on the video fusion feature.

[0089] Step 504: Obtain a channel feature matrix according to the video fusion features after the deformation operation and the video fusion features after the deformation displacement operation.

[0090] Step 506, perform softmax processing on the channel feature matrix to obtain a channel attention map.

[0091] Step 508, obtaining a mixed attention feature map based on the video fusion features and the channel attention map.

[0092] Specifically, the feature map of the video fusion feature can be reshaped and reshaped and transposed accordingly, and then the inner product operation is performed on the video fusion feature after the deformation operation and the video fusion feature after the deformation and displacement operation to obtain the inter-channel feature correlation matrix, that is, the channel feature matrix, and then the softmax operation is performed on the channel feature matrix to obtain the inter-channel attention map, that is, the channel attention map, which is used to assign weights to images in different channels. Then, by performing the inner product and addition operations on the video fusion feature and the channel attention map, the video fusion feature map adjusted by the attention mechanism, that is, the mixed attention feature map, can be obtained.

[0093] In one embodiment, step 207 includes: obtaining the action category corresponding to the video to be identified by using a classification loss function, and obtaining the action position corresponding to the video to be identified by using a regression loss function.

[0094] Specifically, the classification and regression process can be performed by a classification loss function and a regression loss function, respectively. The classification loss function is used to determine the category of the action, and the regression loss function is used to regress the position information of the action. The combined action category and action position information can be used as the final result of action recognition. In this embodiment, the action category and action position corresponding to the video to be recognized can be obtained more accurately through the classification loss function and the regression loss function.

[0095] In one embodiment, after step 207, the method further includes: when it is detected that the action in the to-be-identified video is an illegal action according to the action category, generating illegal detection information according to the action category and the action position.

[0096] Specifically, the action category is used to determine whether the action in the current video to be identified is an illegal action. The action category in the action recognition process can specifically be an illegal action category and a non-illegal action category. When it is determined that the action category in the video to be identified is an illegal action, the extracted video frame, the detected action category and the action position can be used as evidence, and the corresponding violation detection information is generated. In another embodiment, the illegal action can be subdivided according to the category, and the action category and violation detection obtained at this time are more accurate violation information. In this embodiment, the corresponding violation detection can be performed based on the action category, and when a violation is detected, the corresponding violation detection information is generated based on the action category and the action position, so that the recognition results of action recognition can be more effectively utilized.

[0097] It should be understood that although Figure 2-5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0098] In one embodiment, Figure 6 As shown, an action recognition device is provided, including: a video frame extraction module 601, a convolution processing module 603, an attention mechanism module 605 and a classification regression module 607, wherein:

[0099] The video frame extraction module 601 is used to obtain the video to be identified, and extract the video frame sequence and the current video frame in the video to be identified;

[0100] The convolution processing module 603 is used to perform a first convolution operation on the video frame sequence to obtain the time series features in the video to be identified, and perform a second convolution operation on the current video frame to obtain the spatial features in the video to be identified;

[0101] An attention mechanism module 605 is used to adjust the attention mechanism of the time series features and the spatial features to obtain mixed action features;

[0102] The classification and regression module 607 is used to perform classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame.

[0103] In one of the embodiments, the video frame extraction module is specifically used to: obtain a video to be identified; determine the number of video frames in a video frame sequence based on the video length of the video to be identified; determine the video frame interval according to the number of video frames and the video length; based on the video frame interval, capture each video frame in the video frame sequence from the video to be identified, and use the last video frame in the video frame sequence as the current video frame.

[0104] In one embodiment, the attention mechanism module 605 is specifically used to: concatenate time series features and spatial features; perform convolution processing on the concatenated time series features and spatial features to obtain video fusion features; obtain a mixed attention feature map based on the video fusion features; perform convolution operation on the mixed attention feature map to obtain mixed action features.

[0105] In one embodiment, the attention mechanism module 605 is specifically used to: perform deformation operations and deformation displacement operations on video fusion features; obtain a channel feature matrix based on the video fusion features after the deformation operation and the video fusion features after the deformation displacement operation; perform softmax processing on the channel feature matrix to obtain a channel attention map; obtain a mixed attention feature map based on the video fusion features and the channel attention map.

[0106] In one embodiment, the classification regression module 607 is specifically used to: obtain the action category corresponding to the video to be identified through a classification loss function, and obtain the action position corresponding to the video to be identified through a regression loss function.

[0107] In one of the embodiments, a violation detection module is further included, which is specifically used to: when an action in the to-be-identified video is detected as a violation action according to the action category, generate violation detection information according to the action category and the action position.

[0108] For the specific definition of the action recognition device, please refer to the definition of the action recognition method above, which will not be repeated here. The various modules in the above-mentioned action recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0109] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store action recognition data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an action recognition method is implemented.

[0110] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0111] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0112] Obtain the video to be identified, and extract the video frame sequence and the current video frame in the video to be identified;

[0113] Performing a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and performing a second convolution operation on the current video frame to obtain spatial features in the video to be identified;

[0114] Adjust the attention mechanism for time series features and spatial features to obtain mixed action features;

[0115] The mixed action features are classified and regressed to obtain the action category and the action position in the current video frame.

[0116] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining a video to be identified; determining the number of video frames in a video frame sequence based on the video duration of the video to be identified; determining a video frame interval based on the number of video frames and the video duration; based on the video frame interval, intercepting each video frame in a video frame sequence from the video to be identified, and taking the last video frame in the video frame sequence as the current video frame.

[0117] In one embodiment, when the processor executes the computer program, the following steps are also implemented: concatenating the time series features and the spatial features; performing convolution processing on the concatenated time series features and the spatial features to obtain video fusion features; obtaining a mixed attention feature map based on the video fusion features; performing convolution operation on the mixed attention feature map to obtain mixed action features.

[0118] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing deformation operations and deformation replacement operations on the video fusion features; obtaining a channel feature matrix based on the video fusion features after the deformation operation and the video fusion features after the deformation replacement operation; performing softmax processing on the channel feature matrix to obtain a channel attention map; obtaining a mixed attention feature map based on the video fusion features and the channel attention map.

[0119] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining the action category corresponding to the video to be identified through a classification loss function, and obtaining the action position corresponding to the video to be identified through a regression loss function.

[0120] In one embodiment, when the processor executes the computer program, the following steps are further implemented: when the action in the to-be-identified video is detected as an illegal action according to the action category, illegal detection information is generated according to the action category and the action position.

[0121] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0122] Obtain the video to be identified, and extract the video frame sequence and the current video frame in the video to be identified;

[0123] Performing a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and performing a second convolution operation on the current video frame to obtain spatial features in the video to be identified;

[0124] Adjust the attention mechanism for time series features and spatial features to obtain mixed action features;

[0125] The mixed action features are classified and regressed to obtain the action category and the action position in the current video frame.

[0126] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining a video to be identified; determining the number of video frames in a video frame sequence based on the video duration of the video to be identified; determining a video frame interval based on the number of video frames and the video duration; intercepting each video frame in a video frame sequence from the video to be identified based on the video frame interval, and taking the last video frame in the video frame sequence as the current video frame.

[0127] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: concatenating time series features and spatial features; performing convolution processing on the concatenated time series features and spatial features to obtain video fusion features; obtaining a mixed attention feature map based on the video fusion features; performing convolution operation on the mixed attention feature map to obtain mixed action features.

[0128] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing deformation operations and deformation replacement operations on the video fusion features; obtaining a channel feature matrix based on the video fusion features after the deformation operation and the video fusion features after the deformation replacement operation; performing softmax processing on the channel feature matrix to obtain a channel attention map; obtaining a mixed attention feature map based on the video fusion features and the channel attention map.

[0129] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining the action category corresponding to the video to be identified through a classification loss function, and obtaining the action position corresponding to the video to be identified through a regression loss function.

[0130] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: when it is detected that the action in the to-be-identified video is an illegal action according to the action category, illegal detection information is generated according to the action category and the action position.

[0131] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for motion recognition, the method comprising: Acquire a video to be identified, extract a video frame sequence and a current video frame in the video to be identified, wherein the current video frame is the last frame in the video to be identified, and the video frame sequence is composed of a plurality of video frames extracted from the video stream to be identified; Performing a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and performing a second convolution operation on the current video frame to obtain spatial features in the video to be identified; splicing the feature graph corresponding to the time series feature and the feature graph corresponding to the spatial feature to connect the time series feature and the spatial feature in series; Performing convolution processing on the concatenated time series features and the spatial features to obtain video fusion features; Perform deformation operation and deformation displacement operation on the video fusion feature; perform inner product operation on the video fusion feature after deformation operation and the video fusion feature after deformation displacement operation to obtain a channel feature matrix; perform softmax processing on the channel feature matrix to obtain a channel attention map; perform inner product and addition operations on the video fusion feature and the channel attention map to obtain a mixed attention feature map; Performing a convolution operation on the mixed attention feature map to obtain a mixed action feature; Classification and regression processing is performed on the mixed action features to obtain the action category and the action position in the current video frame.

2. The method according to claim 1, characterized in that The step of obtaining a video to be identified and extracting a video frame sequence and a current video frame in the video to be identified comprises: Get the video to be identified; Determine the number of video frames in the video frame sequence based on the video duration of the video to be identified; Determine a video frame interval according to the number of video frames and the video duration; Each video frame in the video frame sequence is intercepted from the video to be identified based on the video frame interval, and the last video frame in the video frame sequence is used as the current video frame.

3. The method according to claim 1, characterized in that The performing classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame comprises: The action category corresponding to the video to be identified is obtained through a classification loss function, and the action position corresponding to the video to be identified is obtained through a regression loss function.

4. The method according to claim 1, characterized in that: After the mixed action features are classified and regressed to obtain the action category and the action position in the current video frame, the method further includes: When the action in the to-be-identified video is detected as an illegal action according to the action category, illegal detection information is generated according to the action category and the action position.

5. A motion recognition device, characterized in that: The device comprises: A video frame extraction module is used to obtain a video to be identified, extract a video frame sequence and a current video frame in the video to be identified, wherein the current video frame is the last frame in the video to be identified, and the video frame sequence is composed of multiple video frames extracted from the video stream to be identified; A convolution processing module, configured to perform a first convolution operation on the video frame sequence to obtain time series features in the video to be identified, and perform a second convolution operation on the current video frame to obtain spatial features in the video to be identified; The attention mechanism module is used to splice the feature map corresponding to the time series feature and the feature map corresponding to the spatial feature to connect the time series feature and the spatial feature in series; perform convolution processing on the connected time series feature and the spatial feature to obtain video fusion features; perform deformation operation and deformation displacement operation on the video fusion feature; perform inner product operation on the video fusion feature after deformation operation and the video fusion feature after deformation displacement operation to obtain a channel feature matrix; perform softmax processing on the channel feature matrix to obtain a channel attention map; perform inner product and addition operations on the video fusion feature and the channel attention map to obtain a mixed attention feature map; perform convolution operation on the mixed attention feature map to obtain mixed action features; The classification and regression module is used to perform classification and regression processing on the mixed action features to obtain the action category and the action position in the current video frame.

6. The device according to claim 5, characterized in that The video frame extraction module is specifically used for: Get the video to be identified; Determine the number of video frames in the video frame sequence based on the video duration of the video to be identified; Determine a video frame interval according to the number of video frames and the video duration; Each video frame in the video frame sequence is intercepted from the video to be identified based on the video frame interval, and the last video frame in the video frame sequence is used as the current video frame.

7. The device according to claim 5, characterized in that The classification and regression module is specifically used to: obtain the action category corresponding to the video to be identified through a classification loss function, and obtain the action position corresponding to the video to be identified through a regression loss function.

8. The device according to claim 5, characterized in that It also includes a violation detection module, which is used to: when the action in the to-be-identified video is detected as a violation action according to the action category, generate violation detection information according to the action category and the action position.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images