A method, apparatus, storage medium, and electronic device for video action recognition.

Abstract

This specification discloses a method, apparatus, storage medium, and electronic device for video action recognition, comprising: inputting a single-frame image and optical flow information of the video to be recognized into the ResNeXt3D subnet of a video action recognition model to determine the image features of the single-frame image and the optical flow features of the optical flow information; inputting the image features and optical flow features into the bidirectional attention layer of the video action recognition model to determine a first weighted feature after weighting the image features and a second weighted feature after weighting the optical flow features; inputting the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to determine the risk level of user actions corresponding to users contained in the video to be recognized; sending alert information to monitoring personnel based on the risk level, enabling timely intervention and prevention of dangerous user actions; and improving the accuracy of the determined risk level by using the video action recognition model to recognize user actions in the video to be recognized.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and electronic device for video motion recognition. Background Technology

[0002] With the continuous development of technology, video action recognition is being used more and more widely, especially in the field of public safety.

[0003] Currently, video capture devices, such as cameras, are deployed in communities or streets to collect videos of users' actions in the area. Based on the captured videos, motion recognition is performed on the users to determine whether their actions are dangerous. When a user's action is deemed dangerous, timely intervention and intervention can be implemented to prevent them from endangering public safety. Therefore, how to perform motion recognition on users in videos is an important issue.

[0004] Based on this, this specification provides a method for video action recognition. Summary of the Invention

[0005] This specification provides a method, apparatus, storage medium, and electronic device for video motion recognition, in order to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a method for video action recognition, including:

[0008] Identify the video to be recognized;

[0009] Based on the video to be identified, determine the single frame image corresponding to the video to be identified, and determine the optical flow information corresponding to the video to be identified;

[0010] The single-frame image is input into the ResNeXt3D subnet of the pre-trained video action recognition model to determine image features, and the optical flow information is input into the ResNeXt3D subnet of the video action recognition model to determine optical flow features.

[0011] The image features are input into the bidirectional attention layer of the video action recognition model to determine the first weighted feature, and the optical flow features are input into the bidirectional attention layer of the video action recognition model to determine the second weighted feature;

[0012] The first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to determine the risk status of the user actions corresponding to the user contained in the video to be recognized.

[0013] Based on the risk situation, a prompt message is sent to the monitoring personnel so that they can monitor the user in the video to be identified.

[0014] Optionally, a video action recognition model is pre-trained, specifically including:

[0015] The pre-collected videos are selected as training samples, wherein the videos include at least one user;

[0016] Determine the single-frame image corresponding to the training sample, and determine the optical flow information corresponding to the training sample;

[0017] The single-frame image is input into the ResNeXt3D subnet of the video action recognition model to be trained to determine image features, and the optical flow information is input into the ResNeXt3D subnet of the video action recognition model to be trained to determine optical flow features.

[0018] The image features are input into the bidirectional attention layer of the video action recognition model to be trained to determine the first weighted feature, and the optical flow features are input into the bidirectional attention layer of the video action recognition model to be trained to determine the second weighted feature.

[0019] The first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to be trained to determine the recognition result;

[0020] The risk status of user actions corresponding to users contained in the video is used as a label. Based on the label and the recognition result, the video action recognition model to be trained is trained. The trained video action recognition model is used to determine the risk status of user actions corresponding to users contained in the video to be recognized.

[0021] Optionally, the video action recognition model includes a first ResNeXt3D subnetwork and a second ResNeXt3D subnetwork, wherein the ResNeXt3D subnetwork includes an input layer, a convolutional layer, a residual layer, an attention layer, and a fully connected layer.

[0022] Optionally, the first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to determine the risk of user actions corresponding to users contained in the video to be recognized, specifically including:

[0023] The first weighted feature is linearly transformed to obtain a first feature of a specified dimension, and the second weighted feature is linearly transformed to obtain a second feature of the specified dimension.

[0024] The first feature and the second feature are concatenated, and the concatenated feature is input into the recognition layer of the video action recognition model to determine the risk status of the user actions corresponding to the user contained in the video to be recognized.

[0025] Optionally, the video to be identified includes at least two frames of images;

[0026] Based on the video to be identified, determine the single-frame image corresponding to the video to be identified, and determine the optical flow information corresponding to the video to be identified, specifically including:

[0027] From the at least two frames of the video to be identified, one frame is selected as the single frame image corresponding to the video to be identified;

[0028] The optical flow extraction layer is pre-trained and inputs at least two frames of the video to be identified into it to determine the optical flow information corresponding to the video to be identified output by the optical flow extraction layer.

[0029] Optionally, based on the aforementioned risk situation, a notification message may be sent to the monitoring personnel, specifically including:

[0030] When the risk situation is considered risky, users whose actions in the video to be identified are deemed to be risky users are identified as such.

[0031] From the multiple frames of images contained in the video to be identified, determine the images containing the risky user, and mark the risky user in the images;

[0032] The image of the user with the risk tag is sent to the monitoring personnel so that the monitoring personnel can monitor the user with the risk.

[0033] Optionally, the video to be identified is determined, specifically including:

[0034] Acquire the video captured by the video capture device and use it as the raw video;

[0035] The original video is cropped to identify videos containing at least one user.

[0036] This specification provides a device for video action recognition, comprising:

[0037] The first determining module is used to determine the video to be identified;

[0038] The second determining module is used to determine, based on the video to be identified, a single frame image corresponding to the video to be identified, and optical flow information corresponding to the video to be identified;

[0039] The feature extraction module is used to input the single-frame image into the ResNeXt3D subnet of the pre-trained video action recognition model to determine image features, and to input the optical flow information into the ResNeXt3D subnet of the video action recognition model to determine optical flow features;

[0040] The feature weighting module is used to input the image features into the bidirectional attention layer of the video action recognition model to determine the first weighted feature, and to input the optical flow features into the bidirectional attention layer of the video action recognition model to determine the second weighted feature;

[0041] The recognition module is used to input the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to determine the risk status of the user actions corresponding to the user contained in the video to be recognized;

[0042] The monitoring module is used to send alerts to monitoring personnel based on the risk situation, so that the monitoring personnel can monitor the users in the video to be identified.

[0043] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described video motion recognition method.

[0044] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described video motion recognition method.

[0045] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0046] The video action recognition method provided in this manual determines the video to be recognized, then determines the corresponding single-frame image and optical flow information of the video. The single-frame image and optical flow information are input into the ResNeXt3D subnet of the video action recognition model to determine the image features corresponding to the single-frame image and the optical flow features corresponding to the optical flow information. Then, the image features and optical flow features are input into the bidirectional attention layer of the video action recognition model, and attention weights are applied to the image features and optical flow features respectively to determine the first weighted feature (weighted by the image features) and the second weighted feature (weighted by the optical flow features). Finally, the first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to determine the risk level of the user actions contained in the video to be recognized. Based on the risk level, a prompt message is sent to the monitoring personnel.

[0047] As can be seen from the above method, this application, when performing video action recognition, first determines the video to be recognized, then determines the corresponding single-frame image and the corresponding optical flow information. The single-frame image and optical flow information are input into the ResNeXt3D subnet of the video action recognition model to determine the image features corresponding to the single-frame image and the optical flow features corresponding to the optical flow information. Then, the image features and optical flow features are input into the bidirectional attention layer of the video action recognition model, and attention weighting is applied to the image features and optical flow features respectively to determine the first weighted feature after weighting the image features and the second weighted feature after weighting the optical flow features. Then, the first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to determine the risk level of the user actions corresponding to the user contained in the video to be recognized. Based on the risk level, a prompt message is sent to the monitoring personnel. By using the video action recognition model to recognize user actions in the video to be recognized and determining the risk level of the user actions corresponding to the user contained in the video, the accuracy of the determined risk level is improved. Furthermore, based on the determined risk level, timely intervention and prevention of dangerous user actions can be achieved, ensuring public safety. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0049] Figure 1 This is a flowchart illustrating a video action recognition method provided in this specification;

[0050] Figure 2 This is a schematic diagram of the structure of a ResNeXt3D subnet provided in this specification;

[0051] Figure 3 This is a schematic diagram illustrating a video action recognition process provided in this specification;

[0052] Figure 4 This is a schematic diagram illustrating the training process of a video action recognition model provided in this specification;

[0053] Figure 5 This is a schematic diagram of a video motion recognition device provided in this specification;

[0054] Figure 6 This specification provides a corresponding Figure 1 A schematic diagram of the structure of an electronic device. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0056] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0057] Figure 1 This is a flowchart illustrating a video action recognition method provided in this specification, including the following steps:

[0058] S100: Identify the video to be recognized.

[0059] In this manual, the device used for video action recognition can determine the video to be recognized. This device can be a server or an electronic device such as a desktop computer or laptop. For ease of description, the video action recognition method provided in this manual will be described below using a server as the execution entity.

[0060] The video to be identified can be input by monitoring personnel. Specifically, the server can respond to the input operation of the monitoring personnel and determine the video to be identified. The video to be identified can also be captured by video capture devices deployed in communities or streets, etc. This manual does not make specific limitations. For ease of explanation, the following description uses video captured by an image capture device as an example. This video capture device can be a camera, recorder, or pan-tilt unit, etc. The video to be identified must include at least the environment of the area where the video capture device is located, such as buildings and trees. Action recognition of the video to be identified mainly involves identifying the user's actions in the video to determine whether the user is performing a dangerous action, i.e., whether the user's action is dangerous. Therefore, the video to be identified must include at least one user.

[0061] Furthermore, since a video is composed of multiple frames, some frames may not include the user, only showing the environment of the area where the video capture device is located. Therefore, the server can acquire the video captured by the video capture device and use it as the raw video. The raw video is then cropped to identify the video containing at least one user. Specifically, when cropping the raw video to identify the video containing at least one user, the server can first identify the frames in the raw video that do not include the user as cropped images. If the cropped images are located at the beginning or end of the raw video, the raw video can be directly cropped, removing the cropped images, and the cropped video can be used as the video to be identified. Since a video can be an image sequence composed of multiple frames, the beginning of the video is the first frame, that is, the image at the very first position in the image sequence. Similarly, the end of the video is the last frame, that is, the image at the very end of the image sequence.

[0062] If the cropped image is located outside the beginning and end of the original video, its position can be determined within the original video. By cropping the original video at the determined position, it can be divided into several segments, each containing at least one user. Therefore, each segment can be used as the video to be identified. For example, in a 10-second video, the images at the 3rd and 6th seconds do not include the user; these are the cropped images. Therefore, the video can be cropped at the 3rd and 6th seconds, dividing it into three sub-videos: 1-2 seconds, 4-5 seconds, and 7-10 seconds. Each sub-video can be used as the video to be identified.

[0063] In the process of cropping the original video, the monitoring personnel can view the original video, crop it, and send the cropped video to the server. The server then determines that the video sent by the monitoring personnel is the video to be identified.

[0064] When determining that a frame in the original video that does not include a user is a cropped image, the server can identify multiple frames corresponding to the original video. For each frame, a pre-trained image classification model is used to determine whether the frame includes a user. This involves inputting the frame into the image classification model and determining the classification result. If the classification result indicates the presence of a user, then the frame contains a user. If the classification result indicates the absence of a user, then the frame is considered a cropped image. The image classification model can be a model pre-trained by the server based on image data, which includes images of users, or any existing image classification model; this specification does not impose specific limitations.

[0065] S102: Based on the video to be identified, determine the single-frame image corresponding to the video to be identified, and determine the optical flow information corresponding to the video to be identified.

[0066] The server can determine the single-frame image corresponding to the video to be identified, as well as the optical flow information corresponding to the video. Since a video is composed of multiple frames, the video to be identified includes at least two frames. Each frame is an RGB image. The optical flow information represents the motion speed and direction of each pixel in two adjacent frames. The optical flow information can be determined based on the multiple frames of the video to be identified; specifically, based on the multiple frames, the motion speed and direction of each pixel in the image are calculated.

[0067] Specifically, the server selects one frame from at least two frames of the video to be identified as the single frame corresponding to the video. Simultaneously, the at least two frames of the video are input into a pre-trained optical flow extraction layer to determine the optical flow information corresponding to the video output by the optical flow extraction layer. The optical flow extraction layer can be pre-trained by the server using supervised learning based on multiple frames of images and their corresponding optical flow information. Of course, the optical flow extraction layer can also be any existing network layer for extracting optical flow information from multiple frames of images; this specification does not impose specific limitations.

[0068] When selecting one frame from at least two frames of the video to be identified as the single frame corresponding to the video to be identified, the server can randomly select one frame from the at least two frames of the video to be identified as the single frame corresponding to the video to be identified. Alternatively, since the at least two frames of the video to be identified may include images where the user is not present, the server can first determine the frames containing the user from the at least two frames of the video to be identified as candidate images, and then randomly select one frame from each candidate image as the single frame corresponding to the video to be identified.

[0069] S104: Input the single-frame image into the ResNeXt3D subnet of the pre-trained video action recognition model to determine image features, and input the optical flow information into the ResNeXt3D subnet of the video action recognition model to determine optical flow features.

[0070] The server can input single-frame images into the ResNeXt3D subnet of a pre-trained video action recognition model to determine image features, and input optical flow information into the same subnet to determine optical flow features. The video action recognition model is a pre-trained model used to identify the risk level of user actions in a video; that is, whether a user's action is dangerous. If the action is dangerous, it poses a risk. Similarly, if the action is not dangerous, it poses no risk.

[0071] The aforementioned video action recognition model includes two ResNeXt3D subnetworks: a first ResNeXt3D subnetwork and a second ResNeXt3D subnetwork. Each ResNeXt3D subnetwork is used for feature extraction. Each ResNeXt3D subnetwork includes an input layer, a convolutional layer, a residual layer, an attention layer, and a fully connected layer, as shown below. Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of a ResNeXt3D subnet provided in this specification.

[0072] Specifically, the server can input a single frame image into the first ResNeXt3D subnet of the pre-trained video action recognition model to determine the image features output by the first ResNeXt3D subnet. Simultaneously, optical flow information is input into the second ResNeXt3D subnet of the video action recognition model to determine the optical flow features output by the second ResNeXt3D subnet.

[0073] S106: Input the image features into the bidirectional attention layer of the video action recognition model to determine the first weighted feature, and input the optical flow features into the bidirectional attention layer of the video action recognition model to determine the second weighted feature.

[0074] The server inputs image features into the bidirectional attention layer of the video action recognition model to determine the first weighted feature, and inputs optical flow features into the bidirectional attention layer of the video action recognition model to determine the second weighted feature. The video action recognition model can include two bidirectional attention layers: a first bidirectional attention layer and a second bidirectional attention layer. Each bidirectional attention layer is a BERT (Bidirectional Encoder Representation from Transformers) structured bidirectional attention layer.

[0075] Specifically, the server can input image features into the first bidirectional attention layer of the video action recognition model, apply attention weighting to the image features, and determine the first weighted feature output by the first bidirectional attention layer. Simultaneously, the server can input optical flow features into the second bidirectional attention layer of the video action recognition model, apply attention weighting to the optical flow features, and determine the second weighted feature output by the second bidirectional attention layer.

[0076] S108: Input the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to determine the risk status of the user actions corresponding to the user contained in the video to be recognized.

[0077] The server inputs the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to determine the risk level of the user actions corresponding to the user contained in the video to be recognized. The risk level can be either risky or risk-free, or it can be the probability of risk existing; this specification does not specify a particular risk level.

[0078] Specifically, the server can concatenate the first weighted feature and the second weighted feature, and then input the concatenated feature into the recognition layer of the video action recognition model to determine the risk level of the user actions corresponding to the user contained in the video to be recognized. Since the dimensions of the first weighted feature and the second weighted feature may be inconsistent, the server can first transform the first weighted feature and the second weighted feature to the same feature dimension, then concatenate the transformed features, and input the concatenated feature into the recognition layer of the video action recognition model to determine the risk level of the user actions corresponding to the user contained in the video to be recognized, as output by the recognition layer.

[0079] Based on this, the server can perform a linear transformation on the first weighted feature to obtain a first feature of a specified dimension, and perform a linear transformation on the second weighted feature to obtain a second feature of a specified dimension. Then, the first and second features are concatenated, and the concatenated feature is input into the recognition layer of the video action recognition model to determine the risk level of user actions corresponding to the user contained in the video to be recognized. Here, the specified dimension is a pre-set dimension.

[0080] S110: Based on the risk situation, send a prompt message to the monitoring personnel so that the monitoring personnel can monitor the user in the video to be identified.

[0081] The server can send alerts to monitoring personnel based on risk assessment, enabling them to monitor users in the video to be identified. These alerts can be text or image-based. For example, if the risk is no, the alert might read "User actions in the video are risk-free," while if the risk is high, the alert might read "User actions in the video are risky." Monitoring personnel can then use these alerts to monitor users in the video. If the alert is no-risk, monitoring can continue. However, if the alert is risky, monitoring personnel can monitor the user in the video and intervene promptly to stop their actions, ensuring public safety.

[0082] Additionally, when the risk situation is deemed risky, the server can identify users whose actions in the video to be identified are risky. It identifies images containing risky users from multiple frames in the video and marks these images. The marked images are then sent to monitoring personnel so they can monitor these users and intervene promptly to stop their dangerous actions.

[0083] As can be seen from the above method, when performing video action recognition, the server first determines the video to be recognized, then determines the single-frame image corresponding to the video and the optical flow information corresponding to the video. Next, the single-frame image and optical flow information are input into the ResNeXt3D subnet of the video action recognition model to determine the image features corresponding to the single-frame image and the optical flow features corresponding to the optical flow information. By using the ResNeXt3D subnet to extract image features and optical flow features, the extraction speed and accuracy of features are improved, thereby improving the efficiency of video action recognition. Then, the image features and optical flow features are input into the bidirectional attention layer of the video action recognition model, and attention weighting is applied to the image features and optical flow features respectively, determining the first weighted feature after weighting the image features and the second weighted feature after weighting the optical flow features. By applying attention weighting to the extracted features through the bidirectional attention layer, features that have a greater impact on the identified risk situation have higher weights. Then, the first weighted feature and the second weighted feature are input into the recognition layer of the video action recognition model to determine the risk situation of the user actions corresponding to the user contained in the video to be recognized. Finally, based on the risk assessment, alerts are sent to monitoring personnel. By using a video action recognition model to identify user actions in the video, the risk level of the user actions depicted in the video is determined, improving the accuracy of the identified risk levels. Furthermore, based on the identified risk levels, timely intervention and prevention of dangerous user actions can be implemented, ensuring public safety.

[0084] Furthermore, when inputting the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to determine the risk status of user actions corresponding to users contained in the video to be recognized, the first weighted feature and the second weighted feature are first linearly transformed to obtain the first feature and the second feature of the same dimension. Then, the first feature and the second feature are concatenated and the concatenated feature is input into the recognition layer of the video action recognition model to determine the risk status of user actions corresponding to users contained in the video to be recognized. By concatenating the multimodal features (i.e., the first feature and the second feature) and inputting them into the recognition layer to recognize user actions in the video to be recognized, the accuracy of the identified risk status is improved.

[0085] In this specification, since the video to be identified may include multiple users, when all user actions in the video are not dangerous, the risk level identified by the video action recognition model is no risk. However, if any user in the video has a dangerous action, the risk level identified by the video action model may be risky, or it may be that a particular user's action in the video is risky. For example, if there are two users in the video, and user 1's action is not dangerous, then user 1's action is risk-free. However, user 2's action is dangerous, therefore user 2's action is risky. Therefore, the risk level identified by the video action recognition model for the video is that user 2's action is risky.

[0086] Furthermore, in step S102 above, when determining the single-frame image corresponding to the video to be identified, since the video to be identified may include multiple users, but each user may not appear in every frame of the video to be identified, that is, a certain frame of the video to be identified may not include all the users included in the video. For example, users 1 to 3 may appear in a 1-minute video to be identified, but users 1 to 3 may appear in different time periods of the video to be identified. User 1 appears in the 1st to 20th second of the video to be identified, and users 2 and 3 appear in the 15th to 60th second of the video to be identified. Therefore, for such videos to be identified, when determining the single-frame image corresponding to the video to be identified, the server first determines each frame containing users from the multiple frames included in the video to be identified as candidate images, and then determines the frame containing the most users from the candidate images as the single-frame image corresponding to the video to be identified.

[0087] In this instruction manual, such as Figure 3 As shown, Figure 3This is a schematic diagram illustrating a video action recognition process provided in this specification. The server first determines the video to be recognized, then determines the corresponding single-frame image and optical flow information. Next, the single-frame image is input into a ResNeXt3D subnet of the video action recognition model to determine image features. Simultaneously, the optical flow information is input into another ResNeXt3D subnet of the video action recognition model to determine optical flow features. Then, the image features are input into a bidirectional attention layer of the video action recognition model to determine the first weighted feature. Simultaneously, the optical flow features are input into another bidirectional attention layer of the video action recognition model to determine the second weighted feature. Next, the first weighted feature undergoes a linear transformation to obtain a first feature of a specified dimension. Simultaneously, the second weighted feature undergoes a linear transformation to obtain a second feature of a specified dimension. Finally, the first and second features are concatenated, and the concatenated feature is input into the recognition layer of the video action recognition model to determine the risk level of the user actions corresponding to the user in the video to be recognized. Based on the risk level, a prompt message is sent to monitoring personnel to enable them to monitor the user in the video to be recognized.

[0088] In this specification, the server can pre-train a video recognition model, such as Figure 4 As shown, Figure 4 This is a flowchart illustrating the training process of a video action recognition model provided in this specification, specifically including the following steps:

[0089] S200: Determine the pre-collected videos as training samples, wherein the videos include at least one user.

[0090] The server can use pre-collected videos as training samples. These videos can be captured by video capture devices deployed in communities or streets, and must include at least the environment of the area where the video capture device is located, such as buildings and trees. Since the trained video action recognition model is used to identify user actions in the video to determine whether the user is performing a dangerous action, videos containing at least one user are used as training samples. Videos that only include the environment are not considered as training samples for the video action recognition model.

[0091] Furthermore, a video is composed of multiple frames. Some frames in the video may not include the user, but only the environment of the area where the video capture device is located. The server can crop the video to determine the video containing at least one user as training samples. The specific cropping process is the same as in step S100 above, and will not be repeated here.

[0092] S202: Determine the single-frame image corresponding to the training sample, and determine the optical flow information corresponding to the training sample.

[0093] The server can determine the single-frame image corresponding to the training sample, as well as the optical flow information corresponding to the training sample. Since the video consists of multiple frames, the training sample includes at least two frames. Each frame is an RGB image. The optical flow information represents the motion speed and direction of each pixel in two adjacent frames. The optical flow information can be determined from the multiple frames corresponding to the training sample. The specific process for determining the single-frame image and optical flow information corresponding to the training sample is the same as the process for determining the single-frame image and optical flow information of the video to be identified in step S102 above, and will not be repeated here.

[0094] S204: Input the single-frame image into the ResNeXt3D subnet of the video action recognition model to be trained to determine image features, and input the optical flow information into the ResNeXt3D subnet of the video action recognition model to be trained to determine optical flow features.

[0095] The server can input single-frame images into the ResNeXt3D subnet of the video action recognition model to determine image features, and input optical flow information into the ResNeXt3D subnet of the video action recognition model to determine optical flow features. The video action recognition model to be trained can include two ResNeXt3D subnets: a first ResNeXt3D subnet and a second ResNeXt3D subnet. Each ResNeXt3D subnet includes an input layer, a convolutional layer, a residual layer, an attention layer, and a fully connected layer. The specific process for determining the image features and optical flow features corresponding to the training samples is consistent with the process for determining the image features and optical flow features corresponding to the video to be recognized in step S104 above, and will not be repeated here.

[0096] S206: Input the image features into the bidirectional attention layer of the video action recognition model to be trained to determine the first weighted feature, and input the optical flow features into the bidirectional attention layer of the video action recognition model to be trained to determine the second weighted feature.

[0097] The server can input image features into the bidirectional attention layer of the video action recognition model to be trained to determine the first weighted feature, and input optical flow features into the bidirectional attention layer of the same model to determine the second weighted feature. The video action recognition model to be trained can include two bidirectional attention layers, each of which is a BERT (Bidirectional Encoder Representation from Transformers) structure. The specific process for determining the first and second weighted features corresponding to the training samples is consistent with the process for determining the first and second weighted features corresponding to the video to be recognized in step S106 above, and will not be repeated here.

[0098] S208: Input the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to be trained, and determine the recognition result.

[0099] The server can input the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to be trained, and determine the recognition result. The recognition result represents the risk level of the user actions corresponding to the users included in the training samples. The specific process for determining the recognition result corresponding to the training samples is consistent with the process for determining the risk level of the video to be recognized in step S108 above, and will not be repeated here.

[0100] S210: The risk status of the user actions corresponding to the user contained in the video is used as a label. Based on the label and the recognition result, the video action recognition model to be trained is trained. The trained video action recognition model is used to determine the risk status of the user actions corresponding to the user contained in the video to be recognized based on the video to be recognized.

[0101] The server can label the risk level of user actions within the video as annotations. Based on these labels and the recognition results, it trains the video action recognition model. The risk level can be pre-labeled by monitoring personnel using the video samples as training data. The risk level is determined based on the type of user action. Actions such as fighting, vandalism, and violence are considered dangerous and pose a risk to public safety. Actions such as standing still, walking, and running are not considered dangerous and do not pose a risk to public safety.

[0102] Of course, the risk level of a user's actions is also related to the environment in the video. Some types of user actions may be dangerous in one environment but not in another. For example, a user running quickly in a train station might collide with others, so running is a risky action in a train station environment. However, when a user is competing on a track and field, running quickly is not considered a dangerous action. Therefore, the type of user action may differ in different scenarios, the risk level of different types of user actions may differ, and the risk level of the same type of user action may also differ. It is necessary to determine the risk level of a user's action based on its type and the environment in which the user is located (i.e., the environment of the area where the video capture device is located).

[0103] Specifically, the server can use the risk level of user actions corresponding to users contained in the video as annotations, and train the video action recognition model with the goal of minimizing the difference between the annotations and the recognition results. The trained video action recognition model is then used to determine the risk level of user actions corresponding to users contained in the video to be recognized.

[0104] The above describes one or more implementations of the methods described in this specification. Based on the same concept, this specification also provides corresponding video action recognition devices, such as... Figure 5 As shown.

[0105] Figure 5 This is a schematic diagram of a video motion recognition device provided in this specification, including:

[0106] The first determining module 300 is used to determine the video to be identified;

[0107] The second determining module 302 is used to determine a single frame image corresponding to the video to be identified, and to determine optical flow information corresponding to the video to be identified, based on the video to be identified.

[0108] The feature extraction module 304 is used to input the single-frame image into the ResNeXt3D subnet of the pre-trained video action recognition model to determine image features, and to input the optical flow information into the ResNeXt3D subnet of the video action recognition model to determine optical flow features;

[0109] The feature weighting module 306 is used to input the image features into the bidirectional attention layer of the video action recognition model to determine the first weighted feature, and to input the optical flow features into the bidirectional attention layer of the video action recognition model to determine the second weighted feature;

[0110] The recognition module 308 is used to input the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to determine the risk status of the user action corresponding to the user contained in the video to be recognized;

[0111] The monitoring module 310 is used to send a prompt message to the monitoring personnel based on the risk situation, so that the monitoring personnel can monitor the user in the video to be identified.

[0112] Optionally, the device further includes:

[0113] Training module 312 is used to determine pre-collected videos as training samples, wherein the videos include at least one user; determine the single-frame image corresponding to the training sample, and determine the optical flow information corresponding to the training sample; input the single-frame image into the ResNeXt3D subnet of the video action recognition model to be trained to determine image features, and input the optical flow information into the ResNeXt3D subnet of the video action recognition model to be trained to determine optical flow features; input the image features into the bidirectional attention layer of the video action recognition model to be trained to determine a first weighted feature, and input the optical flow features into the bidirectional attention layer of the video action recognition model to be trained to determine a second weighted feature; input the first weighted feature and the second weighted feature into the recognition layer of the video action recognition model to be trained to determine the recognition result; use the risk status of user actions corresponding to the user contained in the video as a label, and train the video action recognition model to be trained according to the label and the recognition result. The trained video action recognition model is used to determine the risk status of user actions corresponding to the user contained in the video to be recognized.

[0114] Optionally, the video action recognition model includes a first ResNeXt3D subnetwork and a second ResNeXt3D subnetwork, wherein the ResNeXt3D subnetwork includes an input layer, a convolutional layer, a residual layer, an attention layer, and a fully connected layer.

[0115] Optionally, the recognition module 308 is specifically used to: perform a linear transformation on the first weighted feature to obtain a first feature of a specified dimension, and perform a linear transformation on the second weighted feature to obtain a second feature of the specified dimension; concatenate the first feature and the second feature, and input the concatenated feature into the recognition layer of the video action recognition model to determine the risk status of the user action corresponding to the user contained in the video to be recognized.

[0116] Optionally, the video to be identified includes at least two frames of images;

[0117] The second determining module 302 is specifically used to: select one frame image from at least two frames of images included in the video to be identified as the single frame image corresponding to the video to be identified; input the at least two frames of images included in the video to be identified into a pre-trained optical flow extraction layer, and determine the optical flow information corresponding to the video to be identified output by the optical flow extraction layer.

[0118] Optionally, the monitoring module 310 is specifically used to: when the risk situation is risky, determine that the user whose actions in the video to be identified are risky; determine the image containing the risky user from the multiple frames of images contained in the video to be identified, and mark the risky user in the image; send the image marked with the risky user to the monitoring personnel so that the monitoring personnel can monitor the risky user.

[0119] Optionally, the first determining module 300 is specifically used to: acquire the video captured by the video acquisition device and use it as the original video; perform cropping processing on the original video and determine the video containing at least one user as the video to be identified.

[0120] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 This provides a method for video action recognition.

[0121] This instruction manual also provides, for example, Figure 6 The one shown corresponds to Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 6 As shown, at the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above. Figure 1 The aforementioned video action recognition method.

[0122] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0123] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0124] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0125] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0126] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0127] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0132] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0133] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0135] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0137] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0138] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method of video action recognition, characterized by, The method comprises: determining a video to be identified; determining a single-frame image corresponding to the video to be identified and determining optical flow information corresponding to the video to be identified according to the video to be identified; inputting the single-frame image into a ResNeXt3D subnet of a pre-trained video action recognition model to determine image features and inputting the optical flow information into the ResNeXt3D subnet of the video action recognition model to determine optical flow features; inputting the image features into a bidirectional attention layer of the video action recognition model to determine first weighted features and inputting the optical flow features into the bidirectional attention layer of the video action recognition model to determine second weighted features; inputting the first weighted features and the second weighted features into a recognition layer of the video action recognition model to determine a risk situation of a user action of a user included in the video to be identified; sending prompt information to a monitoring personnel according to the risk situation to enable the monitoring personnel to monitor the user in the video to be identified; the determination of the single-frame image corresponding to the video to be identified comprises: selecting a single-frame image from at least two images included in the video to be identified as the single-frame image corresponding to the video to be identified.

2. The method of claim 1, wherein, The pre-training of the video action recognition model comprises: determining a video collected in advance as a training sample, wherein the video includes at least one user; determining a single-frame image corresponding to the training sample and determining optical flow information corresponding to the training sample; inputting the single-frame image into a ResNeXt3D subnet of a video action recognition model to be trained to determine image features and inputting the optical flow information into the ResNeXt3D subnet of the video action recognition model to be trained to determine optical flow features; inputting the image features into a bidirectional attention layer of the video action recognition model to be trained to determine first weighted features and inputting the optical flow features into the bidirectional attention layer of the video action recognition model to be trained to determine second weighted features; inputting the first weighted features and the second weighted features into a recognition layer of the video action recognition model to be trained to determine a recognition result; determining a risk situation of a user action of a user included in the video as a label, training the video action recognition model to be trained according to the label and the recognition result, and using the trained video action recognition model to determine a risk situation of a user action of a user included in a video to be identified according to the video to be identified.

3. The method of claim 1, wherein, The video action recognition model comprises a first ResNeXt3D subnet and a second ResNeXt3D subnet, and the ResNeXt3D subnet comprises an input layer, a convolution layer, a residual layer, an attention layer, and a fully connected layer.

4. The method of claim 1, wherein, inputting the first weighted features and the second weighted features into a recognition layer of the video action recognition model to determine a risk situation of a user action of a user included in the video to be identified, specifically comprising: linearly transforming the first weighted feature to obtain a first feature of a specified dimension, and linearly transforming the second weighted feature to obtain a second feature of the specified dimension; splicing the first feature and the second feature, and inputting the spliced feature into a recognition layer of the video action recognition model to determine a risk situation of a user action corresponding to a user included in the to-be-identified video.

5. The method of claim 1, wherein, The to-be-identified video includes at least two images. According to the to-be-identified video, the single-frame image corresponding to the to-be-identified video is determined, and the optical flow information corresponding to the to-be-identified video is determined, specifically including: selecting an image from the at least two images included in the to-be-identified video as the single-frame image corresponding to the to-be-identified video; inputting the at least two images included in the to-be-identified video into a pre-trained optical flow extraction layer to determine the optical flow information corresponding to the to-be-identified video output by the optical flow extraction layer.

6. The method of claim 1, wherein, According to the risk situation, sending prompt information to a monitoring personnel, specifically including: when the risk situation is risky, determining that the user action in the to-be-identified video is a risky user; determining an image in which the risky user exists from the multiple images included in the to-be-identified video, and marking the risky user in the image; sending the image in which the risky user is marked to the monitoring personnel, so that the monitoring personnel monitors the risky user.

7. The method of claim 1, wherein, determining a to-be-identified video, specifically including: acquiring a video collected by a video collection device and taking the video as an original video; cropping the original video to determine a video including at least one user as a to-be-identified video.

8. An apparatus for video action recognition, the apparatus comprising: including: a first determination module configured to determine a to-be-identified video; a second determination module configured to determine a single-frame image corresponding to the to-be-identified video and determine optical flow information corresponding to the to-be-identified video according to the to-be-identified video; a feature extraction module configured to input the single-frame image into a ResNeXt3D subnetwork of a pre-trained video action recognition model to determine an image feature, and input the optical flow information into the ResNeXt3D subnetwork of the video action recognition model to determine an optical flow feature; a feature weighting module configured to input the image feature into a bidirectional attention layer of the video action recognition model to determine a first weighted feature, and input the optical flow feature into the bidirectional attention layer of the video action recognition model to determine a second weighted feature; a recognition module configured to input the first weighted feature and the second weighted feature into a recognition layer of the video action recognition model to determine a risk situation of a user action corresponding to a user included in the to-be-identified video; a monitoring module configured to send prompt information to a monitoring personnel according to the risk situation, so that the monitoring personnel monitors the user in the to-be-identified video; the second determination module is specifically configured to select an image from the at least two images included in the to-be-identified video as the single-frame image corresponding to the to-be-identified video when determining the single-frame image corresponding to the to-be-identified video.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by a processor to implement the method in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the method in any one of claims 1-7.

Images