Behavior recognition model construction method and behavior recognition method

Abstract

The present application relates to a method for constructing a behavior recognition model and a behavior recognition method. The method includes: obtaining a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, the model including a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network; inputting each video frame sequence in the video frame sequence set into the spatiotemporal feature extraction network to obtain the spatiotemporal features corresponding to the video frame sequence, and inputting the video frame sequence into the spatial feature extraction network to obtain the spatial features corresponding to the video frame sequence; inputting the spatiotemporal features and the spatial features into the feature fusion network to obtain the fusion features corresponding to the video frame sequence; inputting the fusion features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence; adjusting the model parameters of the initial behavior recognition model according to the behavior prediction results and the behavior annotations to obtain the trained behavior recognition model. The present method can achieve accurate behavior recognition.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a behavior recognition model construction method, a behavior recognition method, an apparatus, a computer device, and a storage medium. Background Art

[0002] With the development of computer technology, behavior recognition technology has emerged. Behavior recognition technology is an important research direction in the field of computer vision and has broad application prospects in daily scenarios such as monitoring and autonomous driving. For example, in the field of logistics, behavior recognition technology can be used to accurately and quickly screen out whether there is violent sorting behavior, so as to provide more accurate and timely reminders and guidance.

[0003] In traditional technology, when performing behavior recognition on video data, a common method is to first identify the person in each video frame, then reasonably connect these bounding boxes into an action pipeline, detect the behavior to be recognized, and then perform behavior recognition on the behavior to be recognized.

[0004] However, traditional technologies suffer from the problem of inaccurate behavior recognition. Summary of the invention

[0005] Based on this, it is necessary to provide a behavior recognition model construction method that supports high-accuracy behavior recognition and a behavior recognition method that can achieve high-accuracy behavior recognition in response to the above technical problems.

[0006] A method for constructing a behavior recognition model, the method comprising:

[0007] Obtain a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, which includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network;

[0008] Input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0009] Input the spatiotemporal features and spatial features into the feature fusion network to obtain the fusion features corresponding to the video frame sequence;

[0010] Input the fused features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence;

[0011] According to the behavior prediction results and behavior annotations, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

[0012] In one embodiment, the video frame sequence is input into a spatial feature extraction network, and the spatial features corresponding to the video frame sequence are obtained, including:

[0013] Perform image fusion on the video frame sequence to obtain the trajectory image corresponding to the video frame sequence;

[0014] The trajectory image is input into the spatial feature extraction network to obtain the spatial features corresponding to the video frame sequence.

[0015] In one embodiment, performing image fusion on a video frame sequence to obtain a trajectory image corresponding to the video frame sequence includes:

[0016] Convert each video frame in the video frame sequence into a grayscale image, and divide the converted video frame sequence according to a preset number of image channels to obtain multiple groups of grayscale images to be fused;

[0017] Calculate the pixel average of each group of grayscale images to be fused, and obtain the trajectory image corresponding to the video frame sequence based on the pixel average.

[0018] In one embodiment, the spatiotemporal features and the spatial features are input into a feature fusion network to obtain fusion features corresponding to the video frame sequence, including:

[0019] Input spatiotemporal features and spatial features into the feature fusion network, and determine the feature weight distribution through the feature fusion network;

[0020] According to the feature weight distribution, spatiotemporal features and spatial features, the fusion features corresponding to the video frame sequence are obtained.

[0021] In one embodiment, the fused features are input into a convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence, including:

[0022] Input the fused features into the convolution operation network to obtain the candidate detection box;

[0023] According to the behavior annotations, the candidate detection boxes are threshold-selected to obtain the behavior prediction results corresponding to the video frame sequence.

[0024] In one embodiment, according to the behavior prediction results and the behavior annotations, the model parameters of the initial behavior recognition model are adjusted to obtain the trained behavior recognition model, including:

[0025] Calculate the bounding box loss function based on the predicted detection box and behavior annotation in the behavior prediction result, and calculate the category loss function based on the predicted category and behavior annotation in the behavior prediction result;

[0026] According to the border loss function and the category loss function, the model loss function is obtained;

[0027] According to the model loss function, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

[0028] A behavior recognition method, the method comprising:

[0029] Obtain a video frame sequence to be identified;

[0030] Inputting the video frame sequence to be identified into the trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be identified, wherein the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method;

[0031] Compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0032] A behavior recognition model construction device, the device comprising:

[0033] The sample acquisition module is used to obtain a set of video frame sequences carrying behavior annotations and an initial behavior recognition model. The initial behavior recognition model includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network.

[0034] A feature extraction module is used to input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0035] A feature fusion module is used to input spatiotemporal features and spatial features into a feature fusion network to obtain fusion features corresponding to a video frame sequence;

[0036] The prediction module is used to input the fused features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence;

[0037] The adjustment module is used to adjust the model parameters of the initial behavior recognition model according to the behavior prediction results and the behavior annotations to obtain a trained behavior recognition model.

[0038] A behavior recognition device, comprising:

[0039] A data acquisition module, used to acquire a video frame sequence to be identified;

[0040] A recognition module, used to input the video frame sequence to be recognized into a trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be recognized, wherein the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method;

[0041] The comparison module is used to compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0042] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Obtain a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, which includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network;

[0044] Input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0045] Input the spatiotemporal features and spatial features into the feature fusion network to obtain the fusion features corresponding to the video frame sequence;

[0046] Input the fused features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence;

[0047] According to the behavior prediction results and behavior annotations, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

[0048] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0049] Obtain a video frame sequence to be identified;

[0050] Inputting the video frame sequence to be identified into the trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be identified, wherein the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method;

[0051] Compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0052] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0053] Obtain a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, which includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network;

[0054] Input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0055] Input the spatiotemporal features and spatial features into the feature fusion network to obtain the fusion features corresponding to the video frame sequence;

[0056] Input the fused features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence;

[0057] According to the behavior prediction results and behavior annotations, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

[0058] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0059] Obtain a video frame sequence to be identified;

[0060] Inputting the video frame sequence to be identified into the trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be identified, wherein the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method;

[0061] Compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0062] The above-mentioned behavior recognition model construction method, after obtaining a set of video frame sequences carrying behavior annotations, performs spatiotemporal feature extraction and spatial feature extraction on each video frame sequence to obtain the spatiotemporal features and spatial features corresponding to the video frame sequence, and uses a feature fusion network to fuse the spatiotemporal features and spatial features to obtain fused features with richer feature information, and obtains behavior prediction results corresponding to the video frame sequence by inputting the fused features into a convolution operation network, and adjusts the model parameters of the initial behavior recognition model according to the behavior prediction results and behavior annotations, so as to realize the training of the initial behavior recognition model and obtain a behavior recognition model that can realize accurate behavior recognition, so that the behavior recognition model can be used to improve the accuracy of behavior recognition.

[0063] The above-mentioned behavior recognition method can obtain accurate predicted behavior recognition results by inputting the video frame sequence to be recognized into the trained behavior recognition model after obtaining the video frame sequence to be recognized, so as to obtain the behavior recognition result by comparing the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold, thereby achieving accurate behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1A flowchart of a method for building a behavior recognition model in one embodiment;

[0065] Figure 2 A flowchart of a behavior recognition method in one embodiment;

[0066] Figure 3 A flowchart of a method for building a behavior recognition model in one embodiment;

[0067] Figure 4 A flowchart of a behavior recognition model construction method and a behavior recognition method in one embodiment;

[0068] Figure 5 A structural block diagram of a behavior recognition model building device in one embodiment;

[0069] Figure 6 is a structural block diagram of a behavior recognition device in an embodiment;

[0070] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0072] In one embodiment, Figure 1 As shown, a method for building a behavior recognition model is provided. This embodiment uses the method applied to a server as an example. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0073] Step 102, obtaining a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, wherein the initial behavior recognition model includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network.

[0074] Among them, the video frame sequence refers to the video frame sequence intercepted in time order from the original surveillance video in multiple scenes, and the number of video frames intercepted in the video frame sequence can be set as needed. Behavior annotation is used to characterize the preset behavior category corresponding to the video frame sequence. For example, in the field of logistics, the behavior label can be violent sorting, non-violent sorting, etc. The initial behavior recognition model refers to the behavior recognition model that has not yet been adjusted. The initial behavior recognition model includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network. The spatiotemporal feature extraction network is used to extract the spatiotemporal features of each video frame in the video frame sequence. The spatiotemporal features refer to the temporal spatial features. For example, the spatiotemporal feature extraction network can be a 3D-CNN (three dimensional-Convolutional Neural Network) network, with a 3D network extended by the residual network Resnet (Residual Network) as the basic skeleton. The spatial feature extraction network is used to extract the spatial features of each video frame in the video frame sequence. For example, the spatial feature extraction network can specifically be a 2D-CNN (two dimensional-Convolutional Neural Network) network with Darknet as the basic skeleton.

[0075] Specifically, after obtaining the original surveillance video, the video frame sequence can be intercepted from the original surveillance video in chronological order by presetting the number of intercepted video frames, and behavior annotations can be set for the video frame sequence, and the video frame sequence carrying the behavior annotations can be stored in a preset database. When it is necessary to build a behavior recognition model, the server will obtain a set of video frame sequences carrying behavior annotations and an initial behavior recognition model from the preset database. Among them, setting behavior annotations means marking the area containing behavioral actions with a rectangular frame and marking the category of the behavioral action.

[0076] Step 104, input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence.

[0077] Specifically, the server will input each video frame sequence in the video frame sequence set into the spatiotemporal feature extraction network to obtain the spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into the spatial feature extraction network to obtain the spatial features corresponding to the video frame sequence, so as to achieve the synchronous acquisition of the spatiotemporal features and spatial features corresponding to the video frame sequence. Before inputting each video frame sequence into the spatiotemporal feature extraction network and the spatial feature extraction network, the server will perform data enhancement processing on each video frame in each video frame sequence. The data enhancement processing here includes cropping, translation, rotation, brightness change, and noise addition, etc. Cropping refers to cropping the video frame according to the rectangular box in the behavior annotation to make the rectangular box of the behavior annotation more prominent.

[0078] For example, when the spatial feature extraction network is a 3D-CNN network, by inputting bn*6 pictures (the tensor is represented by (bn, 6, w, h, c), where bn refers to the number of pictures, w and h are the picture sizes, and c is the number of image channels, and for example, the tensor can be (16, 6, 608, 416, 3)) into the 3D-CNN network, a feature with an output size of (bn, c2, 1, w, h) can be obtained. When the spatial feature extraction network is a 2D-CNN network, by inputting bn pictures (the tensor is represented by (bn, w, h, c), corresponding to the above tensor, specifically (16, 608, 416, 3)) into the 2D-CNN network, a feature with an output size of (bn, c1, w, h) can be obtained.

[0079] Step 106, input the spatiotemporal features and the spatial features into a feature fusion network to obtain fusion features corresponding to the video frame sequence.

[0080] The feature fusion network refers to a network that fuses spatiotemporal features and spatial features. For example, the feature fusion network can be a module that combines spatial and channel attention mechanisms, which can learn feature weight distributions from spatiotemporal features and spatial features, and then fuse spatiotemporal features and spatial features according to the feature weight distributions.

[0081] Specifically, the server will input the spatiotemporal features and spatial features into the feature fusion network, learn the spatiotemporal features and spatial features through the feature fusion network, obtain the corresponding feature weight distribution, and use the feature weight distribution to fuse the spatiotemporal features and spatial features. In this way, the distribution of the original features can be changed, effective features can be enhanced, and invalid features or noise can be suppressed.

[0082] Step 108: Input the fused features into a convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence.

[0083] Among them, the convolution operation network refers to a network used to predict behavior based on fused features. For example, the convolution operation network can be specifically composed of continuous convolution layers and a number of shortcut connections, and can predict the detection box, the detection box confidence, and the category. The behavior prediction results include the predicted detection box, the detection box confidence, and the predicted category, where the predicted detection box can be represented by four frame coordinates, where the frame coordinates refer to the pixel coordinates on the video frame.

[0084] Specifically, the server will input the fused features into the convolution operation network, and obtain the behavior prediction results corresponding to the video frame sequence through the analysis of the fused features by the convolution operation network.

[0085] Step 110, adjusting the model parameters of the initial behavior recognition model according to the behavior prediction results and the behavior annotations to obtain a trained behavior recognition model.

[0086] Specifically, the server will calculate the model loss function based on the predicted detection box, detection box confidence, predicted category, and behavior annotation in the behavior prediction results, and back-propagate the initial behavior recognition model to update the model parameters based on the model loss function to obtain the trained behavior recognition model. The model loss function includes the border loss function and the category loss function, which need to be calculated separately.

[0087] The above-mentioned behavior recognition model construction method, after obtaining a set of video frame sequences carrying behavior annotations, performs spatiotemporal feature extraction and spatial feature extraction on each video frame sequence to obtain the spatiotemporal features and spatial features corresponding to the video frame sequence, and uses a feature fusion network to fuse the spatiotemporal features and spatial features to obtain fused features with richer feature information, and obtains behavior prediction results corresponding to the video frame sequence by inputting the fused features into a convolution operation network, and adjusts the model parameters of the initial behavior recognition model according to the behavior prediction results and behavior annotations, so as to realize the training of the initial behavior recognition model and obtain a behavior recognition model that can realize accurate behavior recognition, so that the behavior recognition model can be used to improve the accuracy of behavior recognition.

[0088] In one embodiment, the video frame sequence is input into a spatial feature extraction network, and the spatial features corresponding to the video frame sequence are obtained, including:

[0089] Perform image fusion on the video frame sequence to obtain the trajectory image corresponding to the video frame sequence;

[0090] The trajectory image is input into the spatial feature extraction network to obtain the spatial features corresponding to the video frame sequence.

[0091] The track image refers to an image obtained by fusion of each video frame in the video frame sequence, and the track image can represent the behavior track corresponding to each video frame. For example, the track image can be an image optical flow track image.

[0092] Specifically, the server will perform image fusion on each video frame in the video frame sequence, fuse each video frame into a trajectory image, and then use the spatial feature extraction network to extract features from the trajectory image to obtain spatial features corresponding to the video frame sequence.

[0093] In this embodiment, by performing image fusion on the video frame sequence, a trajectory image corresponding to the video frame sequence is obtained, and the trajectory image is input into the spatial feature extraction network to obtain the spatial features corresponding to the video frame sequence, thereby realizing the acquisition of the spatial features corresponding to the video frame sequence.

[0094] In one embodiment, performing image fusion on a video frame sequence to obtain a trajectory image corresponding to the video frame sequence includes:

[0095] Convert each video frame in the video frame sequence into a grayscale image, and divide the converted video frame sequence according to a preset number of image channels to obtain multiple groups of grayscale images to be fused;

[0096] Calculate the pixel average of each group of grayscale images to be fused, and obtain the trajectory image corresponding to the video frame sequence based on the pixel average.

[0097] The preset number of image channels refers to the number of image channels of a preset trajectory image. For example, when the trajectory image is an RGB (red, green, blue) image, the preset number of image channels may be specifically 3.

[0098] Specifically, the server converts each video frame in the video frame sequence into a grayscale image, and randomly divides the converted video frame sequence according to the preset number of image channels. On the premise that the number of images in each group of grayscale images to be fused is as even as possible, multiple groups of grayscale images to be fused are obtained, and then the pixel average of each group of grayscale images to be fused is calculated respectively. The pixel average of each group of grayscale images to be fused is used as the image channel pixel of the trajectory image corresponding to the video frame sequence, and the trajectory image is obtained according to the image channel pixel.

[0099] Specifically, taking the trajectory image as an RGB image and the video frame sequence including 6 video frames as an example, the server will first convert the 6 consecutive video frames into grayscale images, and then divide the converted video frame sequence according to the preset number of image channels (i.e., 3) to obtain 3 groups of grayscale images to be fused, where the first and second grayscale images are a group, the third and fourth grayscale images are a group, and the fifth and sixth grayscale images are a group according to the time sequence of the video frames. The server will calculate the pixel average of each group of grayscale images to be fused by adding the pixels of the first and second images and dividing them by 2, adding the pixels of the third and fourth images and dividing them by 2, and adding the pixels of the fifth and sixth images and dividing them by 2, and use the above three pixel averages as the RGB three channels of the fused trajectory image.

[0100] In this embodiment, by converting each video frame in the video frame sequence into a grayscale image, and dividing the converted video frame sequence according to a preset number of image channels, a plurality of groups of grayscale images to be fused are obtained, and the pixel average value of each group of grayscale images to be fused is calculated. According to the pixel average value, a trajectory image corresponding to the video frame sequence is obtained, thereby realizing the acquisition of the trajectory image.

[0101] In one embodiment, the spatiotemporal features and the spatial features are input into a feature fusion network to obtain fusion features corresponding to the video frame sequence, including:

[0102] Input spatiotemporal features and spatial features into the feature fusion network, and determine the feature weight distribution through the feature fusion network;

[0103] According to the feature weight distribution, spatiotemporal features and spatial features, the fusion features corresponding to the video frame sequence are obtained.

[0104] Among them, the feature fusion network can specifically refer to the convolutional attention module, which is a simple and effective attention module for feedforward convolutional neural networks. Given an intermediate feature map, the convolutional attention module will infer the attention map along two independent dimensions (channel and space) in sequence, and then multiply the attention map with the input feature map for adaptive feature optimization. It can be seamlessly integrated into any CNN architecture and can be trained end-to-end with the basic CNN.

[0105] Specifically, the server will input the spatiotemporal features and spatial features into the feature fusion network, and the spatiotemporal features and spatial features will be learned through a series of convolution and pooling operations through the feature fusion network, and the feature weight distribution of the spatiotemporal features and spatial features will be determined. The feature weights corresponding to the spatiotemporal features and spatial features will be determined according to the feature weight distribution, and the fusion features corresponding to the video frame sequence will be obtained according to the feature weights, spatiotemporal features and spatial features. For example, when the spatial features are (bn, c1, w, h) and the spatiotemporal features are (bn, c2, 1, w, h), the fusion features after feature fusion can be obtained as (bn, c1+c2, w, h) through the feature fusion network.

[0106] In this embodiment, by inputting the spatiotemporal features and spatial features into the feature fusion network, determining the feature weight distribution through the feature fusion network, and obtaining the fusion features corresponding to the video frame sequence based on the feature weight distribution, spatiotemporal features and spatial features, the acquisition of the fusion features can be achieved.

[0107] In one embodiment, the fused features are input into a convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence, including:

[0108] Input the fused features into the convolution operation network to obtain the candidate detection box;

[0109] According to the behavior annotations, the candidate detection boxes are threshold-selected to obtain the behavior prediction results corresponding to the video frame sequence.

[0110] Specifically, the server inputs the fused features into the convolution operation network, and can use the convolution operation network to predict the behavior based on the fused features, and obtain the candidate detection boxes, the confidence of each candidate detection box, and the behavior category. After obtaining the candidate detection boxes and the confidence of each candidate detection box, the server will perform threshold selection on the candidate detection boxes according to the behavior annotations to obtain the behavior prediction results corresponding to the video frame sequence.

[0111] Specifically, threshold selection refers to using a preset detection box threshold, an IOU (Intersection-over-Union) threshold, and the confidence of the detection box to select. Through this selection method, the detection box with a confidence that meets the requirements can be screened out to achieve accurate prediction, wherein the preset detection box threshold can be set as needed. Further, the selection process can be that the server first screens the candidate detection boxes according to the preset detection box threshold and the confidence of the candidate detection boxes to obtain the detection boxes to be screened for the second time, and then performs IOU threshold selection on the detection boxes to be screened for the second time according to the confidence of the detection boxes to be screened for the second time to obtain the detection boxes that meet the requirements, and obtain the behavior prediction results according to the detection boxes that meet the requirements, as well as the corresponding confidence and behavior categories.

[0112] Furthermore, the method of performing IOU threshold selection can be: sorting the detection frames to be screened for the second time according to their confidence, selecting the detection frame to be screened for the second time with the highest score, traversing the remaining detection frames to be screened for the second time according to the detection frame to be screened for the second time with the highest score, if there is a detection frame in the remaining detection frames to be screened for the second time with an overlap area with the detection frame to be screened for the second time with the highest score greater than a preset ratio threshold, it will be deleted, and the detection frame to be screened for the second time with the highest score will continue to be selected from the remaining detection frames to be screened for the second time that have not been deleted, and the step of traversing the remaining detection frames to be screened for the second time according to the detection frame to be screened for the second time with the highest score will be returned until all the detection frames to be screened for the second time are selected and the detection frames that meet the requirements are obtained. Among them, the preset ratio threshold can be set as needed. In this way, redundant detection frames in the candidate detection frames can be selected for the second time.

[0113] In this embodiment, by inputting the fused features into the convolution operation network to obtain the candidate detection boxes, the candidate detection boxes are threshold-selected according to the behavior annotations to obtain the behavior prediction results corresponding to the video frame sequence, thereby achieving the acquisition of the behavior prediction results.

[0114] In one embodiment, according to the behavior prediction results and the behavior annotations, the model parameters of the initial behavior recognition model are adjusted to obtain the trained behavior recognition model, including:

[0115] Calculate the bounding box loss function based on the predicted detection box and behavior annotation in the behavior prediction result, and calculate the category loss function based on the predicted category and behavior annotation in the behavior prediction result;

[0116] According to the border loss function and the category loss function, the model loss function is obtained;

[0117] According to the model loss function, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

[0118] Among them, the border loss function includes the border coordinate loss function and the border confidence loss function. The border coordinate loss function can be obtained by comparing the coordinates of the predicted detection box in the behavior prediction result with the coordinates of the real border in the behavior annotation. The border confidence loss function can be obtained by the confidence of the predicted detection box in the behavior prediction result. The category loss function is used to characterize the loss between the predicted category in the behavior prediction result and the behavior category in the behavior annotation.

[0119] Specifically, the server will calculate the border coordinate loss function based on the coordinates of the predicted detection box in the behavior prediction result and the coordinates of the real border in the behavior annotation, and then perform mean square loss calculation on the confidence of the predicted detection box in the behavior prediction result to obtain the border confidence loss function. According to the border coordinate loss function and the border confidence loss function, the border loss function is obtained, and according to the predicted category in the behavior prediction result and the behavior category in the behavior annotation, Focal Loss is performed on each predicted category to obtain the category loss function. Finally, according to the border loss function, the category loss function and the preset loss function weight, the model loss function is obtained, and back propagation is performed according to the model loss function to adjust the model parameters of the initial behavior recognition model until the model loss function converges or is less than the preset loss function threshold, and the trained behavior recognition model is obtained. Among them, the preset loss function weight and the preset loss function threshold can be set as needed.

[0120] For example, the bounding box coordinate loss function can be expressed as: l1loss(x i ,y i )=|x i -y i |Get, where x i and i They represent the coordinates of the predicted detection box and the coordinates of the real box in the behavior annotation. The box confidence loss function can be expressed by the formula: MSE loss(x i ,y i )=(x i -y i ) 2 Get, where x i and i They represent the confidence of the predicted detection box and the confidence of the real box in the behavior annotation respectively. The category loss function can be expressed by the formula: Get, where y is the predicted category.

[0121] Furthermore, when back propagation is performed according to the model loss function and the model parameters of the initial behavior recognition model are adjusted, the model gradient is calculated according to the model loss function and back propagation is performed using the model gradient. In the process of back propagation using the model gradient, for the feature fusion network, the model gradient is divided into two parts and parameters are updated through the spatiotemporal feature extraction network and the spatial feature extraction network respectively.

[0122] In this embodiment, the border loss function is calculated based on the predicted detection box and behavior annotation in the behavior prediction results, and the category loss function is calculated based on the predicted category and behavior annotation in the behavior prediction results. The model loss function is obtained based on the border loss function and the category loss function. According to the model loss function, the model parameters of the initial behavior recognition model are adjusted to obtain the trained behavior recognition model, thereby realizing the training of the behavior recognition model.

[0123] In one embodiment, Figure 2 As shown, a behavior recognition method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0124] Step 202: Obtain a video frame sequence to be identified.

[0125] Step 204, input the video frame sequence to be identified into the trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be identified, and the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method.

[0126] Step 206, comparing the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0127] Specifically, when behavior recognition is required, the server obtains a video frame sequence to be recognized, inputs the video frame sequence to be recognized into a trained behavior recognition model, and obtains a predicted behavior recognition result corresponding to the video frame sequence to be recognized. The predicted behavior recognition result includes a detection box, a confidence level of the detection box, and a behavior category. The server can screen out detection boxes that meet the requirements by comparing the confidence level of the detection box in the predicted behavior recognition result with a preset detection box confidence threshold, and obtains a behavior recognition result based on the detection boxes that meet the requirements and the corresponding behavior categories.

[0128] The above-mentioned behavior recognition method can obtain accurate predicted behavior recognition results by inputting the video frame sequence to be recognized into the trained behavior recognition model after obtaining the video frame sequence to be recognized, so as to obtain the behavior recognition result by comparing the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold, thereby achieving accurate behavior recognition.

[0129] In one embodiment, if Figure 3 As shown, the behavior recognition model construction method of the present application is described through an embodiment, and the method specifically includes the following steps:

[0130] 1. Data processing stage: The server obtains a set of video frame sequences carrying behavior annotations, performs data enhancement processing on each video frame sequence in the set of video frame sequences, and performs image fusion on the video frame sequences after data enhancement processing to obtain a fused image (i.e., trajectory image);

[0131] 2. Feature extraction stage: The server inputs the video frame sequence after data enhancement processing into the 3D-CNN network (i.e., spatiotemporal feature extraction network) in the initial behavior recognition model to obtain the spatiotemporal features corresponding to the video frame sequence, and inputs the fused image into the 2D-CNN network (i.e., spatial feature extraction network) in the initial behavior recognition model to obtain the spatial features corresponding to the video frame sequence;

[0132] 3. Feature fusion stage: Input the spatiotemporal features and spatial features into the feature fusion network to obtain the fusion features corresponding to the video frame sequence;

[0133] 4. Processing output stage: Input the fused features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence. According to the behavior prediction results and behavior annotations, adjust the model parameters of the initial behavior recognition model to obtain the trained behavior recognition model.

[0134] In one embodiment, if Figure 4 As shown, the behavior recognition model construction method and the behavior recognition method of the present application are described by an embodiment, and the method specifically includes the following steps:

[0135] Step 402, obtaining a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, wherein the initial behavior recognition model includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network;

[0136] Step 404, input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0137] Step 406, input the spatiotemporal features and the spatial features into a feature fusion network to obtain fusion features corresponding to the video frame sequence;

[0138] Step 408, inputting the fused features into a convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence;

[0139] Step 410, adjusting the model parameters of the initial behavior recognition model according to the behavior prediction results and the behavior annotations to obtain a trained behavior recognition model.

[0140] Step 412, obtaining a sequence of video frames to be identified;

[0141] Step 414, inputting the to-be-recognized video frame sequence into the trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the to-be-recognized video frame sequence;

[0142] Step 416, compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0143] It should be understood that, although the steps in each flow chart involved in the above-described embodiment are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a portion of the steps in each flow chart involved in the above-described embodiment may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0144] In one embodiment, Figure 5 As shown, a behavior recognition model construction device is provided, including: a sample acquisition module 502, a feature extraction module 504, a feature fusion module 506, a prediction module 508 and an adjustment module 510, wherein:

[0145] A sample acquisition module 502 is used to acquire a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, wherein the initial behavior recognition model includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network;

[0146] A feature extraction module 504 is used to input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0147] A feature fusion module 506 is used to input the spatiotemporal features and the spatial features into a feature fusion network to obtain fusion features corresponding to the video frame sequence;

[0148] A prediction module 508, used to input the fused features into a convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence;

[0149] The adjustment module 510 is used to adjust the model parameters of the initial behavior recognition model according to the behavior prediction results and the behavior annotations to obtain a trained behavior recognition model.

[0150] The above-mentioned behavior recognition model construction device, after obtaining a set of video frame sequences carrying behavior annotations, performs spatiotemporal feature extraction and spatial feature extraction on each video frame sequence to obtain spatiotemporal features and spatial features corresponding to the video frame sequence, and uses a feature fusion network to fuse the spatiotemporal features and spatial features to obtain fused features with richer feature information. By inputting the fused features into a convolution operation network, a behavior prediction result corresponding to the video frame sequence is obtained. According to the behavior prediction results and behavior annotations, the model parameters of the initial behavior recognition model are adjusted, and the initial behavior recognition model can be trained to obtain a behavior recognition model that can realize accurate behavior recognition, so that the behavior recognition model can be used to improve the accuracy of behavior recognition.

[0151] In one of the embodiments, the feature extraction module is also used to perform image fusion on the video frame sequence to obtain a trajectory image corresponding to the video frame sequence, and input the trajectory image into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence.

[0152] In one of the embodiments, the feature extraction module is also used to convert each video frame in the video frame sequence into a grayscale image, and divide the converted video frame sequence according to a preset number of image channels to obtain multiple groups of grayscale images to be fused, calculate the pixel average value of each group of grayscale images to be fused, and obtain a trajectory image corresponding to the video frame sequence based on the pixel average value.

[0153] In one of the embodiments, the feature fusion module is also used to input the spatiotemporal features and spatial features into the feature fusion network, determine the feature weight distribution through the feature fusion network, and obtain the fusion features corresponding to the video frame sequence based on the feature weight distribution, spatiotemporal features and spatial features.

[0154] In one of the embodiments, the prediction module is further used to input the fused features into a convolution operation network to obtain candidate detection boxes, perform threshold selection on the candidate detection boxes according to the behavior annotations, and obtain behavior prediction results corresponding to the video frame sequence.

[0155] In one of the embodiments, the adjustment module is also used to calculate a border loss function based on the predicted detection box and behavior annotation in the behavior prediction results, and calculate a category loss function based on the predicted category and behavior annotation in the behavior prediction results, and obtain a model loss function based on the border loss function and the category loss function, and adjust the model parameters of the initial behavior recognition model based on the model loss function to obtain a trained behavior recognition model.

[0156] In one embodiment, Figure 6 As shown, a behavior recognition device is provided, including: a data acquisition module 602, a recognition module 604 and a comparison module 606, wherein:

[0157] The data acquisition module 602 is used to acquire a video frame sequence to be identified;

[0158] The recognition module 604 is used to input the video frame sequence to be recognized into the trained behavior recognition model to obtain the predicted behavior recognition result corresponding to the video frame sequence to be recognized, and the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method;

[0159] The comparison module 606 is used to compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0160] The above-mentioned behavior recognition device can obtain accurate predicted behavior recognition results by inputting the video frame sequence to be recognized into the trained behavior recognition model after obtaining the video frame sequence to be recognized, so as to obtain the behavior recognition result by comparing the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold, thereby realizing accurate behavior recognition.

[0161] For the specific limitations of the behavior recognition model building device and the behavior recognition device, please refer to the limitations of the behavior recognition model building method and the behavior recognition method above, which will not be repeated here. The various modules in the above-mentioned behavior recognition model building device and the behavior recognition device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0162] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video frame sequences carrying behavior annotations, etc. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a behavior recognition model construction method and a behavior recognition method are implemented.

[0163] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0165] Obtain a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, which includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network;

[0166] Input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0167] Input the spatiotemporal features and spatial features into the feature fusion network to obtain the fusion features corresponding to the video frame sequence;

[0168] Input the fused features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence;

[0169] According to the behavior prediction results and behavior annotations, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

[0170] In one embodiment, when the processor executes the computer program, the following steps are also implemented: performing image fusion on the video frame sequence to obtain a trajectory image corresponding to the video frame sequence, and inputting the trajectory image into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence.

[0171] In one embodiment, when the processor executes the computer program, the following steps are also implemented: converting each video frame in the video frame sequence into a grayscale image, and dividing the converted video frame sequence according to a preset number of image channels to obtain multiple groups of grayscale images to be fused, calculating the pixel average of each group of grayscale images to be fused, and obtaining a trajectory image corresponding to the video frame sequence based on the pixel average.

[0172] In one embodiment, when the processor executes the computer program, the following steps are also implemented: the spatiotemporal features and the spatial features are input into the feature fusion network, the feature weight distribution is determined through the feature fusion network, and the fusion features corresponding to the video frame sequence are obtained based on the feature weight distribution, the spatiotemporal features and the spatial features.

[0173] In one embodiment, when the processor executes the computer program, the following steps are also implemented: inputting the fused features into the convolution operation network to obtain candidate detection boxes, performing threshold selection on the candidate detection boxes according to the behavior annotations, and obtaining behavior prediction results corresponding to the video frame sequence.

[0174] In one embodiment, when the processor executes the computer program, the following steps are also implemented: a border loss function is calculated based on the predicted detection box and behavior annotation in the behavior prediction result, and a category loss function is calculated based on the predicted category and behavior annotation in the behavior prediction result, a model loss function is obtained based on the border loss function and the category loss function, and model parameters of the initial behavior recognition model are adjusted based on the model loss function to obtain a trained behavior recognition model.

[0175] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0176] Obtain a video frame sequence to be identified;

[0177] Inputting the video frame sequence to be identified into the trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be identified, wherein the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method;

[0178] Compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0179] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0180] Obtain a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, which includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network;

[0181] Input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and input the video frame sequence into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence;

[0182] Input the spatiotemporal features and spatial features into the feature fusion network to obtain the fusion features corresponding to the video frame sequence;

[0183] Input the fused features into the convolution operation network to obtain the behavior prediction results corresponding to the video frame sequence;

[0184] According to the behavior prediction results and behavior annotations, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

[0185] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: performing image fusion on the video frame sequence to obtain a trajectory image corresponding to the video frame sequence, and inputting the trajectory image into a spatial feature extraction network to obtain spatial features corresponding to the video frame sequence.

[0186] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: converting each video frame in the video frame sequence into a grayscale image, and dividing the converted video frame sequence according to a preset number of image channels to obtain multiple groups of grayscale images to be fused, calculating the pixel average value of each group of grayscale images to be fused, and obtaining a trajectory image corresponding to the video frame sequence based on the pixel average value.

[0187] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: the spatiotemporal features and spatial features are input into a feature fusion network, feature weight distribution is determined through the feature fusion network, and fusion features corresponding to the video frame sequence are obtained based on the feature weight distribution, spatiotemporal features and spatial features.

[0188] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: inputting the fused features into the convolution operation network to obtain candidate detection boxes, performing threshold selection on the candidate detection boxes according to the behavior annotations, and obtaining behavior prediction results corresponding to the video frame sequence.

[0189] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: a border loss function is calculated based on the predicted detection box and behavior annotation in the behavior prediction results, and a category loss function is calculated based on the predicted category and behavior annotation in the behavior prediction results, a model loss function is obtained based on the border loss function and the category loss function, and model parameters of the initial behavior recognition model are adjusted based on the model loss function to obtain a trained behavior recognition model.

[0190] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0191] Obtain a video frame sequence to be identified;

[0192] Inputting the video frame sequence to be identified into the trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be identified, wherein the trained behavior recognition model is constructed according to the above-mentioned behavior recognition model construction method;

[0193] Compare the confidence of the detection box in the predicted behavior recognition result with the preset detection box confidence threshold to obtain the behavior recognition result.

[0194] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0195] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0196] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a behavior recognition model, characterized in that: The method comprises: Acquire a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, wherein the initial behavior recognition model includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network; Inputting each video frame sequence in the video frame sequence set into the spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, converting each video frame in the video frame sequence into a grayscale image, and dividing the converted video frame sequence according to a preset number of image channels to obtain multiple groups of grayscale images to be fused; Calculating the pixel average of each group of grayscale images to be fused, and obtaining a trajectory image corresponding to the video frame sequence according to the pixel average; Inputting the trajectory image into the spatial feature extraction network to obtain spatial features corresponding to the video frame sequence; Inputting the spatiotemporal features and the spatial features into the feature fusion network to obtain fusion features corresponding to the video frame sequence; Inputting the fused features into the convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence; According to the behavior prediction result and the behavior annotation, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

2. The method according to claim 1, characterized in that: The step of inputting the spatiotemporal features and the spatial features into the feature fusion network to obtain fusion features corresponding to the video frame sequence comprises: Inputting the spatiotemporal features and the spatial features into the feature fusion network, and determining feature weight distribution through the feature fusion network; A fusion feature corresponding to the video frame sequence is obtained according to the feature weight distribution, the spatiotemporal feature and the spatial feature.

3. The method according to claim 1, characterized in that The step of inputting the fusion features into the convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence includes: Inputting the fused features into the convolution operation network to obtain an alternative detection box; The candidate detection boxes are threshold-selected according to the behavior annotations to obtain behavior prediction results corresponding to the video frame sequence.

4. The method according to claim 1, characterized in that The step of adjusting the model parameters of the initial behavior recognition model according to the behavior prediction result and the behavior annotation to obtain a trained behavior recognition model includes: Calculating a border loss function based on the predicted detection box in the behavior prediction result and the behavior annotation, and calculating a category loss function based on the predicted category in the behavior prediction result and the behavior annotation; Obtaining a model loss function according to the border loss function and the category loss function; According to the model loss function, the model parameters of the initial behavior recognition model are adjusted to obtain a trained behavior recognition model.

5. A behavior recognition method, characterized in that: The method comprises: Obtain a video frame sequence to be identified; Inputting the to-be-recognized video frame sequence into a trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the to-be-recognized video frame sequence, wherein the trained behavior recognition model is constructed according to the method described in any one of claims 1 to 4 above; The confidence of the detection box in the predicted behavior recognition result is compared with a preset detection box confidence threshold to obtain the behavior recognition result.

6. A behavior recognition model construction device, characterized in that: The device comprises: A sample acquisition module, used to acquire a set of video frame sequences carrying behavior annotations and an initial behavior recognition model, wherein the initial behavior recognition model includes a spatiotemporal feature extraction network, a spatial feature extraction network, a feature fusion network, and a convolution operation network; A feature extraction module, used to input each video frame sequence in the video frame sequence set into a spatiotemporal feature extraction network to obtain spatiotemporal features corresponding to the video frame sequence, and convert each video frame in the video frame sequence into a grayscale image, and divide the converted video frame sequence according to a preset number of image channels to obtain multiple groups of grayscale images to be fused, calculate the pixel average of each group of grayscale images to be fused, obtain a trajectory image corresponding to the video frame sequence according to the pixel average, and input the trajectory image into the spatial feature extraction network to obtain spatial features corresponding to the video frame sequence; A feature fusion module, used for inputting the spatiotemporal features and the spatial features into the feature fusion network to obtain fusion features corresponding to the video frame sequence; A prediction module, used for inputting the fusion features into the convolution operation network to obtain a behavior prediction result corresponding to the video frame sequence; The adjustment module is used to adjust the model parameters of the initial behavior recognition model according to the behavior prediction result and the behavior annotation to obtain a trained behavior recognition model.

7. The device according to claim 6, characterized in that The feature fusion module is also used to input the spatiotemporal features and the spatial features into the feature fusion network, determine the feature weight distribution through the feature fusion network, and obtain the fusion features corresponding to the video frame sequence according to the feature weight distribution, the spatiotemporal features and the spatial features.

8. A behavior recognition device, characterized in that: The device comprises: A data acquisition module, used to acquire a video frame sequence to be identified; A recognition module, used for inputting the video frame sequence to be recognized into a trained behavior recognition model to obtain a predicted behavior recognition result corresponding to the video frame sequence to be recognized, wherein the trained behavior recognition model is constructed according to the method described in any one of claims 1 to 4 above; The comparison module is used to compare the confidence of the detection box in the predicted behavior recognition result with a preset detection box confidence threshold to obtain the behavior recognition result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images