Human behavior recognition method and device, computer device, and storage medium

Abstract

Embodiments of the present application disclose a human behavior recognition method and device, computer equipment and a storage medium, the method comprising: acquiring image data in a detection area; performing frame cutting processing on the image data to obtain multiple frames of static pictures; inputting the multiple frames of static pictures in the form of a picture sequence into a human behavior prediction model for processing to obtain a human behavior recognition result. The present application can well recognize multi-person behavior through the semantic segmentation model and the optical flow prediction model, based on the feature information fusion mode of human semantic mask features and human optical flow features. By combining a tracking model, the semantic segmentation features and the optical flow features are fused to perform multi-angle multi-person real-time behavior detection. In addition, the Residual GRU module is used in the optical flow network to update and iterate the optical flow fused with multiple features, so that the loss of key information is avoided and the prediction accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to methods, devices, computer equipment, and storage media for human behavior recognition. Background Technology

[0002] With rapid societal development, human behavior prediction is currently a research hotspot and challenge in both industry and academia, and it has significant practical applications. Several methods are currently available for human behavior prediction:

[0003] The first method involves processing the acquired visible light and infrared images to obtain the target area for tracking. Next, it detects whether the target area includes pedestrians. If pedestrians are included, they are tracked, and during tracking, pedestrian edges are detected to obtain the pedestrian identification area from the target area. This pedestrian identification area is then input into a recognition model to obtain the pedestrian behavior recognition result. This patent uses infrared tracking, which requires additional equipment costs. Furthermore, it relies on traditional histogram-based feature extraction to determine pedestrian targets, which is severely affected by lighting conditions and has poor performance. Additionally, this method only recognizes the outline of the person, ignoring spatial-temporal fusion information, resulting in poor recognition performance.

[0004] The second approach involves training a 3D convolutional neural network using multi-task deep learning. This method uses a fixed number of consecutive frames from a background video as input to various human behavioral attributes, and the network is trained to perform the recognition task. However, this method only recognizes human behavior at fixed locations and is limited to a single individual, which has significant limitations. Furthermore, behavior recognition based on 3D convolution is inefficient, and the trained model is heavily influenced by background factors, resulting in poor recognition performance.

[0005] The third method involves acquiring video clips, extracting features and reducing dimensionality from the image frame sequences within those clips, encoding the reduced feature vectors, and then expanding these expanded feature vectors to obtain a predetermined number of expanded feature vectors. These expanded feature vectors and the encoded feature vectors are then input into a three-layer single-layer decoder for decoding. The final decoded feature vector is then input into a single-layer fully connected feedforward network for calculation, yielding multiple predicted values. These predicted values ​​are then input into a logistic regression network to obtain the corresponding predicted probabilities. The category corresponding to the highest probability value is selected as the human action category of the bounding box corresponding to the final decoded feature vector. This method first extracts spatial information and then extracts temporal features using spatial features from different time periods. However, this process easily loses spatial information during temporal feature extraction, making it difficult to achieve a fusion of spatial and temporal features, thus reducing the accuracy of action recognition. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, device, computer equipment and storage medium for human behavior recognition.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect is human behavior recognition methods, including:

[0009] Acquire image data within the detection area;

[0010] The image data is sliced ​​into frames to obtain multiple static images;

[0011] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0012] The further technical solution is as follows: The step of inputting multiple static images as an image sequence into a human behavior prediction model for processing to obtain human behavior recognition results includes:

[0013] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0014] Different human images with ID indices are input into a semantic segmentation model for processing to obtain human semantic mask features;

[0015] Human images with ID indexes are input into an optical flow prediction model for processing to obtain human optical flow features;

[0016] Human semantic mask features and human optical flow features are input into the behavior recognition classification model for processing to obtain human behavior recognition results.

[0017] The further technical solution is as follows: The step of inputting different human images with ID indices into a semantic segmentation model for processing to obtain human semantic mask features includes:

[0018] The first, second, and third convolutions are performed on the person images with id indexes to obtain the first, second, and third convolution features.

[0019] The first convolutional feature is input into the Switch conv for convolution operation to obtain the first processed convolutional feature;

[0020] The second convolutional feature is input into the Dilated conv for processing to obtain the second processed convolutional feature;

[0021] The third convolutional feature is input into the Coord conv for processing to obtain the third processed convolutional feature;

[0022] The first-processed convolutional features are input into a dimension-wise convolutional function and processed by the ReLU activation function to obtain the first-processed features.

[0023] The convolutional features after the second processing are input into Dynamic conv for convolution processing to obtain intermediate processing features;

[0024] The intermediate processing features are input into a 3*3 convolution and a BatchNormalization operation to obtain the second processing features;

[0025] The convolutional features after the third processing are input into Mix conv for convolution operation to obtain the third processed features;

[0026] The result of the first processed feature is obtained by convolution, followed by deconvolution upsampling and convolution again to obtain the first processed feature.

[0027] The first processed features are subjected to convolution and BatchNormalization operations to obtain the second processed feature;

[0028] The second processing result features are subjected to unpooling upsampling and convolutional block operations to obtain unpooling upsampling features and convolutional block features;

[0029] The second processed feature is added to the convolutional block feature to obtain the added feature;

[0030] Perform a sigmoid operation on the additive features to obtain sigmoid features;

[0031] The sigmoid features are multiplied with the unpooled upsampled features to obtain the multiplied features;

[0032] The multiplied features are processed by the ReLU function to obtain the first result feature;

[0033] The result of subtracting the sigmoid feature from 1 is multiplied with the third processing feature to obtain the second result feature;

[0034] The result obtained by adding the second result feature to the first result feature is then upsampled and convolved to obtain the third result feature.

[0035] The third result feature is input into the residual network after three processing steps to obtain the fourth result feature;

[0036] The fourth result feature is concatenated with the third result feature to obtain the fused feature;

[0037] The fused features are processed through 3*3 convolution, upsampling, and 1*1 convolution to obtain human semantic mask features.

[0038] The further technical solution is as follows: The step of inputting a person image with an ID index into an optical flow prediction model for processing to obtain human optical flow features includes:

[0039] The first network processes two consecutive frames of character images with ID indices to obtain the first optical flow features.

[0040] The first optical flow feature is used to perform bilinear interpolation on the previous frame of a person image with an id index between two consecutive frames to obtain bilinear interpolation features.

[0041] The results of processing the two consecutive frames of character images with ID indices using AsymOFMM are concatenated with the bilinear interpolation features to obtain the concatenated features.

[0042] The concatenated feature is stacked with the next frame of the two frames of character images with id indexes to obtain a stacked image.

[0043] The stacked image is input into the second network for processing to obtain the second optical flow feature;

[0044] The second optical flow feature is input into the Residual GRU module for iterative updating.

[0045] When the iteration is complete, the human body optical flow feature results are output.

[0046] The further technical solution is as follows: The step of inputting two consecutive frames of character images with ID indices into a first network for processing to obtain the first optical flow feature includes:

[0047] The two frames of character images with id indices are processed by 4 layers of Convolution+ReLU to obtain the first feature map and the second feature map.

[0048] After downsampling the first and second feature maps respectively, a Max pooling layer is used to process them to obtain the third and fourth feature maps;

[0049] The third and fourth feature maps are processed through three residual layers to obtain the fifth and sixth feature maps;

[0050] The fifth feature map is used as the kernel to perform convolution with the sixth feature map to obtain the matching feature map;

[0051] The matching feature map is fed into three convolutional layers and one average pooling layer for advanced feature extraction to obtain advanced features;

[0052] Four layers of deconvolution are used on high-level features to obtain high-level semantic information;

[0053] The high-level semantic information is fused with the fifth and sixth feature maps, and then upsampled to obtain the first optical flow feature.

[0054] A further technical solution is as follows: the step of inputting the stacked image into the second network for processing to obtain the second optical flow feature includes:

[0055] The stacked image is fed into a three-layer Convolution+BatchNorm layer for processing to obtain the seventh feature map;

[0056] After performing average pooling on the seventh feature map, the feature map is then multiplied by three 1x1 convolutions to obtain the extracted feature map.

[0057] The extracted feature maps are processed using deep learning interpolation methods to obtain the second optical flow features.

[0058] The further technical solution is as follows: the human semantic mask features and human optical flow features are input into the behavior recognition classification model for processing to obtain human behavior recognition results, including:

[0059] The semantic mask features and optical flow features of the human body are concatenated and merged to obtain the merged features;

[0060] The merged features are input into a lightweight video classification network for processing to obtain the probabilities corresponding to different human behavior categories;

[0061] The human behavior category with the highest probability is selected as the predicted human behavior recognition result.

[0062] Secondly, human behavior recognition devices include:

[0063] The acquisition unit is used to acquire image data within the detection area;

[0064] The first processing unit is used to perform frame-slicing processing on the image data to obtain multiple frames of static images;

[0065] The second processing unit is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain human behavior recognition results.

[0066] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the human behavior recognition method steps as described above.

[0067] Fourthly, a computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the steps of the human behavior recognition method described above.

[0068] The advantages of this invention compared to existing technologies are as follows: This invention uses a semantic segmentation model and an optical flow prediction model, based on a feature fusion method of human semantic mask features and human optical flow features, to perform human behavior recognition, which can effectively identify multiple people's behavior. By combining a tracking model and fusing semantic segmentation features and optical flow features, multi-angle, real-time multi-person behavior detection can be performed. Furthermore, a Residual GRU module is used in the optical flow network to update and iterate the optical flow that has fused multiple features, ensuring that key information is not lost and improving prediction accuracy.

[0069] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are described in detail below. Attached Figure Description

[0070] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a schematic diagram illustrating an application scenario of the human behavior recognition method provided in a specific embodiment of the present invention.

[0072] Figure 2 A flowchart of a human behavior recognition method provided in a specific embodiment of the present invention;

[0073] Figure 3 A schematic block diagram of a human behavior recognition device provided in a specific embodiment of the present invention;

[0074] Figure 4 This is a schematic block diagram of a computer device provided for a specific embodiment of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0076] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0077] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0078] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0079] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the human behavior recognition method provided in an embodiment of the present invention. Figure 2 The flowchart illustrates a human behavior recognition method provided in a specific embodiment of the present invention. This human behavior recognition method is applied in a server and is executed by application software installed on the server.

[0080] like Figure 2 As shown, the human behavior recognition method includes the following steps: S10-S30.

[0081] S10. Acquire image data within the detection area.

[0082] Use surveillance equipment, such as surveillance cameras, to monitor and capture video data (image data) of the area to be detected. Any commercially available surveillance equipment is acceptable; this application does not impose any restrictions on its use.

[0083] S20. Perform frame segmentation on the image data to obtain multiple static images.

[0084] The detected image data is sliced ​​into individual RGB static images, which are then used as input data for the human behavior prediction model.

[0085] S30. Input multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0086] The human behavior prediction model includes a tracking model, a semantic segmentation model, and an optical flow prediction model. The sample data input into the human behavior prediction model is processed by the tracking model, the semantic segmentation model, and the optical flow prediction model to obtain the human behavior recognition result.

[0087] In one embodiment, step S30 specifically includes the following steps: S301-S304.

[0088] S301. Input multiple static images as an image sequence into the tracking model for processing to obtain a person image with an ID index.

[0089] In this embodiment, the tracking model used is ByteTrack, which uses the YOLOv5 model for human target detection. The coordinates of the human target bounding box and the confidence information of the human category can be obtained. In addition, the corresponding person ID index information can also be obtained.

[0090] In one embodiment, step S301 specifically includes the following steps: S3011-S3012.

[0091] S3011. Input multiple static images as an image sequence into the tracking model to detect different human target boxes.

[0092] In this embodiment, the obtained multiple frames of static images (frame video stream) are input into the tracking model in the form of an image sequence, which can detect different human target boxes. The target box is represented by the image position coordinates, that is, the coordinates of the target box relative to the upper left corner of the image and the coordinates of the target box relative to the lower right corner of the image.

[0093] S3012. Assign an index id to the different human target boxes detected to obtain human images with id indexes.

[0094] S302. Input different human images with id indexes into the semantic segmentation model for processing to obtain human semantic mask features.

[0095] Semantic segmentation models primarily perform semantic segmentation on human images output by tracking models to obtain segmented human mask features.

[0096] In one embodiment, step S302 specifically includes the following steps: S3021-S3040.

[0097] S3021. Perform first convolution, second convolution and third convolution on the person image with id index to obtain first convolution features, second convolution features and third convolution features.

[0098] The images of people with ID indices are processed by 3*3 convolution, 5*5 convolution and 7*7 convolution respectively to obtain features and characteristics.

[0099] S3022. Input the first convolutional feature into Switch conv for convolution operation to obtain the first processed convolutional feature.

[0100] In this embodiment, the features are input into a Switch conv to perform a convolution operation to obtain the features.

[0101] S3023. Input the second convolutional feature into Dilated conv for processing to obtain the second processed convolutional feature.

[0102] In this embodiment, the features are input into the Dilated conv to obtain the features.

[0103] S3024. Input the third convolutional feature into Coord conv for processing to obtain the third processed convolutional feature.

[0104] In this embodiment, the features are input into the Coord conv and the convolution operation is performed to obtain the features.

[0105] S3025. Input the first processed convolutional features into Dimension-wise conv for convolution and process them through the ReLU activation function to obtain the first processed features.

[0106] In this embodiment, the first processed convolutional features are input into a dimension-wise convolution function for convolution and the features are obtained through the ReLU activation function.

[0107] S3026. Input the second-processed convolutional features into Dynamic conv for convolution processing to obtain intermediate-processed features.

[0108] In this embodiment, the second-processed convolutional features are input into Dynamic conv for convolution to obtain intermediate-processed features.

[0109] S3027. Input the intermediate processing features into a 3*3 convolution and BatchNormalization operation to obtain the second processing features.

[0110] In this embodiment, the intermediate processing features are input into a 3*3 convolution and a BatchNormalization operation to obtain the second processing features.

[0111] S3028. Input the convolutional features after the third processing into Mix conv for convolution operation to obtain the third processing features.

[0112] In this embodiment, the third-processed convolutional features are input into Mix conv for convolution to obtain the third-processed features.

[0113] S3029. The result obtained by convolution processing of the first processed features is then subjected to deconvolution upsampling and convolution processing again to obtain the first processed result features.

[0114] In this embodiment, the first processed feature is obtained by performing a 5*5 convolution, and then the feature is obtained by performing deconvolution upsampling and a 1*1 convolution to obtain the first processed result feature.

[0115] S3030. The first processed features are subjected to convolution and BatchNormalization operations to obtain the second processed feature.

[0116] In this embodiment, the first processed feature is subjected to 3*3 convolution and BatchNormalization to obtain the second processed feature.

[0117] S3031. Perform unpooling upsampling and convolutional block operations on the second processing result features to obtain unpooling upsampling features and convolutional block features.

[0118] In this embodiment, unpooling upsampling and convolutional block operations are performed on the features resulting from the second processing to obtain unpooling upsampling features and convolutional block features. It should be noted that the convolutional block operation here is convolution plus BatchNormalization and upsampling.

[0119] S3032. Add the second processed feature to the convolutional block feature to obtain the added feature.

[0120] In this embodiment, the second processed feature is added to the convolutional block feature to obtain the summed feature.

[0121] S3033. Perform a sigmoid operation on the additive features to obtain sigmoid features.

[0122] In this embodiment, the sigmoid feature is obtained by performing a sigmoid operation on the additive features.

[0123] S3034. Multiply the sigmoid features with the unpooling upsampling features to obtain the multiplied features.

[0124] In this embodiment, the sigmoid feature is multiplied by the unpooling upsampling feature to obtain the multiplied feature.

[0125] S3035. Process the multiplied features using the ReLU function to obtain the first result feature.

[0126] In this embodiment, the multiplied features are processed by the ReLU function to obtain the first result feature.

[0127] S3036. Multiply the result obtained by subtracting the sigmoid feature from 1 with the third processing feature to obtain the second result feature.

[0128] In this embodiment, the result obtained by subtracting the sigmoid feature from 1 is multiplied by the third processing feature to obtain the second result feature.

[0129] S3037. The result obtained by adding the second result feature to the first result feature is then subjected to upsampling and convolution to obtain the third result feature.

[0130] In this embodiment, the second result feature is added to the first result feature and then upsampled and convolved with 1*1 to obtain the third result feature.

[0131] S3038. Input the third result feature into the residual network after three processing steps to obtain the fourth result feature.

[0132] In this embodiment, the third result feature is passed through three residual network blocks to obtain the fourth result feature.

[0133] S3039. Concatenate the fourth result feature with the third result feature to obtain the fused feature.

[0134] S3040. The fused features are processed by 3*3 convolution, upsampling and 1*1 convolution to obtain human semantic mask features.

[0135] For S3039-S3040, in this embodiment, the fourth result feature and the third result feature are concatenated and then subjected to 3*3 convolution, upsampling and 1*1 convolution to obtain the human semantic mask feature.

[0136] S303. Input the human image with id index into the optical flow prediction model for processing to obtain human optical flow features.

[0137] The optical flow prediction model consists of three modules. The first two modules are composed of two stacked CNN-based optical flow estimation networks, named CFNet1 and CFNet2. The optical flow obtained from the first two networks is then fed into the third part of the optical flow prediction model for optimization. The third part uses a Residual GRU module to iteratively update the optical flow. Both the first and second networks include Encoder and Decoder components. The Encoder extracts features, and the Decoder upsamples and amplifies the extracted features to obtain the final optical flow prediction.

[0138] In one embodiment, step S303 specifically includes the following steps: S3031-S3037.

[0139] S3031. Input the two consecutive frames of character images with ID indexes into the first network for processing to obtain the first optical flow features.

[0140] In one embodiment, step S3031 specifically includes the following steps: S30311-S30317.

[0141] S30311. Process the two frames of character images with id indices through 4 layers of Convolution+ReLU to obtain the first feature map and the second feature map.

[0142] In this embodiment, the Encoder part of CFNet1 first obtains feature maps by passing the two consecutive frames of character images with id indices output from the tracking model through 4 layers of Convolution+ReLU.

[0143] S30312. After performing downsampling operations on the first feature map and the second feature map respectively, Max pooling is applied to obtain the third feature map and the fourth feature map.

[0144] In this embodiment, a downsampling operation is performed on each feature map, which is then passed through a Max pooling layer to obtain the feature map.

[0145] S30313. The third and fourth feature maps are processed through three residual layers to obtain the fifth and sixth feature maps.

[0146] In this embodiment, the image resolution decreases as the number of channels increases through three residual layers. The output consists of two feature maps, one from the previous frame and the other from the next two frames of a person image with an ID index.

[0147] S30314. The fifth feature map is used as a convolution kernel to perform convolution with the sixth feature map to obtain a matching feature map.

[0148] In this embodiment, local correlation calculation is performed on the two feature maps, which are then used as convolution kernels to perform convolution (multiplying and summing the corresponding pixel values) to generate a matching feature map.

[0149] S30315. Input the matching feature map into three convolutional layers and one average pooling layer for advanced feature extraction to obtain advanced features.

[0150] In this embodiment, the input is fed into three convolutional layers and one average pooling layer for advanced feature extraction, at which point the Encoder operation is complete.

[0151] S30316. Use four layers of deconvolution operation on high-level features to obtain high-level semantic information.

[0152] In this embodiment, the Decoder part of CFNet1 uses four layers of deconvolution operation (deconv) to output high-level semantic information.

[0153] S30317. The high-level semantic information is fused with the fifth and sixth feature maps, and then upsampled to obtain the first optical flow feature.

[0154] In this embodiment, two Bilinear interpolations and one Nearest interpolation are used to upsample the optical flow of the output coarse from the previous convolutional layer. Simultaneously, CFNet1 fuses the obtained high-level semantic information with the feature map (low-level local information) obtained from the encoder, introducing a Coarse-to-Fine (CF) mechanism. The optical flow is then output via the decoder.

[0155] Optical flow consists of two channels: the first channel represents the displacement of each pixel on the x-axis, and the second channel represents the displacement of each pixel on the y-axis.

[0156] S3032. The first optical flow feature is used to perform bilinear interpolation on the previous frame of the character image with id index in the two frames to obtain the bilinear interpolation feature.

[0157] S3033. The results of processing the two consecutive frames of character images with id indexes using AsymOFMM are concatenated with the bilinear interpolation features to obtain the concatenated features.

[0158] For S3032-S3033, in this embodiment, the optical flow output by CFNet1 is used to perform bilinear interpolation (warp operation) on the previous frame of the two frames of character images with id indexes to obtain a warped image. Then, an occlusion mask is generated using an AsymOFMM module to concatenate it with the warped image to obtain a masked image.

[0159] S3034. Stack the concatenated feature with the next frame of the two frames of character images with id indexes to obtain a stacked image.

[0160] In this embodiment, the masked image is stacked to generate a new image, which is then input into CFNet2 for a new round of learning.

[0161] S3035. Input the stacked image into the second network for processing to obtain the second optical flow feature.

[0162] In this embodiment, step S3034 specifically includes the following steps: S30341-S30343.

[0163] S30341. Input the stacked image into a three-layer Convolution+BatchNorm layer for processing to obtain the seventh feature map.

[0164] The input is fed into a 3-layer Convolution+BatchNorm layer to obtain the feature map.

[0165] S30342. After performing average pooling on the seventh feature map, the feature map is then subjected to three 1x1 convolutions to increase its dimensionality, so as to obtain the extracted feature map.

[0166] After performing an average pooling operation, the features are then upscaled using three 1x1 convolutional layers to change the number of channels. At this point, the feature extraction part of CFNet2 is complete, and the feature map is output.

[0167] S30343. The extracted feature map is processed by a deep learning interpolation method to obtain the second optical flow feature.

[0168] In this embodiment, the decoder operation is performed, using the same decoder architecture as CFNet1, but all upsampling methods are replaced with the deep learning-based interpolation method Transpose convolution. CFNet2 will decode the optical flow result Flow2.

[0169] S3036. Input the second optical flow feature into the Residual GRU module for cyclical update and iteration.

[0170] In this embodiment, the optical flow result Flow2 decoded by CFNet2 and the stacked image of the first and second frames as context information will be input into the Residual GRU module for cyclical update iteration.

[0171] S3037. When the iteration is complete, the human body optical flow feature results are output.

[0172] In this embodiment, the concepts of GRU and residuals are combined. Each iteration generates a new optical flow, and the pixel shuffle upsampling method is used in this process. After the iteration is completed, the final optical flow result will be output.

[0173] S304. Input the human semantic mask features and human optical flow features into the behavior recognition classification model for processing to obtain the human behavior recognition results.

[0174] Human semantic mask features and human optical flow features are concatenated and input into the behavior classification network. In this embodiment, the behavior recognition classification model uses the lightweight video classification network Moviet.

[0175] In one embodiment, step S304 specifically includes the following steps: S3041-S3043.

[0176] S3041. Concatenate and merge the human semantic mask features and the human optical flow features to obtain the merged features.

[0177] S3042. Input the merged features into a lightweight video classification network for processing to obtain the probabilities corresponding to different human behavior categories.

[0178] S3043. Select the human behavior category with the highest probability as the predicted human behavior recognition result.

[0179] For S3041-S3043, in this embodiment, video frames from different indexers are first merged, with the number of merged frames controlled to 60. Before being input into the network, each frame's input is convolved with a 1*1 convolution to make its channels 3. After being input into the moviet video classification network, the probabilities corresponding to different categories are obtained. The category with the highest probability is selected as the predicted category for the behavior. There are 5 predicted categories, corresponding to jumping, running, sitting, standing, and walking.

[0180] In addition, the loss functions used in the human behavior prediction model include the tracking model loss function, the semantic segmentation model loss function, the optical flow loss function, and the video behavior classification loss function, among which:

[0181] The tracking model uses the YOLOv5 loss function, which mainly consists of classification loss, localization loss, and confidence loss function.

[0182] .

[0183] The semantic segmentation model's loss function is based on the Dice coefficient. For each class's mask, a Dice loss is calculated. The Dice losses for each class are summed and averaged to obtain the final Dice soft loss. The loss function in this patented model mainly consists of , , , that is, .

[0184] here , =0.3, .

[0185] The optical flow loss function consists of the loss of , the loss of , and the loss of . The endpoint error (EPE) is used uniformly as the training loss function, which represents the average Euclidean distance per pixel between the predicted flow vector and the ground truth.

[0186] The video behavior classification loss function uses the cross-entropy loss of classification for model training.

[0187] This invention utilizes a semantic segmentation model and an optical flow prediction model, fusing feature information from human semantic mask features and human optical flow features for human behavior recognition, enabling effective multi-person behavior identification. By combining a tracking model with semantic segmentation and optical flow features, real-time multi-person behavior detection from multiple angles is possible. Furthermore, a Residual GRU module is used in the optical flow network to update and iterate the optical flow data fused with multiple features, ensuring no loss of key information and improving prediction accuracy.

[0188] Figure 3 This is a schematic block diagram of the human behavior recognition device 100 provided in an embodiment of the present invention. Corresponding to the above-described human behavior recognition method, a specific embodiment of the present invention also provides a human behavior recognition device 100. The human behavior recognition device 100 includes units and modules for executing the above-described human behavior recognition method, and the device can be configured in a server.

[0189] like Figure 3As shown, the human behavior recognition device 100 includes:

[0190] The acquisition unit 110 is used to acquire image data within the detection area.

[0191] The first processing unit 120 is used to perform frame-slicing processing on the image data to obtain multiple frames of static images.

[0192] The second processing unit 130 is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain human behavior recognition results.

[0193] In one embodiment, the second processing unit 130 includes:

[0194] The first processing module is used to input multiple frames of static images into the tracking model in the form of an image sequence for processing, so as to obtain a person image with an ID index.

[0195] In one embodiment, the first processing module includes:

[0196] The first processing submodule is used to input multiple frames of static images into the tracking model in the form of an image sequence in order to detect different human target boxes.

[0197] The second processing submodule is used to assign index IDs to the different detected human target boxes in order to obtain human images with ID indexes.

[0198] The second processing module is used to input different human images with ID indices into the semantic segmentation model for processing to obtain human semantic mask features.

[0199] In one embodiment, the second processing module includes:

[0200] The convolution submodule is used to perform first, second, and third convolution processes on the person image with id index to obtain first, second, and third convolution features.

[0201] The first post-processing convolutional submodule is used to input the first convolutional features into the Switch conv for convolution operations to obtain the first post-processing convolutional features.

[0202] The second post-processing convolutional submodule is used to input the second convolutional features into the Dilated conv for processing to obtain the second post-processing convolutional features.

[0203] The third post-processing convolutional submodule is used to input the third convolutional features into the Coord conv for processing to obtain the third post-processing convolutional features.

[0204] The third processing submodule is used to input the first-processed convolutional features into a dimension-wise conv for convolution and to process them through the ReLU activation function to obtain the first-processed features.

[0205] The intermediate processing submodule is used to input the second-processed convolutional features into Dynamic conv for convolution processing to obtain intermediate processed features.

[0206] The fourth processing submodule is used to input the intermediate processing features into a 3*3 convolution and BatchNormalization operation to obtain the second processing features.

[0207] The fifth processing submodule is used to input the convolutional features after the third processing into Mix conv for convolution operation to obtain the third processed features.

[0208] The first processing result submodule is used to perform deconvolution upsampling and reconvolution processing on the result obtained by convolution of the first processing feature to obtain the first processing result feature.

[0209] The second processing result submodule is used to process the first processed features through convolution and BatchNormalization operations to obtain the second processed result features.

[0210] The unpooling upsampling and convolutional block submodule is used to perform unpooling upsampling and convolutional block operations on the second processing result features to obtain unpooling upsampling features and convolutional block features.

[0211] The additive feature submodule is used to add the second processed feature to the convolutional block feature to obtain the additive feature.

[0212] The sigmoid feature submodule is used to perform a sigmoid operation on the additive features to obtain sigmoid features.

[0213] The multiplication feature submodule is used to multiply the sigmoid feature with the unpooling upsampled feature to obtain the multiplied feature.

[0214] The first result submodule is used to process the multiplied features using a ReLU function to obtain the first result feature.

[0215] The second result submodule is used to multiply the result obtained by subtracting the sigmoid feature from 1 with the third processing feature to obtain the second result feature.

[0216] The third result submodule is used to add the second result feature to the first result feature and then perform upsampling and convolution processing to obtain the third result feature.

[0217] The fourth result submodule is used to input the third result features into the residual network processed three times to obtain the fourth result features.

[0218] The fusion submodule is used to concatenate the fourth result feature with the third result feature to obtain the fused feature.

[0219] The human semantic submodule is used to process the fused features through 3*3 convolution, upsampling, and 1*1 convolution to obtain human semantic mask features.

[0220] The third processing module is used to input the human image with ID index into the optical flow prediction model for processing to obtain human optical flow features.

[0221] In one embodiment, the third processing module includes:

[0222] The sixth processing submodule is used to input the two consecutive frames of character images with ID indexes into the first network for processing to obtain the first optical flow features.

[0223] In one embodiment, the sixth processing submodule includes:

[0224] The tenth processing submodule is used to process the two consecutive frames of character images with id indices through 4 layers of Convolution+ReLU to obtain the first feature map and the second feature map.

[0225] The eleventh processing submodule is used to perform downsampling operations on the first feature map and the second feature map respectively, followed by Maxpooling layer processing to obtain the third feature map and the fourth feature map.

[0226] The twelfth processing submodule is used to process the third and fourth feature maps through three residual layers to obtain the fifth and sixth feature maps.

[0227] The thirteenth processing submodule is used to convolve the fifth feature map with the sixth feature map as a convolution kernel to obtain a matching feature map.

[0228] The advanced feature extraction submodule is used to input the matched feature map into three convolutional layers and one average pooling layer for advanced feature extraction to obtain advanced features.

[0229] The convolution operation submodule is used to perform four-layer deconvolution operations on high-level features to obtain high-level semantic information.

[0230] The feature fusion submodule is used to fuse high-level semantic information with the fifth and sixth feature maps, and then perform upsampling to obtain the first optical flow feature.

[0231] The seventh processing submodule is used to perform bilinear interpolation on the previous frame of a person image with an id index in two consecutive frames using the first optical flow feature, so as to obtain bilinear interpolation features.

[0232] The concatenating submodule is used to concatenate the results of AsymOFMM processing of two consecutive frames of character images with ID indices with bilinear interpolation features to obtain concatenated features.

[0233] The eighth processing submodule is used to stack the concatenated feature with the next frame of the two frames of character images with id indexes to obtain a stacked image.

[0234] The ninth processing submodule is used to input the stacked map into the second network for processing to obtain the second optical flow feature.

[0235] In one embodiment, the ninth processing submodule includes:

[0236] The fourteenth processing submodule is used to input the stacked map into a three-layer Convolution+BatchNorm layer for processing to obtain the seventh feature map.

[0237] The dimension-up processing submodule is used to perform average pooling on the seventh feature map and then perform dimension-up processing on the features through three layers of 1x1 convolution to obtain the extracted feature map.

[0238] The fifteenth processing submodule is used to process the deep learning interpolation method of the extracted feature map to obtain the second optical flow feature.

[0239] The update iteration submodule is used to input the second optical flow feature into the Residual GRU module for cyclic update iteration.

[0240] The output submodule is used to output the human optical flow feature results when the iteration is complete.

[0241] The fourth processing module is used to input human semantic mask features and human optical flow features into the behavior recognition classification model for processing, so as to obtain human behavior recognition results.

[0242] In one embodiment, the fourth processing module includes:

[0243] The merging submodule is used to concatenate and merge human semantic mask features and human optical flow features to obtain merged features.

[0244] The classification processing submodule is used to input the merged features into a lightweight video classification network for processing, so as to obtain the probabilities corresponding to different human behavior categories.

[0245] The selection submodule is used to select the human behavior category with the highest probability as the predicted human behavior recognition result.

[0246] The aforementioned human behavior recognition device can be implemented in the form of a computer program, which can, for example... Figure 4 It runs on the computer device shown.

[0247] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0248] like Figure 4 As shown, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the human behavior recognition method steps described above.

[0249] The computer device 700 can be a terminal or a server. The computer device 700 includes a processor 720, a memory, and a network interface 750 connected via a system bus 710, wherein the memory may include a non-volatile storage medium 730 and internal memory 740.

[0250] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. When the computer program 732 is executed, it enables the processor 720 to perform any human behavior recognition method.

[0251] The processor 720 provides computing and control capabilities to support the operation of the entire computer device 700.

[0252] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute any human behavior recognition method.

[0253] This network interface 750 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 700 to which the present application is applied. The specific computer device 700 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. The processor 720 is used to run program code stored in memory to implement the following steps:

[0254] The first aspect is human behavior recognition methods, including:

[0255] Acquire image data within the detection area;

[0256] The image data is sliced ​​into frames to obtain multiple static images;

[0257] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0258] In one embodiment: the step of inputting multiple frames of static images as an image sequence into a human behavior prediction model for processing to obtain human behavior recognition results includes:

[0259] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0260] Different human images with ID indices are input into a semantic segmentation model for processing to obtain human semantic mask features;

[0261] Human images with ID indexes are input into an optical flow prediction model for processing to obtain human optical flow features;

[0262] Human semantic mask features and human optical flow features are input into the behavior recognition classification model for processing to obtain human behavior recognition results.

[0263] In one embodiment: the step of inputting different human images with ID indices into a semantic segmentation model for processing to obtain human semantic mask features includes:

[0264] The first, second, and third convolutions are performed on the person images with id indexes to obtain the first, second, and third convolution features.

[0265] The first convolutional feature is input into Switch conv for convolution operation to obtain the first processed convolutional feature;

[0266] The second convolutional feature is input into the Dilated conv for processing to obtain the second processed convolutional feature;

[0267] The third convolutional feature is input into the Coord conv for processing to obtain the third processed convolutional feature;

[0268] The first-processed convolutional features are input into a dimension-wise convolutional function and processed by the ReLU activation function to obtain the first-processed features.

[0269] The convolutional features after the second processing are input into Dynamic conv for convolution processing to obtain intermediate processing features;

[0270] The intermediate processing features are input into a 3*3 convolution and a BatchNormalization operation to obtain the second processing features;

[0271] The convolutional features after the third processing are input into Mix conv for convolution operation to obtain the third processed features;

[0272] The result of the first processed feature is obtained by convolution, followed by deconvolution upsampling and convolution again to obtain the first processed feature.

[0273] The first processed features are subjected to convolution and BatchNormalization operations to obtain the second processed feature;

[0274] The second processing result features are subjected to unpooling upsampling and convolutional block operations to obtain unpooling upsampling features and convolutional block features;

[0275] The second processed feature is added to the convolutional block feature to obtain the added feature;

[0276] Perform a sigmoid operation on the additive features to obtain sigmoid features;

[0277] The sigmoid features are multiplied with the unpooled upsampled features to obtain the multiplied features;

[0278] The multiplied features are processed by the ReLU function to obtain the first result feature;

[0279] The result of subtracting the sigmoid feature from 1 is multiplied with the third processing feature to obtain the second result feature;

[0280] The result obtained by adding the second result feature to the first result feature is then upsampled and convolved to obtain the third result feature.

[0281] The third result feature is input into the residual network after three processing steps to obtain the fourth result feature;

[0282] The fourth result feature is concatenated with the third result feature to obtain the fused feature;

[0283] The fused features are processed through 3*3 convolution, upsampling, and 1*1 convolution to obtain human semantic mask features.

[0284] In one embodiment: the step of inputting a person image with an ID index into an optical flow prediction model for processing to obtain human optical flow features includes:

[0285] The first network processes two consecutive frames of character images with ID indices to obtain the first optical flow features.

[0286] The first optical flow feature is used to perform bilinear interpolation on the previous frame of a person image with an id index between two consecutive frames to obtain bilinear interpolation features.

[0287] The results of processing the two consecutive frames of character images with ID indices using AsymOFMM are concatenated with the bilinear interpolation features to obtain the concatenated features.

[0288] The concatenated feature is stacked with the next frame of the two frames of character images with id indexes to obtain a stacked image.

[0289] The stacked image is input into the second network for processing to obtain the second optical flow feature;

[0290] The second optical flow feature is input into the Residual GRU module for iterative updating.

[0291] When the iteration is complete, the human body optical flow feature results are output.

[0292] In one embodiment: the step of inputting two consecutive frames of character images with ID indices into a first network for processing to obtain the first optical flow feature includes:

[0293] The two frames of character images with id indices are processed by 4 layers of Convolution+ReLU to obtain the first feature map and the second feature map.

[0294] After downsampling the first and second feature maps respectively, a Max pooling layer is used to process them to obtain the third and fourth feature maps;

[0295] The third and fourth feature maps are processed through three residual layers to obtain the fifth and sixth feature maps;

[0296] The fifth feature map is used as the kernel to perform convolution with the sixth feature map to obtain the matching feature map;

[0297] The matching feature map is fed into three convolutional layers and one average pooling layer for advanced feature extraction to obtain advanced features;

[0298] Four layers of deconvolution are used on high-level features to obtain high-level semantic information;

[0299] The high-level semantic information is fused with the fifth and sixth feature maps, and then upsampled to obtain the first optical flow feature.

[0300] In one embodiment: the step of inputting the stacked map into a second network for processing to obtain a second optical flow feature includes:

[0301] The stacked image is fed into a three-layer Convolution+BatchNorm layer for processing to obtain the seventh feature map;

[0302] After performing average pooling on the seventh feature map, the feature map is then multiplied by three 1x1 convolutions to obtain the extracted feature map.

[0303] The extracted feature maps are processed using deep learning interpolation methods to obtain the second optical flow features.

[0304] In one embodiment: the step of inputting human semantic mask features and human optical flow features into a behavior recognition classification model for processing to obtain human behavior recognition results includes:

[0305] The semantic mask features and optical flow features of the human body are concatenated and merged to obtain the merged features;

[0306] The merged features are input into a lightweight video classification network for processing to obtain the probabilities corresponding to different human behavior categories;

[0307] The human behavior category with the highest probability is selected as the predicted human behavior recognition result.

[0308] It should be understood that, in the embodiments of this application, the processor 720 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0309] Those skilled in the art will understand that Figure 4 The structure of the computer device 700 shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0310] In another embodiment of the present invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the human behavior recognition method disclosed in the embodiments of the present invention.

[0311] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0312] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0313] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0314] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0315] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0316] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for recognizing human behavior, characterized in that, include: Acquire image data within the detection area; The image data is sliced ​​into frames to obtain multiple static images; Multiple static images are input into the human behavior prediction model as an image sequence for processing to obtain human behavior recognition results. The step of inputting multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results includes: Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes; Different human images with ID indices are input into a semantic segmentation model for processing to obtain human semantic mask features; Human images with ID indexes are input into an optical flow prediction model for processing to obtain human optical flow features; Human semantic mask features and human optical flow features are input into the behavior recognition and classification model for processing to obtain human behavior recognition results; The process of inputting different human images with ID indices into a semantic segmentation model for processing to obtain human semantic mask features includes: The first, second, and third convolutions are performed on the person images with id indexes to obtain the first, second, and third convolution features. The first convolutional feature is input into Switch conv for convolution operation to obtain the first processed convolutional feature; The second convolutional feature is input into the Dilated conv for processing to obtain the second processed convolutional feature; The third convolutional feature is input into the Coord conv for processing to obtain the third processed convolutional feature; The first-processed convolutional features are input into a dimension-wise convolutional function and processed by the ReLU activation function to obtain the first-processed features. The convolutional features after the second processing are input into Dynamic conv for convolution processing to obtain intermediate processing features; The intermediate processing features are input into a 3*3 convolution and a BatchNormalization operation to obtain the second processing features; The convolutional features after the third processing are input into Mix conv for convolution operation to obtain the third processed features; The result of the first processed feature is obtained by convolution, followed by deconvolution upsampling and convolution again to obtain the first processed feature. The first processed features are subjected to convolution and BatchNormalization operations to obtain the second processed feature; The second processing result features are subjected to unpooling upsampling and convolutional block operations to obtain unpooling upsampling features and convolutional block features; The second processed feature is added to the convolutional block feature to obtain the added feature; Perform a sigmoid operation on the additive features to obtain sigmoid features; The sigmoid features are multiplied with the unpooled upsampled features to obtain the multiplied features; The multiplied features are processed by the ReLU function to obtain the first result feature; The result of subtracting the sigmoid feature from 1 is multiplied with the third processing feature to obtain the second result feature; The result obtained by adding the second result feature to the first result feature is then upsampled and convolved to obtain the third result feature. The third result feature is input into the residual network after three processing steps to obtain the fourth result feature; The fourth result feature is concatenated with the third result feature to obtain the fused feature; The fused features are processed through 3*3 convolution, upsampling, and 1*1 convolution to obtain human semantic mask features.

2. The human behavior recognition method according to claim 1, characterized in that, The process of inputting a person image with an ID index into an optical flow prediction model for processing to obtain human optical flow features includes: The first network processes two consecutive frames of character images with ID indices to obtain the first optical flow features. The first optical flow feature is used to perform bilinear interpolation on the previous frame of a person image with an id index between two consecutive frames to obtain bilinear interpolation features. The results of processing the two consecutive frames of character images with ID indices using AsymOFMM are concatenated with the bilinear interpolation features to obtain the concatenated features. The concatenated feature is stacked with the next frame of the two frames of character images with id indexes to obtain a stacked image. The stacked image is input into the second network for processing to obtain the second optical flow feature; The second optical flow feature is input into the Residual GRU module for iterative updating. When the iteration is complete, the human body optical flow feature results are output.

3. The human behavior recognition method according to claim 2, characterized in that, The step of inputting two consecutive frames of character images with ID indices into the first network for processing to obtain the first optical flow feature includes: The two frames of character images with id indices are processed by 4 layers of Convolution+ReLU to obtain the first feature map and the second feature map. After downsampling the first and second feature maps respectively, a Max pooling layer is used to process them to obtain the third and fourth feature maps; The third and fourth feature maps are processed through three residual layers to obtain the fifth and sixth feature maps; The fifth feature map is used as the kernel to perform convolution with the sixth feature map to obtain the matching feature map; The matching feature map is fed into three convolutional layers and one average pooling layer for advanced feature extraction to obtain advanced features; Four layers of deconvolution are used on high-level features to obtain high-level semantic information; The high-level semantic information is fused with the fifth and sixth feature maps, and then upsampled to obtain the first optical flow feature.

4. The human behavior recognition method according to claim 3, characterized in that, The step of inputting the stacked map into the second network for processing to obtain the second optical flow feature includes: The stacked image is fed into a three-layer Convolution+BatchNorm layer for processing to obtain the seventh feature map; After performing average pooling on the seventh feature map, the feature map is then multiplied by three 1x1 convolutions to obtain the extracted feature map. The extracted feature maps are processed using deep learning interpolation methods to obtain the second optical flow features.

5. The human behavior recognition method according to claim 1, characterized in that, The process of inputting human semantic mask features and human optical flow features into a behavior recognition and classification model for processing to obtain human behavior recognition results includes: The semantic mask features and optical flow features of the human body are concatenated and merged to obtain the merged features; The merged features are input into a lightweight video classification network for processing to obtain the probabilities corresponding to different human behavior categories; The human behavior category with the highest probability is selected as the predicted human behavior recognition result.

6. A human behavior recognition device, which, when operating, executes the human behavior recognition method according to any one of claims 1-5, characterized in that, include: The acquisition unit is used to acquire image data within the detection area; The first processing unit is used to perform frame-slicing processing on the image data to obtain multiple frames of static images; The second processing unit is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain human behavior recognition results.

7. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the human behavior recognition method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor performs the steps of the human behavior recognition method as described in any one of claims 1 to 5.

Images