Human behavior recognition method, device and equipment based on key points and optical flow

Abstract

This invention discloses a method, apparatus, and device for human behavior recognition based on key points and optical flow. The method includes: acquiring image data within a detection area; performing frame segmentation on the image data to obtain multiple static images; and inputting the multiple static images as an image sequence into a human behavior prediction model for processing to obtain human behavior recognition results. This invention integrates optical flow features into the skeletal key point location features, and then uses a graph convolutional neural network to perform behavior recognition on the fused features. This effectively combines the long and short time series features of key points, improving the recognition rate. The optical flow model can effectively extract optical flow information. Furthermore, SPO-GCN can simultaneously extract original image information, skeletal key point location information, and optical flow information, and fuse these three pieces of information. It also performs behavior prediction in both spatial and temporal dimensions, making the model more robust and improving its accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method, apparatus, and device for human behavior recognition based on key points and optical flow. Background Technology

[0002] With rapid societal development, human behavior prediction is currently a research hotspot and challenge in both industry and academia, and it has significant practical applications. Several methods are currently available for human behavior prediction:

[0003] The first method involves processing the acquired visible light and infrared images to obtain the target area for tracking. Next, it detects whether the target area includes pedestrians. If pedestrians are included, they are tracked, and during tracking, pedestrian edges are detected to obtain the pedestrian identification area from the target area. This pedestrian identification area is then input into a recognition model to obtain the pedestrian behavior recognition result. This patent uses infrared tracking, which requires additional equipment costs. Furthermore, it relies on traditional histogram-based feature extraction to determine pedestrian targets, which is severely affected by lighting conditions and has poor performance. Additionally, this method only recognizes the outline of the person, ignoring spatial-temporal fusion information, resulting in poor recognition performance.

[0004] The second approach involves training a 3D convolutional neural network using multi-task deep learning. This method uses a fixed number of consecutive frames from a background video as input to various human behavioral attributes, and the network is trained to perform the recognition task. However, this method only recognizes human behavior at fixed locations and is limited to a single individual, which has significant limitations. Furthermore, behavior recognition based on 3D convolution is inefficient, and the trained model is heavily influenced by background factors, resulting in poor recognition performance.

[0005] The third method involves acquiring video clips, extracting features and reducing dimensionality from the image frame sequences within those clips, encoding the reduced feature vectors, and then expanding these expanded feature vectors to obtain a predetermined number of expanded feature vectors. These expanded feature vectors and the encoded feature vectors are then input into a three-layer single-layer decoder for decoding. The final decoded feature vector is then input into a single-layer fully connected feedforward network for calculation, yielding multiple predicted values. These predicted values ​​are then input into a logistic regression network to obtain the corresponding predicted probabilities. The category corresponding to the highest probability value is selected as the human action category of the bounding box corresponding to the final decoded feature vector. This method first extracts spatial information and then extracts temporal features using spatial features from different time periods. However, this process easily loses spatial information during temporal feature extraction, making it difficult to achieve a fusion of spatial and temporal features, thus reducing the accuracy of action recognition. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, device and equipment for human behavior recognition based on key points and optical flow.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect is a human behavior recognition method based on key points and optical flow, including:

[0009] Acquire image data within the detection area;

[0010] The image data is sliced ​​into frames to obtain multiple static images;

[0011] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0012] The further technical solution is as follows: The step of inputting multiple static images as an image sequence into a human behavior prediction model for processing to obtain human behavior recognition results includes:

[0013] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0014] Human images with ID indexes are input into a target detection and keypoint detection model for processing to obtain human skeleton keypoint information for people with different IDs.

[0015] The image of a person with an ID index is input into the optical flow model for processing to obtain a two-dimensional optical flow feature map;

[0016] Human skeletal key point information and two-dimensional optical flow feature map are input into the graph model for processing to obtain human behavior recognition results.

[0017] The further technical solution is as follows: The step of inputting multiple frames of static images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes:

[0018] Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes;

[0019] Different human target bounding boxes are assigned an index ID to obtain human images with ID indexes.

[0020] The further technical solution is as follows: The process of inputting the person image with ID index into the target detection and keypoint detection model for processing to obtain the human skeleton keypoint information of people with different IDs includes:

[0021] Add a keypoints branch to the image of a person with an id index to perform keypoint detection, so as to obtain the human skeleton keypoint information of the person with the id.

[0022] The further technical solution is as follows: The step of inputting the image of a person with an ID index into the optical flow model for processing to obtain a two-dimensional optical flow feature map includes:

[0023] The feature descriptor operator is used to extract motion salient features from two consecutive frames of human images with id indexes to obtain the first feature map and the second feature map.

[0024] The correlation between the first feature map and the second feature map is calculated pixel by pixel to obtain the correlation features.

[0025] After the correlation features are input into the convolutional layer and the residual layer for processing, the processed features are obtained.

[0026] The processed features are upsampled and amplified to obtain the optical flow information of the first stage;

[0027] The extracted optical flow feature map is warped using the optical flow information from the first stage to obtain the warp feature map.

[0028] The Subtracting Operator is used to perform element-wise subtraction on the second feature map and the Warp feature map to obtain the feature difference map;

[0029] The feature difference map and the first feature map are input into the FlowNetCorr network for optical flow estimation to obtain a two-dimensional optical flow feature map.

[0030] The further technical solution is as follows: The step of performing a warp operation on the extracted optical flow feature map using the optical flow information from the first stage to obtain a warp feature map includes:

[0031] The motion boundary histogram of the feature description operator is used to extract the feature information of the moving object boundary from two consecutive frames of human images with id indexes.

[0032] HOG features are calculated on the optical flow maps in the x and y directions of the extracted moving object boundary information to obtain optical flow feature maps in the x and y directions;

[0033] The optical flow feature maps in the x and y directions are merged to obtain the merged optical flow feature map;

[0034] The merged optical flow feature map is subjected to bilinear interpolation to obtain the Warp feature map.

[0035] The further technical solution is as follows: The process of inputting key point information of the human skeleton and two-dimensional optical flow feature map into a graph model for processing to obtain human behavior recognition results includes:

[0036] Construct a skeletal node diagram based on key point information of the human skeleton;

[0037] Construct a pixel map based on the pixel information of static images;

[0038] Constructing an optical flow map based on a two-dimensional optical flow feature map;

[0039] Temporal and spatial features are extracted from the skeletal node map, pixel map, and optical flow map to obtain a first spatiotemporal feature map, a second spatiotemporal feature map, and a third spatiotemporal feature map;

[0040] The first, second, and third spatiotemporal feature maps are fused together to obtain a fused feature map.

[0041] The fused feature maps are processed through a fully connected layer and SoftMax to obtain human behavior recognition results.

[0042] Secondly, a human behavior recognition device based on key points and optical flow includes:

[0043] The acquisition unit is used to acquire image data within the detection area;

[0044] The first processing unit is used to perform frame-slicing processing on the image data to obtain multiple frames of static images;

[0045] The second processing unit is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain human behavior recognition results.

[0046] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the human behavior recognition method based on key points and optical flow as described above.

[0047] Fourthly, a computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the steps of the human behavior recognition method based on key points and optical flow as described above.

[0048] The beneficial effects of this invention compared with the prior art are: this invention integrates optical flow features into the location features of skeletal key points, and then uses a graph convolutional neural network to perform behavior recognition on the fused features, which effectively combines the features of key points in both long and short time series, thereby improving the recognition rate.

[0049] Optical flow models can effectively extract optical flow information. Based on traditional SIFT and MBH operators, spatial and temporal information can be extracted respectively, and optical flow detection can be performed well by combining deep learning modules.

[0050] In addition, SPO-GCN can simultaneously extract original image information, skeletal keypoint location information, and optical flow information, and fuse these three types of information. It also performs behavior prediction in spatial and temporal dimensions, making the model more robust and improving its accuracy.

[0051] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are described in detail below. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A schematic diagram illustrating an application scenario of the human behavior recognition method based on key points and optical flow provided in a specific embodiment of the present invention;

[0054] Figure 2 A flowchart of a human behavior recognition method based on key points and optical flow provided for a specific embodiment of the present invention;

[0055] Figure 3 A schematic block diagram of a human behavior recognition device based on key points and optical flow provided for a specific embodiment of the present invention;

[0056] Figure 4 This is a schematic block diagram of a computer device provided for a specific embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0059] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0060] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0061] Please see Figure 1 and Figure 2 , Figure 1 A schematic diagram illustrating an application scenario of the human behavior recognition method based on key points and optical flow provided in an embodiment of the present invention; Figure 2 The flowchart illustrates a human behavior recognition method based on key points and optical flow, provided in a specific embodiment of the present invention. This method is applied in a server and is executed by application software installed on the server.

[0062] like Figure 2 As shown, the human behavior recognition method based on key points and optical flow includes the following steps: S10-S30.

[0063] S10. Acquire image data within the detection area.

[0064] Use surveillance equipment, such as surveillance cameras, to monitor and capture video data (image data) of the area to be detected. Any commercially available surveillance equipment is acceptable; this application does not impose any restrictions on its use.

[0065] S20. Perform frame segmentation on the image data to obtain multiple static images.

[0066] The detected image data is sliced ​​into individual RGB static images, which are then used as input data for the human behavior prediction model.

[0067] S30. Input multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0068] In one embodiment, step S30 specifically includes the following steps: S301-S304.

[0069] S301. Input multiple static images as an image sequence into the tracking model for processing to obtain a person image with an ID index.

[0070] In this embodiment, the tracking model used is StrongSORT, which uses the YOLOv5 model for human target detection. This allows the coordinates of the human target bounding box and the confidence information of the human category to be obtained. In addition, the corresponding person ID index information can also be obtained.

[0071] In one embodiment, step S301 specifically includes the following steps: S3011-S3012.

[0072] S3011. Input multiple static images as an image sequence into the tracking model to detect different human target boxes.

[0073] In this embodiment, the obtained multiple frames of static images (frame video stream) are input into the tracking model as an image sequence, which can detect different human target boxes. These target boxes are identified by their image position coordinates (x, y, z). min ,y min (x) max ,y max The coordinates are represented by the coordinates of the target bounding box relative to the top left corner of the image and the coordinates of the target bounding box relative to the bottom right corner of the image.

[0074] S3012. Assign an index id to the different human target boxes detected to obtain human images with id indexes.

[0075] In this embodiment, the image is cropped based on the target bounding box, resulting in a single-person image with a unique ID.

[0076] S302. Input the human image with id index into the target detection and key point detection model for processing to obtain the human skeleton key point information of people with different id.

[0077] In this embodiment, the target detection and key point detection model adopts the YOLOpose-based key point detection model. Compared with the top-down method, which performs target detection first and then key point detection, this model can detect the bounding box of pedestrians and the key point coordinates of people simultaneously using only one model.

[0078] In one embodiment, step S302 specifically includes the following steps: S3021.

[0079] S3021. Add a keypoints branch to the image of a person with an id index to perform keypoint detection, so as to obtain the human skeleton keypoint information of the person with the id.

[0080] In this embodiment, keypoint detection is performed by adding keypoint branches to head1, head2, head3, and head4 of the YOLOv5 output. The model first crops the original image based on multiple human bounding boxes obtained from the tracking model, and then performs attribute and keypoint recognition on each cropped human image. Attribute recognition here is based on object detection, which effectively enhances the model's supervision information. For example, scenarios such as whether a helmet or mask is worn can be identified using this model. Simultaneously, the model can also predict the skeletal keypoint coordinates of each pedestrian with a different ID. Based on multiple frames of pedestrian images with different ID indices and keypoint coordinates, a time-frame sequence of comprehensive feature information for different pedestrians is constructed. It should be noted that the input images for the optical flow model and graph model in this patent are single-person images with IDs, cropped from the original image based on the target bounding boxes detected by the tracking model.

[0081] S303. Input the image of the person with the id index into the optical flow model for processing to obtain a two-dimensional optical flow feature map.

[0082] In this embodiment, the optical flow model is named ST-Net because it integrates the spatial and temporal features of the two frames before and after the pedestrian's movement corresponding to each ID in the tracking model.

[0083] In one embodiment, step S302 specifically includes the following steps: S3021-S3026.

[0084] S3021. Use the feature description operator to extract motion salient features from the two frames of the character image with id index, respectively, to obtain the first feature map and the second feature map.

[0085] In this embodiment, the spatial network S-Net first uses the feature description operator SIFT (Scale Invariable Feature Transform) to extract motion salient features from the input previous and next frame images I1 and I2 respectively to obtain F1 and F2.

[0086] S3022. Perform correlation calculation on a pixel-by-pixel basis on the first feature map and the second feature map to obtain the correlation features.

[0087] In this embodiment, feature map I3 is obtained by performing pixel-by-pixel correlation calculation on F1 and F2.

[0088] S3022. After inputting the correlation features into the convolutional layer and residual layer for processing, the processed features are obtained.

[0089] In this embodiment, I3 is input into two Conv7x7 convolutional layers for convolution, and then input into two residual layers to obtain feature map F3.

[0090] S3023. The processed features are upsampled and amplified to obtain the optical flow information of the first stage.

[0091] In this embodiment, F3 is upsampled and amplified to obtain the first-stage optical flow output Flow1.

[0092] S3024. Use the optical flow information from the first stage to perform a warp operation on the extracted optical flow feature map to obtain a warp feature map.

[0093] In one embodiment, step S3024 specifically includes the following steps: S30241-S30244.

[0094] S30241. Use the feature description operator motion boundary histogram to extract the feature information of the moving object boundary from two consecutive frames of human images with id index.

[0095] S30242. Perform HOG feature calculation on the optical flow maps in the x and y directions of the extracted moving object boundary information to obtain optical flow feature maps in the x and y directions.

[0096] S30243. Merge the optical flow feature maps in the x and y directions to obtain the merged optical flow feature map.

[0097] S30244. Perform bilinear interpolation on the merged optical flow feature map to obtain the Warp feature map.

[0098] For S30241-S30244, in this embodiment, the temporal network N-Net uses the feature description operator Motion Boundary Histogram (MBH) to extract features of the moving object boundary information for I1 and I2. MBH performs HOG (Histogram of Oriented Gradient) feature calculation on the optical flow maps in the x and y directions, resulting in optical flow feature maps F in the x and y directions. x F y F x F y By merging the samples, we can obtain an optical flow feature map F. flow It contains the motion and optical flow information of the original image. At this point, Flow1 obtained from S-Net will be used to process F...flow Performing a warp operation, which is essentially a bilinear interpolation, yields F. warp Feature map.

[0099] S3025. Use the Subtracting Operator to perform element-wise subtraction on the second feature map and the Warp feature map to obtain the feature difference map.

[0100] In this embodiment, a subtracting operator is used to extract the original second frame image I2 and the image after changing its dimensions and size F. warp Performing element-wise subtraction yields a graph F representing the feature difference. difference It can effectively describe motion information.

[0101] S3026. Input the feature difference map and the first feature map into the FlowNetCorr network to perform optical flow estimation to obtain a two-dimensional optical flow feature map.

[0102] In this embodiment, F difference The I1 input is fed into a FlowNetCorr network for optical flow estimation, yielding the final optical flow result Flow. out This is a two-dimensional optical flow feature map with velocity and direction. The upsampling operation in FlowNetCorr ensures that the size of the output feature map is the same as the size of the original image.

[0103] S304. Input the key point information of the human skeleton and the two-dimensional optical flow feature map into the graph model for processing to obtain the human behavior recognition result.

[0104] In this embodiment, the graph model is the SPO-GCN model, which integrates three different features to achieve behavior prediction. The three features are extracted based on skeleton keypoints, original image pixels, and optical flow.

[0105] In one embodiment, step S304 specifically includes the following steps: S3041-S3046.

[0106] S3041. Construct a skeletal node diagram based on key point information of the human skeleton.

[0107] In one embodiment, the model's input consists of a time-frame sequence of integrated feature information of pedestrians with the same ID, output by the tracking model, and an optical flow sequence output by the optical flow model. For each frame, three 25x25 graphs are generated based on the SPO information. First, based on skeletal keypoint information, joints (each keypoint) are used as nodes in the graph, and physiological connections between joints are used as edges to construct a graph with 25 nodes and 24 edges. s In addition, identical nodes between consecutive frames are connected by an edge, representing connection information in the time dimension. Since the NTU-RGB-120 dataset is used, all 25 joint nodes are represented. A 25x25 adjacency matrix A is used through a predefined topology. s It can be constructed naturally. Adjacency matrices typically use 1s and 0s to represent graph connectivity, but here all adjacency matrices A are normalized to increase the weight of low-degree nodes and reduce the influence of high-degree nodes. It's worth mentioning that throughout the entire process of skeletal feature extraction and learning, A... s It will not be updated.

[0108] S3042. Construct a pixel map based on the pixel information of a static image.

[0109] S3043. Construct an optical flow map based on a two-dimensional optical flow feature map.

[0110] For S3042 and S3043, in this embodiment, both the optical flow graph and the original image (based on the original pixels of the static image) need to be converted into a graph representation by dividing them into patches. This patent divides them into 25 patches respectively, performs feature transformation, and then represents the feature-bearing patches as nodes of the graph. It is important to note that the node connection methods for graphs constructed based on pixels and those based on optical flow are different; that is, the methods for constructing the adjacency matrix are different. Considering that optical flow does not contain specific RGB information, the positional information in the original image is used as an index to define its adjacency information. That is, adjacent patches are neighboring nodes. Based on this idea, a graph containing 25 nodes and 44 edges is generated. O A 25x25 adjacency matrix A O It is obtained. Similarly, throughout the process of optical flow feature extraction and learning, A OIt will not be updated. Next comes the pixel-based graph construction process. The similarity between patches is calculated by comparing pixel variances. For each patch, the five patches with the smallest variance differences are selected as neighboring nodes. Therefore, patches with similar color textures are clustered together, thus defining the neighboring nodes of each patch. Therefore, a graph containing 25 nodes and 125 edges is formed. P A 25x25 adjacency matrix A P To be obtained. And A S A O The difference is that A P It will be updated at each layer of the graph convolutional network.

[0111] S3044. Perform temporal and spatial feature extraction on the skeletal node map, pixel map, and optical flow map to obtain a first spatiotemporal feature map, a second spatiotemporal feature map, and a third spatiotemporal feature map.

[0112] In this embodiment, Graph s and A s Graph O and A O Graph P and A P The three sets of data will be fed into an 8-layer Graph Convolutional Network (GCN) + Temporal Convolutional Network (TCN) for feature updates. During this process, neighboring nodes will be aggregated, and all nodes will be updated. In the 8-layer network, the input data first passes through the first layer of GCN + Residual Units and TCN + Residual Units, then the output of the first layer is fed into the second layer of GCN + Residual Units and TCN + Residual Units. This process of layer-by-layer network stacking extracts features, with the number of output channels increasing from 64 to 256.

[0113] In the GCN network, 1x1 Conv is used to change channels, and the Einstein summation convention (einsum) is used to implement matrix multiplication. After the spatial feature extraction module, two Residual Units are added, and after passing through the residual layer, three updated graphs are obtained. s Graph O Graph PAt this point, time-dimensional feature extraction will be performed. The Temporal Convolutional Network (TCN) consists of six 3x1 Convs, two 1x1 Convs, and one Max-Pooling layer. After passing through the final layer, the 8th layer of the Temporal Convolutional Network, and the residual unit, a global pooling process is used to summarize the features, resulting in 256-dimensional feature maps F. s F O F P .

[0114] S3045. The first spatiotemporal feature map, the second spatiotemporal feature map, and the third spatiotemporal feature map are fused to obtain a fused feature map.

[0115] In this embodiment, the obtained 256-dimensional feature map F s F O F P Feature merging is performed to obtain a graph F that integrates the three features. all .

[0116] S3046. The fused feature map is processed through a fully connected layer and SoftMax to obtain the human behavior recognition result.

[0117] In this embodiment, F all The predicted categories of human behavior outcomes are obtained through a fully connected layer and SoftMax.

[0118] In this embodiment, there are five categories: jumping, running, sitting, standing, and walking. This allows for the prediction of the behaviors of different people in the surveillance video.

[0119] In addition, the loss functions used in the human behavior prediction model include the object detection loss function, the YOLOpose loss function, the optical flow loss function, and the graph model-based behavior recognition loss function, among which:

[0120] The loss function used for object detection is the YOLOv5 loss function, which mainly consists of classification loss, localization loss, and confidence loss function.

[0121] Loss = Loss 分类 +Loss 定位 +Loss 置信度 .

[0122] The YOLOpose loss function consists of an object detection loss function and a keypoint loss function. Here, the keypoint loss function is composed of the OKS loss function and the keypoint confidence loss function.

[0123] Loss total =δcls Loss cls +δ box Loss box +δ kpts Loss kpts +δ kpts conf Loss kpts conf

[0124] The parameter δ here cls =0.5, δ box =0.5, δ kpts =0.5, δ kpts conf =0.5. Loss cls For object detection classification loss function, Loss box Loss is the loss function for the location of the target bounding box in target detection. kpts Keypoint loss function, Loss kpts conf The keypoint confidence loss function.

[0125] The optical flow loss function is the optical flow loss estimated in S-Net. Optical flow loss estimated in T-Net The system uniformly uses endpoint error (EPE) as the loss function for training, which represents the average Euclidean distance per pixel between the predicted flow vector and the ground truth.

[0126] The graph-based behavior recognition loss function uses the classification cross-entropy loss for model training.

[0127] This invention integrates optical flow features into the location features of skeletal keypoints, and then uses a graph convolutional neural network to perform behavior recognition on the fused features. This effectively combines the long and short time series features of keypoints, improving the recognition rate. The optical flow model effectively extracts optical flow information, using traditional SIFT and MBH operators to extract spatial and temporal information respectively, and combining this with a deep learning module for efficient optical flow detection. Furthermore, SPO-GCN can simultaneously extract original image information, skeletal keypoint location information, and optical flow information, fusing these three pieces of information and performing behavior prediction in both spatial and temporal dimensions, making the model more robust and improving its accuracy.

[0128] Figure 3This is a schematic block diagram of a human behavior recognition device 100 based on key points and optical flow provided in an embodiment of the present invention. Corresponding to the above-described human behavior recognition method based on key points and optical flow, a specific embodiment of the present invention also provides a human behavior recognition device 100 based on key points and optical flow. This human behavior recognition device 100 based on key points and optical flow includes units and modules for executing the above-described human behavior recognition method based on key points and optical flow, and the device can be configured in a server.

[0129] like Figure 3 As shown, the human behavior recognition device 100 based on key points and optical flow includes:

[0130] The acquisition unit 110 is used to acquire image data within the detection area.

[0131] The first processing unit 120 is used to perform frame-slicing processing on the image data to obtain multiple frames of static images.

[0132] The second processing unit 130 is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain human behavior recognition results.

[0133] In one embodiment, the second processing unit 130 includes:

[0134] The first processing module is used to input multiple frames of static images into the tracking model in the form of an image sequence for processing, so as to obtain a person image with an ID index.

[0135] In one embodiment, the first processing module includes:

[0136] The detection submodule is used to input multiple frames of static images as an image sequence into the tracking model in order to detect different human target boxes.

[0137] The submodule is used to assign index IDs to different detected human target boxes in order to obtain human images with ID indexes.

[0138] The second processing module is used to input the human images with ID indexes into the target detection and key point detection model for processing, so as to obtain the human skeleton key point information of people with different IDs.

[0139] In one embodiment, the second processing module includes:

[0140] The keypoint detection submodule is used to add a keypoints branch to the image of a person with an ID index to perform keypoint detection, so as to obtain the human skeleton keypoint information of the person with the ID.

[0141] The third processing module is used to input the image of the person with the ID index into the optical flow model for processing to obtain a two-dimensional optical flow feature map.

[0142] In one embodiment, the third processing module includes:

[0143] The motion feature extraction submodule is used to extract salient motion features from two consecutive frames of a person image with an ID index using feature descriptor operators, so as to obtain a first feature map and a second feature map.

[0144] The correlation processing submodule is used to perform pixel-by-pixel correlation processing on the first feature map and the second feature map to obtain correlation features.

[0145] The first processing submodule is used to input the relevant features into the convolutional layer and the residual layer for processing, and then obtain the processed features.

[0146] The amplification processing submodule is used to upsample and amplify the processed features to obtain the optical flow information of the first stage.

[0147] The second processing submodule is used to perform a warp operation on the extracted optical flow feature map using the optical flow information from the first stage to obtain a warp feature map.

[0148] In one embodiment, the second processing submodule includes:

[0149] The feature extraction submodule is used to extract the feature information of the moving object boundary from two consecutive frames of a person image with an ID index using the motion boundary histogram of the feature description operator.

[0150] The feature calculation submodule is used to perform HOG feature calculation on the optical flow maps in the x and y directions of the extracted moving object boundary information to obtain optical flow feature maps in the x and y directions.

[0151] The merging submodule is used to merge the optical flow feature maps in the x and y directions to obtain the merged optical flow feature map.

[0152] The interpolation processing submodule is used to perform bilinear interpolation on the merged optical flow feature map to obtain the Warp feature map.

[0153] The third processing submodule is used to perform element-wise subtraction on the second feature map and the Warp feature map using the Subtracting Operator to obtain the feature difference map.

[0154] The optical flow estimation submodule is used to input the feature difference map and the first feature map into the FlowNetCorr network for optical flow estimation to obtain a two-dimensional optical flow feature map.

[0155] The fourth processing module is used to input the key point information of the human skeleton and the two-dimensional optical flow feature map into the graph model for processing to obtain the human behavior recognition results.

[0156] In one embodiment, the fourth processing module includes:

[0157] The first construction submodule is used to construct a skeletal node graph based on key point information of the human skeleton.

[0158] The second construction submodule is used to construct a pixel map based on the pixel information of a static image.

[0159] The third construction submodule is used to construct optical flow maps based on two-dimensional optical flow feature maps.

[0160] The spatiotemporal feature extraction submodule is used to extract temporal and spatial features from the skeletal node map, pixel map, and optical flow map to obtain a first spatiotemporal feature map, a second spatiotemporal feature map, and a third spatiotemporal feature map.

[0161] The fusion submodule is used to fuse the first spatiotemporal feature map, the second spatiotemporal feature map, and the third spatiotemporal feature map to obtain a fused feature map.

[0162] The fourth processing submodule is used to process the fused feature map through a fully connected layer and SoftMax to obtain the human behavior recognition result.

[0163] The aforementioned human behavior recognition device based on key points and optical flow can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.

[0164] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0165] like Figure 4 As shown, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the human behavior recognition method based on key points and optical flow as described above.

[0166] The computer device 700 can be a terminal or a server. The computer device 700 includes a processor 720, a memory, and a network interface 750 connected via a system bus 710, wherein the memory may include a non-volatile storage medium 730 and internal memory 740.

[0167] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. When the computer program 732 is executed, it enables the processor 720 to execute any human behavior recognition method based on key points and optical flow.

[0168] The processor 720 provides computing and control capabilities to support the operation of the entire computer device 700.

[0169] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute any human behavior recognition method based on key points and optical flow.

[0170] This network interface 750 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 700 to which the present application is applied. The specific computer device 700 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. The processor 720 is used to run program code stored in memory to implement the following steps:

[0171] Human behavior recognition methods based on key points and optical flow include:

[0172] Acquire image data within the detection area;

[0173] The image data is sliced ​​into frames to obtain multiple static images;

[0174] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0175] In one embodiment: the step of inputting multiple frames of static images as an image sequence into a human behavior prediction model for processing to obtain human behavior recognition results includes:

[0176] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0177] Human images with ID indexes are input into a target detection and keypoint detection model for processing to obtain human skeleton keypoint information for people with different IDs.

[0178] The image of a person with an ID index is input into the optical flow model for processing to obtain a two-dimensional optical flow feature map;

[0179] Human skeletal key point information and two-dimensional optical flow feature map are input into the graph model for processing to obtain human behavior recognition results.

[0180] In one embodiment: the step of inputting multiple frames of still images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes:

[0181] Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes;

[0182] Different human target bounding boxes are assigned an index ID to obtain human images with ID indexes.

[0183] In one embodiment: the step of inputting a person image with an ID index into a target detection and keypoint detection model for processing to obtain human skeletal keypoint information for people with different IDs includes:

[0184] Add a keypoints branch to the image of a person with an id index to perform keypoint detection, so as to obtain the human skeleton keypoint information of the person with the id.

[0185] In one embodiment: the step of inputting the image of a person with an ID index into an optical flow model for processing to obtain a two-dimensional optical flow feature map includes:

[0186] The feature descriptor operator is used to extract motion salient features from two consecutive frames of human images with id indexes to obtain the first feature map and the second feature map.

[0187] The correlation between the first feature map and the second feature map is calculated pixel by pixel to obtain the correlation features.

[0188] After the correlation features are input into the convolutional layer and the residual layer for processing, the processed features are obtained.

[0189] The processed features are upsampled and amplified to obtain the optical flow information of the first stage;

[0190] The extracted optical flow feature map is warped using the optical flow information from the first stage to obtain the warp feature map.

[0191] The Subtracting Operator is used to perform element-wise subtraction on the second feature map and the Warp feature map to obtain the feature difference map;

[0192] The feature difference map and the first feature map are input into the FlowNetCorr network for optical flow estimation to obtain a two-dimensional optical flow feature map.

[0193] In one embodiment: the step of performing a warp operation on the extracted optical flow feature map using the optical flow information from the first stage to obtain a warp feature map includes:

[0194] The motion boundary histogram of the feature description operator is used to extract the feature information of the moving object boundary from two consecutive frames of human images with id indexes.

[0195] HOG features are calculated on the optical flow maps in the x and y directions of the extracted moving object boundary information to obtain optical flow feature maps in the x and y directions;

[0196] The optical flow feature maps in the x and y directions are merged to obtain the merged optical flow feature map;

[0197] The merged optical flow feature map is subjected to bilinear interpolation to obtain the Warp feature map.

[0198] In one embodiment: the step of inputting human skeletal key point information and two-dimensional optical flow feature map into a graph model for processing to obtain human behavior recognition results includes:

[0199] Construct a skeletal node diagram based on key point information of the human skeleton;

[0200] Construct a pixel map based on the pixel information of static images;

[0201] Constructing an optical flow map based on a two-dimensional optical flow feature map;

[0202] Temporal and spatial features are extracted from the skeletal node map, pixel map, and optical flow map to obtain a first spatiotemporal feature map, a second spatiotemporal feature map, and a third spatiotemporal feature map;

[0203] The first, second, and third spatiotemporal feature maps are fused together to obtain a fused feature map.

[0204] The fused feature maps are processed through a fully connected layer and SoftMax to obtain human behavior recognition results.

[0205] It should be understood that in the embodiments of this application, the processor 720 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0206] Those skilled in the art will understand that Figure 4 The structure of the computer device 700 shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0207] In another embodiment of the present invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when executed by a processor, the computer program implements the human behavior recognition method based on key points and optical flow disclosed in the embodiments of the present invention.

[0208] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0209] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0210] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0211] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0212] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0213] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A human behavior recognition method based on key points and optical flow, characterized in that, include: Acquire image data within the detection area; The image data is sliced ​​into frames to obtain multiple static images; Multiple static images are input into the human behavior prediction model as an image sequence for processing to obtain human behavior recognition results. The step of inputting multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results includes: Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes; Human images with ID indexes are input into a target detection and keypoint detection model for processing to obtain human skeleton keypoint information for people with different IDs. The image of a person with an ID index is input into the optical flow model for processing to obtain a two-dimensional optical flow feature map; The key points of the human skeleton and the two-dimensional optical flow feature map are input into the graph model for processing to obtain the human behavior recognition results. The step of inputting the image of a person with an ID index into the optical flow model for processing to obtain a two-dimensional optical flow feature map includes: The feature descriptor operator is used to extract motion salient features from two consecutive frames of human images with id indexes to obtain the first feature map and the second feature map. The correlation between the first feature map and the second feature map is calculated pixel by pixel to obtain the correlation features. After the correlation features are input into the convolutional layer and the residual layer for processing, the processed features are obtained. The processed features are upsampled and amplified to obtain the optical flow information of the first stage; The extracted optical flow feature map is warped using the optical flow information from the first stage to obtain the warp feature map. The Subtracting Operator is used to perform element-wise subtraction on the second feature map and the Warp feature map to obtain the feature difference map; The feature difference map and the first feature map are input into the FlowNetCorr network for optical flow estimation to obtain a two-dimensional optical flow feature map.

2. The human behavior recognition method based on key points and optical flow according to claim 1, characterized in that, The step of inputting multiple static images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes: Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes; Different human target bounding boxes are assigned an index ID to obtain human images with ID indexes.

3. The human behavior recognition method based on key points and optical flow according to claim 1, characterized in that, The process of inputting a person image with an ID index into a target detection and keypoint detection model for processing to obtain human skeletal keypoint information for people with different IDs includes: Add a keypoints branch to the image of a person with an id index to perform keypoint detection, so as to obtain the human skeleton keypoint information of the person with the id.

4. The human behavior recognition method based on key points and optical flow according to claim 1, characterized in that, The step of performing a warp operation on the extracted optical flow feature map using the optical flow information from the first stage to obtain a warp feature map includes: The motion boundary histogram of the feature description operator is used to extract the feature information of the moving object boundary from two consecutive frames of human images with id indexes. HOG features are calculated on the optical flow maps in the x and y directions of the extracted moving object boundary information to obtain optical flow feature maps in the x and y directions; The optical flow feature maps in the x and y directions are merged to obtain the merged optical flow feature map; The merged optical flow feature map is subjected to bilinear interpolation to obtain the Warp feature map.

5. The human behavior recognition method based on key points and optical flow according to claim 1, characterized in that, The process of inputting key point information of the human skeleton and two-dimensional optical flow feature map into the graph model for processing to obtain human behavior recognition results includes: Construct a skeletal node diagram based on key point information of the human skeleton; Construct a pixel map based on the pixel information of static images; Constructing an optical flow map based on a two-dimensional optical flow feature map; Temporal and spatial features are extracted from the skeletal node map, pixel map, and optical flow map to obtain a first spatiotemporal feature map, a second spatiotemporal feature map, and a third spatiotemporal feature map; The first, second, and third spatiotemporal feature maps are fused together to obtain a fused feature map. The fused feature maps are processed through a fully connected layer and SoftMax to obtain human behavior recognition results.

6. A human behavior recognition device based on key points and optical flow, which, when operating, executes the human behavior recognition method based on key points and optical flow as described in any one of claims 1-5, characterized in that... include: The acquisition unit is used to acquire image data within the detection area; The first processing unit is used to perform frame-slicing processing on the image data to obtain multiple frames of static images; The second processing unit is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain human behavior recognition results.

7. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the human behavior recognition method based on key points and optical flow as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor performs the steps of the human behavior recognition method based on key points and optical flow as described in any one of claims 1 to 5.

Images