Human behavior prediction method and device based on optical flow and graph, and equipment

Abstract

This invention discloses a two-branch human behavior prediction method, apparatus, and device based on optical flow and graph. The method includes: acquiring image data within a detection area; performing frame segmentation on the image data to obtain multiple static images; and inputting the multiple static images as an image sequence into a human behavior prediction model for processing to obtain human behavior prediction results. This invention combines instantaneous optical flow information and long-sequence information of human behavior in the temporal dimension, and uses a graph convolutional network in the spatial dimension to effectively learn local body information of pedestrians. By combining spatial and temporal information, the prediction accuracy is improved. Iterative refinement of the output optical flow clearly distinguishes the differences between moving pedestrians and the background. Simultaneously, the predicted optical flow direction and velocity information can also provide certain supervisory information for the model to predict human behavior categories.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method, apparatus, and device for predicting human behavior based on optical flow and graphs. Background Technology

[0002] With rapid societal development, human behavior prediction is currently a research hotspot and challenge in both industry and academia, and it has significant practical applications. Several methods are currently available for human behavior prediction:

[0003] The first method involves processing the acquired visible light and infrared images to obtain the target area for tracking. Next, it detects whether the target area includes pedestrians. If pedestrians are included, they are tracked, and during tracking, pedestrian edges are detected to obtain the pedestrian identification area from the target area. This pedestrian identification area is then input into a recognition model to obtain the pedestrian behavior recognition result. This patent uses infrared tracking, which requires additional equipment costs. Furthermore, it relies on traditional histogram-based feature extraction to determine pedestrian targets, which is severely affected by lighting conditions and has poor performance. Additionally, this method only recognizes the outline of the person, ignoring spatial-temporal fusion information, resulting in poor recognition performance.

[0004] The second approach involves training a 3D convolutional neural network using multi-task deep learning. This method uses a fixed number of consecutive frames from a background video as input to various human behavioral attributes, and the network is trained to perform the recognition task. However, this method only recognizes human behavior at fixed locations and is limited to a single individual, which has significant limitations. Furthermore, behavior recognition based on 3D convolution is inefficient, and the trained model is heavily influenced by background factors, resulting in poor recognition performance.

[0005] The third method involves acquiring video clips, extracting features and reducing dimensionality from the image frame sequences within those clips, encoding the reduced feature vectors, and then expanding these expanded feature vectors to obtain a predetermined number of expanded feature vectors. These expanded feature vectors and the encoded feature vectors are then input into a three-layer single-layer decoder for decoding. The final decoded feature vector is then input into a single-layer fully connected feedforward network for calculation, yielding multiple predicted values. These predicted values ​​are then input into a logistic regression network to obtain the corresponding predicted probabilities. The category corresponding to the highest probability value is selected as the human action category of the bounding box corresponding to the final decoded feature vector. This method first extracts spatial information and then extracts temporal features using spatial features from different time periods. However, this process easily loses spatial information during temporal feature extraction, making it difficult to achieve a fusion of spatial and temporal features, thus reducing the accuracy of action recognition. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus and device for predicting human behavior based on optical flow and graph.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect is a two-branch human behavior prediction method based on optical flow and graphs, including:

[0009] Acquire image data within the detection area;

[0010] The image data is sliced ​​into frames to obtain multiple static images;

[0011] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior prediction results.

[0012] The further technical solution is as follows: The step of inputting multiple frames of static images as an image sequence into the human behavior prediction model for processing to obtain human behavior prediction results includes:

[0013] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0014] The image of a person with an ID index is input into the optical flow model for processing to obtain a two-dimensional optical flow feature map;

[0015] The image of a person with an ID index is fused with a two-dimensional optical flow feature map to obtain a fused feature map.

[0016] The fused feature map is input into the graph model for processing to obtain human behavior prediction results.

[0017] The further technical solution is as follows: The step of inputting multiple frames of static images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes:

[0018] Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes;

[0019] Different human target bounding boxes are assigned an index ID to obtain human images with ID indexes.

[0020] The further technical solution is as follows: The step of inputting the image of a person with an ID index into an optical flow model for processing to obtain a two-dimensional optical flow feature map includes:

[0021] Dilated convolution kernels are used to extract features from two consecutive frames of character images with ID indices, and then processed by ReLU activation layer and Max Pooling layer to obtain the first feature map and the second feature map.

[0022] The first feature map and the second feature map are subjected to full-pixel correlation calculation to obtain the correlation feature map;

[0023] Construct the optical flow information for the 0th cycle;

[0024] The optical flow information from the 0th cycle is used to perform a warp operation on the second feature map to obtain the warp result from the 1st cycle.

[0025] The results of processing the two consecutive frames of character images with ID indices using AsymOFMM with occlusion masking are merged with the Warp results from the first loop to obtain the first merged feature.

[0026] The result obtained by merging the first merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the first cycle;

[0027] The optical flow information from the first cycle is used to perform a warp operation on the second feature map to obtain the warp result from the second cycle.

[0028] The results of processing the two consecutive frames of character images with ID indices using AsymOFMM with occlusion masking are merged with the Warp results from the second loop to obtain the second merged feature.

[0029] The result obtained by merging the second merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the second cycle.

[0030] Based on the set number of loops, the optical flow information of the first stage is obtained after all loops are completed;

[0031] The optical flow information from the first stage is input into ConvGRU for processing to obtain the final two-dimensional optical flow feature map.

[0032] The further technical solution is as follows: the construction of the optical flow information for the 0th cycle includes:

[0033] The first feature map, the second feature map, and the correlation feature map are merged to obtain a merged feature map;

[0034] The merged feature map is input into the encoding network for processing to obtain the optical flow information for the 0th cycle.

[0035] The further technical solution is as follows: the feature fusion of the person image with the id index and the two-dimensional optical flow feature map to obtain the fused feature map includes:

[0036] The Sobel operator is used to extract edge features from the image of a person with an id index to obtain an initial edge feature map;

[0037] Laplacian filtering is used to sharpen and refine the edge feature map of the initial edge feature map.

[0038] The refined edge feature map is processed by 3x3 convolution and then downsampled using Average Pooling to obtain the downsampled feature map.

[0039] Use bilinear interpolation to adjust the downsampled feature map to an adjusted feature map of the same size as the original image;

[0040] The two-dimensional optical flow feature map and the adjusted feature map are fused to obtain a fused feature map.

[0041] The further technical solution is as follows: the step of inputting the fused feature map into the graph model for processing to obtain the human behavior prediction result includes:

[0042] The fused feature map corresponding to each frame is stitched together with the original image to obtain the stitched image;

[0043] The mosaic image is divided into three different patch sizes;

[0044] The mosaic is divided into a corresponding number of equal-sized patches based on the patch size, resulting in 3 patch blocks;

[0045] Perform feature transformation on the three Patches blocks to obtain three sets of Graph nodes;

[0046] The perceptual hashing algorithm is used to calculate the similarity of the three Patches blocks based on the RGB information in the stitched graph, so as to form a dynamic adjacency matrix of the three Patches blocks.

[0047] The dynamic adjacency matrix of the three Patches blocks is input into the stacked block of the 12-layer graph convolutional network and the temporal convolutional network for node learning and feature updating. The output of the stacked block of the 12-layer graph convolutional network and the temporal convolutional network is then aggregated through the GlobalPooling layer to obtain a 512-dimensional feature vector for each sequence.

[0048] The 512-dimensional feature vector of each sequence is input into a fully connected layer for processing to generate preliminary predicted probabilities of human behavior categories for each patches block.

[0049] The initial predicted probability of the human behavior category for each generated Patches block is processed by a consensus function to obtain a consensus prediction.

[0050] The consensus prediction is fed into the SoftMax function for processing to obtain the final human behavior prediction result.

[0051] Secondly, a dual-branch human behavior prediction device based on optical flow and graphs includes:

[0052] The acquisition unit is used to acquire image data within the detection area;

[0053] The frame-slicing processing unit is used to slice the image data into frames to obtain multiple static images.

[0054] The prediction unit is used to input multiple frames of static images as an image sequence into the human behavior prediction model for processing, so as to obtain the human behavior prediction results.

[0055] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the optical flow and graph-based two-branch human behavior prediction method described above.

[0056] Fourthly, a computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the steps of the above-described optical flow and graph-based two-branch human behavior prediction method.

[0057] The advantages of this invention compared to existing technologies are as follows: This invention combines instantaneous optical flow information and long-sequence information of human behavior in the temporal dimension, and uses graph convolutional networks in the spatial dimension to effectively learn local body information of pedestrians. By combining spatial and temporal information, the prediction accuracy is improved. Iterative refinement of the output optical flow clearly distinguishes the differences between moving pedestrians and the background. Simultaneously, the predicted optical flow direction and velocity information can also provide some supervision information for the model to predict human behavior categories.

[0058] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are described in detail below. Attached Figure Description

[0059] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 A schematic diagram illustrating an application scenario of the optical flow and graph-based bi-branch human behavior prediction method provided in a specific embodiment of the present invention;

[0061] Figure 2 A flowchart of a two-branch human behavior prediction method based on optical flow and graphs provided for a specific embodiment of the present invention;

[0062] Figure 3 A schematic block diagram of a dual-branch human behavior prediction device based on optical flow and graphs, provided for a specific embodiment of the present invention;

[0063] Figure 4 This is a schematic block diagram of a computer device provided for a specific embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0066] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0067] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0068] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the two-branch human behavior prediction method based on optical flow and graphs provided in an embodiment of the present invention. Figure 2 The flowchart illustrates a two-branch human behavior prediction method based on optical flow and graphs, provided in a specific embodiment of the present invention. This two-branch human behavior prediction method based on optical flow and graphs is applied in a server and is executed by application software installed on the server.

[0069] like Figure 2 As shown, the two-branch human behavior prediction method based on optical flow and graph includes the following steps: S10-S30.

[0070] S10. Acquire image data within the detection area.

[0071] Use surveillance equipment, such as surveillance cameras, to monitor and capture video data (image data) of the area to be detected. Any commercially available surveillance equipment is acceptable; this application does not impose any restrictions on its use.

[0072] S20. Perform frame segmentation on the image data to obtain multiple static images.

[0073] The detected image data is sliced ​​into individual RGB static images, which are then used as input data for the human behavior prediction model.

[0074] S30. Input multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior prediction results.

[0075] In one embodiment, step S30 specifically includes the following steps: S301-S304.

[0076] S301. Input multiple static images as an image sequence into the tracking model for processing to obtain a person image with an ID index.

[0077] In this embodiment, the tracking model used is the BoT-SORT model, which employs the YOLOv7 model as the pedestrian detector. To reduce ID switching caused by person occlusion, it not only uses human features as the matching criterion but also incorporates optical flow features from consecutive frames. The human features and optical flow features derived from the pedestrian detector are fused together to determine whether the pedestrian indices match between consecutive frames, adding auxiliary information. The optical flow feature extraction here primarily uses the dense optical flow extraction method Farneback, which provides richer reference information for the model's prediction of human index IDs, making it particularly suitable for fast-moving people and improving the model's target tracking accuracy.

[0078] In one embodiment, step S301 specifically includes the following steps: S3011-S3012.

[0079] S3011. Input multiple static images as an image sequence into the tracking model to detect different human target boxes.

[0080] In this embodiment, the obtained multiple frames of static images (frame video stream) are input into the tracking model as an image sequence, which can detect different human target boxes. These target boxes are identified by their image position coordinates (x, y, z). min ,y min (x) max ,y max The coordinates are represented by the coordinates of the target bounding box relative to the top left corner of the image and the coordinates of the target bounding box relative to the bottom right corner of the image.

[0081] S3012. Assign an index id to the different human target boxes detected to obtain human images with id indexes.

[0082] In this embodiment, the detected human target bounding boxes are assigned index IDs. The same index ID represents the same person, and different people are tracked in real time based on different person ID indices. It should be noted that the input images of the optical flow model and the graph model are single-person images with IDs, which are cropped from the original image (i.e., the image data detected by the monitoring equipment is cut into frames of RGB static images) based on the target bounding boxes detected by the tracking model.

[0083] S302. Input the image of the person with the id index into the optical flow model for processing to obtain a two-dimensional optical flow feature map.

[0084] In one embodiment, step S302 specifically includes the following steps: S3021-S30292.

[0085] S3021. Features are extracted from two consecutive frames of character images with id indices using dilated convolution kernels, and then processed through ReLU activation layer and MaxPooling layer to obtain the first feature map and the second feature map.

[0086] In this embodiment, four dilated convolutional kernels with an expansion ratio of 2 and a size of 3x3 are used to extract features from the two consecutive frames of character images I1I2 with id indexes output by the tracking model. Then, after passing through a ReLU activation layer and a MaxPooling layer, feature maps F1F2 are obtained.

[0087] S3022. Perform full-pixel correlation calculation on the first feature map and the second feature map to obtain the correlation feature map.

[0088] In this embodiment, the optical flow model reuses the output optical flow for each iteration according to a set number of iterations N. i (i∈N), perform a warp operation on F2. Therefore, in the 0th iteration, without any usable optical flow information, directly perform a full-pixel correlation calculation operation on F1F2 to obtain the feature map F. match .

[0089] S3023, Construct the optical flow information for the 0th cycle.

[0090] In one embodiment, step S3023 specifically includes the following steps: S30231-S30232.

[0091] S30231. Merge the first feature map, the second feature map, and the correlation feature map to obtain a merged feature map.

[0092] S30232. Input the merged feature map into the encoding network for processing to obtain the optical flow information of the 0th cycle.

[0093] For S30231 and S30232, in this embodiment, during the 0th cycle, since there is no usable optical flow Flow0, F1F2 and F are directly used. match The merging process yields a new feature map F. 0_concat (The concat result representing the 0th iteration) is input into an encoding network consisting of an encoder and a decoder for optical flow estimation. The encoding network comprises 5 Convolution+BatchNorm+ReLU layers, 2 residual layers, and 3 MaxPooling layers. The decoding layer consists of 4 deconvolution layers and 3 upsampling operations, which also incorporate low-level feature information from the encoding layer. The upsampling method used is nearest neighbor interpolation. After passing through the decoding layer, the optical flow output Flow0 for the 0th iteration is obtained.

[0094] S3024. Use the optical flow information from the 0th cycle to perform a warp operation on the second feature map to obtain the warp result from the 1st cycle.

[0095] S3025. The results of processing the two consecutive frames of character images with id indexes using AsymOFMM with occlusion mask are merged with the Warp results of the first loop to obtain the first merged feature.

[0096] S3026. The result obtained by merging the first merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the first cycle.

[0097] S3027. Use the optical flow information from the first cycle to perform a warp operation on the second feature map to obtain the warp result for the second cycle.

[0098] S3028. The results of processing the two consecutive frames of character images with id indexes using AsymOFMM with occlusion mask are merged with the Warp results of the second loop to obtain the second merged feature.

[0099] S3029. The result obtained by merging the second merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the second cycle.

[0100] S30291. After completing all loops according to the set number of loops, the optical flow information of the first stage is obtained.

[0101] S30292. Input the optical flow information of the first stage into ConvGRU for processing to obtain the final two-dimensional optical flow feature map.

[0102] For S3024-S30292, in this embodiment, Flow0 is not directly output as the result. Instead, Flow0 is used to perform a warp operation on the previous F2 to obtain I. 1_warp This is recorded as the Warp result of the first iteration. Then, using I1I2, an occlusion mask is generated through an Asymmetric Feature Matching Module (AsymOFMM) with an occlusion mask. 1_warp Merge to obtain a MaskedImageI 1_mask According to the previous description, in the 0th cycle, F 0_concat It consists of two consecutive frames of character images F1F2 and F1F2, each with an ID index. match We obtain it, and in the subsequent loops, we will use I. i_mask (i∈N) is merged with the first feature map and the correlation feature map. This achieves iterative utilization of the optical flow from the previous output in each loop. Based on the set number of loops, the optical flow of the first stage is calculated after all loops are completed. out1 This will be output. Subsequently, based on the idea of ​​iterative updates, a ConvGRU recurrent neural network was added to the optical flow model. This outputs the first-stage optical flow as Flow. out1 The I1 input, used as context information, is fed into the ConvGRU, which outputs the final optical flow estimation result, Flow. output Flow output It is a two-dimensional optical flow feature map that includes both velocity and direction characteristics.

[0103] S303. The image of the person with the id index is fused with the two-dimensional optical flow feature map to obtain the fused feature map.

[0104] In one embodiment, step S303 specifically includes the following steps: S3031-S3035.

[0105] S3031. Use the Sobel operator to extract edge features from the image of a person with an id index to obtain an initial edge feature map.

[0106] S3032. Use Laplacian filtering to sharpen and refine the edge feature map of the initial edge feature map.

[0107] S3033. After processing the refined edge feature map through 3x3 convolution, use Average Pooling to perform downsampling to obtain the downsampled feature map.

[0108] S3034. Use bilinear interpolation to adjust the downsampled feature map to an adjusted feature map of the same size as the original image.

[0109] S3035. Perform feature fusion between the two-dimensional optical flow feature map and the adjusted feature map to obtain a fused feature map.

[0110] For S3031-S3035, in this embodiment, the Sobel operator is applied to image I to extract edge features, resulting in feature map I. s Then, a Laplacian filter was used to apply I... s Edge sharpening is performed to obtain a refined edge feature map (I) with more distinct edge information. l . Will I l After a 3x3 convolution, average pooling is used to perform downsampling to obtain F. a Then, bilinear interpolation was used to adjust F. a Adjust the feature map F to the same size as the original image. b The two-dimensional optical flow feature map output from the optical flow model is compared with F... b Feature fusion is performed to obtain a new feature map. Preprocessing and optical flow feature fusion are performed on each frame of a time-frame sequence to obtain the feature sequence F1, F2…F… N This refers to the fusion of feature maps, which are the inputs to the graph model.

[0111] S304. Input the fused feature map into the graph model for processing to obtain the human behavior prediction results.

[0112] In this embodiment, the graph model used is the Feature-GCN model. The Feature-GCN model converts the sequence of feature maps into a graph representation and inputs it into a graph convolutional network and a temporal convolutional network to extract features of different dimensions, thereby predicting human behavior categories.

[0113] In one embodiment, step S304 specifically includes the following steps: S3041-S3049.

[0114] S3041. The fused feature map corresponding to each frame is stitched together with the original image to obtain a stitched image.

[0115] S3042. Divide the mosaic into three different PatchSize images.

[0116] S3043. Divide the mosaic into a corresponding number of equal-sized Patches according to the Patch Size to obtain 3 Patches blocks.

[0117] S3044. Perform feature transformation on the three Patches blocks to obtain three sets of Graph nodes. S3045. Use a perceptual hashing algorithm to calculate the similarity of the three Patches blocks based on the RGB information in the stitched graph, to construct a dynamic adjacency matrix for the three Patches blocks.

[0118] S3046. Input the dynamic adjacency matrix of the three Patches blocks into the stacked block of the 12-layer graph convolutional network and the temporal convolutional network for node learning and feature updating. Then, summarize the features through the Global Pooling layer to obtain a 512-dimensional feature vector for each sequence.

[0119] S3047. Input the 512-dimensional feature vector of each sequence into a fully connected layer for processing to generate a preliminary predicted probability of the human behavior category for each patches block.

[0120] S3048. The preliminary predicted probabilities of human behavior categories for each generated Patches block are processed using a consensus function to obtain consensus predictions.

[0121] S3049. Input the consensus prediction into the SoftMax function for processing to obtain the final human behavior prediction result.

[0122] For S3041-S3049, in this embodiment, the feature map F corresponding to each frame (where F represents any feature map) is concatenated with the original image to obtain F. concat The data is input into the graphical model. Considering the varying sizes of people in the image, F...concat The graph was divided into three different patch sizes (P = 9, 16, 25), and then F was assigned according to the size of P. concat Divide the image into a corresponding number of equal-sized patches, resulting in 3 patch blocks (3 patches per frame, 3N patches for N frames), denoted as P. s ,P m ,P l The letters represent small, middle, and large, respectively, indicating the size of the patch. For example, P=9 means there are only 9 patches in this patch, which is small. After performing feature transformation on each patch, we obtain 3 sets of features X=[x s1 ,…x s9 ],X=[x m1 ,…x m16 ],X=[x l1 ,…x l25 Then, the patches with features are represented as nodes in a graph, resulting in 3 sets of V = [v s1 ,…v s9 ],V=[v m1 ,…v m16 ],V=[v l1 ,…v l25 Because prediction of time frame sequences is required, dynamic temporal information needs to be considered. Therefore, each graph node will have two types of edges: connections to other patches and connections to patches at the same position in preceding and following frames, representing neighboring nodes in the time dimension. Subsequently, a perceptual hash algorithm based on F... concat The RGB information in the dataset is used to calculate the similarity of patches. A fingerprint string is generated for each patch, and then the fingerprints of different patches are compared. Through this process, for each patch, the N patches with the closest similarity are found as its neighbor nodes to construct an adjacency matrix. Based on this, three initial dynamic adjacency matrices A are constructed. s A m A l The adjacency matrices are 9x9, 16x16, and 25x25 in size, respectively. In each subsequent feature update layer, the adjacency matrix is ​​also dynamically updated. Therefore, in the initial layers, the graph model tends to determine neighboring nodes based on low-level features such as color, while in the later layers, the features extracted by the graph model will have stronger semantic information. Based on feature X, node V, adjacency matrix A, and three GraphG...s G m G l It will be constructed. First, G will be... s G m G l After performing batch normalization, then G s G m G l and adjacency matrix A s A m A l The inputs are respectively fed into a stacked block of a 12-layer graph convolutional network and a temporal convolutional network for node learning and feature updating. This means each layer of the main network has two sub-networks, for a total of 12 layers. The number of output channels changes every three layers: 64, 128, 256, and 512. Taking the first layer as an example, the feature map G is obtained after passing through a single graph convolutional network. s_s G s_m G s_l This represents the features in the spatial dimension. In each layer of the GCN network, a 1x1 Conv convolution operation is performed with a stride of 1 and padding of 0. The purpose is to change the channels. Subsequently, based on the principles of GCN, the Einstein summation convention is used; the corresponding feature X and the adjacency matrix A are multiplied to aggregate the neighbor information of the nodes and update the node features. Then, the feature map G... s_s G s_m G s_l Input into a temporal convolutional network to extract temporal dimension features.

[0123] In addition, three 6x1 convolutional kernels were used for each feature map to extract temporal information, with a stride of 1. Then, a Max Pooling layer was applied to obtain the feature map G. T_s G T_m G T_l This represents the features in the temporal dimension. After passing through a 12-layer network, the output of the last temporal convolutional network is aggregated through a global pooling layer to obtain a 512-dimensional feature vector for each sequence. This is then input into a fully connected layer. At this point, each original patch block generates its own preliminary predicted probability for the behavior category. A consensus function is then used to generate a consensus prediction, which is then input into a SoftMax function to obtain a final human behavior prediction result. This mainly includes five human behavior categories: running, walking, jumping, falling, and standing. The graphical model ultimately predicts the pedestrian's actions, obtaining one of these five categories.

[0124] In addition, the loss functions used in the human behavior prediction model include the target detection loss function (mainly the target detection loss function), the optical flow loss function, and the graph model-based behavior recognition loss function, among which:

[0125] For the loss function used in object detection, the YOLOv7 loss function is employed. This loss function mainly consists of classification loss, localization loss, and confidence loss function, i.e.,

[0126] Loss = Loss 分类 +Loss 定位 +Loss 置信度 .

[0127] For the optical flow loss function, all recurrent units will output an optical flow prediction sequence Flow0, Flow1, Flow2, ..., Flow N The total loss is the sum of the losses between the predicted and true values ​​of each loop block output. Here, the optical flow loss function uniformly uses the average endpoint error (EPE) as the training loss function, which represents the average Euclidean distance per pixel between the predicted flow vector and the ground truth.

[0128] For the behavior recognition loss function based on graph models, the cross-entropy loss of classification is used for model training.

[0129] This invention combines instantaneous optical flow information and long-sequence information of human behavior in the temporal dimension, and uses graph convolutional networks in the spatial dimension to effectively learn local body information of pedestrians. By combining spatial and temporal information, the prediction accuracy is improved. Iterative refinement of the output optical flow clearly distinguishes the differences between moving pedestrians and the background. Furthermore, the predicted optical flow direction and velocity information can provide some supervision for the model to predict human behavior categories.

[0130] Figure 3 This is a schematic block diagram of a two-branch human behavior prediction device 100 based on optical flow and graphs provided in an embodiment of the present invention. Corresponding to the above-described two-branch human behavior prediction method based on optical flow and graphs, a specific embodiment of the present invention also provides a two-branch human behavior prediction device 100 based on optical flow and graphs. This two-branch human behavior prediction device 100 based on optical flow and graphs includes units and modules for executing the above-described two-branch human behavior prediction method based on optical flow and graphs, and the device can be configured in a server.

[0131] like Figure 3 As shown, the dual-branch human behavior prediction device 100 based on optical flow and graphs includes:

[0132] The acquisition unit 110 is used to acquire image data within the detection area.

[0133] The frame-slicing processing unit 120 is used to slice the image data to obtain multiple frames of static images.

[0134] The prediction unit 130 is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain the human behavior prediction result.

[0135] In one embodiment, the prediction unit 130 includes...

[0136] The first processing module is used to input multiple frames of static images into the tracking model in the form of an image sequence for processing, so as to obtain a person image with an ID index.

[0137] The second processing module is used to input the image of the person with the ID index into the optical flow model for processing to obtain a two-dimensional optical flow feature map.

[0138] The feature fusion module is used to fuse the image of a person with an ID index with a two-dimensional optical flow feature map to obtain a fused feature map.

[0139] The third processing module is used to input the fused feature map into the graph model for processing to obtain the human behavior prediction results.

[0140] In one embodiment, the first processing module includes:

[0141] The detection submodule is used to input multiple frames of static images as an image sequence into the tracking model in order to detect different human target boxes.

[0142] The submodule is used to assign index IDs to different detected human target boxes in order to obtain human images with ID indexes.

[0143] In one embodiment, the second processing module includes:

[0144] The feature extraction submodule is used to extract features from two consecutive frames of character images with ID indices using dilated convolution kernels, and then process them through ReLU activation layers and Max Pooling layers to obtain the first feature map and the second feature map.

[0145] The correlation processing submodule is used to perform full-pixel correlation processing on the first feature map and the second feature map to obtain a correlation feature map.

[0146] Construct a submodule to build the optical flow information for the 0th loop.

[0147] The first Warp processing submodule uses the optical flow information from the 0th cycle to perform a Warp operation on the second feature map to obtain the Warp result from the 1st cycle.

[0148] The first merging submodule is used to merge the results of the AsymOFMM processing of the two consecutive frames of character images with id indices through occlusion mask with the Warp result of the first loop to obtain the first merged feature.

[0149] The first merging processing submodule is used to input the result obtained by merging the first merging feature with the first feature map and the correlation feature map into the encoding network for processing, so as to obtain the optical flow information of the first cycle.

[0150] The second Warp processing submodule is used to perform Warp operations on the second feature map using the optical flow information from the first cycle, so as to obtain the Warp result for the second cycle.

[0151] The second merging submodule is used to merge the results of the AsymOFMM processing of the two consecutive frames of character images with id indices through occlusion mask with the Warp result of the second loop to obtain the second merged feature.

[0152] The second merging processing submodule is used to input the result obtained by merging the second merging feature with the first feature map and the correlation feature map into the encoding network for processing, so as to obtain the optical flow information of the second cycle.

[0153] The loop processing submodule is used to obtain the optical flow information of the first stage after completing all loops according to the set number of loops.

[0154] The centralized processing submodule is used to input the optical flow information from the first stage into the ConvGRU for processing to obtain the final two-dimensional optical flow feature map.

[0155] In one embodiment, the construction submodule includes:

[0156] The merging submodule is used to merge the first feature map, the second feature map, and the correlation feature map to obtain a merged feature map.

[0157] The encoding submodule is used to input the merged feature map into the encoding network for processing to obtain the optical flow information of the 0th cycle.

[0158] In one embodiment, the feature fusion module includes:

[0159] The edge feature extraction submodule is used to extract edge features from the person image with id index using the Sobel operator to obtain an initial edge feature map.

[0160] The edge sharpening submodule is used to sharpen and refine the initial edge feature map using Laplacian filtering.

[0161] The downsampling processing submodule is used to process the refined edge feature map through 3x3 convolution and then perform downsampling using Average Pooling to obtain a downsampled feature map.

[0162] The resizing submodule is used to resize the downsampled feature map to the same size as the original image using bilinear interpolation.

[0163] The feature fusion submodule is used to fuse the two-dimensional optical flow feature map with the adjusted feature map to obtain a fused feature map.

[0164] In one embodiment, the third processing module includes:

[0165] The stitching submodule is used to stitch the fused feature map corresponding to each frame with the original image to obtain the stitched image.

[0166] The first partitioning submodule is used to divide the stitched image into three different patch sizes.

[0167] The second partitioning submodule is used to divide the mosaic into a corresponding number of equal-sized patches according to the patch size, so as to obtain 3 patch blocks.

[0168] The feature transformation submodule is used to perform feature transformation on the three Patches blocks to obtain three sets of Graph nodes.

[0169] The feature transformation submodule is used to calculate the similarity of the three Patches blocks based on the RGB information in the stitched graph using a perceptual hash algorithm, so as to form a dynamic adjacency matrix of the three Patches blocks.

[0170] The aggregation submodule is used to input the dynamic adjacency matrix of the three Patches blocks into the stacked block of the 12-layer graph convolutional network and the temporal convolutional network for node learning and feature updating, and to aggregate the features through the Global Pooling layer to obtain a 512-dimensional feature vector for each sequence.

[0171] The fully connected processing submodule is used to input the 512-dimensional feature vector of each sequence into the fully connected layer for processing to generate a preliminary predicted probability of the human behavior category for each patches block.

[0172] The consensus prediction submodule is used to process the preliminary predicted probabilities of human behavior categories for each generated Patches block using a consensus function to obtain a consensus prediction.

[0173] The final processing submodule is used to input the consensus prediction into the SoftMax function for processing to obtain the final human behavior prediction result.

[0174] The aforementioned optical flow and graph-based dual-branch human behavior prediction device can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.

[0175] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0176] like Figure 4 As shown, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the two-branch human behavior prediction method based on optical flow and graph as described above.

[0177] The computer device 700 can be a terminal or a server. The computer device 700 includes a processor 720, a memory, and a network interface 750 connected via a system bus 710, wherein the memory may include a non-volatile storage medium 730 and internal memory 740.

[0178] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. When the computer program 732 is executed, it enables the processor 720 to execute any two-branch human behavior prediction method based on optical flow and graphs.

[0179] The processor 720 provides computing and control capabilities to support the operation of the entire computer device 700.

[0180] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute any two-branch human behavior prediction method based on optical flow and graph.

[0181] This network interface 750 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 700 to which the present application is applied. The specific computer device 700 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. The processor 720 is used to run program code stored in memory to implement the following steps:

[0182] A two-branch human behavior prediction method based on optical flow and graphs includes:

[0183] Acquire image data within the detection area;

[0184] The image data is sliced ​​into frames to obtain multiple static images;

[0185] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior prediction results.

[0186] In one embodiment: the step of inputting multiple frames of static images as an image sequence into a human behavior prediction model for processing to obtain human behavior prediction results includes:

[0187] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0188] The image of a person with an ID index is input into the optical flow model for processing to obtain a two-dimensional optical flow feature map;

[0189] The image of a person with an ID index is fused with a two-dimensional optical flow feature map to obtain a fused feature map.

[0190] The fused feature map is input into the graph model for processing to obtain human behavior prediction results.

[0191] In one embodiment: the step of inputting multiple frames of still images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes:

[0192] Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes;

[0193] Different human target bounding boxes are assigned an index ID to obtain human images with ID indexes.

[0194] In one embodiment: the step of inputting the image of a person with an ID index into an optical flow model for processing to obtain a two-dimensional optical flow feature map includes:

[0195] Dilated convolution kernels are used to extract features from two consecutive frames of character images with ID indices, and then processed by ReLU activation layer and Max Pooling layer to obtain the first feature map and the second feature map.

[0196] The first feature map and the second feature map are subjected to full-pixel correlation calculation to obtain the correlation feature map;

[0197] Construct the optical flow information for the 0th cycle;

[0198] The optical flow information from the 0th cycle is used to perform a warp operation on the second feature map to obtain the warp result from the 1st cycle.

[0199] The results of processing the two consecutive frames of character images with ID indices using AsymOFMM with occlusion masking are merged with the Warp results from the first loop to obtain the first merged feature.

[0200] The result obtained by merging the first merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the first cycle;

[0201] The optical flow information from the first cycle is used to perform a warp operation on the second feature map to obtain the warp result from the second cycle.

[0202] The results of processing the two consecutive frames of character images with ID indices using AsymOFMM with occlusion masking are merged with the Warp results from the second loop to obtain the second merged feature.

[0203] The result obtained by merging the second merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the second cycle.

[0204] Based on the set number of loops, the optical flow information of the first stage is obtained after all loops are completed;

[0205] The optical flow information from the first stage is input into ConvGRU for processing to obtain the final two-dimensional optical flow feature map.

[0206] In one embodiment: constructing the optical flow information for the 0th cycle includes:

[0207] The first feature map, the second feature map, and the correlation feature map are merged to obtain a merged feature map;

[0208] The merged feature map is input into the encoding network for processing to obtain the optical flow information for the 0th cycle.

[0209] In one embodiment: the feature fusion of the person image with the ID index and the two-dimensional optical flow feature map to obtain the fused feature map includes:

[0210] The Sobel operator is used to extract edge features from the image of a person with an id index to obtain an initial edge feature map;

[0211] Laplacian filtering is used to sharpen and refine the edge feature map of the initial edge feature map.

[0212] The refined edge feature map is processed by 3x3 convolution and then downsampled using Average Pooling to obtain the downsampled feature map.

[0213] Use bilinear interpolation to adjust the downsampled feature map to an adjusted feature map of the same size as the original image;

[0214] The two-dimensional optical flow feature map and the adjusted feature map are fused to obtain a fused feature map.

[0215] In one embodiment: the step of inputting the fused feature map into a graph model for processing to obtain human behavior prediction results includes:

[0216] The fused feature map corresponding to each frame is stitched together with the original image to obtain the stitched image;

[0217] The mosaic image is divided into three different patch sizes;

[0218] The mosaic is divided into a corresponding number of equal-sized patches based on the patch size, resulting in 3 patch blocks;

[0219] Perform feature transformation on the three Patches blocks to obtain three sets of Graph nodes;

[0220] The perceptual hashing algorithm is used to calculate the similarity of the three Patches blocks based on the RGB information in the stitched graph, so as to form a dynamic adjacency matrix of the three Patches blocks.

[0221] The dynamic adjacency matrix of the three Patches blocks is input into the stacked block of the 12-layer graph convolutional network and the temporal convolutional network for node learning and feature updating. The output of the stacked block of the 12-layer graph convolutional network and the temporal convolutional network is then aggregated through the Global Pooling layer to obtain a 512-dimensional feature vector for each sequence.

[0222] The 512-dimensional feature vector of each sequence is input into a fully connected layer for processing to generate preliminary predicted probabilities of human behavior categories for each patches block.

[0223] The initial predicted probability of the human behavior category for each generated Patches block is processed by a consensus function to obtain a consensus prediction.

[0224] The consensus prediction is fed into the SoftMax function for processing to obtain the final human behavior prediction result.

[0225] It should be understood that in the embodiments of this application, the processor 720 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0226] Those skilled in the art will understand that Figure 4 The structure of the computer device 700 shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0227] In another embodiment of the present invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when executed by a processor, the computer program implements the optical flow and graph-based two-branch human behavior prediction method disclosed in this embodiment of the present invention.

[0228] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0229] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0230] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0231] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0232] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0233] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for human action prediction based on optical flow and graph with dual branches, characterized in that, include: Acquire image data within the detection area; The image data is sliced ​​into frames to obtain multiple static images; Multiple static images are input into the human behavior prediction model as an image sequence for processing to obtain human behavior prediction results. The step of inputting multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior prediction results includes: Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes; The image of a person with an ID index is input into the optical flow model for processing to obtain a two-dimensional optical flow feature map; The image of a person with an ID index is fused with a two-dimensional optical flow feature map to obtain a fused feature map. The fused feature map is input into the graph model for processing to obtain human behavior prediction results; The step of inputting the fused feature map into the graph model for processing to obtain the human behavior prediction result includes: The fused feature map corresponding to each frame is stitched together with the original image to obtain the stitched image; The mosaic image is divided into three different patch sizes; The mosaic is divided into a corresponding number of equal-sized patches based on the patch size, resulting in 3 patch blocks; Perform feature transformation on the three Patches blocks to obtain three sets of Graph nodes; The perceptual hashing algorithm is used to calculate the similarity of the three Patches blocks based on the RGB information in the stitched graph, so as to form a dynamic adjacency matrix of the three Patches blocks. The dynamic adjacency matrix of the three Patches blocks is input into the stacked block of the 12-layer graph convolutional network and the temporal convolutional network for node learning and feature updating. The output of the stacked block of the 12-layer graph convolutional network and the temporal convolutional network is then aggregated through the Global Pooling layer to obtain a 512-dimensional feature vector for each sequence. The 512-dimensional feature vector of each sequence is input into a fully connected layer for processing to generate preliminary predicted probabilities of human behavior categories for each patches block. The initial predicted probability of the human behavior category for each generated Patches block is processed by a consensus function to obtain a consensus prediction. The consensus prediction is fed into the SoftMax function for processing to obtain the final human behavior prediction result.

2. The dual-branch human behavior prediction method based on optical flow and graph according to claim 1, characterized in that, The step of inputting multiple static images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes: Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes; Different human target bounding boxes are assigned an index ID to obtain human images with ID indexes. 3.The dual-branch human behavior prediction method based on optical flow and graph according to claim 1, wherein, The step of inputting the image of a person with an ID index into the optical flow model for processing to obtain a two-dimensional optical flow feature map includes: Dilated convolution kernels are used to extract features from two consecutive frames of character images with ID indices, and then processed by ReLU activation layer and Max Pooling layer to obtain the first feature map and the second feature map. The first feature map and the second feature map are subjected to full-pixel correlation calculation to obtain the correlation feature map; Construct the optical flow information for the 0th cycle; The optical flow information from the 0th cycle is used to perform a warp operation on the second feature map to obtain the warp result from the 1st cycle. The results of processing the two consecutive frames of character images with ID indices using AsymOFMM with occlusion masking are merged with the Warp results from the first loop to obtain the first merged feature. The result obtained by merging the first merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the first cycle; The optical flow information from the first cycle is used to perform a warp operation on the second feature map to obtain the warp result from the second cycle. The results of processing the two consecutive frames of character images with ID indices using AsymOFMM with occlusion masking are merged with the Warp results from the second loop to obtain the second merged feature. The result obtained by merging the second merged feature with the first feature map and the correlation feature map is input into the coding network for processing to obtain the optical flow information of the second cycle. Based on the set number of loops, the optical flow information of the first stage is obtained after all loops are completed; The optical flow information from the first stage is input into ConvGRU for processing to obtain the final two-dimensional optical flow feature map.

4. The dual-branch human behavior prediction method based on optical flow and graph according to claim 3, characterized in that, The construction of the optical flow information for the 0th cycle includes: The first feature map, the second feature map, and the correlation feature map are merged to obtain a merged feature map; The merged feature map is input into the encoding network for processing to obtain the optical flow information for the 0th cycle. 5.The dual-branch human behavior prediction method based on optical flow and graph according to claim 1, characterized in that, The step of fusing the image of a person with an ID index with a two-dimensional optical flow feature map to obtain a fused feature map includes: The Sobel operator is used to extract edge features from the image of a person with an id index to obtain an initial edge feature map; Laplacian filtering is used to sharpen and refine the initial edge feature map. The refined edge feature map is processed by 3x3 convolution and then downsampled using Average Pooling to obtain the downsampled feature map. Use bilinear interpolation to adjust the downsampled feature map to an adjusted feature map of the same size as the original image; The two-dimensional optical flow feature map and the adjusted feature map are fused to obtain a fused feature map.

6. An optical flow and graph based dual-branch human action prediction apparatus, which, in operation, performs the optical flow and graph based dual-branch human action prediction method of any one of claims 1-5, wherein include: The acquisition unit is used to acquire image data within the detection area; The frame-slicing processing unit is used to slice the image data into frames to obtain multiple static images. The prediction unit is used to input multiple frames of static images as an image sequence into the human behavior prediction model for processing, so as to obtain the human behavior prediction results.

7. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the optical flow and graph-based bi-branch human behavior prediction method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor performs the steps of the two-branch human behavior prediction method based on optical flow and graph as described in any one of claims 1 to 5.

Images