Method and device for recognizing two-person interaction behavior based on graph network, equipment and medium

Abstract

Embodiments of the present application disclose a two-person interactive behavior recognition method and device based on a graph network, equipment and a medium, wherein the method comprises: acquiring image data in a detection area; performing frame cutting processing on the image data to obtain multiple frames of static pictures; inputting the multiple frames of static pictures in the form of a picture sequence into a human behavior prediction model for processing to obtain a human behavior recognition result. The present application processes based on a static and dynamic multi-branch graph convolutional neural network, fuses a CBAM attention mechanism, strengthens the motion representation sensitivity of the model to the local joints of the human body in the spatial and temporal dimensions, improves the overall two-person behavior prediction accuracy of the human behavior prediction model, performs more in-depth time dimension feature extraction on single-person actions without losing interactive information, effectively performs real-time prediction, effectively avoids the problem of easy loss of spatial information, ensures the integrity of the fusion of spatial features and temporal features, and improves the recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method, apparatus, device, and medium for recognizing two-person interactive behavior based on graph networks. Background Technology

[0002] With rapid societal development, human behavior prediction is currently a research hotspot and challenge in both industry and academia, and it has significant practical applications. Several methods are currently available for human behavior prediction:

[0003] The first method involves processing the acquired visible light and infrared images to obtain the target area for tracking. Next, it detects whether the target area includes pedestrians. If pedestrians are included, they are tracked, and during tracking, pedestrian edges are detected to obtain the pedestrian identification area from the target area. This pedestrian identification area is then input into a recognition model to obtain the pedestrian behavior recognition result. This patent uses infrared tracking, which requires additional equipment costs. Furthermore, it relies on traditional histogram-based feature extraction to determine pedestrian targets, which is severely affected by lighting conditions and has poor performance. Additionally, this method only recognizes the outline of the person, ignoring spatial-temporal fusion information, resulting in poor recognition performance.

[0004] The second approach involves training a 3D convolutional neural network using multi-task deep learning. This method uses a fixed number of consecutive frames from a background video as input to various human behavioral attributes, and the network is trained to perform the recognition task. However, this method only recognizes human behavior at fixed locations and is limited to a single individual, which has significant limitations. Furthermore, behavior recognition based on 3D convolution is inefficient, and the trained model is heavily influenced by background factors, resulting in poor recognition performance.

[0005] The third method involves acquiring video clips, extracting features and reducing dimensionality from the image frame sequences within those clips, encoding the reduced feature vectors, and then expanding these expanded feature vectors to obtain a predetermined number of expanded feature vectors. These expanded feature vectors and the encoded feature vectors are then input into a three-layer single-layer decoder for decoding. The final decoded feature vector is then input into a single-layer fully connected feedforward network for calculation, yielding multiple predicted values. These predicted values ​​are then input into a logistic regression network to obtain the corresponding predicted probabilities. The category corresponding to the highest probability value is selected as the human action category of the bounding box corresponding to the final decoded feature vector. This method first extracts spatial information and then extracts temporal features using spatial features from different time periods. However, this process easily loses spatial information during temporal feature extraction, making it difficult to achieve a fusion of spatial and temporal features, thus reducing the accuracy of action recognition. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, device, equipment and medium for recognizing two-person interactive behavior based on graph networks.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect is a method for recognizing two-person interaction behavior based on graph networks, including:

[0009] Acquire image data within the detection area;

[0010] The image data is sliced ​​into frames to obtain multiple static images;

[0011] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0012] The further technical solution is as follows: The step of inputting multiple static images as an image sequence into a human behavior prediction model for processing to obtain human behavior recognition results includes:

[0013] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0014] The human image with an ID index is input into the graph model for processing to obtain the human behavior recognition result. The graph model includes a static branch network and two dynamic branch networks with the same network structure.

[0015] The further technical solution is as follows: The step of inputting multiple frames of static images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes:

[0016] Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes;

[0017] The human body bounding box is cropped to obtain a human image with an ID index.

[0018] The further technical solution is as follows: The human image with an ID index is input into a graph model for processing to obtain human behavior recognition results. The graph model includes one static branch network and two dynamic branch networks. The two dynamic branch networks have the same network structure, including:

[0019] The human images with ID indexes are input into the OpenPose model for skeletal keypoint detection to obtain the skeletal keypoint information of each human figure.

[0020] The key skeletal information of the character is input into two dynamic branch networks for processing to obtain the first dynamic feature data and the second dynamic feature data.

[0021] The skeletal key point information of the character is input into a static branch network for processing to obtain static feature data;

[0022] The first dynamic feature data, the second dynamic feature data, and the static feature data are processed through a fully connected layer to obtain the first processed feature, the second processed feature, and the third processed feature.

[0023] The first, second, and third processed features are concatenated to obtain the fused features;

[0024] The results obtained by processing the fused features through the CBAM attention mechanism model are input into SoftMax for classification to obtain human behavior recognition results.

[0025] The further technical solution is as follows: the skeletal key point information of the character is input into two dynamic branch networks for processing to obtain first dynamic feature data and second dynamic feature data, including:

[0026] The diagram is constructed based on the key skeletal information of the character.

[0027] For each skeletal key point in the constructed graph, feature transformation is performed to obtain a first transformed feature set and a second transformed feature set.

[0028] The first and second image features are created based on the connection relationships of the skeletal nodes of the character in the spatial dimension and the changes of the same skeletal nodes in the temporal dimension.

[0029] The first transformation feature group, the second transformation feature group, the first graph feature, and the second graph feature are respectively input into the spatiotemporal convolution modules of the two dynamic branch networks for feature extraction and updating, so as to obtain the first spatial dimension node graph and the second spatial dimension node graph.

[0030] Multiple convolution processes are performed on the skeletal nodes of the character in the two dynamic branch networks to obtain the first time dimension node map and the second time dimension node map.

[0031] The first spatial dimension node map, the second spatial dimension node map, the first time dimension node map, and the second time dimension node map are subjected to feature fusion and addition to obtain the first dynamic feature data and the second dynamic feature data.

[0032] The further technical solution is as follows: the skeletal key point information of the figure is input into a static branch network for processing to obtain static feature data, including:

[0033] The image data is processed by frame extraction to obtain the first frame data, the middle frame data, and the last frame data;

[0034] The first frame data, the middle frame data, and the last frame data are concatenated to obtain the frame sequence;

[0035] The frame sequence is input into the OpenPose model for processing to obtain the skeletal keypoint information of the two individuals in the two-person model;

[0036] Based on the spatial dimension, a first single-image feature and a second single-image feature are created using the skeletal keypoint information of two individuals.

[0037] Connect the first single-image features and the second single-image features with nodes to generate a graph containing key point information of both people;

[0038] Static feature data is obtained by extracting static features and residual information from the graph of key points of two people.

[0039] Secondly, a graph network-based dual-person interaction behavior recognition device includes:

[0040] The acquisition unit is used to acquire image data within the detection area;

[0041] The frame-slicing processing unit performs frame-slicing processing on the image data to obtain multiple frames of static images;

[0042] The recognition unit is used to input multiple frames of static images as an image sequence into the human behavior prediction model for processing, so as to obtain human behavior recognition results.

[0043] The further technical solution is as follows: the identification unit includes:

[0044] The first processing module is used to input multiple frames of static images into the tracking model in the form of an image sequence for processing, so as to obtain a person image with an ID index.

[0045] The second processing module is used to input the human image with ID index into the graph model for processing to obtain the human behavior recognition result. The graph model includes a static branch network and two dynamic branch networks with the same network structure.

[0046] Thirdly, a computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the graph network-based two-person interaction behavior recognition method described above.

[0047] Fourthly, a computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the steps of the graph network-based two-person interaction behavior recognition method described above.

[0048] The advantages of this invention compared to existing technologies are as follows: This invention uses static and dynamic multi-branch graph convolutional neural networks for two-person interaction behavior detection, incorporating the characteristics of the CBAM attention mechanism to enhance the model's sensitivity to the spatial and temporal motion representations of local joints in the human body, thereby improving the overall accuracy of two-person behavior prediction. This human behavior prediction model considers the features of both single-person and two-person scenes simultaneously, extracting more in-depth temporal features from single-person actions without losing interaction information, thus enabling effective real-time prediction.

[0049] This human behavior prediction model does not use a spatial-then-temporal feature extraction method, but extracts spatiotemporal features in parallel. This effectively avoids the problem of easy loss of spatial information, ensures the complete integration of spatial and temporal features, and improves recognition accuracy.

[0050] This human behavior prediction model is based on skeletal key points for behavior recognition and prediction, and is not affected by factors such as background in the image, thus improving robustness and generalization.

[0051] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are described in detail below. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A schematic diagram illustrating an application scenario of the graph network-based two-person interaction behavior recognition method provided in a specific embodiment of the present invention;

[0054] Figure 2 A flowchart illustrating a graph network-based method for recognizing two-person interaction behaviors, provided as a specific embodiment of the present invention;

[0055] Figure 3 A schematic block diagram of a graph network-based dual-person interactive behavior recognition device provided for a specific embodiment of the present invention;

[0056] Figure 4 This is a schematic block diagram of a computer device provided for a specific embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0059] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0060] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0061] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the graph network-based two-person interaction behavior recognition method provided in this embodiment of the invention. Figure 2 The flowchart illustrates a graph network-based method for recognizing two-person interactive behaviors, which is applied to a server and executed by application software installed on the server.

[0062] like Figure 2 As shown, the method for recognizing two-person interaction behavior based on graph networks includes the following steps: S10-S30.

[0063] S10. Acquire image data within the detection area.

[0064] Use surveillance equipment, such as surveillance cameras, to monitor and capture video data (image data) of the area to be detected. Any commercially available surveillance equipment is acceptable; this application does not impose any restrictions on its use.

[0065] S20. Perform frame segmentation on the image data to obtain multiple static images.

[0066] The detected image data is sliced ​​into individual RGB static images, which are then used as input data for the human behavior prediction model.

[0067] S30. Input multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0068] In one embodiment, step S30 specifically includes the following steps: S301-S302.

[0069] S301. Input multiple static images as an image sequence into the tracking model for processing to obtain a person image with an ID index.

[0070] In this embodiment, the tracking model uses the ByteTrack model, which employs the YOLOv6 model for human target detection. This allows the coordinates of the human target bounding box and the confidence information of the human category to be obtained. Furthermore, the corresponding person ID index information can also be obtained. It should be noted that the input image for the graph model is a single-person image with an ID, cropped from the original image based on the target bounding box detected by the tracking model.

[0071] In one embodiment, step S301 specifically includes the following steps: S3011-S3012.

[0072] S3011. Input multiple static images as an image sequence into the tracking model to detect different human target boxes.

[0073] In this embodiment, the obtained multi-frame static images (frame video stream) are input into the tracking model as an image sequence, which can detect different human target boxes. These target boxes are identified by their image position coordinates. The coordinates are used to represent the position of the target bounding box relative to the top left corner of the image and the bottom right corner of the image.

[0074] S3012. Cropping the human body target bounding box to obtain a human image with an id index.

[0075] In this embodiment, the image is cropped based on the target bounding box, resulting in a single-person image with a unique ID.

[0076] S302. Input the human image with id index into the graph model for processing to obtain the human behavior recognition result. The graph model includes a static branch network and two dynamic branch networks with the same network structure.

[0077] In this embodiment, the graph model is Multi-Branch-GCN, which divides the two-person interaction behavior recognition task into two recognition branches: static and dynamic.

[0078] The static branch handles two-person behavior recognition, while the dynamic branch handles single-person behavior recognition. This approach preserves interaction information while ensuring recognition accuracy. Multi-Branch-GCN primarily relies on human skeletal keypoints for behavior recognition. The static branch mainly extracts spatial features from the instantaneous images of two people, without considering continuous motion changes across the time frame sequence. In contrast to the static branch, which only considers a portion of the time frames, the dynamic branch uses a temporal convolutional network to extract skeletal keypoint information from the complete time frame sequence of a single person, preserving complete motion features in the temporal dimension.

[0079] Since the recognition is for two-person actions, the dynamic branch consists of two sub-branches, one for each person. The two branches have identical network structures and perform the same feature extraction operations on different inputs.

[0080] In one embodiment, step S302 specifically includes the following steps: S3021-3026.

[0081] S3021. Input the human image with id index into the OpenPose model to perform skeletal key point detection in order to obtain the skeletal key point information of each human.

[0082] In this embodiment, the skeletal keypoint information for each person is a 3D matrix consisting of 2D coordinates and 1 confidence score, denoted as ( Therefore, the input to each branch comes from the skeletal keypoint information of the person with different ID indices in a frame of image. For an N-frame video, the input data is a 3D matrix of size (3, N, 18), where 3 corresponds to 2D skeletal keypoint coordinate information and a confidence score, N represents that the video has N frames, and 18 represents that OpenPose outputs 18 skeletal keypoints.

[0083] S3022. Input the key skeletal information of the character into two dynamic branch networks for processing to obtain the first dynamic feature data and the second dynamic feature data.

[0084] In one embodiment, step S3022 specifically includes the following steps:

[0085] S30221. Construct a diagram based on the key skeletal information of the character;

[0086] S30222. Perform feature transformation on each skeletal key point in the constructed graph to obtain the first transformed feature group and the second transformed feature group.

[0087] S30223. Create first and second image features based on the connection relationship of the skeletal nodes of the character in the spatial dimension and the change relationship of the same skeletal nodes in the time dimension.

[0088] S30224. Input the first transformation feature group, the second transformation feature group, the first graph feature and the second graph feature into the spatiotemporal convolution module of the two dynamic branch networks respectively for feature extraction and updating, so as to obtain the first spatial dimension node graph and the second spatial dimension node graph.

[0089] S30225. Perform multiple convolutions on the skeletal nodes of the character in the two dynamic branch networks to obtain the first time dimension node map and the second time dimension node map.

[0090] S30226. Perform feature fusion and addition on the first spatial dimension node map, the second spatial dimension node map, the first time dimension node map, and the second time dimension node map to obtain the first dynamic feature data and the second dynamic feature data.

[0091] For S30221-S30226, in this embodiment, two 3D matrices of size (3, N, 18) are input into the corresponding small branches of the two dynamic branch networks to construct graphs. Each small branch constructs a graph with 18 nodes, corresponding to 18 skeletal keypoints. After feature transformation of each node, two sets of features are obtained. The dynamic branching incorporates temporal information, resulting in two types of edges, referred to in this embodiment as S-edges and T-edges. S-edges represent connections in the spatial dimension, while T-edges represent connections in the temporal dimension. S-edges connect skeletal nodes based on the natural connections of human joints, while T-edges connect identical skeletal nodes at different times, reflecting the changes in each node over time. Based on this design approach, the graph... Created. Based on spatial connectivity information, an 18x18 adjacency matrix is ​​generated. Then The features are then fed into the spatiotemporal convolution module TS-Block in the corresponding sub-branch for feature extraction and updating.

[0092] In the dynamic branching, each sub-branch is connected by 10 identical TS-Block layers, updating the input map layer by layer. Within each TS-Block layer, temporal and spatial convolutional networks work in parallel to extract and update features in both the temporal and spatial dimensions. The spatial convolutional network, based on the principles of graph convolutional networks, first uses a 1x1 convolution to perform channel transformation on the input feature map in each layer, and then uses the Einstein summation convention to... besides Perform matrix multiplication. The resulting feature map combines information from all neighboring nodes for each node, completing one layer's update. This yields a node graph in the spatial dimension. .

[0093] In a temporal convolutional network, each layer first uses a 9x1 convolutional kernel with a stride of 1 to convolve all nodes. Then, a Max Pooling layer is applied to obtain the feature map. This represents the output of each sub-branch at this point. Then, for... Two 6x1 convolutional kernels with a stride of 2 are used to convolve all nodes, followed by a Max Pooling layer to obtain the feature map. Finally, let's talk about... Using four 3x1 convolutional kernels with a stride of 4, convolution is performed on all nodes to obtain the feature map. Finally, through a global pooling process, a node graph with a time dimension is obtained. In the time dimension, this frame-skipping method effectively extracts temporal features, both over long and short periods. Within each TS-Block layer, the two sub-branches ultimately combine the obtained data... and Feature fusion and addition are performed, and each branch outputs a new feature map. , subscript This represents the output after the first TS-Block. After 10 TS-Blocks, the output is... ,Right now This is the first dynamic feature data. This is the second dynamic feature data.

[0094] S3023. Input the key skeletal information of the character into the static branch network for processing to obtain static feature data.

[0095] In one embodiment, step S3023 specifically includes the following steps: S30231-S30236.

[0096] S30231. Perform frame extraction processing on the image data to obtain the first frame data, the middle frame data, and the last frame data.

[0097] S30232. The first frame data, the middle frame data and the last frame data are concatenated to obtain a frame sequence.

[0098] S30233. Input the frame sequence into the OpenPose model for processing to obtain the skeletal key point information of the two single people in the two-person model.

[0099] S30234. Based on the spatial dimension, create the first single-image feature and the second single-image feature using the skeletal keypoint information of two individuals.

[0100] S30235. Connect the first single-image features and the second single-image features with nodes to generate a graph containing key point information of both people.

[0101] S30236. Static feature extraction and residual information extraction are performed on the graph of key point information of the two people to obtain static feature data.

[0102] For S30231-S30236, in the static branch network, the image is first processed by frame extraction, extracting the first frame, middle frame, and last frame, and then concatenating them to obtain a new frame sequence. .Will The input to the OpenPose model yields two individual skeletal keypoints. These two keypoints are then concatenated and their coordinates transformed to obtain a 3D matrix of size (3, 3, 36). The first 3 corresponds to the 2D skeletal keypoint coordinates and a confidence score; the second 3 represents the size of the input frame sequence (3); and 36 represents the number of skeletal keypoints for each person (18 keypoints per person). Based on the definition of the S-edge mentioned in the dynamic branching section, two graphs can be created spatially based on the skeletal keypoints of the two individuals. The coordinates of each node in each image correspond to its position in the original image, which is consistent with... The difference lies in the position of the cropped image. To ensure that the input to the subsequent model is a single image, [the following is omitted]. Connect the nodes corresponding to the neck positions to generate a graph containing keypoint information for both individuals. After performing feature transformation on each node, the features are obtained. Based on spatial connectivity information, a 36x36 adjacency matrix is ​​generated. Then , The input is fed into a 10-layer spatial convolutional module to extract static features. The static branch does not contain a temporal convolutional module and uses the same graph convolutional structure as the dynamic branch. Perform node updates. After every two layers of the network, add a residual link to extract residual information. Finally, pass through the 10th layer of the network to obtain... That is, static feature data.

[0103] S3024. The first dynamic feature data, the second dynamic feature data, and the static feature data are processed through a fully connected layer to obtain the first processed feature, the second processed feature, and the third processed feature.

[0104] In this embodiment, Each is obtained through a fully connected layer. .

[0105] S3025. Concatenate the first processing feature, the second processing feature, and the third processing feature to obtain the fused feature.

[0106] In this embodiment, Features are obtained by concatenating .

[0107] S3026. The result obtained by processing the fused features through the CBAM attention mechanism model is input into SoftMax for classification to obtain the human behavior recognition result.

[0108] In this embodiment, The CBAM attention mechanism model is used to input the data into SoftMax for classification to obtain human behavior recognition results. The human behavior recognition results include five categories: handshake, high-five, hug, push, and fight.

[0109] In one embodiment, if pushing and fighting are detected, an alarm can be set to be issued as needed for the scenario.

[0110] In addition, the loss functions used in the human behavior prediction model include the object detection loss function and the graph-based behavior recognition loss function, where:

[0111] The loss function used for object detection is the YOLOv6 loss function, which mainly consists of classification loss, localization loss, and confidence loss function.

[0112] .

[0113] The graph-based behavior recognition loss function uses the classification-based cross-entropy loss function to train the human behavior prediction model.

[0114] This invention utilizes static and dynamic multi-branch graph convolutional neural networks for two-person interactive behavior detection, incorporating the characteristics of the CBAM attention mechanism to enhance the model's sensitivity to the spatial and temporal motion representations of local joints in the human body, thereby improving the overall accuracy of two-person behavior prediction. This human behavior prediction model considers features from both single-person and two-person scenes simultaneously, extracting more in-depth temporal features from single-person actions without losing interaction information, enabling effective real-time prediction. Instead of using a spatial-then-temporal feature extraction method, this model extracts spatiotemporal features in parallel, effectively avoiding the problem of easily lost spatial information and ensuring the complete fusion of spatial and temporal features, thus improving recognition accuracy. Based on skeletal keypoints, this human behavior prediction model is unaffected by background factors in the image, improving robustness and generalization.

[0115] Figure 3 This is a schematic block diagram of a graph network-based two-person interaction behavior recognition device 100 provided in an embodiment of the present invention. Corresponding to the above-described graph network-based two-person interaction behavior recognition method, a specific embodiment of the present invention also provides a graph network-based two-person interaction behavior recognition device 100. This graph network-based two-person interaction behavior recognition device 100 includes units and modules for executing the above-described graph network-based two-person interaction behavior recognition method, and the device can be configured in a server.

[0116] like Figure 3 As shown, the graph network-based dual-person interaction behavior recognition device 100 includes:

[0117] The acquisition unit 110 is used to acquire image data within the detection area.

[0118] The frame-slicing processing unit 120 performs frame-slicing processing on the image data to obtain multiple frames of static images.

[0119] The recognition unit 130 is used to input multiple frames of static images into the human behavior prediction model in the form of an image sequence for processing, so as to obtain human behavior recognition results.

[0120] In one embodiment, the identification unit 130 includes:

[0121] The first processing module is used to input multiple frames of static images into the tracking model in the form of an image sequence for processing, so as to obtain a person image with an ID index.

[0122] The second processing module is used to input the human image with ID index into the graph model for processing to obtain the human behavior recognition result. The graph model includes a static branch network and two dynamic branch networks with the same network structure.

[0123] In one embodiment, the first processing module includes:

[0124] The detection module is used to input multiple static images as an image sequence into the tracking model in order to detect different human target boxes.

[0125] The cropping module is used to crop the human body target bounding box to obtain a human image with an ID index.

[0126] In one embodiment, the second processing module includes:

[0127] The keypoint detection submodule is used to input the human image with ID index into the OpenPose model to detect skeletal keypoints in order to obtain the skeletal keypoint information of each human.

[0128] The first processing submodule is used to input the skeletal key point information of the character into two dynamic branch networks for processing, so as to obtain the first dynamic feature data and the second dynamic feature data.

[0129] In one embodiment, the first processing submodule includes:

[0130] The construction submodule is used to construct the graph based on the key skeletal information of the character.

[0131] The feature transformation submodule is used to perform feature transformation on each skeletal keypoint in the constructed graph to obtain a first transformed feature set and a second transformed feature set.

[0132] The first creation submodule is used to create first and second graph features based on the connection relationship of the skeletal nodes of the character in the spatial dimension and the change relationship of the same skeletal nodes in the time dimension.

[0133] The feature extraction and update submodule is used to input the first transformation feature group, the second transformation feature group, the first graph feature, and the second graph feature into the spatiotemporal convolution modules of the two dynamic branch networks for feature extraction and update, so as to obtain the first spatial dimension node graph and the second spatial dimension node graph.

[0134] The convolution processing submodule is used to perform multiple convolution processes on the skeletal nodes of the character in the two dynamic branch networks to obtain the first time dimension node map and the second time dimension node map.

[0135] The feature fusion submodule is used to perform feature fusion and addition processing on the first spatial dimension node map, the second spatial dimension node map, the first time dimension node map, and the second time dimension node map to obtain the first dynamic feature data and the second dynamic feature data.

[0136] The second processing submodule is used to input the skeletal key point information of the character into the static branch network for processing to obtain static feature data.

[0137] In one embodiment, the second processing submodule includes:

[0138] The frame extraction processing submodule is used to extract frames from image data to obtain the first frame data, the middle frame data, and the last frame data.

[0139] The splicing submodule is used to splice the first frame data, the middle frame data, and the last frame data to obtain a frame sequence.

[0140] The OpenPose model processing submodule is used to input frame sequences into the OpenPose model for processing to obtain the skeletal keypoint information of the two individuals in a two-person model.

[0141] The second creation submodule is used to create a first single-image feature and a second single-image feature based on the skeletal keypoint information of two individuals in terms of spatial dimension.

[0142] The node connection submodule is used to connect the first single-graph feature and the second single-graph feature with nodes to generate a graph containing key point information of both people.

[0143] The extraction submodule is used to extract static features and residual information from the graph of key points of two people to obtain static feature data.

[0144] The fully connected layer processing submodule is used to process the first dynamic feature data, the second dynamic feature data, and the static feature data through the fully connected layer to obtain the first processed feature, the second processed feature, and the third processed feature.

[0145] The fusion submodule is used to concatenate the first, second, and third processed features to obtain the fused features.

[0146] The classification and recognition submodule is used to input the results of the fused features processed by the CBAM attention mechanism model into SoftMax for classification in order to obtain human behavior recognition results.

[0147] The aforementioned graph network-based dual-person interactive behavior recognition device can be implemented as a computer program, which can, for example, Figure 4 It runs on the computer device shown.

[0148] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 700 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0149] like Figure 4 As shown, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the graph network-based two-person interaction behavior recognition method described above.

[0150] The computer device 700 can be a terminal or a server. The computer device 700 includes a processor 720, a memory, and a network interface 750 connected via a system bus 710, wherein the memory may include a non-volatile storage medium 730 and internal memory 740.

[0151] The non-volatile storage medium 730 can store an operating system 731 and a computer program 732. When the computer program 732 is executed, it enables the processor 720 to execute any graph network-based method for recognizing two-person interactive behaviors.

[0152] The processor 720 provides computing and control capabilities to support the operation of the entire computer device 700.

[0153] The internal memory 740 provides an environment for the operation of the computer program 732 in the non-volatile storage medium 730. When the computer program 732 is executed by the processor 720, the processor 720 can execute any graph network-based method for recognizing two-person interactive behavior.

[0154] This network interface 750 is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 700 to which the present application is applied. The specific computer device 700 may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. The processor 720 is used to run program code stored in memory to implement the following steps:

[0155] A graph network-based method for recognizing two-person interaction behaviors includes:

[0156] Acquire image data within the detection area;

[0157] The image data is sliced ​​into frames to obtain multiple static images;

[0158] Multiple static images are input as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results.

[0159] In one embodiment: the step of inputting multiple frames of static images as an image sequence into a human behavior prediction model for processing to obtain human behavior recognition results includes:

[0160] Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes;

[0161] The human image with an ID index is input into the graph model for processing to obtain the human behavior recognition result. The graph model includes a static branch network and two dynamic branch networks with the same network structure.

[0162] In one embodiment: the step of inputting multiple frames of still images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes:

[0163] Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes;

[0164] The human body bounding box is cropped to obtain a human image with an ID index.

[0165] In one embodiment: the human image with an ID index is input into a graph model for processing to obtain human behavior recognition results. The graph model includes one static branch network and two dynamic branch networks, the two dynamic branch networks having the same network structure, including:

[0166] The human images with ID indexes are input into the OpenPose model for skeletal keypoint detection to obtain the skeletal keypoint information of each human figure.

[0167] The key skeletal information of the character is input into two dynamic branch networks for processing to obtain the first dynamic feature data and the second dynamic feature data.

[0168] The skeletal key point information of the character is input into a static branch network for processing to obtain static feature data;

[0169] The first dynamic feature data, the second dynamic feature data, and the static feature data are processed through a fully connected layer to obtain the first processed feature, the second processed feature, and the third processed feature.

[0170] The first, second, and third processed features are concatenated to obtain the fused features;

[0171] The results obtained by processing the fused features through the CBAM attention mechanism model are input into SoftMax for classification to obtain human behavior recognition results.

[0172] In one embodiment: the step of inputting the skeletal key point information of a person into two dynamic branch networks for processing to obtain first dynamic feature data and second dynamic feature data includes:

[0173] The diagram is constructed based on the key skeletal information of the character.

[0174] For each skeletal key point in the constructed graph, feature transformation is performed to obtain a first transformed feature set and a second transformed feature set.

[0175] The first and second image features are created based on the connection relationships of the skeletal nodes of the character in the spatial dimension and the changes of the same skeletal nodes in the temporal dimension.

[0176] The first transformation feature group, the second transformation feature group, the first graph feature, and the second graph feature are respectively input into the spatiotemporal convolution modules of the two dynamic branch networks for feature extraction and updating, so as to obtain the first spatial dimension node graph and the second spatial dimension node graph.

[0177] Multiple convolution processes are performed on the skeletal nodes of the character in the two dynamic branch networks to obtain the first time dimension node map and the second time dimension node map.

[0178] The first spatial dimension node map, the second spatial dimension node map, the first time dimension node map, and the second time dimension node map are subjected to feature fusion and addition to obtain the first dynamic feature data and the second dynamic feature data.

[0179] In one embodiment: the step of inputting the skeletal key point information of a person into a static branch network for processing to obtain static feature data includes:

[0180] The image data is processed by frame extraction to obtain the first frame data, the middle frame data, and the last frame data;

[0181] The first frame data, the middle frame data, and the last frame data are concatenated to obtain the frame sequence;

[0182] The frame sequence is input into the OpenPose model for processing to obtain the skeletal keypoint information of the two individuals in the two-person model;

[0183] Based on the spatial dimension, a first single-image feature and a second single-image feature are created using the skeletal keypoint information of two individuals.

[0184] Connect the first single-image features and the second single-image features with nodes to generate a graph containing key point information of both people;

[0185] Static feature data is obtained by extracting static features and residual information from the graph of key points of two people.

[0186] It should be understood that, in the embodiments of this application, the processor 720 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0187] Those skilled in the art will understand that Figure 4 The structure of the computer device 700 shown does not constitute a limitation on the computer device 700, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0188] In another embodiment of the present invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when executed by a processor, the computer program implements the graph network-based two-person interaction behavior recognition method disclosed in this embodiment of the present invention.

[0189] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0190] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0191] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0192] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0193] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0194] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for recognizing two-person interaction behavior based on graph networks, characterized in that, include: Acquire image data within the detection area; The image data is sliced ​​into frames to obtain multiple static images; Multiple static images are input into the human behavior prediction model as an image sequence for processing to obtain human behavior recognition results. The step of inputting multiple static images as an image sequence into the human behavior prediction model for processing to obtain human behavior recognition results includes: Multiple static images are input into the tracking model as an image sequence for processing to obtain human images with ID indexes; The human image with ID index is input into the graph model for processing to obtain the human behavior recognition result. The graph model includes a static branch network and two dynamic branch networks with the same network structure. The process involves inputting a person image with an ID index into a graph model for processing to obtain human behavior recognition results. The graph model includes one static branch network and two dynamic branch networks. The two dynamic branch networks have the same network structure, including: The human images with ID indexes are input into the OpenPose model for skeletal keypoint detection to obtain the skeletal keypoint information of each human figure. The key skeletal information of the character is input into two dynamic branch networks for processing to obtain the first dynamic feature data and the second dynamic feature data. The skeletal key point information of the character is input into a static branch network for processing to obtain static feature data; The first dynamic feature data, the second dynamic feature data, and the static feature data are processed through a fully connected layer to obtain the first processed feature, the second processed feature, and the third processed feature. The first, second, and third processed features are concatenated to obtain the fused features; The results obtained by processing the fused features through the CBAM attention mechanism model are input into SoftMax for classification to obtain human behavior recognition results; The process of inputting the skeletal key point information of the character into two dynamic branch networks for processing to obtain first dynamic feature data and second dynamic feature data includes: The diagram is constructed based on the key skeletal information of the character. For each skeletal key point in the constructed graph, feature transformation is performed to obtain a first transformed feature set and a second transformed feature set. The first and second image features are created based on the connection relationships of the skeletal nodes of the character in the spatial dimension and the changes of the same skeletal nodes in the temporal dimension. The first transformation feature group, the second transformation feature group, the first graph feature, and the second graph feature are respectively input into the spatiotemporal convolution modules of the two dynamic branch networks for feature extraction and updating, so as to obtain the first spatial dimension node graph and the second spatial dimension node graph. Multiple convolution processes are performed on the skeletal nodes of the character in the two dynamic branch networks to obtain the first time dimension node map and the second time dimension node map. The first spatial dimension node map, the second spatial dimension node map, the first time dimension node map, and the second time dimension node map are subjected to feature fusion and addition to obtain the first dynamic feature data and the second dynamic feature data.

2. The method for recognizing two-person interaction behavior based on graph networks according to claim 1, characterized in that, The step of inputting multiple static images as an image sequence into the tracking model for processing to obtain a person image with an ID index includes: Multiple static images are input into the tracking model as an image sequence to detect different human target bounding boxes; The human body bounding box is cropped to obtain a human image with an ID index.

3. The method for recognizing two-person interaction behavior based on graph networks according to claim 1, characterized in that, The process of inputting the skeletal key point information of the character into a static branch network for processing to obtain static feature data includes: The image data is processed by frame extraction to obtain the first frame data, the middle frame data, and the last frame data; The first frame data, the middle frame data, and the last frame data are concatenated to obtain the frame sequence; The frame sequence is input into the OpenPose model for processing to obtain the skeletal keypoint information of the two individuals in the two-person model; Based on the spatial dimension, a first single-image feature and a second single-image feature are created using the skeletal keypoint information of two individuals. Connect the first single-image features and the second single-image features with nodes to generate a graph containing key point information of both people; Static feature data is obtained by extracting static features and residual information from the graph of key points of two people.

4. A graph network-based dual-person interaction behavior recognition device, which, when operating, executes the graph network-based dual-person interaction behavior recognition method according to any one of claims 1-3, characterized in that, include: The acquisition unit is used to acquire image data within the detection area; The frame-slicing processing unit performs frame-slicing processing on the image data to obtain multiple frames of static images; The recognition unit is used to input multiple frames of static images as an image sequence into the human behavior prediction model for processing, so as to obtain human behavior recognition results.

5. The graph network-based dual-person interaction behavior recognition device according to claim 4, characterized in that, The identification unit includes: The first processing module is used to input multiple frames of static images into the tracking model in the form of an image sequence for processing, so as to obtain a person image with an ID index. The second processing module is used to input the human image with ID index into the graph model for processing to obtain the human behavior recognition result. The graph model includes a static branch network and two dynamic branch networks with the same network structure.

6. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the graph network-based two-person interaction behavior recognition method as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the processor performs the steps of the graph network-based two-person interaction behavior recognition method as described in any one of claims 1 to 3.

Images