Human-robot team formation intention recognition method and system based on skeleton trajectory and graph convolution

Abstract

The application provides a human-machine marshalling intention recognition method and system based on skeleton trajectory and graph convolution, and comprises the following steps: acquiring real-time personnel behavior video stream data in a human-machine marshalling scene; extracting human skeleton sequence information from the real-time personnel behavior video stream data, and constructing a personnel skeleton space-time feature sequence graph; performing joint-limb-behavior three-level feature coding on the personnel skeleton space-time feature sequence graph by using a hierarchical graph convolution network, and outputting a predicted personnel behavior intention.

Description

Technical Field

[0001] This invention belongs to the field of human behavior intention recognition technology, and particularly relates to a human-computer grouping intention recognition method and system based on skeletal trajectory and graph convolution. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the widespread application of human-robot teams in collaborative operations, urban search and rescue, target capture, and environmental monitoring, the real-time response and collaboration capabilities of robots in complex and dynamic tasks have become a key technological requirement. Especially in high-pressure and complex scenarios involving emergency tasks, robot systems need to quickly respond to the needs of interacting personnel and flexibly adjust their strategies, which places higher demands on robots to autonomously recognize the intentions of human activities.

[0004] Vision-based intent recognition is one of the core technologies for achieving efficient human-computer collaboration. Existing research constructs recognition models using multimodal features (such as RGB sequences, optical flow, audio signals, and skeleton data). Among these, skeleton data-based methods have become mainstream due to their advantages such as lightweight design and resistance to background interference. Graph Convolutional Networks (GCNs) have become a typical solution by modeling the topological relationships of skeletons to extract spatiotemporal dynamic features. However, existing methods face limitations in human-computer collaboration scenarios, including response latency, insufficient accuracy in multi-view action recognition, and robustness to dynamic interference, which restricts their practical application in real-time collaborative tasks. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a human-robot grouping behavior intention recognition method based on spatiotemporal modeling of skeletal trajectory and augmented graph convolutional networks. By optimizing the skeletal trajectory extraction algorithm, designing a multi-node information aggregation mechanism, and integrating a channel attention module, combined with multi-angle data augmentation strategies, the feature extraction effect and recognition speed are improved. This enables robots to quickly and accurately recognize human behavior intentions and adapt to changes in perspective under different angles, thereby improving collaborative efficiency.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] Firstly, a method for human-computer grouping intent recognition based on skeletal trajectory and graph convolution is disclosed, including:

[0008] Acquire real-time video stream data of human activities in human-machine grouping scenarios;

[0009] Human skeleton sequence information is extracted from real-time video stream data of human activities to generate a 2D skeleton sequence image of the human body.

[0010] A graph convolutional network model is used to process 2D skeleton sequence graphs of people and output the predicted behavioral intentions of people.

[0011] As a further technical solution, extracting human skeletal sequence information from real-time human activity video stream data specifically includes:

[0012] A lightweight MobileNet V1 was used as the backbone network to extract feature maps from real-time video stream data of people's activities.

[0013] Predict skeletal key points based on extracted feature maps;

[0014] A partial association field is generated based on the extracted feature map to describe the spatial connection relationship between various key points;

[0015] Based on the predicted skeletal key points and some related fields, a 2D skeleton sequence diagram of the person is assembled.

[0016] As a further technical solution, feature map extraction specifically includes:

[0017] The lightweight MobileNet V1 performs convolution operations independently on the input real-time people activity video stream data on each input channel, and then convolves the output of all channels;

[0018] After processing by MobileNet V1, the generated feature map contains high-level features related to human pose in the image.

[0019] As a further technical solution, a graph convolutional network model is used to process the 2D skeleton sequence of personnel to output the predicted activity intentions of the personnel, specifically including:

[0020] Batch normalization processing was performed on the 2D skeleton sequence images of personnel to obtain normalized standard data;

[0021] Spatiotemporal graph convolution operations are used to extract high-level features in the skeletal pose space and time dimension from normalized standard data.

[0022] To extract high-level features, a global average pooling layer and a fully connected layer are called, and then a Softmax classifier is used to obtain the corresponding action classification.

[0023] As a further technical solution, spatiotemporal graph convolution operations are used on the normalized standard data to extract high-level features in the skeletal pose space and time dimension, specifically including:

[0024] The spatiotemporal graph convolutional network model includes: ST-GCN unit; the ST-GCN unit alternately uses GCN graph convolutional network, spatial extension module SEM, TCN temporal convolutional network and channel attention module SENet, and extracts features from both spatial and temporal dimensions;

[0025] GCN is used to process the spatial features of skeleton data, that is, the spatial relationship between each skeleton point. Through graph convolution operation, GCN can fuse the spatial information of each skeleton point with the information of its neighboring points, thereby obtaining richer spatial features.

[0026] The convolution operation in the Spatial Extension Module (SEM) not only handles the local relationships between adjacent nodes, but also captures the interaction of the first non-adjacent nodes by expanding the skeleton graph, thereby enhancing the model's ability to capture long-range dependencies between skeleton points.

[0027] TCN effectively captures dynamic dependencies in time series data through convolution operations, and is used to handle the temporal dimension of skeleton data, i.e. how the model understands the changes in the skeleton at different time steps.

[0028] As a further technical solution, the graph convolutional network model introduces SENet as a channel attention mechanism to optimize the network's attention to key information;

[0029] SENet enhances the model's focus on key information and ignores irrelevant features by adaptively learning the importance of each channel.

[0030] Specifically, SENet compresses each channel using global average pooling (GAP) and generates a global description from the feature map of each channel through pooling operations.

[0031] As a further technical solution, after extracting high-level features, a global average pooling layer and a fully connected layer are called, and a Softmax classifier is used to obtain the corresponding action classification, specifically including:

[0032] After passing through the spatiotemporal graph convolutional layer, the generated feature map is input into the global average pooling (GAP) layer. The GAP layer operation averages the features at each time step and compresses the feature vector into a fixed-length representation.

[0033] Next, the features output by GAP are fed into a fully connected layer, which performs a linear transformation through the weight matrix and bias terms to map the input features to a higher-dimensional space and prepare appropriate outputs for the classification task.

[0034] Finally, the Softmax classifier is used to transform the feature vector output by the fully connected layer into a probability distribution of categories. Softmax returns the category with the highest probability as the predicted intent category.

[0035] Secondly, a human-computer grouping intent recognition system based on skeletal trajectory and graph convolution is disclosed, including:

[0036] The data acquisition module is configured to acquire real-time video stream data of human activities in human-machine grouping scenarios;

[0037] The 2D skeleton sequence diagram generation module is configured to extract human skeleton sequence information from real-time human activity video stream data and generate a 2D skeleton sequence diagram of the personnel.

[0038] The personnel activity intention prediction module is configured to: use a graph convolutional network model to process the 2D skeleton sequence graph of personnel and output the predicted personnel activity intention.

[0039] The above one or more technical solutions have the following beneficial effects:

[0040] This invention significantly improves the inference speed of a lightweight human pose estimation model while maintaining accuracy. This optimization enables the model to run in real-time in resource-constrained dynamic environments, meeting the needs of rapidly changing behavior recognition. Particularly on edge devices, the optimized model can efficiently handle complex action recognition tasks, thereby enhancing the practical application value of human pose estimation.

[0041] The Spatial Extension Module (SEM) designed in this invention enhances the feature extraction capability of spatial skeleton sequences, especially in complex and rapidly changing dynamic environments. By introducing information from non-adjacent nodes, this module enables the network to not only focus on skeleton points in the local neighborhood but also effectively capture spatial dependencies between distant nodes. This design improves the recognition accuracy of complex activities and effectively avoids the information loss problem in traditional methods, thereby greatly enhancing the accurate recognition of complex activity patterns.

[0042] This invention significantly improves the model's attention to key information by introducing the SENet module as a channel attention mechanism. This mechanism adaptively adjusts the weights of each channel, enabling the model to more accurately identify and extract important skeleton features, especially when dealing with complex dynamic activities. By optimizing the transmission of channel information, the channel attention mechanism enhances the model's accuracy and robustness, significantly improving the accuracy and reliability of behavioral intent recognition.

[0043] This invention addresses the perspective sensitivity issue in human-computer collaboration scenarios by proposing a robust training framework based on a multi-angle data augmentation strategy. By constructing a multi-view data space incorporating geometric rotation (±45°), dynamic lighting, and complex background perturbations, it systematically solves the generalization capability deficiencies caused by traditional single-view training data. This strategy integrates saliency feature extraction of limb movements with cross-view feature alignment constraints, enabling the model to maintain consistency in action representation under diverse observation conditions such as frontal / side views, thereby reducing recognition errors caused by angle changes.

[0044] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0046] Figure 1 This is a flowchart of the method of the present invention;

[0047] Figure 2 This is a schematic diagram of the network structure of the lightweight OpenPose according to the present invention;

[0048] Figure 3 This is a schematic diagram of the optimized ST-GCN network structure according to the present invention;

[0049] Figure 4 This is a schematic diagram of the structural design principle of the spatial expansion module SEM of the present invention;

[0050] Figure 5 This is a schematic diagram of the network structure of SENet based on the channel attention mechanism of this invention.

[0051] Figure 6 This is the robot experimental platform used in this invention. Detailed Implementation

[0052] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0053] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0054] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0055] Due to the high degree of uncertainty in human-robot teaming scenarios under complex dynamic environments, interactions between humans and robots in practice are often extremely complex. Therefore, it is necessary to improve the adaptability of algorithm models in such scenarios, making the recognition of human behavioral intentions more accurate and practical. Based on this, this embodiment proposes a method for recognizing human behavioral intentions in complex environments. This method can accurately identify human behavioral intentions in real time and provide precise operational instructions to the robot, thereby achieving efficient human-robot collaboration. The invention will be described in detail below with reference to specific embodiments.

[0056] Example 1

[0057] This embodiment discloses a human-computer grouping intent recognition method based on skeletal trajectory and graph convolution, including:

[0058] S1: Captures real-time video streams of people's activities in complex environments using the RealSense D435i camera.

[0059] Step 101: Use a RealSense D435i camera to capture a color video stream of the scene where the device is located at 640×480 resolution and 30 frames per second.

[0060] Step 102: The acquired video stream data is transmitted in real time to a processing system based on the NVIDIA Jetson AGX Orin platform and running the Ubuntu 20.04 operating system, so that it can be used for subsequent pose estimation and behavior recognition.

[0061] Step 103: Determine the pixel coordinate system of the video stream based on its resolution. This provides a foundation for the subsequent generation and standardization of the 2D skeleton sequence map, ensuring the accuracy of bone point localization.

[0062] S2: In this embodiment, the optimized lightweight OpenPose human pose estimation model is used to process the video stream data captured in step S1, perform bottom-up skeletal point prediction on the person, extract the two-dimensional coordinates and confidence of the skeletal points, and generate a 2D skeleton sequence map of the person.

[0063] In this step, the lightweight OpenPose model is trained using the COCO dataset released by the Microsoft team. This dataset contains over 200,000 labeled images, covering multi-angle human pose, object localization, and semantic segmentation information. The human keypoint annotation data is specifically used to improve the accuracy of skeletal joint detection. After training, the model outputs a neural network weight file (.pth) under the PyTorch framework. This PyTorch model undergoes structured quantization processing and is then converted to the TensorRT inference engine (.trt) via the ONNX intermediate format, providing high real-time performance assurance for subsequent deployment on the NVIDIA Jetson AGX Orin embedded platform.

[0064] This network is based on MobileNet v1 as its backbone and uses dilated convolution to improve the efficiency of feature extraction while reducing computational complexity. During training, a preliminary two-stage network (including an initialization stage and a refinement stage) is adopted, and the network architecture and training process are continuously optimized to ultimately obtain a model that can run efficiently on edge devices.

[0065] Real-time human pose estimation is performed using the obtained TensorRT model files. The TensorRT engine accelerates the inference process, significantly improving inference speed and efficiency. Specifically, the model first generates preliminary keypoint heatmaps through a keypoint prediction branch. This branch optimizes the feature maps using convolutional layers, concentrating the probability values ​​of each skeletal point in its corresponding region. A softmax activation function is then used to form the probability distribution of keypoints, thereby locating key areas such as the nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles. Simultaneously, the model generates vector fields describing the spatial connectivity between joints through a part affinity fields (PAFs) prediction branch. This branch uses convolutional layers to efficiently capture the orientation and affinity strength between joints, providing a spatial dependency basis for subsequent skeletal point connections.

[0066] In the refinement stage, the model iteratively optimizes the initially generated keypoint heatmap and PAFs. Through multi-scale feature fusion and confidence weighting, redundant detections are eliminated and mismatched keypoint connections are corrected. Finally, combining the optimized heatmap and PAFs, the model achieves globally optimal keypoint matching using the Hungarian algorithm, assembling a complete and accurate 2D human skeleton sequence map, providing high-precision data support for subsequent behavioral intent recognition.

[0067] Specifically, such as Figure 2As shown, the steps for extracting human skeleton points using the lightweight OpenPose human pose estimation model include:

[0068] S2.1: A lightweight MobileNet V1 is used as the backbone network for feature map extraction. This embodiment significantly reduces model complexity by introducing depthwise separable convolution instead of traditional convolution. Specifically, depthwise separable convolution consists of two parts: depthwise convolution and pointwise convolution. First, depthwise convolution is performed independently on each channel of the input feature map (multi-channel image features after preprocessing such as image normalization and resizing) to extract spatial dimension features. Then, pointwise convolution is used to linearly combine the features between channels to further fuse semantic information. Through this structure, MobileNetV1 significantly reduces the computational cost and parameter count while maintaining feature extraction capabilities. After processing by the backbone network, the generated high-level feature map contains spatial and semantic information related to human pose, providing a robust data foundation for subsequent keypoint prediction.

[0069] S2.2: Predicting Skeletal Keypoints. In the keypoint prediction branch, the convolutional layers within the lightweight OpenPose model further extract and optimize the feature maps output by the backbone network. Multi-level convolutional operations enhance the response features of keypoint regions, concentrating the probability value of each skeletal point at its corresponding node's spatial location. Specifically, the network outputs the confidence level of a keypoint's existence for each pixel location in the feature map and normalizes the confidence level across the entire map using the Softmax activation function, generating a two-dimensional probability distribution map. This probability distribution map characterizes the predicted probability density of keypoints at different locations; for example, the probability peak region represents the predicted joint coordinates (such as elbow, knee, etc.). In subsequent steps, the model uses non-maximum suppression (NMS) or thresholding to extract the coordinates with the highest confidence from the probability distribution map as the final detection result, thereby accurately locating human skeletal keypoints.

[0070] S2.3: Generating Partial Association Fields (PAFs). Parallel to the keypoint prediction branch, the model generates partial association fields (PAFs) describing the spatial connectivity between joints through the PAF prediction branch. Specifically, this branch, based on the backbone network feature map, uses depthwise separable convolutions to efficiently extract directional association features between joints (such as the spatial orientation of the shoulder-elbow connection) and outputs multi-channel PAFs. Each PAF channel corresponds to a predefined pair of joints, and its pixel position's two-dimensional vector and confidence level characterize the geometric connection strength and direction between adjacent joints. For example, the PAF channels for the elbow and wrist will generate high-confidence direction vectors along the path connecting them, providing spatial constraints for subsequent skeletal point connections and ensuring the coherence of the human body's topology.

[0071] S2.4: Connecting Skeletal Points and Assembling the Skeletal Map. Based on the predicted keypoint heatmap and Partial Association Fields (PAFs), the model first performs association matching on predefined joint pairs (such as shoulder-elbow, hip-knee): by integrating the direction vectors in the PAFs along the joint connection path, a geometric consistency score is calculated, and valid connections are filtered in conjunction with the confidence scores of joints in the heatmap. Subsequently, the Hungarian algorithm is used to globally optimize the matching results, eliminating contradictory connections such as limb crossings, ensuring that the skeletal topology conforms to human anatomy. Finally, the model outputs a 2D skeletal sequence map containing the two-dimensional coordinates of each joint and their connection relationships. Its spatiotemporal information provides high-precision structured data support for subsequent human behavior intention recognition.

[0072] S3: Convert the skeletal sequence map (original joint coordinate sequence) extracted in step S2 into a spatiotemporal graph structure, and input it into the improved spatiotemporal graph convolutional network (ST-GCN) model. See the specific model structure below. Figure 3 , Figure 4 and Figure 5 Based on the above optimized features, the model can accurately predict the activity intentions of personnel in a human-robot group setting, thereby enabling the robot to effectively perceive and predict personnel activities.

[0073] It should be noted that a skeletal sequence graph refers to raw skeletal motion data, such as the coordinates of joints in a time series, like (x, y). A human skeleton graph, on the other hand, is a structured model of skeletal data, representing the connections between nodes (joints) and edges in graph form, used to adapt to the input requirements of Graph Convolutional Networks (ST-GCN).

[0074] Therefore, a human skeleton diagram is constructed: the skeletal sequence is typically represented by the coordinates of each joint in each frame (e.g., 2D (x,y) or 3D (x,y,z)). Through a spatiotemporal graph structure, the skeletal sequence can form a hierarchical topological representation. The specific method is as follows:

[0075] For a skeleton sequence containing N joints and T frames, construct an undirected spatiotemporal graph G = (V, E).

[0076] Where: V is the set of nodes, representing the coordinates of key points in all time frames, defined as... v ti Let d be the coordinates of the i-th joint in frame t, where d is the coordinate dimension (2D or 3D).

[0077] E is a set of edges, divided into spatial edges (connections of joints within the same frame) and temporal edges (connections of the same joints across frames). Spatial edge E S Spatial connections between joints within the same frame are determined by a predefined anatomical relationship H (e.g., ... Time edge E T For the temporal connection of the same key point across consecutive time frames (e.g.) Specifically, it is expressed as:

[0078]

[0079] Among them, v ti v represents the i-th key point in time frame t. tj Let v represent the j-th joint in time frame t, H represent the anatomical connections between human joints, and v (t+1)i This represents the i-th key point in time frame t+1.

[0080] Key points of the human skeleton are represented by (x, y) coordinates in a spatial coordinate system. During graph convolution, the skeletal information of the human body is processed through spatiotemporal graph convolution, as shown in the following formula:

[0081]

[0082] Among them, f ω (v ti ) represents node v ti The output features, N(v) ti ) represents node v ti The neighborhood (including the spatial domain E) S and Time Domain E T ), m(v tj ) represents the feature information of neighboring nodes, and ω is a learnable weight function used to adjust the importance of different connections.

[0083] Regarding the Spatial Extension Module: In traditional Graph Convolutional Networks (GCNs), spatial graph convolutional models typically only focus on the information of a node itself and its neighboring nodes. This can lead to the loss of skeleton information, thus affecting the accuracy of behavior recognition. To address this issue, this invention designs a Spatial Extension Module (SEM). When calculating the spatial features of a node, this module not only focuses on the skeletal information of the node itself and its neighboring nodes, but also simultaneously incorporates the skeletal information of the first non-adjacent node. In this way, the Spatial Extension Module can capture dependencies between skeletal points at greater distances, thereby enhancing the spatial modeling capability between nodes in the skeleton graph, avoiding information waste, and improving the accuracy of behavior recognition.

[0084] In its implementation, the spatial expansion module considers information about non-adjacent nodes by extending the traditional graph convolution calculation method. By incorporating information about non-adjacent nodes, the spatial expansion module expands the calculation formula to:

[0085]

[0086] Where N(v) i ) is node v i The set of adjacent nodes, f in (v j ) is node v j Input features, W ij This represents the weights between nodes. The second term represents the weight of the first non-adjacent node v. k Information, W ik For node v i Non-adjacent node v k The weights between nodes are determined. In this way, the module can not only focus on information from neighboring nodes, but also effectively transmit information from distant nodes, enhancing the ability to model spatial features.

[0087] Furthermore, to further optimize the feature extraction capabilities of the spatial expansion module, a skeletal point weighting mechanism was designed to adjust the influence of each node during the computation process. This weighting mechanism dynamically allocates weights by calculating the distance between nodes, and its formula is as follows:

[0088]

[0089] Where, α ik Represents node v i Non-adjacent node v k The weights between them, d ik For node v i and v k The distance between them is μ, where μ is the center value of the distance, and β is a hyperparameter controlling the weight distribution. Through this weighting mechanism, the model can adjust the transmission of information based on the distance, thereby extracting spatial features more accurately.

[0090] Regarding the channel attention mechanism (SENet module): The SENet module is introduced as a channel attention mechanism to enhance the network's focus on important features and ignore the influence of irrelevant channels. The SENet module uses a "compression-activation" mechanism, first compressing the global information of each channel through global average pooling, and then calculating the global description z of the c-th channel. c :

[0091]

[0092] Where H and W are the height and width of the feature map, respectively, c represents the number of channels, and f c (n) (i,j) represents the feature value of the c-th channel in the n-th layer at position (i,j).

[0093] Then, through the first fully connected layer (weight matrix W1∈R) C×C / r Bias term b1) for global descriptor z c Dimensionality reduction is performed, and the ReLU activation function is applied:

[0094] u c =ReLU(W1·z) c +b1)

[0095] Where r is the reduction ratio (usually set to 16), and C / r is the number of channels after dimensionality reduction.

[0096] Through the second fully connected layer (weight matrix W2∈R) C / r×C Bias term b2) restores the channel dimension and applies the Sigmoid function to generate attention weights:

[0097] s c =σ(W2·u c +b2)

[0098] Where σ is the Sigmoid function, s c ∈[0,1] represents the importance weight of the c-th channel.

[0099] Finally, the attention weights s c The optimized feature map is obtained by multiplying it channel-by-channel with the original feature map:

[0100]

[0101] In this way, the SENet module enables the network to adaptively adjust the channel weights, thereby capturing key information more effectively and improving the accuracy of behavioral intent recognition.

[0102] The improved ST-GCN model, by combining a spatial extension module and a channel attention mechanism, enhances both its spatial modeling capabilities and its dynamic processing capabilities in time series data. The improved network structure comprises nine main spatiotemporal units with output channel numbers of 64, 64, 64, 128, 128, 128, 256, 256, and 256, respectively. Convolutional operations are performed at each layer to extract more refined features. This improved network structure not only enhances the model's computational efficiency but also improves the accuracy of predicting activity intent. The formula is as follows:

[0103]

[0104] Among them, f out W represents the output feature. k A represents the weights of the convolution operation. k M represents the spatial feature map obtained through convolution operation. k Indicates the characteristics of the activity. This indicates a convolution operation.

[0105] Optimization and Prediction: Through the optimized ST-GCN model described above, the network can more accurately capture the spatial and temporal relationships between different skeleton points. In the refinement stage, the network further optimizes the initially estimated skeleton point positions to ensure accurate identification of each skeleton point. Finally, the model outputs an accurate 2D skeleton sequence map, and uses a spatiotemporal graph convolutional network to predict the activity intent.

[0106] Specifically, such as Figure 3 As shown, the optimized GCN model includes the following steps for predicting people's activity intentions:

[0107] S3.1: Batch normalization of skeleton input data. Before the data is input into the model, the skeleton data is first subjected to batch normalization, which is divided into the following stages:

[0108] During the training phase, the mean μ is calculated for each batch of skeleton data. B and variance in m is the batch size, x i Let i be the i-th sample in the batch.

[0109] In the above batch normalization, 'batch' refers to the data processing method during the training phase. During training, the model updates the normalization parameters by calculating the mean and variance using mini-batch samples; while in the prediction phase, the average parameters accumulated during training are used directly, regardless of the number of batches of input data.

[0110] Subsequently, the data is standardized and scaled to restore its expressive power, using the following formula:

[0111]

[0112] Where ò is the numerical stability constant, and γ (scaling factor) and β (offset factor) are the parameters learned during training.

[0113] During the prediction phase, the global mean μ accumulated during training is used. global and variance (Updated via moving average), rather than the current batch statistic, the formula is:

[0114]

[0115] Through the above operations, batch normalization ensures the stability of the input distribution of the model during the training and prediction stages, thereby improving the feature extraction efficiency and generalization ability of the spatiotemporal graph convolutional network.

[0116] S3.2: Use a series of spatiotemporal graph convolution operations to extract high-level features in the skeletal pose space and time dimension.

[0117] ST-GCN units with added space extension modules, such as Figure 4 As shown, after batch normalization of the input data, it is fed into the ST-GCN unit with added spatial expansion module. This unit alternately uses GCN (Graph Convolutional Network), spatial expansion module (SEM), and TCN (Temporal Convolutional Network) to extract features from both spatial and temporal dimensions.

[0118] Specifically, GCN is used to process the spatial features of skeleton data, that is, the spatial relationships between various skeleton points. The specific graph convolution operation can be represented as:

[0119]

[0120] in, It is the normalized adjacency matrix, H (l) W is the input feature matrix of the l-th layer. (l) σ is the weight matrix for learning, and σ is the activation function. Through graph convolution operations, GCN can fuse the spatial information of each skeleton point with the information of its neighboring points, thereby obtaining richer spatial features.

[0121] The Spatial Extension Module (SEM) enhances the model's ability to capture long-range dependencies between skeleton nodes. In the SEM, graph convolution operations not only handle local relationships between nodes but also capture interactions between distant nodes by extending the skeleton graph. Specifically, for each skeleton node v... i Its characteristics are determined by neighboring nodes v j and the first non-adjacent node v k The information is jointly determined. The formula can be expressed as:

[0122]

[0123] Among them, f in (v j ) and f in (v k ) is node v j and v k Input features, W ij and W ik It is the connection weight, N (vi) Represents node v i The set of adjacent nodes. In this way, the spatial expansion module can capture global information between skeletal points in the spatial dimension, thereby improving the limitation of traditional methods that only have local dependencies.

[0124] TCN is used to handle the temporal dimension of skeleton data, i.e., how the model understands the changes in the skeleton at different time steps. TCN effectively captures the dynamic dependencies in time series through one-dimensional convolution operations. The specific convolution operation formula is as follows:

[0125]

[0126] Where, x t-k+1 The input features at time step t-k+1, w k These are the weights of the convolution kernel, y t These are the output features. TCN captures the dynamic evolution of skeletal points through this sliding window method, identifies temporal patterns of actions, and ensures that the model can handle long-range dependencies in time-series data.

[0127] ST-GCN units that incorporate spatial extension modules and channel attention mechanisms, such as Figure 5 As shown. Building upon the spatial expansion module, this embodiment introduces SENet as a channel attention mechanism to further optimize the network's focus on key information. SENet adaptively learns the importance of each channel, strengthening the model's focus on key information and ignoring irrelevant features. Specifically, SENet compresses each channel using Global Average Pooling (GAP), reducing the feature map F of each channel... (c) A global description z is generated through pooling operations. c :

[0128]

[0129] Among them, F (c) (i,j) represents the feature map on channel c, and H and W represent the height and width of the image. Then, the attention coefficients z for each channel are calculated using two fully connected layers and activation functions (ReLU and Sigmoid).

[0130] z = σ(W2·ReLU(W1·z) c ))

[0131] Finally, the network weights the features of each channel by generating attention coefficients, thereby improving the robustness and accuracy of the network in action recognition.

[0132] S3.3: The features extracted by the spatiotemporal graph convolutional network are mapped to action class probabilities through a global average pooling layer, a fully connected layer, and a softmax classifier. The specific steps are as follows:

[0133] First, the feature maps generated by the spatiotemporal graph convolutional layer are input into the Global Average Pooling (GAP) layer. The GAP layer operation slices the feature maps at each time step. The averaging process is performed using the following formula:

[0134]

[0135] Here, T is the time dimension, representing the number of time steps. This layer operation eliminates the complexity of the time dimension T.

[0136] The aforementioned spatiotemporal graph convolutional layer refers to the series of spatiotemporal graph convolution operations in S3.2, specifically... Figure 3 The GCN+SEM+TCN block and the GCN+SENet+SEM+TCN block are included.

[0137] Next, the feature F output by GAP is... GAP Through learnable weight matrix and bias terms Perform a linear transformation to map the input features to a higher-dimensional space and prepare an appropriate output for the classification task:

[0138] F E =W·F GAP +b

[0139] Where K is the number of behavior categories in the classification task.

[0140] Finally, the Softmax classifier is used to process the feature vector F output by the fully connected layer. E The probability distribution is converted to categories. The formula for the Softmax function is as follows:

[0141]

[0142] Ultimately, Softmax outputs the category with the highest probability as the predicted intent category.

[0143] Example 2

[0144] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0145] Example 3

[0146] The purpose of this embodiment is to provide a computer-readable storage medium.

[0147] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above-described method.

[0148] Example 4

[0149] The purpose of this embodiment is to provide a human-computer grouping intent recognition system based on skeletal trajectory and graph convolution, including:

[0150] The data acquisition module is configured to acquire real-time video stream data of human activities in human-machine grouping scenarios;

[0151] The 2D skeleton sequence diagram generation module is configured to extract human skeleton sequence information from real-time human activity video stream data and generate a 2D skeleton sequence diagram of the personnel.

[0152] The personnel activity intention prediction module is configured to: use a graph convolutional network model to process the 2D skeleton sequence graph of personnel and output the predicted personnel activity intention.

[0153] This embodiment addresses the problems of slow recognition speed, low accuracy, and susceptibility to the relative angle between the person and the robot in existing behavioral intention recognition methods, proposing an innovative solution. First, this invention optimizes the human pose estimation model, significantly improving estimation speed while maintaining accuracy. Second, it introduces a spatial expansion module and a channel attention mechanism, effectively enhancing the intention recognition algorithm's ability to extract spatial features of skeletal information and optimizing the weight allocation of important features, thereby ignoring irrelevant information and improving recognition accuracy. Furthermore, the optimized dataset composition supports the recognition of human activity intentions from multiple angles, improving the robustness of multi-view recognition. This invention is applicable to human behavioral intention recognition in human-robot group scenarios, possessing significant practical value, particularly with important application prospects in military and industrial robot collaboration fields.

[0154] Example 5

[0155] The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions involved in any of the above embodiments.

[0156] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0157] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0158] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for human-robot marshalling intention recognition based on skeleton trajectory and graph convolution, characterized in that, The method comprises the following steps: acquiring real-time personnel activity video stream data in a man-machine marshalling scene; extracting human body skeleton sequence information from the real-time personnel activity video stream data to generate a personnel 2D skeleton sequence graph; specifically: using a lightweight MobileNet V1 as a backbone network to replace the VGG network in the traditional OpenPose to reduce the computational complexity of feature extraction; further, model deployment optimization is implemented: training the lightweight model to obtain a weight file under the PyTorch framework; then, the PyTorch model is subjected to structured quantization processing and is converted into a TensorRT inference engine through the ONNX intermediate format; finally, the inference engine is deployed on an NVIDIA Jetson AGX Orin embedded platform to extract feature maps and generate skeletons from real-time video streams; in the training process, it is divided into an initialization stage and a refinement stage; in the initialization stage, the obtained TensorRT model file is used for real-time human body posture estimation, and the inference process is accelerated through the TensorRT engine to significantly improve the inference speed and efficiency; specifically, the model first generates a preliminary estimated key point heat map through a key point prediction branch: this branch uses convolutional layers to optimize the feature map, so that the probability value of each skeleton point is concentrated in its corresponding area, and a probability distribution of the key point is formed through a Softmax activation function to locate the key part; at the same time, the model synchronously generates a vector field describing the spatial connection relationship between the joints through a partial association field prediction branch: this branch uses convolutional layers to efficiently capture the direction and association strength between the joints, providing spatial dependency relationship basis for the subsequent skeleton point connection; in the refinement stage, the model iteratively optimizes the preliminary generated key point heat map and PAFs, eliminates redundant detection and corrects the mis-matched joint connection through multi-scale feature fusion and confidence weighting; finally, combined with the optimized heat map and PAFs, the model realizes the global optimal matching of the joints through the Hungarian algorithm, assembles into a complete and accurate 2D human body skeleton sequence graph, and provides high-precision data support for subsequent behavior intention recognition; using a graph convolutional network model to process the personnel 2D skeleton sequence graph to output a predicted personnel activity intention, specifically including: performing batch normalization processing on the personnel 2D skeleton sequence graph to obtain normalized standard data; using a spatio-temporal graph convolution operation to extract high-level features in the skeleton posture space and time dimension on the normalized standard data; after calling a global average pooling layer and a fully connected layer on the extracted high-level features, a Softmax classifier is used to obtain the corresponding action classification; the spatio-temporal graph convolutional network model comprises an ST-GCN unit; the ST-GCN unit alternately uses a GCN graph convolutional network, a spatial expansion module SEM, a TCN time convolutional network and a channel attention module SEnet to extract features from the spatial and temporal dimensions; The convolution operation in the space expansion module SEM not only processes the local relationship between nodes, but also captures the interaction of remote nodes by expanding the skeleton graph, which is used to enhance the model's ability to capture the remote dependency relationship between the skeleton points. Specifically, for each skeleton node The characteristics are jointly determined by the information of the adjacent node And the first non-adjacent node The formula is expressed as: wherein, and are input features of nodes and , and are connection weights, denotes a set of adjacent nodes of node ; in this way, the spatial expansion module can capture the global information between the skeletal points in the spatial dimension, thereby improving the limitation of the traditional method which only depends locally; To further optimize the feature extraction capability of the spatial expansion module, a bone point distribution weight mechanism is designed to adjust the influence of each node in the calculation process; the weight mechanism dynamically distributes the weight by calculating the distance between the nodes, and the formula is: wherein, denotes the weight between nodes and non-adjacent nodes , is the distance between nodes and , is the central value of the distance, is a hyperparameter that controls the weight distribution; The SE-Net as a channel attention mechanism learns the importance of each channel adaptively, strengthens the model's attention to key information, and ignores irrelevant features; specifically, the SE-Net compresses each channel through global average pooling GAP, and generates a global description through the pooling operation of the feature map of each channel; the global descriptor is reduced in dimension through the first fully connected layer, and the ReLU activation function is applied; the channel dimension is restored through the second fully connected layer, and the Sigmoid function is applied to generate attention weights; the attention weights are multiplied with the original feature map channel by channel to obtain the optimized feature map.

2. The method of claim 1, wherein the skeleton trajectory and graph convolution-based human-robot formation intention recognition method is characterized by, When extracting human skeleton sequence information from real-time personnel activity video stream data, it includes: Predicting bone key points based on the extracted feature map; Generating a partial association field based on the extracted feature map, which describes the spatial connection relationship between the nodes; Assembling a personnel 2D skeleton sequence diagram according to the predicted bone key points and the partial association field.

3. The method of claim 1, wherein the skeleton trajectory and graph convolution-based human-robot formation intention recognition method is characterized by, When extracting features, it specifically includes: The lightweight MobileNet V1 independently performs convolution operations on each input channel of the input real-time personnel activity video stream data, and then performs convolution on the outputs of all channels; After processing by MobileNet V1, the generated feature map contains high-level features related to human posture in the image.

4. The method of claim 1, wherein the skeleton trajectory and graph convolution-based human-robot formation intention recognition method is characterized by, The normalized standard data is used for spatio-temporal graph convolution operation in the skeleton pose space and time dimension to extract high-level features, including: GCN is used to process the spatial features of the skeleton data, i.e. the spatial relationship between the nodes, through graph convolution operation, GCN can fuse the spatial information of each node with the information of its adjacent nodes, thereby obtaining richer spatial features; TCN effectively captures dynamic dependencies in time series through convolution operation, which is used to process the time dimension of the skeleton data, i.e. how the model understands the changes of the skeleton at different time steps.

5. The method of claim 1, wherein the skeleton trajectory and graph convolution-based human-robot formation intention recognition method is characterized by, After extracting high-level features by calling the global average pooling layer and the fully connected layer, the Softmax classifier is used to obtain the corresponding action classification, which specifically includes: After the spatio-temporal graph convolution layer, the generated feature map is input to the global average pooling GAP layer; The GAP layer operation averages each time step feature and compresses the feature vector into a fixed length representation; Then, the GAP output features are passed into the fully connected layer, which performs linear transformation through weight matrix and bias term, maps the input features to a higher dimensional space, and prepares appropriate output for the classification task; Finally, the Softmax classifier is used to convert the feature vector output by the fully connected layer into a probability distribution of classes, and the Softmax returns the class with the highest probability as the predicted intent class.

6. A human-machine marshalling intention recognition system based on skeleton trajectory and graph convolution, adopting the human-machine marshalling intention recognition method based on skeleton trajectory and graph convolution according to any one of claims 1-5, characterized in that, It includes: A data acquisition module configured to acquire real-time personnel activity video stream data in a man-machine marshalling scene; The personnel 2D skeleton sequence graph generation module is configured to extract human skeleton sequence information from real-time personnel activity video stream data and generate a personnel 2D skeleton sequence graph. The personnel activity intention prediction module is configured to perform operation processing on the personnel 2D skeleton sequence graph by using a graph convolution network model and output a predicted personnel activity intention.

7. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1-5.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps of the method of any one of claims 1-5.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, performs the steps of the method of any one of claims 1-5.

Images