Foot contact detection method and system based on multi-scale space-time diagram convolutional network

Abstract

The invention belongs to the technical field of three-dimensional reconstruction in computer vision, and discloses a foot contact detection method and system based on a multi-scale space-time diagram convolutional network, and the method comprises the steps: collecting video data, and converting each motion sequence into an image sequence; the image sequence is processed, and an input sequence with the target detection frame as the center is intercepted; generating a subsequence based on a time scale; processing the sub-sequence of each time scale, and extracting the features of the high-dimensional spatial-temporal feature map; performing weighted aggregation on the joint features through a spatial attention mechanism, and converting the spatial-temporal feature map into global representation; cross-time-scale information exchange and complementation are realized, and spatial-temporal features are fused; and outputting a four-dimensional contact probability vector, and generating a binary contact label after threshold judgment to obtain a foot contact state. According to the invention, accurate single-view foot contact detection can be realized in both fast movement and slow movement scenes, and the method is applied to various scenes such as movement capture optimization and gait analysis.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction in computer vision, and in particular to a foot contact detection method and system based on a multi-scale spatio-temporal graph convolution network. BACKGROUND

[0002] Foot contact is the most common and inevitable human-scene interaction in daily activities, and is widely used in fields such as graphics, behavior understanding, healthcare, VR / AR, robots, etc. Taking graphics as an example, in recent years, foot contact detection has increasingly become a key module in many human-centered tasks. Motion capture methods use foot contact to solve the problems of foot sliding and penetration.

[0003] Current foot contact detection techniques mainly fall into two categories: methods based on 2D keypoint sequences and methods based on single-frame images. The former usually uses binary labels to represent foot contact, and usually uses fixed-length 2D keypoint sequences as input to regress foot contact, but the fixed time window cannot adapt to changes in motion speed, and only focuses on keypoint position information while ignoring the topological relationship of human joints. The latter directly predicts dense contact regions from single-frame RGB images, relying on parameterized human models (such as SMPL), which can provide more rich contact information, but requires a large amount of high-quality labeled data, and is easily disturbed by environmental light, occlusion, etc., and has poor temporal continuity. At the hardware level, some methods solve the problem by increasing the number of cameras or sensing devices, but this greatly increases the cost and complexity of the device.

[0004] Therefore, how to provide a foot contact detection method and system based on a multi-scale spatio-temporal graph convolution network is a problem to be solved at present. SUMMARY

[0005] The embodiments of the present application provide a foot contact detection method and system based on a multi-scale spatio-temporal graph convolution network to solve the above technical problems in the prior art.

[0006] To have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general review, nor is it intended to determine key / important components or delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0007] According to a first aspect of the embodiments of the present application, a foot contact detection method based on a multi-scale spatio-temporal graph convolution network is provided.

[0008] In one embodiment, the foot contact detection method based on the multi-scale spatio-temporal graph convolution network comprises: Collect video data and convert each action sequence in the video data into an image sequence; The image sequence is processed through a pre-trained joint detection deep network model based on the pose estimation coordinate classification method to obtain skeleton data and intercept the input sequence centered on the target detection frame; Perform stratified sampling on the input sequence to generate subsequences based on time scales and obtain multi-interval sliding windows; Using the spatiotemporal graph convolutional network, the subsequences of each time scale are processed to extract high-dimensional spatiotemporal feature graph features; Perform temporal average pooling on the spatiotemporal feature map, weightedly aggregate joint features through the spatial attention mechanism, and convert the spatiotemporal feature map into a global representation; Based on the cross-scale enhancement strategy, information exchange and complementation across time scales are achieved, and spatiotemporal features are integrated; The joint detection deep network model is trained based on the first-order optimization algorithm, and a two-dimensional key point sequence is input to output a four-dimensional contact probability vector. After threshold judgment, a binary contact label is generated to obtain the foot contact status.

[0009] In one embodiment, the processing of the image sequence by a pre-trained joint detection deep network model based on a pose estimation coordinate classification method to obtain skeleton data and intercepting an input sequence centered on a target detection frame includes: A pre-trained joint detection deep network model based on the pose estimation coordinate classification method is used to process the image sequence, obtain the 2D key point sequence of the human body, and select the key points for detection; Taking the target detection frame as the center, 10 frames of images before and after it are intercepted to form the input sequence.

[0010] In one embodiment, the pre-trained joint detection deep network model based on the posture estimation coordinate classification method processes the image sequence to obtain a two-dimensional key point sequence of the human body, and selects the key points for detection, including: Use detector to locate the human body bounding box; The single person area is cropped from the human body bounding box and input into the lightweight posture network for key point prediction; The pose estimation coordinate classification method is introduced to transform the continuous coordinate regression into a discrete classification task.

[0011] In one embodiment, the introduction of the pose estimation coordinate classification method to convert the continuous coordinate regression into a discrete classification task includes: Discretize the horizontal and vertical coordinates into sub-pixel intervals; Through two independent fully connected layers, the distribution probability of each key point in the horizontal and vertical directions is predicted, and Gaussian label smoothing technology is combined to reduce quantization error; Continuous coordinates are decoded based on the maximum probability, and the confidence of human joint points is derived from the classification probability.

[0012] In one embodiment, the processing of each time scale subsequence using a spatiotemporal graph convolutional network to extract high-dimensional spatiotemporal feature graph features includes: Utilizing spatial graph convolution in the spatiotemporal graph convolutional network, the correlation between joints is dynamically learned based on the topological structure of human joints. The temporal convolution in the spatiotemporal graph convolutional network is used to extract motion timing features along the time axis.

[0013] In one embodiment, performing temporal average pooling on the spatiotemporal feature map, weightedly aggregating joint features through a spatial attention mechanism, and converting the spatiotemporal feature map into a global representation includes: Perform mean pooling processing on the spatiotemporal feature map along the time dimension; Based on the spatial attention mechanism, the information of each joint is used to calculate the contribution weight of each joint through a linear layer, and the key joint features are weighted and fused to generate compact spatiotemporal features.

[0014] In one embodiment, the cross-scale enhancement strategy is used to achieve information exchange and complementation across time scales and fuse spatiotemporal features, including: Based on diverse attention patterns, temporal modeling is performed on the extracted multi-time scale features; The enhanced multi-timescale features are aggregated through a fully connected layer consisting of one-dimensional convolutions, and the high-dimensional features are mapped into a low-dimensional contact state space; A multi-layer perceptron is used to integrate multi-time scale features.

[0015] In one embodiment, the integrating multi-time-scale features using a multi-layer perceptron includes: The four-dimensional feature probabilities output at each scale are concatenated and input into a multi-layer perceptron for feature fusion; Through linear transformation and ReLU function, activate and generate intermediate features; The features are mapped back to the four-dimensional space through linear transformation, and the output is normalized based on the Sigmoid function to obtain the final feature probability.

[0016] In one embodiment, the loss function of the first-order optimization algorithm training is: ; Where, Expressed as the loss function, B Denoted as batch size, j Represented as four foot contact areas, is represented as the ground truth value, is represented as an output of prediction.

[0017] According to a second aspect of the embodiments of the present application, a foot contact detection system based on a multi-scale spatio-temporal graph convolution network is provided.

[0018] In one embodiment, the foot contact detection system based on the multi-scale spatio-temporal graph convolution network comprises: a data acquisition module configured to acquire video data and convert each action sequence in the video data into an image sequence; a preprocessing module configured to process the image sequence by using a pre-trained joint detection deep network model based on a pose estimation coordinate classification method, obtain skeleton data, and intercept an input sequence centered on a target detection frame; a window generation module configured to perform hierarchical sampling on the input sequence, generate a sub-sequence based on a time scale, and obtain a multi-interval sliding window; a feature extraction module configured to process the sub-sequence of each time scale by using a spatio-temporal graph convolution network, and extract a high-dimensional spatio-temporal feature map feature; a spatio-temporal average module configured to perform time-series average pooling processing on the spatio-temporal feature map, simultaneously aggregate joint features by using a spatial attention mechanism for weighting, and convert the spatio-temporal feature map into a global representation; a scale fusion module configured to realize information exchange and complementation across time scales based on a cross-scale enhancement strategy, and fuse spatio-temporal features; a training output module configured to train the joint detection deep network model based on a first-order optimization algorithm, input a two-dimensional key point sequence, output a four-dimensional contact probability vector, generate a binary contact label after threshold judgment, and obtain a foot contact state.

[0019] According to a third aspect of the embodiments of the present application, a computer device is provided.

[0020] In some embodiments, the computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0021] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided.

[0022] In one embodiment, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0023] The technical scheme provided by the embodiments of the present application can include the following beneficial effects: This method, based on a spatiotemporal graph convolutional network (STGCN), combines multi-timescale feature fusion to accurately detect foot-ground contact in various motion modes. It employs a detection method based on a 2D keypoint sequence as input. Compared to direct detection using a single-frame RGB image, this method not only integrates temporal information but also avoids interference from the scene environment, significantly improving generalization. Furthermore, by introducing a multi-interval sliding window and layered sampling of motion sequences at different time scales (short-term transient contact, medium-term gait transition, and long-term global trends), the model is able to balance local details with global trends, effectively addressing the limitations of a fixed time window and improving the detection algorithm's adaptability to varying motion speeds. Furthermore, by combining an adaptive spatiotemporal graph convolutional network (ST-AGCN) to dynamically learn inter-joint correlations, leveraging human topology priors to enhance spatial feature expression, and employing a multi-timescale enhancement strategy to fuse features across time dimensions, the detection results significantly improve the spatiotemporal consistency of the detection results. Verified on the AMASS test set and real-scene datasets such as RICH and MMVP, the present invention can achieve accurate single-perspective foot contact detection in both fast and slow motion scenarios, and can be applied to various scenarios such as motion capture optimization and gait analysis.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0026] Figure 1 is a flowchart of a foot contact detection method based on a multi-scale spatiotemporal graph convolutional network according to an exemplary embodiment; Figure 2 is a structural block diagram of a foot contact detection system based on a multi-scale spatiotemporal graph convolutional network according to an exemplary embodiment; Figure 3 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0027] The following description and accompanying drawings sufficiently illustrate the specific embodiments herein to enable those skilled in the art to practice them. Portions and features of some embodiments may be included in or substituted for portions and features of other embodiments. The scope of the embodiments herein includes the entire scope of the claims, including all available equivalents thereof. Herein, the terms "first," "second," and the like are used solely to distinguish one element from another and do not require or imply any actual relationship or order between these elements. In practice, the first element can also be referred to as the second element, and vice versa. Furthermore, the terms "comprise," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a structure, device, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such structure, device, or apparatus. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the structure, device, or apparatus comprising the element. The various embodiments herein are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Similar or identical parts between the various embodiments can be referenced to each other.

[0028] The terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like used herein to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are intended only to facilitate the description of this document and simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the present invention. In the description herein, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, they can be mechanical or electrical connections, or they can be internal connections between two elements, they can be directly connected, or they can be indirectly connected through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0029] As used herein, unless otherwise specified, the term "plurality" means two or more.

[0030] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0031] In this article, the term "and / or" is used to describe the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0032] It should be understood that although the various steps in the flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0033] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software so that the processor can call and execute the operations corresponding to the above modules.

[0034] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0035] Figure 1 An embodiment of a foot contact detection method based on a multi-scale spatiotemporal graph convolutional network of the present invention is shown.

[0036] In this optional embodiment, the foot contact detection method based on a multi-scale spatiotemporal graph convolutional network includes: Step S101: Collect video data and convert each action sequence in the video data into an image sequence; Specifically, collecting video data and converting each action sequence in the video data into an image sequence includes: First, a single industrial camera (such as Azure Kinect or a standard RGB camera) is set up at a test site, approximately 1.2 meters above the ground, ensuring that the camera covers the subject's activity area. Then, the subject is asked to perform a designated action (such as walking, jumping, martial arts, dancing, etc.) approximately 2-3 meters in front of the camera. Finally, the video data is recorded on the PC (i.e., the user end), and each action sequence is converted into an RGB image sequence based on the camera's frame rate. The OpenCV library is used to perform frame parsing on the input action video. The video file is opened using the OpenCV VideoCapture class, which can parse the video frame rate. Each frame is read in a loop until the video ends, and each frame is saved as a JPG image. Step S102: Process the image sequence using a pre-trained joint detection deep network model based on a pose estimation coordinate classification method to obtain skeleton data and intercept an input sequence centered on the target detection frame; Step S103: perform stratified sampling on the input sequence to generate subsequences based on time scales, and obtain a multi-interval sliding window; Specifically, the input sequence is used to generate a multi-interval sliding window, and the input 21-frame 2D key point sequence (dimension: T s × J ×2, T s =21) for stratified sampling: generate subsequences of 5 time scales (window size S ={5,9,13,17,21}), covering short-term transient contact (5 frames), medium-term gait transitions (9-13 frames), and long-term global trends (17-21 frames), adapting to both fast and slow motion patterns. Each subsequence is sampled at equidistant geometric intervals of 5 frames.

[0037] Step S104: using a spatiotemporal graph convolutional network to process the subsequences at each time scale and extract high-dimensional spatiotemporal feature graph features; Step S105: Perform temporal average pooling on the spatiotemporal feature map, weightedly aggregate joint features through a spatial attention mechanism, and convert the spatiotemporal feature map into a global representation; Step S106: Based on the cross-scale enhancement strategy, information exchange and complementation across time scales are realized, and spatiotemporal features are integrated; Step S107: Train the joint detection deep network model based on the first-order optimization algorithm, input a two-dimensional key point sequence, output a four-dimensional contact probability vector, generate a binary contact label after threshold judgment, and obtain the foot contact state.

[0038] In this optional embodiment, the pre-trained joint detection deep network model based on the pose estimation coordinate classification method is used to process the image sequence, obtain skeleton data, and intercept the input sequence centered on the target detection frame, including: A pre-trained joint detection deep network model based on the pose estimation coordinate classification method is used to process the image sequence, obtain the 2D key point sequence of the human body, and select the key points for detection; Taking the target detection frame as the center, 10 frames of images before and after it are intercepted to form the input sequence.

[0039] Specifically, the input sequence is constructed by using a time window interception method centered on the target detection frame, selecting the first 10 frames and the last 10 frames of the frame to form continuous time series data with a total length of 21 frames.

[0040] In this optional embodiment, the pre-trained joint detection deep network model based on the posture estimation coordinate classification method processes the image sequence to obtain a two-dimensional key point sequence of the human body, and selects the key points for detection including: Use detector to locate the human body bounding box; The single person area is cropped from the human body bounding box and input into the lightweight posture network for key point prediction; The pose estimation coordinate classification method is introduced to transform the continuous coordinate regression into a discrete classification task.

[0041] In this optional embodiment, the introduction of the posture estimation coordinate classification method to convert the continuous coordinate regression into a discrete classification task includes: Discretize the horizontal and vertical coordinates into sub-pixel intervals; Through two independent fully connected layers, the distribution probability of each key point in the horizontal and vertical directions is predicted, and Gaussian label smoothing technology is combined to reduce quantization error; Continuous coordinates are decoded based on the maximum probability, and the confidence of human joint points is derived from the classification probability.

[0042] Specifically, the RGB image sequence is processed through the pre-trained open source model RTMPose (i.e., a deep network model for joint detection), and the 2D key point sequence of the human body is captured, and 18 points are taken for detection. RTMPose is an open source human 2D joint point detection framework. Its input is a single-frame RGB image. The deep neural network model regresses the corresponding human 2D joint point position and confidence in the RGB image (RTMPose regresses the human 2D joint point position and confidence in the RGB image through a deep neural network model. Its core process can be divided into three steps: First, an efficient detector (such as RTMDet) is used to locate the human bounding box. The single-person area is then cropped and input into a lightweight posture network for key point prediction. The network is based on the CSPNeXt backbone architecture and is optimized for dense prediction tasks. It reduces redundant calculations through partial connections across stages while maintaining high-resolution feature extraction capabilities. Keypoint prediction incorporates the SimCC coordinate classification method (i.e., pose estimation-based coordinate classification), transforming continuous coordinate regression into a discrete classification task. The horizontal and vertical coordinates are discretized into 512 and 384 sub-pixel intervals, respectively. Two independent fully connected layers are used to predict the distribution probability of each keypoint in the x / y direction. Gaussian label smoothing is then used to reduce quantization error. Finally, the continuous coordinates are decoded based on the maximum probability, and the confidence level is directly derived from the classification probability. Furthermore, the model uses gated attention units to model the spatial dependencies between keypoints. Gated linear units and a lightweight attention mechanism are used to fuse local and global information, significantly improving localization accuracy. (RTMPose is a high-performance, real-time multi-person pose estimation framework proposed by the Shanghai AI Lab. Its functions include detecting the location and confidence of 2D human joints from a single RGB image frame. Based on a deep neural network framework, this model efficiently locates key joints such as the head, limbs, and torso, and outputs their 2D coordinates and detection confidence. This open-source model, when used, simply inputs a single image frame to obtain the 2D coordinates and confidence level of the detected human joints.)

[0043] In this optional embodiment, the use of a spatiotemporal graph convolutional network to process the subsequences of each time scale to extract high-dimensional spatiotemporal feature graph features includes: Using spatial graph convolution in the spatiotemporal graph convolutional network (GCN), the correlation between joints is dynamically learned based on the topological structure of human joints. That is, GCN dynamically learns the correlation between joints based on the topological structure of human joints. The temporal convolution in the spatiotemporal graph convolutional network (TCN) is used to extract motion timing features along the time axis, that is, TCN extracts motion timing features along the time axis, and the network outputs a high-dimensional spatiotemporal feature map.

[0044] Specifically, a spatiotemporal graph convolutional network (ST-AGCN) is introduced to process each scale subsequence and extract high-dimensional spatiotemporal feature graph features: Skeleton data is actually in the form of a vector sequence, which not only records the coordinates of the joint points in each frame, but also implies the direction of propagation from the root node to the end node along the human body hierarchy. This human physical structure is modeled using a spatiotemporal graph, a dynamic adjacency matrix is ​​constructed based on the human joint topology, and the graph convolutional network is used to dynamically learn the correlation between joints: ; Where, is the graph convolution feature map, is a fixed topological adjacency matrix, is the dynamic joint relationship matrix, is the implicit similarity matrix, K is the convolution kernel size of the spatial dimension, according to the joint point division strategy, K Set to 3, k For each core, is the feature input, The weight vectors of multiple output channels are stacked to form a weight matrix, and two 1×1 convolutional layers are used as embedding functions to capture joint similarity, enabling the model to infer implicit relationships between joints. Its calculation can be expressed as: ; Where, and There are two embedded functions and The learnable parameters of , T is the transpose of the matrix, Softmax is an activation function commonly used in multi-classification tasks, which can normalize a real vector into a probability distribution. Based on the graph convolution feature map implemented above , and sequentially apply batch normalization layers, random dropout layers, residual connections, and activation function layers to implement the graph convolution module.

[0045] The temporal convolutional network then extracts motion temporal features by performing convolution using specialized 2D convolutions, specifically designed to adapt to the spatiotemporal structure of skeleton data, by applying 3×1 convolutions only along the temporal dimension while keeping the joint dimension and channels unchanged. Batch normalization layers follow the convolutional layers to ensure stable and efficient training.

[0046] The basic spatiotemporal graph convolution module is a combination of spatial graph convolution, temporal convolution, and activation function layers. Each block contains a residual connection to stabilize training. Three such modules are used in the detection network at each time scale. High-dimensional features are extracted through the spatiotemporal graph convolution module, with feature channel transformations of 2-64, 64-128, and 128-256, respectively.

[0047] In this optional embodiment, performing temporal average pooling on the spatiotemporal feature map, weightedly aggregating joint features through a spatial attention mechanism, and converting the spatiotemporal feature map into a global representation includes: Perform mean pooling on the spatiotemporal feature map along the time dimension to compress dynamic information; Based on the spatial attention mechanism, the information of each joint is used to calculate the contribution weight of each joint through a linear layer, and the key joint features are weighted and fused to generate compact spatiotemporal features.

[0048] Specifically, the spatiotemporal average pooling module is used to convert the spatiotemporal feature map into a global representation (the extracted high-dimensional spatiotemporal feature map): the feature map is average pooled along the time dimension to stabilize the dynamic information in the entire sequence: ; Where, is the feature after mean pooling along the time dimension, is the original input feature; The spatial attention mechanism is applied to weight and compress the information of each joint (the spatial attention mechanism is an important attention mechanism in deep learning, mainly used in image and video processing. It improves model performance by focusing on the importance of different positions in the feature map). That is, the contribution weight of each joint is calculated through the linear layer, and the key joint features are weighted and fused to generate compact spatiotemporal features. The attention score of each joint is generated through the fully connected layer and Softmax: ; in, and b are the parameters of the fully connected layer, Indicates the n The attention weights of the joints are used to perform weighted summation of the original features and enhance important features: ; in, is the low-dimensional global feature finally extracted at each scale, J is the number of joints, is the original input feature, is the attention weight of the corresponding position.

[0049] In this optional embodiment, the cross-scale enhancement strategy is used to achieve information exchange and complementation across time scales and to fuse spatiotemporal features, including: Based on diverse attention patterns, temporal modeling is performed on the extracted multi-time scale features; The enhanced multi-timescale features are aggregated through a fully connected layer consisting of one-dimensional convolutions, and the high-dimensional features are mapped into a low-dimensional contact state space; The multi-time scale features are integrated by using a multi-layer perceptron.

[0050] In the optional embodiment, the integrating the multi-time scale features by using a multi-layer perceptron comprises: The four-dimensional feature probabilities of the scale outputs are spliced and input to the multi-layer perceptron for feature fusion; The intermediate features are activated by linear transformation and ReLU function; The features are mapped back to the four-dimensional space by linear transformation, and the output is normalized based on the Sigmoid function to obtain the final feature probability.

[0051] Specifically, a cross-scale enhancement strategy is used to fuse the multi-time scale features: through a cross-scale multi-head attention mechanism, multiple attention heads independently learn different attention patterns, giving higher weights to important time scale relationships and automatically suppressing time scale information with low relevance. A two-layer MLP (i.e., a multi-layer perceptron) is used to fuse the predictions of different time scales, and the contact predictions of each time scale are fused into a unified output. Each time scale can query information from other scales, thereby realizing information exchange and complementation across multiple time scales. The extracted multi-time scale features are subjected to time dependency modeling: ; wherein, is a multi-scale feature, is a scaling factor used to control the numerical range of attention scores in the multi-head attention mechanism to avoid problems of excessively small or large gradients, especially before Softmax, , and are learnable projection matrices that map the compact features C that have been subjected to spatio-temporal average pooling to query (Query), key (Key), and value (Value) spaces, respectively. The specific mapping process is as follows: ; wherein, is a query vector, is a key vector, is a value vector, and K similarity calculation (i.e., ) is used to model multi-scale time dependency, and V enhanced features are generated by attention weights. The multi-head mechanism learns diversified attention patterns through multiple independent projection matrices , such as differential modeling of short-time transient contact and long-time motion trend.

[0052] The multi-scale features after attention enhancement The full connection layer consisting of one-dimensional convolution is aggregated by a pooling layer, which maps the high-dimensional features (256) into an interpretable low-dimensional contact state space (i.e. 4-dimensional probability vector).

[0053] To fuse the prediction results of different time scales, a two-layer multi-layer perceptron (MLP) is used to integrate the contact predictions of each time scale into a unified output. First, the 4-dimensional contact probability output of each scale is integrated into a splicing feature , where n is the number of time scales, taking 5, input to the first layer MLP for feature fusion, and generate 128-dimensional intermediate features by linear transformation (weight matrix S ) and ReLU activation ; the second layer MLP further maps the features back to the 4-dimensional space through linear transformation , and applies the Sigmoid function to normalize the output to obtain the final contact probability . This design integrates multi-scale information through adaptive weighting, suppresses noise interference, and significantly improves the robustness of the detection result in complex motion scenarios.

[0054] In this optional embodiment, the Adam (i.e. first-order optimization algorithm) optimizer is used for training, with an initial learning rate of , and the learning rate is reduced to 10% of the original value every 5 rounds. The loss function uses mean square error, and the formula is: ; where L represents the loss function, B represents the batch size, B A represents the four foot contact areas, j G represents the ground true value, and O represents the predicted output.

[0055] After the model is trained, the human foot contact state of the center frame can be detected through the two-dimensional key point sequence (i.e. 2D key point sequence). The system outputs a 4-dimensional contact probability vector, which generates a binary contact label after thresholding, corresponding to the contact state of the left toe, left heel, right toe and right heel, respectively.

[0056] Figure 2 An embodiment of a foot contact detection system based on a multi-scale spatio-temporal graph convolutional network is shown.

[0057] In this optional embodiment, the foot contact detection system based on the multi-scale spatio-temporal graph convolutional network comprises: A data acquisition module 201 is configured to acquire video data and convert each action sequence in the video data into an image sequence. ​A pre-processing module 202 is used to process the image sequence using a pre-trained joint detection deep network model based on a pose estimation coordinate classification method, obtain skeleton data, and intercept an input sequence centered on the target detection frame; The window generation module 203 is used to perform stratified sampling on the input sequence to generate subsequences based on the time scale to obtain a multi-interval sliding window; The feature extraction module 204 is used to process the subsequences of each time scale using a spatiotemporal graph convolutional network to extract high-dimensional spatiotemporal feature graph features; The spatiotemporal averaging module 205 is used to perform temporal average pooling processing on the spatiotemporal feature map, and at the same time, weightedly aggregate joint features through the spatial attention mechanism, and convert the spatiotemporal feature map into a global representation; The scale fusion module 206 is used to achieve information exchange and complementation across time scales and fuse spatiotemporal features based on a cross-scale enhancement strategy; The training output module 207 is used to train the joint detection deep network model based on the first-order optimization algorithm, input a two-dimensional key point sequence, output a four-dimensional contact probability vector, generate a binary contact label after threshold judgment, and obtain the foot contact state.

[0058] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment are implemented.

[0059] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0060] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0061] In addition, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0062] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes in the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0063] The present invention is not limited to the structures described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A foot contact detection method based on a multi-scale spatiotemporal graph convolutional network, characterized in that: include: Collect video data and convert each action sequence in the video data into an image sequence; The image sequence is processed through a pre-trained joint detection deep network model based on the pose estimation coordinate classification method to obtain skeleton data and intercept the input sequence centered on the target detection frame; Perform stratified sampling on the input sequence to generate subsequences based on time scales and obtain multi-interval sliding windows; Using the spatiotemporal graph convolutional network, the subsequences of each time scale are processed to extract high-dimensional spatiotemporal feature graph features; Perform temporal average pooling on the spatiotemporal feature map, weightedly aggregate joint features through the spatial attention mechanism, and convert the spatiotemporal feature map into a global representation; Based on the cross-scale enhancement strategy, information exchange and complementation across time scales are achieved, and spatiotemporal features are integrated; The joint detection deep network model is trained based on the first-order optimization algorithm, and a two-dimensional key point sequence is input to output a four-dimensional contact probability vector. After threshold judgment, a binary contact label is generated to obtain the foot contact status.

2. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 1 is characterized in that: The method of processing the image sequence by using the pre-trained joint detection deep network model based on the pose estimation coordinate classification method to obtain skeleton data and intercepting the input sequence centered on the target detection frame includes: A pre-trained joint detection deep network model based on the pose estimation coordinate classification method is used to process the image sequence, obtain the 2D key point sequence of the human body, and select the key points for detection; Taking the target detection frame as the center, 10 frames of images before and after it are intercepted to form the input sequence.

3. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 2 is characterized in that: The pre-trained joint detection deep network model based on the posture estimation coordinate classification method processes the image sequence to obtain a two-dimensional key point sequence of the human body, and selects the key points for detection including: Use detector to locate the human body bounding box; The single person area is cropped from the human body bounding box and input into the lightweight posture network for key point prediction; The pose estimation coordinate classification method is introduced to transform the continuous coordinate regression into a discrete classification task.

4. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 3 is characterized in that: The introduction of the pose estimation coordinate classification method to transform the continuous coordinate regression into a discrete classification task includes: Discretize the horizontal and vertical coordinates into sub-pixel intervals; Through two independent fully connected layers, the distribution probability of each key point in the horizontal and vertical directions is predicted, and Gaussian label smoothing technology is combined to reduce quantization error; Continuous coordinates are decoded based on the maximum probability, and the confidence of human joint points is derived from the classification probability.

5. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 1, characterized in that: The spatiotemporal graph convolutional network is used to process the subsequences of each time scale and extract high-dimensional spatiotemporal feature graph features, including: Utilizing spatial graph convolution in the spatiotemporal graph convolutional network, the correlation between joints is dynamically learned based on the topological structure of human joints. The temporal convolution in the spatiotemporal graph convolutional network is used to extract motion timing features along the time axis.

6. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 1, characterized in that: The temporal average pooling process is performed on the spatiotemporal feature map, the weighted aggregation of joint features is performed through the spatial attention mechanism, and the spatiotemporal feature map is converted into a global representation, including: Perform mean pooling processing on the spatiotemporal feature map along the time dimension; Based on the spatial attention mechanism, the information of each joint is used to calculate the contribution weight of each joint through a linear layer, and the key joint features are weighted and fused to generate compact spatiotemporal features.

7. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 1, characterized in that: The cross-scale enhancement strategy is based on which information exchange and complementation across time scales are achieved and spatiotemporal features are integrated, including: Based on diverse attention patterns, temporal modeling is performed on the extracted multi-time scale features; The enhanced multi-timescale features are aggregated through a fully connected layer consisting of one-dimensional convolutions, and the high-dimensional features are mapped into a low-dimensional contact state space; A multi-layer perceptron is used to integrate multi-time scale features.

8. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 7, characterized in that: The multi-layer perceptron is used to integrate multi-time scale features, including: The four-dimensional feature probabilities output at each scale are concatenated and input into a multi-layer perceptron for feature fusion; Through linear transformation and ReLU function, activate and generate intermediate features; The features are mapped back to the four-dimensional space through linear transformation, and the output is normalized based on the Sigmoid function to obtain the final feature probability.

9. The foot contact detection method based on multi-scale spatiotemporal graph convolutional network according to claim 1, characterized in that: The loss function of the first-order optimization algorithm training is: ； Where, Expressed as the loss function, B Denoted as batch size, j Represented as four foot contact areas, is represented as the ground truth value, Represented as the predicted output.

10. A foot contact detection system based on a multi-scale spatiotemporal graph convolutional network, characterized in that: include: The data acquisition module is used to collect video data and convert each action sequence in the video data into an image sequence; The preprocessing module is used to process the image sequence through a pre-trained joint detection deep network model based on the pose estimation coordinate classification method, obtain skeleton data, and intercept the input sequence centered on the target detection frame; The window generation module is used to perform stratified sampling on the input sequence, generate subsequences based on time scales, and obtain multi-interval sliding windows; The feature extraction module is used to process the subsequences of each time scale using the spatiotemporal graph convolutional network to extract high-dimensional spatiotemporal feature graph features; The spatiotemporal averaging module is used to perform temporal average pooling on the spatiotemporal feature map, while weightedly aggregating joint features through the spatial attention mechanism and converting the spatiotemporal feature map into a global representation; The scale fusion module is used to achieve information exchange and complementarity across time scales and fuse spatiotemporal features based on a cross-scale enhancement strategy; The training output module is used to train the joint detection deep network model based on the first-order optimization algorithm, input a two-dimensional key point sequence, output a four-dimensional contact probability vector, generate a binary contact label after threshold judgment, and obtain the foot contact status.

Images