Somatosensory interaction sailboat steering simulation method and system based on three-dimensional human body posture estimation

Abstract

The application belongs to the technical field of sailboat operation simulation, and discloses a somatosensory interaction sailboat operation simulation method and system based on three-dimensional human body posture estimation. The method uses a camera to capture real scene pictures in real time, and uses a 2D posture detector to obtain a 2D human body posture; the 2D human body posture is input into a three-dimensional human body posture estimation method enhanced by a graph-guided state space, and a 3D human body posture is calculated; the obtained 3D human body posture is mapped to a somatosensory control module in a sailboat movement operation virtual scene, and interactive operation based on somatosensory action is realized. The application combines the local feature extraction advantage of the space-time expanded graph convolution model and the global feature extraction advantage of the hierarchical joint enhanced selective state space model, ensures the real-time performance of the model, and exhibits the accuracy advantage in the test of the public data set, and is more suitable for the sailboat operation simulation task of somatosensory interaction.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of sailboat operation simulation, and particularly relates to a somatosensory interaction sailboat operation simulation method and system based on three-dimensional human posture estimation. BACKGROUND

[0002] Sailboat operation simulation breaks through the constraints of external factors such as seasons, weather, and geographical environment, and becomes an effective tool for promoting sailboat sports. Current research can be divided into sailboat operation simulation based on a keyboard and mouse, sailboat operation simulation based on a physical object, sailboat operation simulation based on a VR headset, and the like according to different interaction modes. For example, a sailboat simulation patent proposed by the prior art combines a sailboat kinematic model to build an interactive desktop sailboat simulation system based on a keyboard and mouse, but the interaction mode of this method is limited to a keyboard and mouse, and cannot form sailboat operation action memory through simulation training; a simulation system based on a physical simulator often has better operation effect, for example, the prior art proposes an external rudder device for sailboat driving simulation training, which provides more accurate rudder angle information for sailboat operation simulation; the prior art proposes an OP-level sailboat operation simulation platform and a control method thereof, which provides a land simulation operation mode for beginners. However, these methods require special equipment and sites, and have high deployment complexity; sailboat simulation based on a VR headset reduces the demand for sites while maintaining strong immersion; the prior art provides a set of adjustable lighting devices for users while using VR technology, but due to the high cost of the VR headset device, popularization is difficult. Somatosensory interaction technology has the advantages of simple deployment, low cost, and natural interaction, and can effectively make up for the shortcomings of the above methods, providing a new development direction for sailboat operation simulation. The core technology of somatosensory interaction is high-quality and rapid estimation of three-dimensional posture. Early research in this field uses wearable sensors, the prior art proposes a somatosensory interaction method based on a wearable smart device, which achieves high estimation accuracy, but limits the freedom of user movement and reduces user experience. Another part of the research uses a Kinect or the like as a three-dimensional posture acquisition method, the prior art designs a somatosensory interactive virtual rehabilitation training method and system based on a Kinect, which guides users to complete specified rehabilitation actions. Although this method proves the effectiveness of the Kinect device in somatosensory interaction tasks, the deployment cost is still high. Therefore, these solutions are not conducive to the promotion and popularization of sailboat sports. With the development of three-dimensional human posture estimation methods based on deep learning, the three-dimensional coordinate positions and angle information of human joints can be estimated from two-dimensional images, which provides an opportunity for more low-cost sailboat sports simulation based on ordinary cameras, but also presents certain challenges.

[0003] Current deep learning-based 3D human pose estimation tasks can be divided into single-stage and two-stage methods according to the completion method. Single-stage methods generally design an end-to-end network to directly extract features from images and predict three-dimensional joint information. Such methods have higher information utilization. However, these network designs are often very complex, especially when predicting volume heat maps or regressing three-dimensional coordinates, requiring extremely high computational cost, making it difficult to combine temporal information, and the problem of temporal consistency is more prominent. Two-stage methods first use 2D pose estimation methods to extract two-dimensional joint information from input images or videos; then, based on the two-dimensional joint information, three-dimensional coordinates are predicted. This method improves the usability and flexibility of the network through phased optimization, and with the continuous development of the field of 2D human pose estimation, 2D-to-3D methods have gradually become the mainstream of research. Among them, methods based on Transformer have shown excellent performance in temporal pose estimation tasks. For example, PoseFormer is a spatio-temporal Transformer structure used to model the relationship between human joints in frames and the temporal correlation between frames; MixSTE method alternately uses time and space Transformer modules to encode the spatio-temporal dependence of joints more finely; MHFormer, MotionAGFormer, PoseFormerV2, and other methods are variants or improvements of the transformer method, but these methods generally have high computational complexity, although subsequent optimization of the Token pruning in the transformer and the use of Uplift and Upsample network architecture to reduce computational complexity, they are difficult to meet the real-time task requirements of tasks such as sailboat manipulation based on human pose estimation.

[0004] To further improve computational efficiency and modeling capability, Mamba, as a representative research work, integrates time-varying parameters into a selective state space model based on input in the state space model (SSM) framework, allowing the model to selectively process information, thereby filtering out irrelevant interference signals while memorizing important information over a long period of time, enhancing the model's dynamic adaptability and reasoning ability; the VMamba method further enhances the modeling capability of Mamba for complex visual tasks by capturing multi-dimensional and multi-directional global spatio-temporal dependencies; Hamba successfully realizes efficient hand three-dimensional pose reconstruction based on VMamba. The subsequent Posemamba method uses a rearrangement scanning strategy related to joint order to improve the performance of three-dimensional human pose estimation, but this method is a sequence-to-sequence estimation strategy that uses both past and future frame information during estimation, which is not advantageous in real-time data processing.

[0005] In addition, existing technologies have proposed a pose estimation method for human-computer interaction, which uses a dual-branch twin supervised network with an efficient channel attention mechanism for 3D human pose estimation and validates its effectiveness on a self-made dataset. Existing technologies have also proposed an intelligent interactive system for intelligent exhibition halls, which includes a regression-based 3D human pose estimation method. Existing technologies have proposed a method that integrates two Transformer attention mechanisms for multi-scale temporal feature extraction, thereby improving model accuracy. However, the model architecture is designed as a sequence-to-sequence estimation method, which cannot meet the needs of real-time action recognition applications. Summary of the Invention

[0006] To overcome the problems existing in related technologies, the present invention discloses an embodiment of a somatosensory interactive sailboat maneuvering simulation method and system based on three-dimensional human posture estimation, specifically relating to a somatosensory interactive sailboat maneuvering simulator based on a three-dimensional human posture estimation method.

[0007] The technical solution is as follows: A somatosensory interactive sailboat maneuvering simulation method based on three-dimensional human posture estimation, comprising the following steps:

[0008] S1 uses a camera to capture real-time images and a 2D human posture detector to obtain 2D human posture.

[0009] S2, input the 2D human pose into the graph-guided state space-enhanced 3D human pose estimation method to calculate the 3D human pose;

[0010] S3 maps the obtained 3D human posture to the motion control module in the virtual scene of sailing operation, realizing interactive operation based on motion sensing.

[0011] Furthermore, in step S1, the 2D human pose is obtained by using a 2D pose detector, which is to extract the 2D human pose from the real-shot image using RTMPose2D.

[0012] In step S2, the graph-guided state-space enhanced 3D human pose estimation method includes:

[0013] S21, Modify the PoseFormerV2 model into a baseline model suitable for real-time estimation, which takes only the t-frame two-dimensional pose sequence of the target frame as input;

[0014] S22, the SGJMamer layer is used to extract local spatial features from the pose of the target frame in the past f frames, and the TGJMamer layer is used to extract global spatiotemporal features Y from the low-frequency information of the complete input and the spatial features extracted by SGJMamer.

[0015] The SGJMamer / TGJMamer layer is composed of an STE-GCN module, a JMamba module and a Transformer module.

[0016] The Transformer module in the SGJMamer / TGJMamer layer is a spatial Transformer module and a temporal Transformer module, respectively; the JMamba module introduces a hierarchical joint-enhanced selective state space model to improve the ability of the model to extract global features; and the STE-GCN module learns the graph structure relationship of the joints through a spatiotemporal extended graph convolution network to enhance the attention to local features.

[0017] S23, constructing a regression head to infer the joint information of the target frame, constructing a loss function to train the target frame joints, and then outputting the three-dimensional pose of the target frame;

[0018] Further, in step S22, the STGJMamer is used to perform spatiotemporal feature extraction of the human pose model to obtain an output Y of fused spatiotemporal features, including:

[0019] S221, constructing a graph state space Transformer layer SGJMamer to perform local spatial feature extraction;

[0020] S222, constructing a graph state time Transformer layer TGJMamer to fuse the low-frequency information of the complete input and the spatial features extracted in step S221, and to perform spatiotemporal feature extraction to obtain an output Y of fused spatiotemporal features;

[0021] Further, step S221 of constructing a graph state space Transformer layer SGJMamer to perform local spatial feature extraction includes:

[0022] The f frame information [x t-f ,x t ] of the target frame in the past is input, sequentially passes through the STE-GCN module and the JMamba module, and obtains features X SpaGJMamba containing rich information; the data is sent to the spatial Transformer module.

[0023] The STE-GCN module predefines the spatial connection between the joints in the same frame and the temporal dependency of the same joint in different frames through a spatiotemporal extended adjacency matrix, and then extracts the spatiotemporal dependency between the joints through graph convolution processing; specifically, the processing process of the STE-GCN module to extract the spatiotemporal dependency between the joints includes:

[0024]

[0025] In the formula, is a normalized spatio-temporal adjacency matrix, D graph is a degree matrix, and the diagonal elements represent the connection degree of each node; A graph is an adjacency matrix, which contains spatial and temporal relationships; H' graph is an output feature matrix, and the dimension is E x T x N x F', and sigma is an activation function, H original is an input feature matrix, and the dimension is E x T x N x F; W is a learnable weight matrix of graph convolution, and Drop() is a random dropout mechanism;

[0026] H' processed by graph convolution contains local information, and a hybrid residual design is introduced to fuse the original features and the graph convolution features using dynamic weighting, and the formula is as follows:

[0027] X graph =α·H' graph +(1-α)·H original

[0028] In the formula, X graph is a feature result processed by a residual mechanism; alpha is a learnable weight parameter for dynamically adjusting the proportion of the two features; H' graph is a feature processed by graph convolution, and H original is an original feature without graph convolution processing.

[0029] The JMamba module, on the basis of the VMamba, uses a hierarchical joint enhancement method based on anatomical and kinematic chain reactions to perform feature extraction in information transmission between related joints and complete pose estimation.

[0030] The hierarchical joint enhancement method based on anatomical and kinematic chain reactions comprises:

[0031] In the joint enhancement sequence, the enhancement sequence is designed based on anatomy and kinematic chain reactions, which not only considers the characteristics of linkage between parent-child joints of the human body, but also considers the joint chain reaction exhibited by the human body in maintaining balance during movement. Through mutual enhancement between joints, information transmission is achieved, thereby improving the ability of the model to model complex movements; a mirror flipping operation is applied to the input data to generate a symmetric action; the poses of the original action and the flipped action are estimated simultaneously during the pose estimation process, and the final output is obtained by averaging the results of the original action and the flipped action, thereby achieving enhancement of data symmetry;

[0032] Specifically, the specific processing method of the JMamba module to complete pose estimation comprises: combining the hierarchical joint enhancement method based on anatomy and kinematic chain reactions, constructing a bidirectional to-be-scanned information X scanand calculate the learnable matrices required for selective state space scanning: state matrix A, control matrix B, output matrix C, instruction matrix D and state variable H;

[0033] X scan = f Scan (W in ·X in +b in )

[0034] where X scan is bidirectional information to be scanned, f Scan is the initialization function of bidirectional scanning, which is used to project the original input to the state space; W in is the original joint sequence, X in is the bias term, and b in is the weight matrix;

[0035] Local bidirectional scanning and global bidirectional scanning are performed, and the state space is constantly updated to capture long and short term dependencies, and the expression is:

[0036] H' = A·H + B·X scan

[0037] where H' is the output feature matrix;

[0038] The dimension is:

[0039] E×T×N×F'

[0040] where E is the batch size, T is the number of input frames, N is the number of joints, and F' is the feature corresponding to each joint;

[0041] X SSM = C·H' + D·X scan

[0042] where X SSM is the intermediate feature generated by selective state space scanning;

[0043] After bidirectional scanning is completed, feature fusion is performed to obtain complete feature expression rich in global and local information:

[0044] X out = f Merge (X SSM )

[0045] where X out is the final output result of this module, and f Merge is the integration of bidirectional scanning results into the original dimension;

[0046] The input features of the space Transformer module are first normalized and processed by multi-head self-attention to capture the internal relationship between joints; then, the features are normalized and processed by an MLP layer to learn the complex nonlinear relationship between the input features by introducing a nonlinear transformation through an activation function GELU.

[0047] The spatial feature extraction includes:

[0048] Q, K, V = linear(LayerNorm(X SpaGJMamba ))

[0049]

[0050] x spatial = x res + Drop(x mlp )

[0051] In the formula, Q is a query vector, K is a key vector, V is a value vector, linear() is a full connection layer, LayerNorm() is a normalization process, X SpaGJMamba is the spatial feature processed by the STE-GCN module and the JMamba module, x res is an intermediate process result, x attn is a feature processed by an attention mechanism, softmax() is a vector normalization to a probability distribution vector, K T is the transpose of K, d k is a scaling factor, is an intermediate processing, Drop() is a random drop processing, x mlp is a feature processed by an MLP mechanism, W1 and W2 are weight matrices of linear transformation, GELU() is an activation function processing, b1 and b2 are bias terms, x spatial is the spatial dimension feature in the target frame estimation process.

[0052] Further, in step S222, a graph state time Transformer layer TGJMamer is constructed to fuse the complete input low-frequency information and the spatial feature extracted in step S221, and to perform spatio-temporal feature extraction to obtain an output Y of fused spatio-temporal features, including:

[0053] The graph state time Transformer layer is composed of an STE-GCN module, a JMamba module and a time Transformer module; the data input is the low-frequency part of the complete action sequence, which captures the change characteristics of the action from the global information and obtains the overall evolution trend of the action; after being processed by the STE-GCN module and the JMamba module, the frequency domain feature X TGJMamba is obtained, and this frequency domain information will be combined with the spatial feature xspatial The spatial-temporal information is combined and sent into the self-attention mechanism for processing; when the processed data passes through the MLP layer, the spatial feature part data will be subjected to frequency domain transformation, the local detail changes are converted into frequency domain information, and the low-frequency time information is effectively supplemented.

[0054] Further, in step S23, a regression head is constructed to infer the joint information of the target frame, a loss function is constructed to train the target frame joint, and the joint error is calculated between the output result of the regression head and the real 3D pose, including:

[0055] The spatial-temporal features Y extracted from the input two-dimensional sequence pose information have a feature dimension of Y represents a set to which Y belongs, f is the frame number of the input feature; the target frame is the last frame of the sequence, and the dimension is The regression head is constructed to infer the joint information of the target frame; the regression head is composed of a weighted average operation and a fully connected layer, and the estimated pose information is calculated under the weighting of the learnable weight parameter to obtain the final estimation Y of the target frame predicate ;

[0056] Y predicate =W head ·LayerNorm(mean(Y))+b head

[0057] In the formula, W head is a trainable weight, mean(Y) is the mean value, b head is a bias term, and LayerNorm() is a layer normalization;

[0058] The output result is compared with the real 3D pose Y target to calculate the joint error MPJPE, and the MPJPE is used as a loss function to train the model;

[0059]

[0060] In the formula, y k , are the real information and the predicted information of the kth joint, respectively.

[0061] Another object of the application is to provide a somatosensory interaction sailboat steering simulation system based on three-dimensional human pose estimation, comprising:

[0062] A 2D human pose capture module is used to capture real scene pictures in real time by using a camera and obtain 2D human poses by using a 2D pose detector;

[0063] The 3D human pose calculation module is used to input 2D human pose into a graph-guided state space-enhanced 3D human pose estimation method to calculate 3D human pose.

[0064] The motion-sensing interaction module is used to map the obtained 3D human posture onto the motion-sensing control module in the virtual scene of sailing operation, so as to realize interactive operation based on motion-sensing movements.

[0065] Combining all the above technical solutions, the beneficial effects of this invention are as follows: Based on the lightweight Transformer model PoseformerV2, this invention integrates the advantages of spatiotemporally extended graph convolutional models in local feature extraction and hierarchical joint-enhanced state-space models in global feature extraction. While ensuring the real-time performance of the model, it demonstrates its accuracy advantage in tests on public datasets, making it more suitable for motion-sensing interactive sailboat handling simulation tasks.

[0066] This invention proposes a 3D human pose estimation network, STGJMamer, which integrates the hierarchical joint-enhanced selective state space module JMamba and the spatiotemporally extended graph convolutional module STE-GCN. This invention also proposes a motion-sensing interactive sailboat maneuvering simulation based on a 3D human pose estimation method. By proposing the hierarchical joint-enhanced selective state space module JMamba, this invention achieves efficient extraction of global human pose features through hierarchical joint enhancement, improving the model's accuracy. Furthermore, by proposing the spatiotemporally extended graph convolutional module STE-GCN, this invention enhances the model's attention to local information through graph convolution, thereby improving the model's estimation accuracy. This invention deploys STGJMamer in a motion-sensing sailboat maneuvering simulation training system, verifying its effectiveness in practical applications. Moreover, this method only uses past frames as input and can run in real time, thus possessing high practical application value. Attached Figure Description

[0067] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0068] Figure 1 This is a flowchart of the motion-sensing interactive sailboat handling simulation method based on three-dimensional human posture estimation provided in the embodiments of the present invention;

[0069] Figure 2 This is a schematic diagram of the somatosensory interactive sailboat maneuvering simulation method based on three-dimensional human posture estimation provided in this embodiment of the invention;

[0070] Figure 3is a principle diagram of a three-dimensional human pose estimation method enhanced by a graph-guided state space provided by an embodiment of the present application, wherein (a) is a STGJMamer network structure diagram, (b) is a SGJMamer / TGJMamer network architecture diagram, (c) is a space Transformer network architecture diagram, and (d) is a time Transformer network architecture diagram;

[0071] Figure 4 is a principle diagram of a hierarchical joint enhancement method based on an anatomical and kinematic chain reaction provided by an embodiment of the present application;

[0072] Figure 5 is a space-time expansion adjacency matrix in a STE-GCN provided by an embodiment of the present application;

[0073] Figure 6 is a qualitative effect of pose estimation of STGJMamer in different input video cases provided by an embodiment of the present application;

[0074] Figure 7 is a somatosensory interaction action control design. DETAILED DESCRIPTION

[0075] In order to make the above-mentioned objects, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the scope of the present application, so the present application is not limited to the specific implementations disclosed below.

[0076] The innovation of the present application is that the present application proposes a three-dimensional pose estimation network enhanced by a graph-guided state space (STGJMamer). It adopts a lightweight Transformer method PoseformerV2 as a benchmark model, combines a space-time expansion graph convolution and a hierarchical joint enhanced selective state space model to enhance the modeling ability of the model to global and local features on the premise of ensuring the real-time performance of the model, so as to improve the pose estimation accuracy of the mobile user.

[0077] Embodiment 1, as shown in Figure 1 The somatosensory interaction sailboat control simulation method based on three-dimensional human pose estimation provided by an embodiment of the present application comprises:

[0078] S1, real-time capturing a real scene picture by using a camera, and obtaining a 2D human pose by using a 2D pose detector;

[0079] S2, inputting the 2D human pose into a three-dimensional human pose estimation method enhanced by a graph-guided state space, and calculating a 3D human pose;

[0080] S3, mapping the obtained 3D human pose to a somatosensory operation control module in a virtual scene of sailboat operation, to realize interactive operation based on somatosensory action.

[0081] For example, in step S1, the 2D human pose is obtained by using a 2D pose detector, which is extracted by RTMPose2D from a real picture.

[0082] For example, in step S2, the calculation of the 3D human pose includes:

[0083] The accuracy of 3D human pose estimation is reflected by different core evaluation indexes on the MPI-INF-3DHP dataset.

[0084] The different core evaluation indexes include: the percentage of correct key points PCK under the threshold of 150mm, the area under the curve AUC, and the average per-joint position error MPJPE.

[0085] For example, the present application designs specific somatosensory interaction actions for various operations of sailboat movement, and the system continuously monitors the 3D human pose mapped into the virtual scene and matches it with the preset action template, if the matching is successful, the sailboat operation function in the scene is triggered to complete the somatosensory interaction. Figure 2 Schematic diagram of somatosensory interaction sailboat operation simulation method based on three-dimensional human pose estimation.

[0086] In embodiment 2, in step S2, the three-dimensional human pose estimation method of graph-guided state space enhancement, namely STGJMamer network; as shown in Figure 3 The network includes:

[0087] Figure 3 (a) in the figure STGJMamer network structure: the model takes a two-dimensional pose sequence of t frames as input. SGJMamer extracts local spatial features from f frames of poses from the target frame, and TGJMamer extracts global spatiotemporal features from the complete input low-frequency information and the spatial features extracted by SGJMamer. Finally, the features are calculated by a regression head to obtain the three-dimensional pose of the target frame. Figure 3 (b) in the figure SGJMamer / TGJMamer network architecture: composed of a STE-GCN module, a JMamba module and a Transformer module. Figure 3 (c) spatial Transformer module, and Figure 3 (d) in the figure time Transformer module.

[0088] The present application is based on the PoseFormerV2 model, which is modified into a model suitable for real-time estimation task as the baseline model of the present application. On this basis, the present application introduces a hierarchical joint enhanced selective state space module JMamba, which improves the global feature extraction capability of the model through the joint level enhanced bidirectional scanning method; in addition, the present application also introduces a spatio-temporal expansion graph convolution module STE-GCN, which learns the graph structure relationship of the joint through the spatio-temporal expansion graph convolution, and enhances the attention degree of the model to the local features; so as to realize the improvement of the model precision.

[0089] Specifically, the three-dimensional human pose estimation method comprises the following steps:

[0090] S21, the three-dimensional human pose estimation method guided by the graph guided state space enhancement is based on the PoseFormerV2 model, but the PoseFormerV2 model utilizes the past information and future information of the target frame at the same time, and is not suitable for real-time task, therefore, the present application first modifies the PoseFormerV2 model into a baseline model suitable for real-time estimation, which only takes the past t-frame two-dimensional pose sequence of the target frame as input;

[0091] S22, the local spatial features are extracted from the f-frame pose of the past target frame by using the SGJMamer layer, and the global spatio-temporal features Y are extracted from the low-frequency information of the complete input and the spatial features extracted by the SGJMamer by using the TGJMamer layer.

[0092] Wherein, the SGJMamer / TGJMamer layer is composed of STE-GCN module, JMamba module and Transformer module;

[0093] The Transformer module in the SGJMamer / TGJMamer layer is a spatial Transformer module and a time Transformer module respectively; the JMamba module introduces a hierarchical joint enhanced selective state space model to improve the global feature extraction capability of the model; the STE-GCN module learns the graph structure relationship of the joint through the spatio-temporal expansion graph convolution network, and enhances the attention degree to the local features.

[0094] S23, the regression head is constructed to infer the joint information of the target frame, the loss function is constructed to train the joint of the target frame, and then the three-dimensional pose of the target frame is output;

[0095] Exemplarily, in step S22, the STGJMamer is adopted to extract the spatio-temporal features of the human pose model, and the output Y of the fused spatio-temporal features is obtained, which includes:

[0096] S221, constructing a graph state space Transformer layer SGJMamer to perform local spatial feature extraction;

[0097] S222, constructing a graph state time Transformer layer TGJMamer to fuse the low-frequency information of the complete input and the spatial features extracted in step S221 and to perform spatiotemporal feature extraction to obtain an output Y of fused spatiotemporal features;

[0098] For example, step S221 of constructing a graph state space Transformer layer SGJMamer to perform local spatial feature extraction includes:

[0099] As shown in FIG. b, Figure 3 As shown in FIG. b, the graph state space Transformer layer is composed of an STE-GCN module, a JMamba module and a space Transformer module. The f frame information [x t-f ,x t ] of the target frame in the past is input, sequentially passes through the STE-GCN module and the JMamba module, obtains the feature X SpaGJMamba containing rich information, and the data is sent to the space Transformer module.

[0100] The STE-GCN module predefines the spatial connection between the joints in the same frame and the temporal dependence of the same joint in different frames through a spatiotemporal expansion adjacency matrix, and then extracts the spatiotemporal dependence between the joints through graph convolution processing. Specifically, the processing process of the STE-GCN module to extract the spatiotemporal dependence between the joints includes:

[0101]

[0102] In the formula, is the normalized spatiotemporal adjacency matrix, D graph is the degree matrix, and the diagonal elements represent the connection degree of each node; A graph is the adjacency matrix, which contains spatial and temporal relationships; H' graph is the output feature matrix, with a dimension of E×T×N×F', σ is an activation function, H original is the input feature matrix, with a dimension of E×T×N×F; W is a learnable weight matrix of graph convolution, and Drop() is a random dropout mechanism.

[0103] The space Transformer module, the input feature first passes through normalization and multi-head self-attention processing to capture the internal relationship between the joints. Then, the feature passes through normalization and an mlp layer to learn the complex nonlinear relationship between the input features through an activation function GELU to introduce nonlinear transformation.

[0104] The spatial feature extraction comprises:

[0105] Q,K,V = linear(LayerNorm(X SpaGJMamba ))

[0106]

[0107] x spatial = x res + Drop(x mlp )

[0108] In the formula, Q is a query vector, K is a key vector, V is a value vector, linear() is a full connection layer, LayerNorm() is a normalization processing, X SpaGJMamba is a spatial feature processed by the STE-GCN module and the JMamba module, x res is an intermediate processing process result, x attn is a feature processed by an attention mechanism, softmax() is a vector normalization to a probability distribution vector, K T is the transpose of K, d k is a scaling factor, is an intermediate processing, Drop() is a random drop processing, x mlp is a feature processed by an mlp mechanism, W1 and W2 are weight matrixes of linear transformation, GELU() is an activation function processing, b1 and b2 are bias terms, x spatial is a spatial dimension feature in a target frame estimation process.

[0109] In the experiment, since the video stream data is continuously input, and there is significant time correlation information between frames, it is necessary to expand the graph convolution to the time dimension to extract more local feature information. The existing spatio-temporal graph convolution model realizes hierarchical modeling of spatio-temporal dynamic relationship by stacking spatio-temporal convolution and graph convolution operations, but this will bring huge additional computational overhead.

[0110] In order to realize efficient graph convolution operation, the application proposes a spatio-temporal expansion adjacency matrix, as Figure 5For the STE-GCN spatio-temporal expansion adjacency matrix, in the figure, Ji (i = 0, 1,..., 16) represents the joint sequence number; tj (j = 0, 1,..., f) represents the frame; in summary, Ji (tj) represents the i th joint in the j th frame. The spatio-temporal expansion graph convolution (STE-GCN) uses this adjacency matrix to predefine the spatial connection between the joints in the same frame and the temporal dependence of the same joint in different frames. Based on this structure, the graph convolution can effectively extract the spatio-temporal dependence between the joints, so that the model pays attention to the internal relationship between the joints in the feature extraction process, and captures more rich local information. This method essentially extracts the temporal features of multiple frames, effectively avoiding redundant high-dimensional time modeling.

[0111] The H' processed by the graph convolution contains more local information. In order to keep the original information from being lost, a hybrid residual design is introduced to fuse the original features and the graph convolution features together using dynamic weighting, as follows:

[0112] X graph = a · H' graph + (1-a) · H original

[0113] In the formula, X graph is the feature result processed by the residual mechanism; a is a learnable weight parameter for dynamically adjusting the proportion of the two features; H' graph is the feature processed by the graph convolution, and H original is the original feature without graph convolution processing.

[0114] Exemplarily, the JMamba module uses an anatomical and kinematic cascade-based hierarchical joint enhancement method on the basis of the VMamba to perform feature extraction in the information transmission between related joints and complete pose estimation.

[0115] The anatomical and kinematic cascade-based hierarchical joint enhancement method comprises:

[0116] In the joint enhancement sequence, the enhancement sequence is designed based on anatomy and kinematic cascade, which not only considers the characteristics of linkage between parent-child joints of the human body, but also considers the joint cascade exhibited by the human body in the process of maintaining balance. Through mutual enhancement between joints, information transmission is achieved, thereby improving the ability of the model to model complex motion; a mirror flip operation is applied to the input data to generate a symmetric action; the poses of the original action and the flipped action are estimated simultaneously in the pose estimation process, and the final output is obtained by averaging the results of the original action and the flipped action, thereby achieving enhancement of data symmetry.

[0117] Exemplarily, VMamba effectively stores and propagates long-term history information through implicit recursive computation and bidirectional scanning mechanism, thereby achieving stable capture of long-term dependency. The present application fuses VMamba to quickly establish a selective state space layer and model joint sequences in a bidirectional scanning manner, but as Figure 4 As shown in the average error analysis of different joints in the hierarchical joint enhancement method based on anatomy and kinematic chain reaction, the method is prone to produce larger deviations in the end joints during pose estimation. Therefore, the present application proposes a hierarchical joint enhancement method based on anatomy and kinematic chain reaction, which strengthens the information transmission between related joints of VMamba, further enhances the feature extraction capability of the model, and improves the accuracy of pose estimation.

[0118] In the design process of the joint enhancement sequence, the present application considers two aspects. On the one hand, the present application considers the anatomical sequence relationship, i.e. physically connected joints generally exhibit interdependent motion during movement. Therefore, enhancing the parent-child node relationship can effectively amplify action features and further extract more rich motion information. On the other hand, the present application also considers the motion chain reaction in complex motion. When one part of the body moves, other parts will also move accordingly in order to maintain the overall coordination of the body. For example, lifting one leg will cause the opposite hip joint to move, and bending the waist will cause the knee joint to move.

[0119] Considering these two aspects, the final joint enhancement sequence is shown in Table 1. In addition, since the human joints have symmetry, the present application applies a mirror flip operation to generate symmetric actions during training and testing. In pose estimation, the present application estimates the poses of both the original action and the flipped action, and obtains the final output by averaging the results of the two, thereby achieving enhancement of data symmetry.

[0120] Table 1 Joint enhancement sequence

[0121] 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 First 0 0 1 2 3 0 4 5 6 8 11 8 11 12 14 15 16 Second 0 0 2 2 3 5 4 7 6 8 11 8 11 12 14 15 16

[0122] As Figure 4 As shown in the hierarchical joint enhancement method demonstration in Figure (b), the right leg joints are taken as an example: during scanning, the model first performs a structural enhancement according to the parent-child relationship of the joints and the correlation characteristics of human motion, i.e. node 0 enhances node 1, node 1 enhances node 2, and node 2 enhances node 3. Subsequently, the end joint 3 is enhanced again, with the parent node 2 as the bridge for information transmission, linking the information of the grandparent node 1, so that the end joint 3 features will contain all the information of nodes 1, 2 and 3, thereby achieving full utilization of information between nodes.

[0123] The joint information contains rich local information after the process of layer-by-layer enhancement. In order to maintain the complete expression of global information, in the modeling process, the present application performs bidirectional scanning according to the local enhanced information on the one hand, and performs global bidirectional scanning according to the information that is not completely enhanced on the other hand, finally, the two are fused to output more rich feature information. The network structure of JMamba is as shown in the (b) figure. Figure 3

[0124] Specifically, the specific processing method of the JMamba module to complete the pose estimation includes: combining the hierarchical joint enhancement method based on anatomy and kinematic cascade reaction, constructing bidirectional scanning information X composed of enhanced joint node sequence and original joint node sequence scan , and calculating the learnable matrix required for selective state space scanning: state matrix A, control matrix B, output matrix C, instruction matrix D and state variable H;

[0125] X scan =f Scan (W in ·X in +b in )

[0126] In the formula, X scan is bidirectional scanning information, f Scan is an initialization function of bidirectional scanning, used to project the original input to the state space; W in is the original joint node sequence, X in is the bias term, and b in is the weight matrix;

[0127] Local bidirectional scanning and global bidirectional scanning are performed, and the state space is constantly updated to capture long and short term dependencies, and the expression is:

[0128] H'=A·H+B·X scan

[0129] In the formula, H' is the output feature matrix;

[0130] The dimension is:

[0131] E×T×N×F'

[0132] In the formula, E is the batch size, T is the input frame number, N is the number of joints, and F' is the feature corresponding to each joint;

[0133] X SSM =C·H'+D·X scan

[0134] In the formula, X SSM ​Intermediate features generated by selective state space scanning;

[0135] After the completion of bidirectional scanning, feature fusion is performed to obtain complete feature expression rich in global and local information:

[0136] X out =f Merge (X SSM )

[0137] In the formula, X out is the final output result of the module, f Merge is the integration of bidirectional scanning results into the original dimension;

[0138] For example, at step S222, a graph state time Transformer layer TGJMamer is constructed to fuse the complete input low-frequency information and the spatial features extracted at step S221, and to perform spatio-temporal feature extraction to obtain the output Y of the fused spatio-temporal features, including:

[0139] The graph state time Transformer layer is composed of an STE-GCN module, a JMamba module and a time Transformer module; the data input is the low-frequency part of the complete action sequence, which captures the change characteristics of the action from the global information and obtains the overall evolution trend of the action; after processing by the STE-GCN module and the JMamba module, the frequency domain feature X TGJMamba is obtained, which will be combined with the spatial feature x spatial to form spatio-temporal information for processing by the self-attention mechanism; when the processed data passes through the mlp layer, the spatial feature part of the data will be frequency domain transformed to convert the local detail changes into frequency domain information, thereby effectively supplementing the low-frequency time information.

[0140] For example, after processing by the STE-GCN module and the JMamba module similar to SGJMamer, the frequency domain feature X TGJMamba is obtained, which will be combined with the spatial feature x spatial to form spatio-temporal information for processing by the self-attention mechanism. The difference is that when the processed data passes through the multi-layer perceptron, the spatial feature part of the data will be frequency domain transformed by discrete cosine transform to convert the local detail changes into frequency domain information, thereby effectively supplementing the low-frequency time information; the discrete cosine transform formula is:

[0141]

[0142] In the formula, k is the frequency index, k = 0, 1, 2, …, N-1; i is the time index, i = 0, 1, 2, …, N-1; N is the length of the input signal;

[0143] The inverse discrete cosine transform formula is:

[0144]

[0145]

[0146] Exemplarily, in step S23, the regression head is constructed to infer the joint information of the target frame, the loss function is constructed to train the target frame joint, and the joint error is calculated between the output result of the regression head and the real 3D pose, including:

[0147] The spatiotemporal feature Y extracted from the input two-dimensional sequence pose information has a feature dimension of Y represents a set to which Y belongs, f is the frame number of the input feature; the target frame is the last frame of the sequence, and the dimension is The regression head is constructed to realize the inference of the joint information of the target frame; the regression head is composed of a weighted average operation and a full connection layer, and the estimated pose information is calculated under the weighting of the learnable weight parameter to obtain the final estimation Y of the target frame predicate ;

[0148] Y predicate =W head ·LayerNorm(mean(Y))+b head

[0149] In the formula, W head is a trainable weight, mean(Y) is the mean value, b head is a bias term, and LayerNorm() is a layer normalization;

[0150] The output result is compared with the real 3D pose Y target to calculate the joint error MPJPE, and the present application follows the setting of the benchmark model and adopts MPJPE as the loss function of the model to train the model.

[0151]

[0152] In the formula, y k , are the real information and the predicted information of the kth joint, respectively.

[0153] In embodiment 2, the present application provides a somatosensory interaction sailboat steering simulation system based on three-dimensional human pose estimation, which comprises:

[0154] A 2D human pose capture module is used to capture real scene pictures in real time by using a camera and obtain 2D human poses by using a 2D pose detector.

[0155] 3D human pose computation module, configured to input the 2D human pose into a 3D human pose estimation method enhanced by graph-guided state space, and compute a 3D human pose;

[0156] Somatosensory interaction module, configured to map the obtained 3D human pose to a somatosensory control module in a virtual scene of sailboat operation, and realize interactive operation based on somatosensory action.

[0157] Application Example 1: Implementation on a public data set.

[0158] Step 1: Data set. The MPI-INF-3DHP data set is significantly different from other similar data sets in its diversified indoor and outdoor scenes, extensive and dynamic action categories, strong action freedom, and fast action rhythm, and exhibits high similarity to the action conditions of subjects in real-time recognition environments. In particular, in the specific application scenario of sailboat operation simulation, the adaptability of the subject's random and aperiodic movement pattern to the model raises higher requirements, and therefore, the performance of the model on the MPI-INF-3DHP data set is considered to be more close to the actual needs of the sailboat operation simulation task. In order to compare with other methods, the data input in the experimental process follows the unified standard of experimental methods in recent years, and real 2D pose input is adopted. The data input length is set to 81 frames, and a 9-frame short sequence is used.

[0159] Step 2: Evaluation index. The present application focuses on three core evaluation indexes: PCK (percentage of correct key points) under the threshold of 150 mm, AUC (area under the curve), and MPJPE (mean per joint position error). These indexes can reflect the accuracy and stability of the model in the human pose estimation task from different dimensions.

[0160] Step 3: Experimental results.

[0161] Table 2: Experimental results compared with other methods on the MPI-INF-3DHP data set

[0162]

[0163]

[0164] As shown in Table 2, the model proposed in the present application exhibits better performance in all evaluation indicators. Specifically, the PCK value is as high as 97.2%, which means that in most cases, the model can accurately predict the position of the human key point within the specified threshold, reflecting its high precision in key point detection. At the same time, the AUC value reaches 77.9%, indicating that the model can maintain high detection accuracy under different thresholds, with good robustness. In addition, the MPJPE value is only 29.5mm, further proving the precision of the model in joint position estimation.

[0165] In contrast, some existing methods (such as VideoPose3D, MHFormer, etc.) perform well in PCK, but perform poorly in AUC and MPJPE, indicating that these methods lack stability. Although these methods can accurately locate key points under ideal conditions (such as unoccluded scenes), they perform poorly under complex conditions such as dynamic occlusion and rapid movement, resulting in low overall stability and large error fluctuations. Some methods (such as GTA-Net, HDFormer, etc.) perform well in AUC, but perform poorly in PCK and MPJPE. This shows that these methods can maintain stable output under complex poses and occlusion conditions, but rely too much on global features, resulting in decreased local key point positioning accuracy and increased error. In addition, some methods (such as GLA-GCN, STCFormer, etc.) perform well in MPJPE with low model error, but often require high computational cost in practical applications.

[0166] The comprehensive performance of the model of the present application in the three evaluation indicators exceeds most existing methods. In addition, the design of the model relies entirely on past information, which also improves its practical application value in real-time pose estimation tasks. In summary, the model of the present application has good performance on the MPI-INF-3DHP dataset, which verifies its human pose estimation ability in complex dynamic scenes, providing strong technical support for subsequent application in the sailboat maneuvering simulation system.

[0167] Step 3, inference speed.

[0168] The model parameter amount of the model of the application is about 14.4M, the operation amount is about 0.177GMac, and the frame-by-frame inference speed is about 52FPS. In actual application, the application adopts RTMPose as a 2D detector and is deployed on an NVIDIA RTX4090 GPU. Experimental results show that the method of the application still maintains an inference speed of 38FPS in the process of complete estimation, and can fully meet the demand of real-time tasks. In the above experiments, the application all follows the commonly used flip mode of experiments in recent years to enhance the data, that is, the same data is estimated twice. In actual application, according to the performance demand of the device, the data flip function can be closed, at this time, the average estimation speed from the picture to the 3D human body posture can reach 60FPS.

[0169] Step 5, ablation experiment.

[0170] The application performs a systematic ablation experiment on the proposed module, and demonstrates the effectiveness of the component by adding the component one by one in the benchmark model. As shown in Table 3.

[0171] Table 3 compares the experimental results of the existing method

[0172]

[0173] Step 6, qualitative demonstration.

[0174] As Figure 6 As shown in the qualitative demonstration effect diagram, the application demonstrates the processing results of STGJMamer in random frames in different videos, which are Figure 6 It can be seen that the method of the application has good estimation of human body posture in various scenes.

[0175] Application example 2, implementation case in somatosensory interaction sailboat steering simulation.

[0176] Step 1, somatosensory interaction sailboat steering simulation.

[0177] The application builds a somatosensory interaction sailboat steering simulation in Unity3D, which is Figure 2The shown sailboat maneuvering simulation system based on somatosensory interaction applies the proposed STGJMamer in somatosensory interaction design to provide a natural interaction operation environment for users. The system includes five modules. Specifically, the user interaction interface provides a simple and intuitive operation interface to help users achieve friendly interaction with the system; the voice control module controls the system UI button function through voice instructions to provide users with a convenient control experience; the navigation environment simulation module is responsible for building a virtual environment to simulate real marine environment and weather conditions to ensure the authenticity and immersion of the training; the sailboat kinematics module simulates the motion feedback of the sailboat under different environmental conditions to provide real-time operation response; the teaching module and the training module provide customized training content for users to help users gradually improve their operation skills through different navigation tasks. Finally, the somatosensory control module realizes the interaction operation based on somatosensory action, accurately captures the user's body action, and converts it into the corresponding sailboat operation instruction.

[0178] Second step, application of STGJMamer in somatosensory interaction.

[0179] After obtaining the three-dimensional human body posture through the STGJMamer algorithm, the application realizes data transmission between Python and Unity3D by using TCP Socket communication, and realizes the binding of posture data and virtual role by using the Animator component of Unity.

[0180] As Figure 7 As shown, according to the sailboat operation task, the application designs six types of actions, including raising the sail cloth, lowering the sail cloth, turning the sail cloth to the left, turning the sail cloth to the right, steering, and resetting, and makes corresponding action essentials feature descriptions (as shown in Table 4), and performs action recognition through feature matching, so as to drive the sailboat in the virtual scene and realize somatosensory interaction.

[0181] Table 4 somatosensory interaction action feature description

[0182]

[0183] In the experiment, the application uses a simple and intuitive performance evaluation index: posture recognition success rate, that is, the number of successfully recognized actions divided by the total number of executed actions. This index is calculated by repeatedly executing each navigation simulation control action.

[0184] The application uses a camera with a resolution of 1920x1080, a frame capture rate of 30FPS, and tests each of the 6 control actions for 200 times. During the experiment, the application uses the RTMPose 2D detector to extract 2D human poses from real-world scenes as data preprocessing. The extracted pose information is input into the STGJMamer for three-dimensional human pose estimation, and then mapped to the sailboat steering simulation system. As shown in Table 4, the application defines specific evaluation criteria for each action at different execution stages. During system operation, the system continuously evaluates whether the current pose meets the predefined criteria. If it meets the criteria, the action is identified as successful; otherwise, it is considered unsuccessful. The experimental results are shown in Table 5, which shows that the lowest recognition accuracy of the posture recognition algorithm designed by the application reaches 86.5%, and the recognition accuracy of the remaining actions is above 90%, fully verifying the good performance of the algorithm.

[0185] Table 5 Action recognition experiment results

[0186] Action Executions Successes Accuracy % Up sail 200 193 96.5 Down sail 200 191 95.5 Right turn sail 200 186 93.0 Left turn sail 200 189 94.5 Steer 200 173 86.5 Reset 200 188 94.0

[0187] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, and any modification, equivalent replacement and improvement made by those skilled in the art within the technical range disclosed by the application and within the spirit and principles of the application shall be covered within the protection scope of the application.

Claims

1. A somatosensory interaction sailboat maneuver simulation method based on three-dimensional human body posture estimation, characterized in that, The method comprises the following steps: S1, real-time capture of real scene pictures by a camera, and obtaining a 2D human body posture by a 2D posture detector; S2, inputting the 2D human body posture into a graph-guided state space enhanced three-dimensional human body posture estimation method to calculate a 3D human body posture; S3, mapping the obtained 3D human body posture to a somatosensory operation module in a virtual scene of a sailboat operation to realize interactive operation based on somatosensory action; In step S2, the graph-guided state space enhanced three-dimensional human body posture estimation method comprises: S21, modifying a PoseFormerV2 model into a benchmark model suitable for real-time estimation, which takes a two-dimensional posture sequence of t frames in the past of a target frame as input; S22, extracting local spatial features from f frame postures in the past of the target frame by an SGJMamer layer, and extracting global spatio-temporal features Y from complete input low-frequency information and spatial features extracted by the SGJMamer layer by a TGJMamer layer; wherein the SGJMamer / TGJMamer layer is composed of a STE-GCN module, a JMamba module and a Transformer module; the Transformer module in the SGJMamer / TGJMamer layer is a spatial Transformer module and a time Transformer module respectively; the JMamba module introduces a hierarchical joint enhanced selective state space model to improve the ability of the model to extract global features; the STE-GCN module learns the graph structure relationship of the joints through a spatio-temporal expanded graph convolution network; S23, constructing a regression head to infer the joint information of the target frame, constructing a loss function to train the joint of the target frame, and then outputting the three-dimensional posture of the target frame; Step S221, constructing a graph state space Transformer layer SGJMamer for local spatial feature extraction, comprising: Target frame past f frame information [x t-f ,x t ] is input, sequentially through STE-GCN module and JMamba module, and feature X SpaGJMamba is obtained; data will be sent to spatial Transformer module; The STE-GCN module predefines the spatial connection between the joints in the same frame and the time dependence of the same joint in different frames through a spatio-temporal expansion adjacency matrix, and then extracts the spatio-temporal dependence relationship between the joints through graph convolution processing; The processing process of the STE-GCN module for extracting the spatio-temporal dependence relationship between the joints is as follows: wherein, is a normalized spatio-temporal adjacency matrix, D graph is a degree matrix, the diagonal elements represent the connection degree of each node; A graph is an adjacency matrix, containing spatial and temporal relationships; H' graph is an output feature matrix, with dimensions E x T x N x F', and σ is an activation function, H original is an input feature matrix, with dimensions E x T x n x F; W is a learnable weight matrix of graph convolution, and Drop() is a random dropout mechanism; After graph convolution processing, H' will contain local information. A hybrid residual design is introduced to fuse the original features and graph convolution features together using dynamic weighting, and the formula is as follows: X graph = a - H' graph + (1 - a) - H original In the formula, X graph is a feature result processed through a residual mechanism; a is a learnable weight parameter, used to dynamically adjust the proportion of the two features; H' graph is a feature processed through a graph convolution; H original is an original feature without graph convolution processing; The JMamba module uses a hierarchical joint enhancement method based on anatomy and kinematic cascade reaction on the basis of VMamba to extract features in the information transmission between related joints and complete posture estimation; The specific processing method of the JMamba module to complete the pose estimation includes: combining a hierarchical joint enhancement method based on anatomy and kinematic chain reaction to construct bidirectional to-be-scanned information X composed of an enhanced joint node sequence and an original joint node sequence scan and calculating a learnable matrix required for selective state space scanning: a state matrix A, a control matrix B, an output matrix C, an instruction matrix D, and a state variable H. X scan = f Scan (w in · X in + b in ) where X scan is the bidirectional scan information, f Scan is the initialization function for the bidirectional scan, which projects the raw input into the state space; W in is the raw junction sequence, X in is the bias term, b in is the weight matrix; Local bidirectional scanning and global bidirectional scanning are performed to constantly update the state space and capture long and short term dependencies, and the expression is as follows: H' = A - H + B - X scan In the formula, H' is an output feature matrix; The dimension is: E×T×N×F' In the formula, E is the batch size, T is the input frame number, N is the number of joints, and F' is the feature corresponding to each joint; X SSM = C - H' + D - X scan wherein X SSM is an intermediate feature generated by selective state space scanning; After bidirectional scanning, feature fusion is performed to obtain complete feature expression rich in global and local information: X out = f Merge (X SSM ) In the formula, X out is the final output result of the module, f Merge is the integration of the bidirectional scanning results into the original dimension; The spatial Transformer module is used for processing the input features, and the features are firstly normalized and processed by multi-head self-attention to capture the internal connection between joints; then, the features are normalized and processed by an MLP layer, and a nonlinear transformation is introduced by using an activation function GELU to learn the complex nonlinear relationship between the input features. The spatial feature extraction includes: Q, K, V = linear(LayerNorm(X SpaGJMamba )) x spatial = x res + Drop(x mlp ) In the formula, Q is a query vector, K is a key vector, V is a value vector, linear() is a full connection layer, LayerNorm() is a normalization process, X SpaGJMamba is a spatial feature processed by the STE-GCN module and the JMamba module, x res is an intermediate process result, x attn is a feature processed by an attention mechanism, softmax() is a vector normalization process to a probability distribution vector, K T is the transpose of K, d k is a scaling factor, is an intermediate processing, Drop() is a random drop processing, x mlp is a feature processed by an mlp mechanism, W1 and W2 are linear transformation weight matrices, GELU() is an activation function processing, b1 and b2 are bias terms, x spatial is a spatial dimension feature in the target frame estimation process.

2. The somatosensory interaction sailboat handling simulation method based on three-dimensional human posture estimation according to claim 1, characterized in that, In step S1, a 2D human pose is obtained by using a 2D pose detector, i.e., the RTMPose2D is used to extract the 2D human pose in a real picture. 3.The somatosensory interaction sailboat maneuver simulation method based on three-dimensional human pose estimation according to claim 1, wherein, In step S22, the STGJMamer is used to extract the spatial-temporal features of the human pose model, and an output Y of the fused spatial-temporal features is obtained, including: S221, a graph state space Transformer layer SGJMamer is constructed to extract local spatial features; S222, a graph state time Transformer layer TGJMamer is constructed to fuse the low-frequency information of the complete input and the spatial features extracted in step S221, and to extract spatial-temporal features to obtain the output Y of the fused spatial-temporal features. 4.The somatosensory interaction sailboat maneuver simulation method based on three-dimensional human pose estimation of claim 1, wherein, The hierarchical joint enhancement method based on anatomical and kinematic chain reaction includes: In the joint enhancement sequence, the enhancement sequence is designed based on anatomy and kinematic chain reaction, the characteristics of linkage between parent-child joints of the human body are used, and the joint chain reaction exhibited by the human body in the process of movement for maintaining balance is combined, the ability of the model feature extraction is improved through mutual enhancement between joints; meanwhile, symmetric enhancement is adopted in the data processing process, i.e., a mirror flip operation is applied to the input data to generate symmetric actions, the poses of the original action and the flipped action are estimated simultaneously in the pose estimation process, the final output is obtained by averaging the results of the original action and the flipped action, and the enhancement of data symmetry is completed. 5.The somatosensory interaction sailboat maneuver simulation method based on three-dimensional human pose estimation according to claim 3, wherein, In step S222, the graph state time Transformer layer TGJMamer is constructed to fuse the low-frequency information of the complete input and the spatial features extracted in step S221, and to extract spatial-temporal features to obtain the output Y of the fused spatial-temporal features. The figure state time Transformer layer is composed of an STE-GCN module, a JMamba module and a time Transformer module; the data input is the low-frequency part of the complete action sequence, which captures the change characteristics of the action from the global information and obtains the overall evolution trend of the action; after being processed by the STE-GCN module and the JMamba module, the frequency domain feature X TGJMamba is obtained spatial The frequency domain feature will be combined with the spatial feature x spatial to form the space-time information and be input into the self-attention mechanism for processing; when the processed data passes through the mlp layer, the spatial feature part of the data will be frequency domain transformed to convert the local detail changes into frequency domain information and effectively supplement the low-frequency time information. 6.The somatosensory interaction sailboat maneuver simulation method based on three-dimensional human pose estimation according to claim 1, wherein, In step S23, a regression head is constructed to infer the joint information of the target frame, a loss function is constructed to train the joint of the target frame, the joint error is calculated by comparing the result output by the regression head with the real 3D pose, and the training of the joint of the target frame is completed. The spatio-temporal features Y extracted from the input two-dimensional sequence pose information have a feature dimension of Y represents a set to which Y belongs, and f is the frame number of the input features; the target frame is the last frame of the sequence, and has a dimension of The regression head is constructed to realize the inference of the target frame joint information. The regression head is composed of a weighted average operation and a fully connected layer. The estimated pose information is calculated under the weighting of the learnable weight parameters to obtain the final estimation Y of the target frame predicate . Y predicate = W head • LayerNorm(mean(Y)) + b head where W head are trainable weights, mean(Y) is the mean value, b head is a bias term, and LayerNorm() is layer normalization. The output result will be compared with the real 3D pose Y target Calculate the joint point error MPJPE, and train the model as the loss function of where y k , are the true and predicted information of the k-th joint respectively.

7. A somatosensory interaction sailboat maneuver simulation system based on three-dimensional human pose estimation, characterized by, The system implements the somatosensory interaction sailboat manipulation simulation method based on three-dimensional human pose estimation according to any one of claims 1-6, and the system includes: A 2D human pose capturing module is configured to capture real scene pictures in real time by using a camera, and to obtain 2D human poses by using a 2D pose detector; A 3D human pose calculation module is configured to input the 2D human poses into the graph-guided state space enhanced three-dimensional human pose estimation method, and to calculate 3D human poses; A somatosensory interaction module is configured to map the obtained 3D human poses to a somatosensory control module in a virtual scene of sailboat movement, and to realize interaction operation based on somatosensory actions.

Images