Three-dimensional motion capture and intelligent analysis system and method based on monocular camera

Abstract

The invention discloses a three-dimensional motion capture and intelligent analysis system and method based on a monocular camera, and relates to the technical field of image processing, and the system comprises an input and preprocessing module, a feature extraction and interaction fusion module, a human body key point detection module, a three-dimensional attitude reconstruction module and an intelligent analysis module. By constructing a multi-scale feature pyramid structure, the system can refine features under different resolutions to obtain a down-sampling feature map, by applying a Mama model to a human motion capture system, the Mama model adaptively fuses the multi-scale features in a time sequence, and a fused multi-scale feature map is generated through convolution and up-sampling operation. The key point detection precision and robustness are improved; the key point sequence is smoothly processed by combining the xLSTM model with the key point information of the front and back frame images, so that the time consistency and coherence of key point detection are ensured, and the accuracy and stability of human body posture detection are further improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a three-dimensional motion capture and intelligent analysis system and method based on a monocular camera. Background Art

[0002] 3D human motion reconstruction has a wide range of applications, including virtual reality, augmented reality, human-computer interaction, etc. Accurate and smooth reconstruction of human motion sequences is crucial for 3D human pose and shape estimation in monocular videos. However, it is challenging to achieve smooth results using image-based human reconstruction methods for videos.

[0003] Traditional methods rely on complex 3D scanning or motion capture equipment, which is expensive. Reconstruction methods based on monocular video have the advantages of low cost and easy deployment. Existing technologies attempt to use image-based methods to reconstruct 3D humans from videos by estimating the 3D human pose and shape on each frame of the video separately. Although these methods can predict reasonable outputs for static images, due to the lack of modeling the continuity of human motion in consecutive frames, the accuracy and robustness of key point detection in existing methods are low, making it difficult to estimate temporally coherent and smooth 3D human pose and shape from image sequences. Summary of the invention

[0004] The purpose of this application is to provide a three-dimensional motion capture and intelligent analysis system and method based on a monocular camera, which can improve the accuracy and robustness of key point detection and estimate temporally coherent and smooth three-dimensional human posture and shape from an image sequence.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a three-dimensional motion capture and intelligent analysis system based on a monocular camera, comprising:

[0007] An input and preprocessing module is used to: preprocess the original image sequence to obtain a target image sequence; the original image sequence is obtained by taking a monocular camera; the target image sequence includes a plurality of continuous frame target images;

[0008] The feature extraction and interactive fusion module is used to: perform feature extraction on each of the target images to obtain an initial feature map corresponding to each of the target images; for each of the initial feature maps, perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales; use the Mamba model to perform fusion serialization processing on the down-sampled feature maps at each scale to obtain efficient feature maps at different scales; perform up-sampling operations on the efficient feature maps at each scale to obtain up-sampled feature maps at different scales; fuse the up-sampled feature maps at different scales to obtain a multi-scale feature map corresponding to each of the target images; perform feature reconstruction on the multi-scale feature map corresponding to each of the target images to obtain a final feature map corresponding to each of the target images;

[0009] The human key point detection module is used to: extract features from the final feature map corresponding to each of the target images to generate a heat map of each key point in the target image; determine the position coordinates of each key point based on the heat map of each key point in the target image; use the xLSTM model to perform smoothing according to the position coordinates of each key point in the target image, the previous frame image and the next frame image of the target image, and obtain a smoothed key point sequence corresponding to each of the target images;

[0010] The three-dimensional posture reconstruction module is used to: project the smoothed key point sequence corresponding to the target image to obtain the initial high-dimensional embedding representation corresponding to each target image; use dynamic spatial graph convolution to process the initial high-dimensional embedding representation corresponding to each target image to obtain the dynamic spatial graph convolution features corresponding to each frame of the target image; use the Transformer model to perform time-series processing on the dynamic spatial graph convolution features corresponding to each target image to obtain the final high-dimensional feature representation of the key points in each target image; perform three-dimensional mapping processing on the final high-dimensional feature representation of the key points in each target image to obtain the three-dimensional reconstruction results of the key points in each target image;

[0011] The intelligent analysis module is used to analyze and evaluate the motion actions in the target image sequence according to the three-dimensional reconstruction results of the key points in the target image to obtain the analysis results.

[0012] Optionally, the monocular camera-based three-dimensional motion capture and intelligent analysis system further includes a motion stage division module, and the motion stage division module includes:

[0013] A dedicated feature extraction unit is used to: use a third-generation deformable convolutional neural network to extract features from a final feature map corresponding to each of the target images, so as to obtain dedicated features corresponding to each of the target images;

[0014] A global modeling unit, used to: input the dedicated features corresponding to each of the target images into a Transformer model to obtain the global features corresponding to each of the target images;

[0015] The classification unit is used to classify the actions in each target image according to the global features corresponding to each target image, so as to obtain the stage classification result of each target image.

[0016] Optionally, the intelligent analysis module is used to analyze and evaluate motion actions in a target image sequence according to each target image stage classification result and a three-dimensional reconstruction result of key points in the target image to obtain an analysis result.

[0017] Optionally, the original image sequence includes several frames of original images; the original images are obtained by photographing the user's golf movement process with a monocular camera.

[0018] Optionally, the feature extraction and interactive fusion module includes a multi-scale feature extraction unit; the multi-scale feature extraction unit is used to: for each of the initial feature maps, use a multi-scale feature pyramid to perform multi-scale feature extraction on the initial feature map to obtain downsampled feature maps at different scales.

[0019] Optionally, the human key point detection module includes a heat map generation unit; the heat map generation unit is used to: use a multi-layer convolutional neural network to extract features from the final feature map corresponding to each of the target images, and generate a heat map of each key point in the target image; the multi-layer convolutional neural network is composed of multiple convolutional neural networks connected in sequence.

[0020] Optionally, the human key point detection module includes a key point position coordinate determination unit; the key point position coordinate determination unit is used to: perform an argmax operation on the heat map of each key point in the target image to obtain the position coordinates of each key point.

[0021] In a second aspect, the present application provides a monocular camera-based 3D motion capture and intelligent analysis method based on the monocular camera-based 3D motion capture and intelligent analysis system according to the first aspect, comprising:

[0022] Preprocessing the original image sequence to obtain a target image sequence; the original image sequence is obtained by taking a monocular camera; the target image sequence includes a plurality of continuous frame target images;

[0023] Perform feature extraction on each of the target images to obtain an initial feature map corresponding to each of the target images; for each of the initial feature maps, perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales; use the Mamba model to perform fusion serialization processing on the down-sampled feature maps at each scale to obtain efficient feature maps at different scales; perform upsampling operations on the efficient feature maps at each scale to obtain up-sampled feature maps at different scales; fuse the up-sampled feature maps at different scales to obtain a multi-scale feature map corresponding to each of the target images; perform feature reconstruction on the multi-scale feature map corresponding to each of the target images to obtain a final feature map corresponding to each of the target images;

[0024] Perform feature extraction on the final feature map corresponding to each of the target images to generate a heat map of each key point in the target image; determine the position coordinates of each key point based on the heat map of each key point in the target image; use the xLSTM model to perform smoothing according to the position coordinates of each key point in the target image, the previous frame image and the next frame image of the target image, and obtain a smoothed key point sequence corresponding to each of the target images;

[0025] Project the smoothed key point sequence corresponding to the target image to obtain the initial high-dimensional embedding representation corresponding to each target image; use dynamic spatial graph convolution to process the initial high-dimensional embedding representation corresponding to each target image to obtain the dynamic spatial graph convolution features corresponding to each frame of the target image; use the Transformer model to perform time-series processing on the dynamic spatial graph convolution features corresponding to each target image to obtain the final high-dimensional feature representation of the key points in each target image; perform three-dimensional mapping processing on the final high-dimensional feature representation of the key points in each target image to obtain the three-dimensional reconstruction results of the key points in each target image;

[0026] The motion actions in the target image sequence are analyzed and evaluated according to the three-dimensional reconstruction results of the key points in the target image to obtain an analysis result.

[0027] Optionally, the monocular camera-based three-dimensional motion capture and intelligent analysis method further includes:

[0028] Using a third-generation deformable convolutional neural network, extracting features from the final feature map corresponding to each of the target images to obtain dedicated features corresponding to each of the target images;

[0029] Inputting the dedicated features corresponding to each of the target images into the Transformer model to obtain the global features corresponding to each of the target images;

[0030] According to the global features corresponding to each of the target images, the actions in each of the target images are classified to obtain a stage classification result for each of the target images.

[0031] Optionally, the motion actions in the target image sequence are analyzed and evaluated according to the three-dimensional reconstruction results of the key points in the target image to obtain the analysis results, which specifically include:

[0032] The motion actions in the target image sequence are analyzed and evaluated according to the stage classification results of each target image and the three-dimensional reconstruction results of the key points in the target image to obtain an analysis result.

[0033] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0034] The present application provides a three-dimensional motion capture and intelligent analysis system and method based on a monocular camera. By constructing a multi-scale feature pyramid structure, the system can refine the features at different resolutions to obtain a downsampled feature map. By applying the Mamba model to the human motion capture system, the Mamba model adaptively fuses multi-scale features in the time series through a selective state space model (Structured Space Model, SSM), and generates a fused multi-scale feature map through convolution and upsampling operations. The accuracy and robustness of key point detection are improved, and it is particularly suitable for processing complex human motion postures; the time series smoothing technology of the xLSTM model is combined with the key point information of the previous and next frame images to smooth the key point sequence, ensuring the time consistency and coherence of the key point detection, and further improving the accuracy and stability of human posture detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of functional modules of a monocular camera-based three-dimensional motion capture and intelligent analysis system provided in one embodiment of the present application;

[0037] Figure 2 A schematic diagram of a three-dimensional motion capture and intelligent analysis system framework based on a monocular camera provided in one embodiment of the present application;

[0038] Figure 3A flowchart of a monocular camera-based three-dimensional motion capture and intelligent analysis method is provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0041] In an exemplary embodiment, Figure 1 and Figure 2 As shown, a three-dimensional motion capture and intelligent analysis system based on a monocular camera is provided, including: an input and preprocessing module, a feature extraction and interactive fusion module, a human key point detection module, a three-dimensional posture reconstruction module, a motion stage division module and an intelligent analysis module.

[0042] Phase 1:

[0043] The input and preprocessing module is used to preprocess the original image sequence to obtain a target image sequence; the original image sequence is obtained by taking pictures with a monocular camera, and the original image sequence includes several frames of original images; the original image is obtained by taking pictures of the user's golfing process with a monocular camera. The target image sequence includes several consecutive frames of target images.

[0044] In the first stage (input and preprocessing module), the posture features of the human body are extracted from the original image sequence captured by the monocular camera. The core task of this stage is to preprocess the original image to ensure that the following key point detection and posture reconstruction stages can obtain accurate input. The input and preprocessing process is as follows:

[0045] 1.1 Input Data

[0046] The input data is a continuous sequence of raw images from a monocular camera, expressed as:

[0047] I={I1,I1,…,I T};

[0048] Among them, I tRepresents a single frame of original image acquired at time t, t = 1, 2, ..., T, T is the total number of frames in the image sequence. The resolution of each frame of the original image is H × W, H and W are the height and width of the original image, respectively, which constitute the dynamic change sequence of the human body posture during the whole movement process.

[0049] 1.2 Data Preprocessing

[0050] In order to eliminate the influence of different lighting conditions and camera settings on the image, the original image is first normalized. Assuming that the pixel value range of the original image is [0, 255], the normalization process maps the pixel value to the range of [0, 1]. The normalization process formula is:

[0051]

[0052] Among them, I' t is the normalized image corresponding to the single frame original image acquired at time t, μ and σ are the mean and standard deviation of the image pixel values, respectively.

[0053] In order to improve the generalization ability of the model and enhance its robustness under different shooting conditions, we perform a series of data augmentation operations on the input image sequence, including random rotation, scaling, cropping, and flipping.

[0054] 1) Random rotation: Randomly rotate the image in the range of [-10°, 10°].

[0055] 2) Scaling: Randomly scale the image in the range [0.8, 1.2].

[0056] 3) Cropping and flipping: Randomly crop or horizontally flip the image to ensure that the model can adapt to different viewpoints.

[0057] The above data augmentation operations ensure that the model can robustly extract human posture information when facing different angles, scales and environmental changes.

[0058] After normalization and data enhancement, the target image sequence is obtained, which is used as the input for the next stage of posture key point detection and 3D posture reconstruction. The input and preprocessing module provides high-quality input data for subsequent posture reconstruction, significantly improving the robustness and accuracy of the overall model.

[0059] Phase 2:

[0060] The feature extraction and interactive fusion module is used to: perform feature extraction on each of the target images to obtain an initial feature map corresponding to each of the target images; for each of the initial feature maps, perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales; use the Mamba model to perform fusion serialization processing on the down-sampled feature maps at each scale to obtain efficient feature maps at different scales; perform upsampling operations on the efficient feature maps at each scale to obtain up-sampled feature maps at different scales; fuse the up-sampled feature maps at different scales to obtain a multi-scale feature map corresponding to each of the target images; perform feature reconstruction on the multi-scale feature map corresponding to each of the target images to obtain a final feature map corresponding to each of the target images.

[0061] The goal of the second stage is to detect the 2D key points of the human body from the images captured by the monocular camera. To this end, the Mamba model is used for multi-scale feature extraction and combined with time series modeling to ensure the accuracy and continuity of key point detection.

[0062] 2.1 Input Data

[0063] The feature extraction and interactive fusion module includes an initial feature extraction unit; the initial feature extraction unit is used to: use the ResNet-50 model to extract features from each of the target images to obtain an initial feature map corresponding to each of the target images.

[0064] First, after the preprocessing stage, the input is a target image sequence I' = {I'1, I'2, ..., I' T}. After these target images are processed by the ResNet-50 model, the initial feature maps obtained are Where H' and W' are the height and width of the initial feature map, and C is the number of channels of the initial feature map.

[0065] 2.2 Feature Engineering Based on Mamba

[0066] Step 1: Input feature map

[0067] The initial feature map F generated by the ResNet-50 model t Input into the Mamba model. At this time, the initial feature map F t Contains high-level features extracted from the target image. Next, we hope to capture detailed information at different scales through multi-scale processing.

[0068] Step 2: Construct a multi-scale feature pyramid

[0069] The feature extraction and interactive fusion module includes a multi-scale feature extraction unit; the multi-scale feature extraction unit is used to: for each of the initial feature maps, use a multi-scale feature pyramid to perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales.

[0070] In order to realize multi-scale feature extraction, the initial feature map F t Perform downsampling operations to generate downsampled feature maps of different scales. Assuming that we want to generate S downsampled feature maps of different scales, we downsample the initial feature map layer by layer to obtain downsampled feature maps at different resolutions. The specific operations of multi-scale feature extraction are as follows:

[0071] (1) First, the initial feature map F t Perform convolution and pooling operations to generate the first downsampled feature map

[0072]

[0073] in, Represents the convolution operation, k1 is the size of the convolution kernel, and Pool represents the downsampling pooling operation.

[0074] (2) Repeat the above operation to further downsample and generate subsequent downsampled feature maps of different scales:

[0075]

[0076] Where s = 2, 3, ..., S represents different scales; It is the downsampled feature map at the sth scale. As the scale s increases, the resolution of the downsampled feature map gradually decreases.

[0077] Through the multi-scale feature pyramid, downsampled feature maps at multiple scales are obtained These downsampled feature maps capture image information at different spatial resolutions. Each scale of the downsampled feature map contains different levels of features extracted from the target image and is suitable for processing key points of different sizes.

[0078] Step 3: Use Mamba (selective state space model) to interact and fuse downsampled feature maps at different scales They interact through the Mamba model (Selective State Space Model (SSM)) to adaptively fuse features at different time points at each scale. The Mamba model dynamically adjusts its state transfer matrix A, B, C, D to serialize multi-scale features and obtain an efficient feature map, thereby retaining efficient information related to key points. The formula is as follows:

[0079]

[0080] In the above formula, It represents the hidden state vector, which is used in the Mamba model (selective state space model, SSM) to describe the feature map of a certain scale s at each time step t Specifically, The information of the previous time step is encoded and combined with the characteristics of the current time step to form a dynamic state update mechanism, which helps the model process time series data at multiple scales. It is a key state representation in time series modeling, helping the model to effectively capture and process the multi-scale features of human motion, thereby improving the accuracy of key point detection.

[0081] Among them, A, B, C, Δ are learnable parameters:

[0082]

[0083] Among them, l B ,l C ,l Δ are three linear maps, P Δ is a learnable parameter.

[0084] After Mamba serializes the multi-scale features, the downsampled feature sequence at scale s Transformed into an efficient feature map sequence

[0085] These feature maps are then restored to the same size as the original resolution H'×W' through an upsampling operation (the upsampling operation is achieved through a convolutional layer):

[0086]

[0087] Among them, Upsample is the upsampling operation, is the upsampled feature map of the sth scale at time t. Next, the efficient feature maps of different scales are Fusion is performed to obtain the multi-scale feature map after fusion at time t

[0088]

[0089] where α s is a learnable weighting coefficient, which adaptively focuses on more important scale features. The fused multi-scale feature map Feature reconstruction is performed through the convolution layer to generate the final feature map corresponding to each target image For subsequent human key point detection and motion segmentation (motion stage recognition) tasks, the feature reconstruction process can be expressed as follows:

[0090]

[0091] Phase 3:

[0092] The human key point detection module is used to: extract features from the final feature map corresponding to each of the target images to generate a heat map of each key point in the target image; determine the position coordinates of each key point based on the heat map of each key point in the target image; use the xLSTM model to perform smoothing according to the position coordinates of each key point in the target image, the previous frame image and the next frame image of the target image, to obtain a smoothed key point sequence corresponding to each of the target images.

[0093] In the second stage, we use the ResNet-50 model and the Mamba model to extract features from the input video (target image sequence). In the previous stage, we not only paid attention to the multi-resolution characteristics of the image, but also used the Mamba model to realize the interaction of features of different time frames at each resolution. Therefore, the final fused features It can effectively serve tasks such as human key point detection and motion stage division.

[0094] 3.1 Generate key point heat map

[0095] The human key point detection module includes a heat map generation unit; the heat map generation unit is used to: use a multi-layer convolutional neural network to extract features from the final feature map corresponding to each of the target images, and generate a heat map of each key point in the target image; the multi-layer convolutional neural network is composed of multiple convolutional neural networks connected in sequence.

[0096] The task head for human key point detection and motion stage division is connected in parallel to the final feature map After that, we first introduce the human key point detection task head DetectionHead .

[0097] f DetectionHead It consists of a multi-layer convolutional neural network (multiple CNNs connected in sequence), which is based on the final feature map Generate a heat map H for each key point i :

[0098]

[0099] Among them, H i It is the heat map of the i-th key point, which indicates the probability distribution of the key point in the image.

[0100] In another exemplary embodiment of the present application, the key points of the human body may be selected from the following key points:

[0101] 1) Head area

[0102] Top of head: The highest point of the head, used for posture detection.

[0103] Eyes (left eye, right eye): used for facial expression recognition and head direction judgment.

[0104] Ears (left ear, right ear): Assist in judging head rotation and tilt.

[0105] 2) Upper body area

[0106] Shoulders (left shoulder, right shoulder): The key points of the shoulders are usually used to detect the movement posture of the arms.

[0107] Elbows (left, right): Used to track the degree of arm bending and direction of movement.

[0108] Wrist (left wrist, right wrist): The end point of the hand, used to accurately track hand movements.

[0109] 3) Trunk area

[0110] Neck: Connects the head and torso and is often used as the reference point for the central axis of human posture.

[0111] Spine Center: A key point in the middle of the torso that helps understand overall posture and curvature.

[0112] Pelvis (left hip, right hip): These key points are used for motion analysis of the lower body, especially when walking or jumping.

[0113] 4) Lower body area

[0114] Knees (left knee, right knee): used to capture the bending state and movement direction of the legs.

[0115] Ankle (left ankle, right ankle): used to capture the landing of footsteps and changes in pace.

[0116] Toes (Left Toe, Right Toe): Further refine the foot motion in gait analysis.

[0117] 5) Hand area

[0118] Fingers (the tips of five fingers): You can determine whether to select the key points of the hand area based on the action application scenario to be analyzed. In the application scenario of hand detail capture (such as gesture recognition or fine operation), it may be necessary to detect the key points of the fingers.

[0119] These key points are widely used in motion capture and posture reconstruction. For example, in golf swing analysis, key points such as shoulders, hips, and knees are crucial for accurately judging the swing posture; in walking analysis, the detection of parts such as ankles and knees helps to evaluate gait stability and balance. By accurately locating these key points of the human body, the system can better complete tasks such as 3D posture reconstruction and motion stage division.

[0120] 3.2 Extracting key point coordinates

[0121] The human key point detection module includes a key point position coordinate determination unit; the key point position coordinate determination unit is used to: perform an argmax operation on the thermal map of each key point in the target image to obtain the position coordinates of each key point.

[0122] Extract the key point coordinates from the heat map and find the maximum probability position in the heat map through the argmax operation:

[0123] k i = argmax(H i );

[0124] Among them, k i =(x i ,y i ) is the predicted two-dimensional coordinate of the i-th key point, x i is the horizontal coordinate of the i-th key point, y i is the ordinate of the i-th key point. The key point set for each frame image is:

[0125] K t ={k1,k2,…,k N};

[0126] Among them, K t are the coordinates of all key points detected in the t-th frame image, which includes N key points.

[0127] 3.4 Time Series Modeling and Inter-frame Smoothing Based on xLSTM

[0128] In this step, we use xLSTM to smooth the key points in the time series to ensure temporal consistency and coherence. xLSTM uses the key point sequence K of the previous and next frames. t-1 , K t , K t+1 Modeling is performed and the output is a smoothed sequence of key points. The formula is as follows:

[0129]

[0130] in, is the key point sequence after time series modeling and smoothing, i.e., the final human key point detection result; K t-1 K is the coordinates of all key points detected in the t-1th frame image; t+1 are the coordinates of all detected key points in the t+1th frame image. By introducing the exponential gating and matrix memory unit of the xLSTM model, the time series smoothing of key points can be performed more effectively; the xLSTM model performs well in processing complex time dependencies and long sequence tasks, ensuring that the smoothing of key points has higher stability and robustness.

[0131] The loss function used in the human key point detection process includes the key point position loss and smoothing loss The calculation formulas are as follows:

[0132]

[0133] Among them, k i,GT is the actual key point location label.

[0134] Phase 4:

[0135] The motion stage division module includes a dedicated feature extraction unit, a global modeling unit and a classification unit.

[0136] Among them, the dedicated feature extraction unit is used to: use the third-generation deformable convolutional neural network to extract features from the final feature map corresponding to each of the target images, so as to obtain dedicated features corresponding to each of the target images.

[0137] The global modeling unit is used to: input the dedicated features corresponding to each of the target images into the Transformer model to obtain the global features corresponding to each of the target images.

[0138] The classification unit is used to classify the actions in each target image according to the global features corresponding to each target image, so as to obtain the stage classification result of each target image.

[0139] The division of motion phases is an important task in motion analysis. In sports such as golf, badminton, tennis, and table tennis, different movements have important roles and significance. For example, in golf, the "swing" is a complex and multi-layered process, and small changes in many places will affect the final hitting effect. In order to more accurately evaluate the athlete's swing technique and guide the athlete's swing, it is particularly important to divide the swing process into different stages. This more detailed division not only helps to improve the understanding of the details of the swing, but also provides coaches and athletes with more specific and effective technical guidance. Here we take the motion segmentation of the golf swing process as an example. Similarly, this method can be applied to the division of other sports.

[0140] A complete golf swing can locate eight key actions, namely: Address, Toe-up, Mid-backswing, Top, Mid-downswing, Impact, Mid-follow-through and Finish. Therefore, for a complete video (i.e., image sequence), we need to determine the action category to which each frame belongs.

[0141] 4.1 Feature processing based on the third-generation deformable convolution DCNv3

[0142] In the previous network architecture, a complete image feature has been constructed in the first stage "feature extraction and interactive fusion" Therefore, the fourth stage is based on the final feature map constructed previously Implement motion phase segmentation tasks (such as swing phase recognition). Considering the final feature map constructed previously It is a common feature that serves different tasks at the same time, so we need to segment the final feature map in the motion stage Specifically, the final feature map is processed using a deformable convolutional network (DCNv3). In order to capture the dynamic motion features in the motion process (such as the swing process), this application introduces the third-generation deformable convolutional neural network (DCNv3), which converts the general features into Transformed into specialized features required for motion stage segmentation tasks

[0143]

[0144] Among them, f DCNv3It is the third generation of deformable convolutional neural network, which divides the input features into G groups and allows each group of convolution kernels to adaptively adjust the sampling points in different spatial regions. By adaptively learning the spatial offset of the convolution kernel, it can better handle local changes in posture and is particularly suitable for capturing motion features. DCNv3 first calculates the offset Δp of each convolution kernel through a convolutional network k The offset is adaptive, according to the input feature map To make dynamic adjustments:

[0145]

[0146] The convolution process of DCNv3 can be expressed as:

[0147]

[0148] Among them, w g is the weight of the g-th group, is the modulation scalar, x g is the feature of the gth group; y(p0) represents the output feature value at position p0, which is the output result obtained after the deformable convolution operation. This value is based on the input feature x of different groups g. g and its corresponding convolution kernel offset and modulation coefficient Calculated after the convolution operation; k represents the index of the convolution kernel, which is used to represent the kth convolution kernel in each group g. Since the convolution operation may contain multiple kernels, k is used to traverse these convolution kernels, perform the convolution operation on each position, and modulate the coefficients To summarize, the y(p0) in the formula is the input feature x through multiple sets of convolution kernels, offsets, and modulation coefficients. g The output result obtained after calculation. And k is responsible for indexing different convolution kernels in each group of convolution kernels.

[0149] 4.2 Modeling the temporal correlation of features based on Transformer

[0150] Dedicated features after DCNv3 processing It can capture subtle changes in space, but the division of the swing phase not only depends on the features of a single frame image, but also requires global modeling of the dependencies of the time series. To achieve this, we introduced the Transformer model. The Transformer model has powerful global modeling capabilities, especially when processing sequence data. The self-attention mechanism can effectively capture the long-distance dependencies between time steps. The input is fed into a multi-layer Transformer model for global modeling of time step features. The self-attention mechanism in the Transformer model captures the association between different time steps by calculating the similarity between time steps. For each time step t, the self-attention mechanism calculates the query matrix Q t , key matrix K t Sum value matrix V t , and obtain the global context information of the time step:

[0151]

[0152] Where Z t It is the global feature obtained after Transformer processing, which contains the dependencies between time steps.

[0153] 4.3 Motion Segmentation Task Head

[0154] After obtaining the global features {Z1, Z2, …, Z T After that, we need to design a task head to perform the motion segmentation task, that is, to determine the category of each frame in the image sequence. Therefore, the task head will use a classifier to classify the image based on the global features Z1, Z2, …, Z T The classification is completed to determine which stage of motion the image is in at each time step.

[0155] Specifically, we use a fully connected network to classify the global features of each time step and output the probability that the image belongs to different motion stages at that moment. The task of this network is to assign a stage label to each time step (e.g., preparation stage, start stage, acceleration stage, swing completion stage, etc.) based on the input features:

[0156]

[0157] in, is the stage classification result at time step t, W c and b c are the weights and biases of the fully connected layers.

[0158] 4.4 Motion Segmentation Boundary Optimization

[0159] The prediction result of the motion stage to which the image belongs at each time step is obtained through the classifier. However, in order to ensure that the boundaries between the different motion stages obtained by segmentation are reasonable and smooth, this application further introduces dynamic programming to optimize the stage boundaries. The dynamic programming algorithm is used to minimize the global cost function of stage classification to ensure that the boundaries between stages are coherent and natural. The goal of this algorithm is to find the optimal stage dividing point so that the classification result has a smooth transition over the entire time series.

[0160]

[0161] in, is a penalty function that uses a second-order difference penalty to ensure smooth transitions of boundary changes:

[0162]

[0163] After the above steps, the final model output is the stage classification result of each time step The motion phase division result can be used for subsequent motion analysis or further swing phase detection and adjustment. Indicates the position of the motion phase boundary, which is the optimal phase demarcation point found by the dynamic programming algorithm. These boundary points ensure the smooth transition of the segmented motion phase over the entire time series and maintain reasonable phase changes. It will affect the classification results of the motion stages at each time step, making the stage division results more natural and continuous. t represents the stage classification result of the image at the tth time step. Specifically, c t It is the motion stage category label at a certain moment output by the classifier, indicating the motion stage to which the current time step belongs (such as preparation stage, swing stage, etc.). This classification result is the output of the final motion stage detection, and after optimized boundary adjustment, it can be used for subsequent motion analysis. In summary, is the boundary position of the motion phase segmentation, and c t is the predicted motion phase category at each time step. This ensures smooth and accurate segmentation during the motion phase.

[0164] Phase 5:

[0165] The three-dimensional posture reconstruction module is used to: project the smoothed key point sequence corresponding to the target image to obtain the initial high-dimensional embedding representation corresponding to each target image; use dynamic spatial graph convolution to process the initial high-dimensional embedding representation corresponding to each target image to obtain the dynamic spatial graph convolution features corresponding to each frame of the target image; use the Transformer model to perform time-series processing on the dynamic spatial graph convolution features corresponding to each target image to obtain the final high-dimensional feature representation of the key points in each target image; perform three-dimensional mapping processing on the final high-dimensional feature representation of the key points in each target image to obtain the three-dimensional reconstruction results of the key points in each target image.

[0166] 3D human pose reconstruction is achieved by recovering 3D human joint coordinates from 2D human key points, which is crucial for applications such as motion analysis and virtual reality. To this end, we use a 3D pose reconstruction algorithm based on graph neural network GNN and Transformer: Dynamic Graph Convolution-Transformer (DGCN-Transformer) architecture.

[0167] 5.1 Input and Initial Embedding

[0168] In the third stage of human key point detection, a two-dimensional key point sequence of each frame image is obtained. In order to embed the two-dimensional key points into high dimensions, this application uses a linear layer to project these key points to obtain the initial high-dimensional embedding representation X of the two-dimensional key points at time t t :

[0169]

[0170] Among them, W0 and b0 are learnable parameters.

[0171] 5.2 Dynamic Spatiotemporal Graph Convolution-Transformer Architecture

[0172] Graph convolutional networks (GCNs) are able to capture local spatial relationships, which makes them very suitable for representing and memorizing the connections between key points of the human body, thereby achieving better 3D pose reconstruction. However, traditional graph convolutional neural networks rely on predefined adjacency matrices, which limits their flexibility. In order to enable the model to dynamically update the graph structure, we use dynamic graph convolution to replace the traditional graph convolution operation, allowing the model to adaptively adjust the structure of the graph according to the input data and adaptively update the adjacency matrix A. Specifically, the adjacency matrix A that represents the relationship between graph nodes is no longer predefined, but is based on the input high-dimensional embedding vector X. t Dynamically calculated:

[0173]

[0174] Among them, φ(·) is a function for calculating feature similarity (such as Euclidean distance, Mahalanobis distance), and here we use cosine similarity. Therefore, we give the dynamic spatial graph convolution formula as follows:

[0175]

[0176] Among them, D dynamic According to the dynamic adjacency matrix The calculated degree matrix.

[0177] Dynamic graph convolution features of each frame after dynamic graph convolution processing It is input into the Transformer architecture for temporal correlation modeling, and the final high-dimensional feature representation X of the two-dimensional key points in each frame of the target image is obtained. final,t :

[0178] X final,t =Attention time (Q time,t ,K time,t ,V time,t )

[0179] in, In our DGCN-Transformer architecture, DGCN is responsible for learning and modeling the spatial correlation of the input data, and Transformer is responsible for learning and modeling the temporal correlation of the input data. final,t The spatial and temporal correlations of the input two-dimensional key point sequence are fully integrated and mined. Next, it is mapped to three-dimensional coordinates through a linear regression head:

[0180]

[0181] Where W r and b r is a learnable parameter, The reconstruction result of the human key points at time frame t.

[0182] In order to enhance the capture of temporal and spatial associations, we introduce adaptive spatio-temporal loss (AdaptiveSpatio-TemporalLoss) into the proposed dynamic spatio-temporal graph convolution-spatial-temporal Transformer architecture. During the model training process, adaptive spatiotemporal loss is used to train the 3D human posture reconstruction model, which consists of a linear layer, dynamic spatiotemporal graph convolution, a Transformer model, and a linear regression head. The loss function can dynamically adjust the weight of each time step and spatial position:

[0183]

[0184] Among them, w t,n is the adaptive weight for each time step and spatial position, which is learned dynamically by the model, X 3D,t,n is the real 3D coordinate of the key points of the human body at time frame t.

[0185] Phase 6:

[0186] The intelligent analysis module is used to analyze and evaluate the motion actions in the target image sequence according to the three-dimensional reconstruction results of the key points in the target image to obtain the analysis results.

[0187] The intelligent analysis module is used to analyze and evaluate the motion actions in the target image sequence according to the stage classification results of each target image and the three-dimensional reconstruction results of the key points in the target image to obtain the analysis results.

[0188] The present application also provides an application scenario, which applies the above-mentioned three-dimensional motion capture and intelligent analysis system based on a monocular camera. Specifically: the three-dimensional motion capture and intelligent analysis system based on a monocular camera provided in this embodiment can be applied in a golf sports analysis scenario.

[0189] In this phase, the aforementioned monocular camera-based human motion capture technology is applied to golf motion analysis, especially to calculate the X-factor in golf swing. The X-factor is an important parameter in golf kinematics, which reflects the rotation angle difference between the shoulder and pelvis during the swing and is usually used to evaluate the athlete's trunk rotation ability and efficiency when hitting the ball.

[0190] 6.1 Calculation of the X-factor

[0191] X-factor refers to the difference in rotation between the shoulders and hips during the golf swing. A larger X-factor is generally associated with more swing power and longer hitting distance. The calculation steps are as follows:

[0192] (1) Key point extraction:

[0193] The key points of the shoulders include the left shoulder point and the right shoulder point:

[0194] Left shoulder point: S L =[x SL ,y SL ,z SL ];

[0195] Right shoulder point: S L =[x SR ,y SR ,z SR ].

[0196] The key points of the hips include the left hip point and the right hip point:

[0197] Left hip point: H L =[x HL ,y HL ,z HL ];

[0198] Right hip point: H R =[x HR ,y HR ,z HR ].

[0199] In the formula, x SL ,ySL ,z SL are the x-, y- and z-coordinates of the left shoulder point respectively; SR ,y SR ,z SR are the x-, y- and z-coordinates of the right shoulder point respectively; HL ,y HL ,z HL are the x-, y- and z-coordinates of the left hip point respectively; HR ,y HR ,z HR They are the x-, y-, and z-coordinates of the right hip point respectively.

[0200] (2) Define the shoulder and hip vectors:

[0201] V shoulder =S R -S L ;

[0202] V hip =H R -H L ;

[0203] Where V shoulder represents the shoulder vector, V hip represents the hip vector.

[0204] (3) Projection to the horizontal plane: To calculate the horizontal rotation angle, project the shoulder vector and hip vector onto the horizontal plane (usually the XY plane):

[0205]

[0206] Among them, v sx =x SR -x SL , v sy =y SR -y SL ;v hx =x HR -x HL , v hy =y HR -y HL ; represents the shoulder vector projection, represents the hip vector projection.

[0207] (4) Calculate the angle of the vector:

[0208] θ shoulder =arctan 2(v sy ,v sx );

[0209] θ hip =arctan 2(v hy ,v hx );

[0210] In the formula, θ shoulder represents the shoulder angle, θ hip Represents the hip angle.

[0211] (5) Calculate the X-factor:

[0212] X factor =θ shoulder -θ hip ;

[0213] When X-Factor X factor When the value is positive, it means that the shoulder rotates backward relative to the hip. factor A negative value means the shoulder is rotated forward relative to the hip.

[0214] (6) Time series analysis:

[0215] For each time frame t, calculate the corresponding X-factor X factor , get the X-factor change curve during the entire swing process. At the same time, calculate the X-factor at the critical moment:

[0216] Top of Backswing: This is usually the moment when the X-factor reaches its maximum.

[0217] Impact: Observe the degree of regression of the X-factor and evaluate the efficiency of energy release.

[0218] 6.2 Motion Analysis

[0219] Through the above content, accurate motion capture results can be obtained, and indicators such as X-factor can be calculated based on the motion capture results. Based on these indicators, a more detailed analysis of the motion process can be carried out. For example, a curve of the X-factor changing over time can be drawn to intuitively show the rotation difference during the swing process; X-factor curves of different swing attempts or different golfers can be compared horizontally; the training model can predict the optimal X-factor range to improve the hitting effect; abnormal patterns can be identified to provide real-time feedback and correction suggestions. Practical applications include but are not limited to:

[0220] (1) Technical evaluation and improvement:

[0221] Power output: Evaluate the player's power output ability by analyzing the maximum value and change rate of the X-factor.

[0222] Movement Coordination: Observe the synchronization of shoulder and hip rotation to identify potential movement incoordination issues.

[0223] Swing Efficiency: Evaluates the efficiency of energy transfer and release through changes in the X-factor at different stages of the swing.

[0224] (2) Injury prevention:

[0225] Risk of over-rotation: Excessive X-factor may increase the risk of lower back injury, and timely adjustments can prevent injuries.

[0226] Asymmetry detection: discover the difference in left and right rotation and guide players to carry out targeted training and balance.

[0227] (3) Development of personalized training program:

[0228] Targeted training: Based on the results of the X-factor analysis, develop a training plan to improve core strength and flexibility.

[0229] Progress Tracking: Evaluate training effects and progress by continuously monitoring changes in X-Factor.

[0230] In summary, by designing a 3D human key point capture method based on a monocular camera and applying it to golf sports analysis, it is possible to capture and analyze the player's sports details with high precision, especially the key X-factor indicators. This provides a scientific basis and effective means for golf technical training, performance evaluation and injury prevention, and promotes the intelligent and data-driven nature of golf sports analysis.

[0231] This application has the following beneficial effects:

[0232] 1. This application applies the Mamba model to the human motion capture system for the first time. By constructing a multi-scale feature pyramid structure, the system can refine features at different resolutions; the Mamba model further uses a selective state space model to adaptively fuse multi-scale features in a time series, and generates a fused feature map through convolution and upsampling operations. This interactive fusion technology of multi-scale features significantly improves the accuracy and robustness of key point detection, and is particularly suitable for processing complex human motion postures.

[0233] 2. In view of the limitation of traditional graph convolutional networks (GCN) that rely on predefined adjacency matrices when processing the spatial relationship between key points of the human body, this application proposes a DGCN-Transformer architecture. This architecture allows the system to dynamically adjust the adjacency matrix according to the input data through dynamic graph convolution, enhancing the ability to model the spatial correlation of human posture. Combined with the global modeling ability of the Transformer model for time series, more accurate 3D posture reconstruction is achieved, which is particularly suitable for applications such as motion analysis and virtual reality.

[0234] 3. This application introduces the third-generation deformable convolutional network (DCNv3) to meet the needs of stage division in complex sports such as swinging. DCNv3 captures local changes in posture during movement by adaptively learning the spatial offset of the convolution kernel, and can accurately handle subtle differences in movement. Combined with Transformer's modeling of temporal correlation, it achieves a fine division of movement stages, providing more accurate technical support for motion analysis.

[0235] 4. To solve the problem of consistency in key point detection in time series, this application introduces a time series smoothing technology based on xLSTM. xLSTM combines the key point information of the previous and next frames and uses its innovative exponential gating and matrix memory unit to smooth the key point sequence, ensuring the time consistency and coherence of key point detection. The application of this technology further improves the accuracy and stability of human posture detection.

[0236] 5. This application adopts a multi-task learning framework, integrating the three core tasks of human key point detection, motion stage division, and 3D posture reconstruction into the same framework. By sharing the underlying feature extraction network (such as ResNet-50 and Mamba models), different tasks can effectively share information and improve the efficiency and performance of the overall system. In addition, by introducing a joint loss function between tasks and optimizing the losses of multiple tasks at the same time, the results of each task are more coordinated and consistent, further improving the robustness and accuracy of the entire system.

[0237] 6. This application forms a complete system framework from input data acquisition, preprocessing, feature extraction, key point detection, motion stage division to 3D posture reconstruction. Adaptive processing mechanisms and dynamic adjustment strategies are introduced at each stage to ensure the adaptability and stability of the system in different usage scenarios. The optimized design of the entire process makes the system not only suitable for a single scenario, but can also be extended to a variety of complex application environments, such as sports training, medical rehabilitation, virtual reality, etc.

[0238] 7. The system architecture of this application adopts a modular design, and each functional module can be operated independently or used in combination with other modules. This design greatly improves the scalability of the system, allowing the system to be flexibly adjusted according to specific application requirements. For example, for different sensor data types, the corresponding feature extraction and alignment modules can be flexibly introduced or replaced. This scalability not only extends the life cycle of the system, but also increases its applicability in different application scenarios, ensuring the system's continued competitiveness in future technological developments.

[0239] 8. In terms of application, this application uses a monocular camera to accurately capture three-dimensional human motion and applies it to the precise and intelligent analysis of golf, thus achieving a comprehensive analysis of the golfer's swing trajectory and posture. The system can not only reconstruct the three-dimensional motion trajectory of key points of the human body through a monocular camera, but also generate key human mechanical parameters such as X-Factor and compression factor. Through this precise motion capture and analysis, the system can provide precise technical guidance for golf beginners and help coaches develop personalized training plans, greatly improving the training efficiency and scientificity of golf.

[0240] Based on the same inventive concept, the embodiment of the present application also provides a method for realizing the above-mentioned three-dimensional motion capture and intelligent analysis system based on a monocular camera and a three-dimensional motion capture and intelligent analysis method based on a monocular camera. The implementation scheme for solving the problem provided by the method is similar to the implementation scheme recorded in the above-mentioned system, so the specific limitations in one or more embodiments of the three-dimensional motion capture and intelligent analysis method based on a monocular camera provided below can be referred to the above limitations on the three-dimensional motion capture and intelligent analysis system based on a monocular camera, and will not be repeated here.

[0241] In an exemplary embodiment, Figure 3 As shown, a three-dimensional motion capture and intelligent analysis method based on a monocular camera is provided, including:

[0242] S1: preprocessing the original image sequence to obtain a target image sequence; the original image sequence is obtained by taking a monocular camera; the target image sequence includes a plurality of continuous frame target images;

[0243] S2: Perform feature extraction on each of the target images to obtain an initial feature map corresponding to each of the target images; for each of the initial feature maps, perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales; use the Mamba model to perform fusion serialization processing on the down-sampled feature maps at each scale to obtain efficient feature maps at different scales; perform upsampling operations on the efficient feature maps at each scale to obtain up-sampled feature maps at different scales; fuse the up-sampled feature maps at different scales to obtain a multi-scale feature map corresponding to each of the target images; perform feature reconstruction on the multi-scale feature map corresponding to each of the target images to obtain a final feature map corresponding to each of the target images;

[0244] S3: extracting features from the final feature map corresponding to each of the target images to generate a heat map of each key point in the target image; determining the position coordinates of each key point based on the heat map of each key point in the target image; using the xLSTM model, performing smoothing according to the position coordinates of each key point in the target image, the previous frame image and the next frame image of the target image, to obtain a smoothed key point sequence corresponding to each of the target images;

[0245] S4: Project the smoothed key point sequence corresponding to the target image to obtain the initial high-dimensional embedding representation corresponding to each target image; use dynamic spatial graph convolution to process the initial high-dimensional embedding representation corresponding to each target image to obtain the dynamic spatial graph convolution features corresponding to each frame of the target image; use the Transformer model to perform time-series processing on the dynamic spatial graph convolution features corresponding to each target image to obtain the final high-dimensional feature representation of the key points in each target image; perform three-dimensional mapping processing on the final high-dimensional feature representation of the key points in each target image to obtain the three-dimensional reconstruction results of the key points in each target image;

[0246] S5: Analyze and evaluate the motion actions in the target image sequence according to the three-dimensional reconstruction results of the key points in the target image to obtain an analysis result.

[0247] As an optional implementation, the three-dimensional motion capture and intelligent analysis method based on a monocular camera also includes:

[0248] Using a third-generation deformable convolutional neural network, extracting features from the final feature map corresponding to each of the target images to obtain dedicated features corresponding to each of the target images;

[0249] Inputting the dedicated features corresponding to each of the target images into the Transformer model to obtain the global features corresponding to each of the target images;

[0250] According to the global features corresponding to each of the target images, the actions in each of the target images are classified to obtain a stage classification result for each of the target images.

[0251] S5 specifically includes: analyzing and evaluating the motion actions in the target image sequence according to each target image stage classification result and the three-dimensional reconstruction result of the key points in the target image to obtain an analysis result.

[0252] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0253] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A three-dimensional motion capture and intelligent analysis system based on a monocular camera, characterized in that: The three-dimensional motion capture and intelligent analysis system based on a monocular camera includes: An input and preprocessing module is used to: preprocess the original image sequence to obtain a target image sequence; the original image sequence is obtained by taking a monocular camera; the target image sequence includes a plurality of continuous frame target images; The feature extraction and interactive fusion module is used to: perform feature extraction on each of the target images to obtain an initial feature map corresponding to each of the target images; for each of the initial feature maps, perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales; use the Mamba model to perform fusion serialization processing on the down-sampled feature maps at each scale to obtain efficient feature maps at different scales; perform up-sampling operations on the efficient feature maps at each scale to obtain up-sampled feature maps at different scales; fuse the up-sampled feature maps at different scales to obtain a multi-scale feature map corresponding to each of the target images; perform feature reconstruction on the multi-scale feature map corresponding to each of the target images to obtain a final feature map corresponding to each of the target images; The human key point detection module is used to: extract features from the final feature map corresponding to each of the target images to generate a heat map of each key point in the target image; determine the position coordinates of each key point based on the heat map of each key point in the target image; use the xLSTM model to perform smoothing according to the position coordinates of each key point in the target image, the previous frame image and the next frame image of the target image, and obtain a smoothed key point sequence corresponding to each of the target images; The three-dimensional posture reconstruction module is used to: project the smoothed key point sequence corresponding to the target image to obtain the initial high-dimensional embedding representation corresponding to each target image; use dynamic spatial graph convolution to process the initial high-dimensional embedding representation corresponding to each target image to obtain the dynamic spatial graph convolution features corresponding to each frame of the target image; use the Transformer model to perform time-series processing on the dynamic spatial graph convolution features corresponding to each target image to obtain the final high-dimensional feature representation of the key points in each target image; perform three-dimensional mapping processing on the final high-dimensional feature representation of the key points in each target image to obtain the three-dimensional reconstruction results of the key points in each target image; The intelligent analysis module is used to analyze and evaluate the motion actions in the target image sequence according to the three-dimensional reconstruction results of the key points in the target image to obtain the analysis results.

2. The three-dimensional motion capture and intelligent analysis system based on a monocular camera according to claim 1, characterized in that: The monocular camera-based three-dimensional motion capture and intelligent analysis system further includes a motion stage division module, and the motion stage division module includes: A dedicated feature extraction unit is used to: use a third-generation deformable convolutional neural network to extract features from a final feature map corresponding to each of the target images, so as to obtain dedicated features corresponding to each of the target images; A global modeling unit, used to: input the dedicated features corresponding to each of the target images into a Transformer model to obtain the global features corresponding to each of the target images; The classification unit is used to classify the actions in each target image according to the global features corresponding to each target image, so as to obtain the stage classification result of each target image.

3. The three-dimensional motion capture and intelligent analysis system based on a monocular camera according to claim 2, characterized in that: The intelligent analysis module is used to analyze and evaluate the motion actions in the target image sequence according to the stage classification results of each target image and the three-dimensional reconstruction results of the key points in the target image to obtain the analysis results.

4. The three-dimensional motion capture and intelligent analysis system based on a monocular camera according to claim 1, characterized in that: The original image sequence includes several frames of original images; the original images are obtained by photographing the golf movement process of the user with a monocular camera.

5. The three-dimensional motion capture and intelligent analysis system based on a monocular camera according to claim 1, characterized in that: The feature extraction and interactive fusion module includes a multi-scale feature extraction unit; the multi-scale feature extraction unit is used to: for each of the initial feature maps, use a multi-scale feature pyramid to perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales.

6. The three-dimensional motion capture and intelligent analysis system based on a monocular camera according to claim 1, characterized in that: The human key point detection module includes a heat map generation unit; the heat map generation unit is used to: use a multi-layer convolutional neural network to extract features from the final feature map corresponding to each of the target images, and generate a heat map of each key point in the target image; the multi-layer convolutional neural network is composed of multiple convolutional neural networks connected in sequence.

7. The three-dimensional motion capture and intelligent analysis system based on a monocular camera according to claim 1, characterized in that: The human key point detection module includes a key point position coordinate determination unit; the key point position coordinate determination unit is used to: perform an argmax operation on the thermal map of each key point in the target image to obtain the position coordinates of each key point.

8. A method for 3D motion capture and intelligent analysis based on a monocular camera according to any one of claims 1 to 7, characterized in that: The three-dimensional motion capture and intelligent analysis method based on a monocular camera includes: Preprocessing the original image sequence to obtain a target image sequence; the original image sequence is obtained by taking a monocular camera; the target image sequence includes a plurality of continuous frame target images; Perform feature extraction on each of the target images to obtain an initial feature map corresponding to each of the target images; for each of the initial feature maps, perform multi-scale feature extraction on the initial feature map to obtain down-sampled feature maps at different scales; use the Mamba model to perform fusion serialization processing on the down-sampled feature maps at each scale to obtain efficient feature maps at different scales; perform upsampling operations on the efficient feature maps at each scale to obtain up-sampled feature maps at different scales; fuse the up-sampled feature maps at different scales to obtain a multi-scale feature map corresponding to each of the target images; perform feature reconstruction on the multi-scale feature map corresponding to each of the target images to obtain a final feature map corresponding to each of the target images; Perform feature extraction on the final feature map corresponding to each of the target images to generate a heat map of each key point in the target image; determine the position coordinates of each key point based on the heat map of each key point in the target image; use the xLSTM model to perform smoothing according to the position coordinates of each key point in the target image, the previous frame image and the next frame image of the target image, and obtain a smoothed key point sequence corresponding to each of the target images; Project the smoothed key point sequence corresponding to the target image to obtain the initial high-dimensional embedding representation corresponding to each target image; use dynamic spatial graph convolution to process the initial high-dimensional embedding representation corresponding to each target image to obtain the dynamic spatial graph convolution features corresponding to each frame of the target image; use the Transformer model to perform time-series processing on the dynamic spatial graph convolution features corresponding to each target image to obtain the final high-dimensional feature representation of the key points in each target image; perform three-dimensional mapping processing on the final high-dimensional feature representation of the key points in each target image to obtain the three-dimensional reconstruction results of the key points in each target image; The motion actions in the target image sequence are analyzed and evaluated according to the three-dimensional reconstruction results of the key points in the target image to obtain an analysis result.

9. The method for three-dimensional motion capture and intelligent analysis based on a monocular camera according to claim 8, characterized in that: The three-dimensional motion capture and intelligent analysis method based on a monocular camera also includes: Using a third-generation deformable convolutional neural network, extracting features from the final feature map corresponding to each of the target images to obtain dedicated features corresponding to each of the target images; Inputting the dedicated features corresponding to each of the target images into the Transformer model to obtain the global features corresponding to each of the target images; According to the global features corresponding to each of the target images, the actions in each of the target images are classified to obtain a stage classification result for each of the target images.

10. The method for three-dimensional motion capture and intelligent analysis based on a monocular camera according to claim 9, characterized in that: The motion actions in the target image sequence are analyzed and evaluated according to the three-dimensional reconstruction results of the key points in the target image to obtain the analysis results, which specifically include: The motion actions in the target image sequence are analyzed and evaluated according to the stage classification results of each target image and the three-dimensional reconstruction results of the key points in the target image to obtain an analysis result.

Images