Motion analysis system based on inertia-visual signal enhancement and fusion

Abstract

The invention discloses a motion analysis system based on inertia-visual signal enhancement and fusion, and relates to the technical field of intelligent perception and fusion. In the system, an inertial sensor module is used for collecting an inertial signal of a moving object; the visual sensor module is used for collecting human body key point information; the signal processing module is used for performing enhancement processing on the inertial signals by using the improved deep learning model, and reconstructing outrange signals in the inertial signals based on the KAN; the inertia-visual information fusion module is used for carrying out fusion processing on the human body key point information and the inertia signal after signal processing based on a multi-modal feature alignment method of an optimal transport theory and an SINkhorn algorithm to obtain a motion track of the moving object; and the motion analysis module is used for analyzing the motion mode and posture change of the motion object based on the motion trail. According to the invention, the motion trail of the moving object can be accurately captured and accurately analyzed.

Description

Technical Field

[0001] The present application relates to the field of intelligent perception and fusion technology, and in particular to a motion analysis system based on inertial-visual signal enhancement and fusion. Background Art

[0002] Inertial sensors play a vital role in modern technological applications, such as in the automotive, aerospace, robotics, and golf fields. Inertial sensors are devices that can measure the acceleration, angular velocity, and magnetic field strength of an object, and usually include accelerometers, gyroscopes, and magnetometers. They are widely used in motion capture, navigation systems, and attitude control by capturing the movement information of objects in space. In motion capture, the role of inertial sensors is particularly important. It can track and record the movement trajectory of objects in real time, providing key data for analyzing motion behavior and building motion models. Through inertial sensors, researchers and engineers can obtain accurate motion data, thereby achieving highly restored motion capture effects in virtual reality, animation production, and biomechanics research.

[0003] In highly dynamic environments, inertial sensors often face the problem of signal saturation, which limits the application scope of low-cost inertial sensors. Traditional signal processing methods usually treat signals as sequences of discrete points, resulting in large information loss during signal reconstruction, especially poor performance in over-range reconstruction. In addition, deep learning models may produce "hallucination" phenomena that are inconsistent with actual conditions during signal reconstruction, affecting the reliability of the model. In applications such as human key point detection, the detection and memory of sparse features is also a challenge. At the same time, multimodal sensor data fusion, such as the alignment of inertial sensor and visual sensor data, is the key to improving motion capture accuracy, but there are currently technical difficulties. Summary of the invention

[0004] The purpose of this application is to provide a motion analysis system based on inertial-visual signal enhancement and fusion, which can effectively solve the problem of inertial sensor signal saturation in high dynamic environments, improve the accuracy and reliability of signal reconstruction, and optimize multimodal sensor data fusion to improve the accuracy of motion capture.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a motion analysis system based on inertial-visual signal enhancement and fusion, comprising: an inertial sensor module, a visual sensor module, a signal processing module, an inertial-visual information fusion module and a motion analysis module.

[0007] The inertial sensor module is used to collect inertial signals of a moving object; the inertial signals include acceleration, angular velocity and magnetic field strength of a set part of the moving object.

[0008] The visual sensor module is a rare memory enhancement module based on Sharma-MIttal entropy, which is used to collect key point information of the human body of a moving object.

[0009] The signal processing module is used to: use the improved deep learning model to enhance the inertial signal, and reconstruct the over-range signal in the inertial signal based on the Kolmogorov-Arnold network; the improved deep learning model is a deep learning model that introduces generative hallucination entropy into the loss function of the deep learning model.

[0010] The inertial-visual information fusion module is used to fuse the key point information of the human body and the inertial signal after signal processing based on the multimodal feature alignment method of the optimal transport theory and the SInkhorn algorithm to obtain the motion trajectory of the moving object.

[0011] The motion analysis module is used to analyze the motion pattern and posture change of the motion object based on the motion trajectory of the motion object.

[0012] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0013] The present application provides a motion analysis system based on inertial-visual signal enhancement and fusion. The system is based on an inertial sensor module and a visual sensor module, and can more accurately collect the inertial signal of a moving object and the key point information of the human body, thereby improving the accuracy of motion capture; a signal processing module is used to enhance the inertial signal using an improved deep learning model, and reconstruct the over-range signal in the inertial signal based on KAN; based on the inertial-visual information fusion module, the collected key point information of the human body is fused with the processed inertial signal, so that the system can more accurately restore the motion trajectory of the moving object. Based on this system, the motion trajectory of the moving object can be accurately described. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1This is a structural diagram of a motion analysis system based on inertial-visual signal enhancement and fusion in one embodiment of the present application.

[0016] Figure 2 A schematic diagram of the overall framework of a motion capture and intelligent analysis system based on inertial-visual signal enhancement and fusion provided in one embodiment of the present application.

[0017] Figure 3 A schematic diagram of over-range signal reconstruction based on KAN provided in one embodiment of the present application.

[0018] Figure 4 A schematic diagram of a Generative Hallucination Entropy (GHE) calculation process provided in one embodiment of the present application. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0021] Embodiment 1

[0022] like Figure 1 As shown, this embodiment provides a motion analysis system based on inertial-visual signal enhancement and fusion, including: an inertial sensor module, a visual sensor module, a signal processing module, an inertial-visual information fusion module and a motion analysis module.

[0023] The inertial sensor module is used to collect inertial signals of a moving object; the inertial signals include acceleration, angular velocity and magnetic field strength of a set part of the moving object.

[0024] The visual sensor module is a rare memory enhancement module based on Sharma-MIttal entropy, which is used to collect key point information of the human body of a moving object.

[0025] The signal processing module is used to: use the improved deep learning model to enhance the inertial signal, and reconstruct the over-range signal in the inertial signal based on the Kolmogorov-Arnold network; the improved deep learning model is a deep learning model that introduces generative hallucination entropy into the loss function of the deep learning model.

[0026] The inertial-visual information fusion module is used to fuse the key point information of the human body and the inertial signal after signal processing based on the multimodal feature alignment method of the optimal transport theory and the SInkhorn algorithm to obtain the motion trajectory of the moving object.

[0027] The motion analysis module is used to analyze the motion pattern and posture change of the motion object based on the motion trajectory of the motion object.

[0028] In some embodiments, the inertial sensor module first has the function of collecting inertial signals of the moving object. These inertial signals cover a variety of physical quantities of the set parts of the moving object, specifically, acceleration, angular velocity, and magnetic field strength. Through the collection of these data, the dynamic characteristics of the moving object during the movement process can be fully understood and analyzed.

[0029] In some embodiments, the visual sensor module is a rare memory enhancement module based on Sharma-MIttal entropy, which is used to collect key point information of the human body of a moving object.

[0030] Among them, the visual sensor module includes a feature extraction submodule, an attention enhancement submodule, a rare memory enhancement submodule based on Sharma-MIttal entropy, a two-dimensional key point detection submodule and a three-dimensional visual reconstruction submodule.

[0031] Specifically, a normal picture taken by a mobile phone has about hundreds of thousands to millions of pixels. However, in such a picture, the key points of the human body are a few very sparse and rare pixels. In order to better detect the key points of the human body in the image, this embodiment proposes a "Rare Memory Enhancement (RME)" module based on Sharma-MIttal entropy, and constructs a deep learning-based visual sensor module (human key point detector): SM-Detector. SM-Detector is mainly composed of the following four modules:

[0032] The feature extraction submodule (FeatureEXtractIon): extracts multi-channel features from the input image.

[0033] The attention enhancement submodule (AttentIonAugmentation): enhances the attention to sparse features through the attention mechanism.

[0034] The Sharma-MIttal entropy-based rare memory enhancer (Rare Memory Enhancement, RME) module: regularizes the feature layer to enhance the model's memory of sparse features.

[0035] The two-dimensional key point detection submodule (KeypointDetection) outputs the positions of the two-dimensional key points of the human body.

[0036] The three-dimensional vision reconstruction submodule (3D VIsIonReconstructIon): outputs the positions of three-dimensional key points of the human body.

[0037] Among them, the specific calculation process of SM-Detector is as follows:

[0038] 1) Feature extraction submodule: First, this embodiment uses the ResNet50 network to extract multi-channel features from the input image: F = ResNet50 (I).

[0039] Where I is the input image, and its dimension is H0 and W0 are the height and width of the input image respectively, and F is the extracted feature with dimension H and W are the height and width of the feature map, and c is the number of feature channels.

[0040] 2) Attention Enhancement Submodule:

[0041] In order to enhance the focus on sparse key points, this embodiment uses an attention mechanism. This module calculates the weight of each position and uses these weights to weight the features. Calculate the attention weight matrix A:

[0042]

[0043] Where Q = W q F, Q is the query matrix, W q is the first linear transformation matrix that can be learned, with dimension K=W k F, K are key value matrices, W k is the learnable second linear transformation matrix with dimension d k is the dimension of the key value matrix, A is the attention weight matrix, which is used to represent the importance of different positions on the feature map, and its dimension is V is the value matrix, W v is the learnable third linear transformation matrix with dimension F' is the weighted feature, and its dimension is

[0044] After obtaining the attention weights, this embodiment uses the attention weights to weight the features to highlight the features at important positions:

[0045] F'=A·V,V=W v F.

[0046] Where V is the value matrix, W v is a learnable linear transformation matrix with dimension F' is the weighted feature, with dimension

[0047] 2) Rare memory enhancement module based on Sharma-MIttal entropy:

[0048] In order to enhance the model's memory of sparse features, this embodiment introduces Sharma-MIttal entropy. This module acts on the weighted feature layer F' and realizes regularization by calculating the probability distribution of each position. For each channel c's feature map F' c , this embodiment directly calculates the probability distribution p of each position in two-dimensional space i,j,c :

[0049]

[0050] Among them, F' i,j,c is the eigenvalue of the feature map at position (I, j). i,j,c is the probability of the eigenvalue of the cth channel at position (I, j). It is the sum of the eigenvalues ​​at all positions on the feature map and is used to normalize the eigenvalues ​​to ensure a valid probability distribution.

[0051] For each channel c, based on the calculated probability distribution p i,j,c , this embodiment calculates the Sharma-MIttal entropy S r,q (F' c ):

[0052]

[0053] Among them, r and q are two parameters of Sharma-MIttal entropy. By choosing r>1 and q<1, the model's ability to remember sparse features can be enhanced. r,q (F' c ) is the Sharma-MIttal entropy on the c-th channel, which acts on the two-dimensional feature map and enhances the model's attention to some sparse features in the feature map.

[0054] Finally, the Sharma-MIttal entropy regularization term is the sum of all channel entropy values:

[0055]

[0056] Among them, λ is the regularization coefficient, which controls the weight of Sharma-MIttal entropy in the total loss function, S r,q (F' c) is the Sharma-MIttal entropy, c represents the cth channel, and c represents the number of channels. This regularization term helps to enhance the model's response to sparse features, especially in focusing on low-probability events. A smaller Sharma-MIttal entropy means that the feature distribution is more concentrated, and the model can focus on certain key points, thereby enhancing the memory ability of sparse features.

[0057] 4) 2D key point detection submodule:

[0058] The features after attention enhancement and rare memory enhancement are input into the key point detection module to predict the location of the key points of the human body:

[0059]

[0060] Among them, W y and b y are learnable parameters of the linear transformation used to map features to the coordinate space of keypoints. is the predicted position of the 2D key points of the human body, the final result output by the 2D key point detection module, F vision These are the characteristics after attention enhancement and rare memory enhancement.

[0061] 5) 3D visual reconstruction submodule:

[0062] In applications such as motion analysis and virtual reality, the three-dimensional coordinates of human key points are crucial. Therefore, in the overall network architecture, this embodiment designs a three-dimensional human key point reconstruction algorithm, which aims to reconstruct three-dimensional human key points using the aforementioned features after "attention enhancement" and "rare memory enhancement" and the detected two-dimensional key points.

[0063] Considering that it is a feature serving two-dimensional key point detection, this embodiment needs to process it appropriately to serve three-dimensional key point reconstruction. To this end, this embodiment uses a deformable attention mechanism to act on multi-channel features. For the predicted feature F' of each joint point and each channel c , perform deformable attention operation on the feature map to obtain feature

[0064]

[0065] in, is the two-dimensional coordinate of joint j. The specific deformable attention DeformAttn formula is as follows:

[0066]

[0067] Where M is the number of attention heads, K is the number of sampling points for each attention head, and Δp c,m,kis the offset of the kth sampling point of the mth attention head relative to , A c,m,k is the corresponding attention weight.

[0068] Subsequently, this embodiment fuses the features processed by the deformable convolution with the two-dimensional coordinate embedding of the joints to form a feature vector with rich spatial context information. Specifically, this embodiment stacks the features of each joint with the joint coordinate embedding:

[0069]

[0070] in, is the fused feature vector of joint j.

[0071] Then, the stacked fusion vector is input into the self-attention layer for feature fusion:

[0072]

[0073] Finally, the fused feature vector is obtained:

[0074]

[0075] After obtaining the feature representation of each joint, this embodiment also needs to model the spatial dependency between different joints so that the model can better understand the overall structure of the human skeleton. Therefore, this embodiment uses the Transformer encoder to globally model the fused joint features. Specifically, this embodiment inputs the features of each joint into the Transformer encoder for spatial modeling:

[0076]

[0077] The obtained output encodes the spatial dependency information between all joints. Therefore, this embodiment maps the spatially modeled features to a three-dimensional coordinate space to generate the final three-dimensional joint coordinates:

[0078]

[0079] in, are the predicted 3D keypoint coordinates.

[0080] In some embodiments, the signal processing module specifically includes the following contents:

[0081] The signal processing module is used to enhance the inertial signal using an improved deep learning model; the improved deep learning model is a deep learning model that introduces generative hallucination entropy into the loss function of the deep learning model, and is used to reconstruct the over-range signal in the inertial signal based on the Kolmogorov-Arnold network.

[0082] Among them, 1D-ResNet is used to extract features of the input inertial signal to obtain a feature vector; each feature in the feature vector is transformed by a learnable first spline parameterized function to generate the first layer of intermediate nodes; the first layer of intermediate nodes are processed by the second spline parameterized function to generate the second layer of intermediate nodes; the process of generating intermediate nodes is repeated, and the reconstructed signal is generated through multi-layer spline function transformation.

[0083] Specifically, since traditional signal processing methods often regard signals as a series of discrete point sequences, this discretized processing method has obvious limitations in capturing the continuity and dynamic change characteristics of signals, especially when dealing with highly dynamic nonlinear signals, the effect is usually not ideal.

[0084] Therefore, this embodiment proposes to introduce the Kolmogorov-Arnold network (KAN) into inertial signal processing. KAN has a built-in learnable spline function, which is suitable for curve fitting. Its core idea is to use a multi-layer network structure embedded with spline functions to express complex multivariable continuous functions. Compared with the traditional multi-layer perceptron (MLP), KAN can more effectively capture the nonlinear characteristics and subtle fluctuations of the signal. This network structure is suitable for processing inertial signals that are prone to saturation in high dynamic environments. The schematic diagram of over-range signal reconstruction based on KAN is shown in the figure. Figure 3 shown.

[0085] Specifically, the reconstruction process is as follows:

[0086] First, the input inertial signal is feature extracted through 1D-ResNet, and the signal is mapped into a feature vector F = [F1, F2, ..., F m ], providing high-quality input for the subsequent KAN network. For each feature F j , transformed through a learnable spline parameterized function to generate the first layer of intermediate nodes Its mathematical expression is:

[0087]

[0088] Among them, g ij It is the first learnable parameterized function with the ability to capture nonlinear relationships in signals. is the first layer intermediate node; m is the number of feature vectors, F j is the jth feature vector after feature extraction of the input inertial signal through 1D-ResNet.

[0089] Through learning, these functions can be dynamically adjusted to more accurately fit the complex curves in the signal. Next, these intermediate nodes are further processed by another set of spline parameterized functions to generate the next layer of intermediate nodes. Its mathematical expression is:

[0090]

[0091] In the formula, f ki is a learnable second spline parameterized function, is the middle node of the second layer; I is the Ith eigenvector of the second layer, and p is the number of eigenvectors of the second layer.

[0092] In the multi-layer propagation process, the output of each layer will be used as the input of the next layer, and the final output layer will generate the reconstructed signal. The motivation for this layer-by-layer structural design is to gradually approach the complex nonlinear characteristics of the original signal through layer-by-layer spline function transformation, thereby achieving high-precision signal reconstruction.

[0093] Therefore, the KAN network overcomes the limitations of the traditional discrete point sequence model by embedding continuous spline functions, and can better capture and reconstruct nonlinear dynamic changes in the signal. Especially when processing inertial signals in highly dynamic environments, the KAN network can effectively reduce the information loss caused by discretization and achieve more accurate signal reconstruction. In addition, by introducing learnable spline parameterized functions into the network, KAN can adaptively adjust the parameters of each layer to adapt to signal characteristics of different complexities, which makes it have strong generalization ability in a variety of application scenarios.

[0094] Specifically, in the application of deep learning models, especially in the field of signal processing, the so-called "hallucination" problem is often encountered, that is, the model generates chaotic or false outputs that do not conform to the actual situation when reconstructing the signal. This phenomenon is particularly important in high-risk fields (such as aerospace, medical devices, and autonomous driving), because even small errors can have serious consequences.

[0095] For this reason, Figure 4 As shown, this embodiment proposes a new measurement method, Generative Hallucination Entropy (GHE), to quantitatively evaluate and reduce the hallucination phenomenon of the model and ensure the stability and reliability of signal reconstruction. The calculation process of GHE can be divided into the following steps:

[0096] (1) Multi-resolution downsampling:

[0097] The input saturated signal is downsampled N times to obtain a series of downsampled signals of different lengths. This step simulates the performance of signals at different scales by generating signal versions with multiple resolutions, so that GHE can detect the reconstruction consistency of the model at different resolutions.

[0098] (2) Reconstruction of KAN model:

[0099] These multi-resolution signals are input into the KAN model for reconstruction to obtain the reconstructed signal

[0100] (3) Pearson correlation coefficient calculation:

[0101] All reconstructed signals are uniformly downsampled to simplify the computational complexity, and then the Pearson correlation coefficient P between these signals is calculated. ij The Pearson correlation coefficient was used to quantify the similarity between signals of different resolutions and thus assess the consistency of the model output.

[0102] (4) Building GHE:

[0103] The final GHE is calculated by the following formula:

[0104]

[0105] Where σ is the SIgmoId function used to normalize the Pearson correlation coefficient, and the weights ensure that signals of similar scales are more similar. GHE By weighted penalty on the consistency of output signals under similar inputs, the hallucination phenomenon generated by the model is reduced. ij is the Pearson correlation coefficient, len (j) is the data of the jth eigenvector, the data of the initial eigenvector, len (i) is the data of the I-th eigenvector.

[0106] Then, GHE is introduced into the loss function. By minimizing GHE, the hallucination phenomenon of the deep learning model in signal reconstruction is reduced, ensuring that the signal generated by the model is more stable and reliable. Compared with the traditional loss function, GHE not only considers the difference between the reconstructed signal and the original signal, but also pays special attention to the consistency of the model at different resolutions, which makes it more applicable and effective in signal processing tasks. The advantage of GHE is that it can effectively punish model outputs that show inconsistency on multi-resolution signals, thereby forcing the model to continuously optimize its reconstruction ability during the training process, and ultimately achieve high-precision reconstruction of overloaded signals.

[0107] The I-KAN model proposed in this embodiment successfully solves the signal overload problem of inertial sensors in high dynamic environments by introducing the Kolmogorov-Arnold network and generating hallucination entropy. The KAN network shows excellent performance in fitting complex curves, and the introduction of GHE greatly improves the output stability and reliability of the model. Experimental verification shows that the I-KAN model can not only effectively reconstruct the overload signal, but also significantly expand the working range of low-cost inertial sensors.

[0108] In some embodiments, the inertial-visual information fusion module is used to fuse the key point information of the human body and the inertial signal after signal processing to obtain the motion trajectory of the moving object, specifically including:

[0109] 1) Inertial sensor feature extraction:

[0110] The inertial sensor captures acceleration and angular velocity data, which are then repaired and enhanced by the I-KAN architecture designed in this embodiment to avoid signal overload caused by range limitation. The final 3D acceleration data is A t =[a x (t),a y (t),a z (t)], the angular velocity data is Ω t =[ω x (t),ω y (t),ω z (t)], where t represents the time step. This embodiment uses a 1D convolutional neural network (conv1D) to extract inertial signal features:

[0111] F inertial =Conv1D(A t ,Ω t ).

[0112] Among them, F inertial is the extracted inertial sensor feature, T is the number of time steps and d is the feature dimension.

[0113] 2) Visual feature extraction:

[0114] In the section “2 AI Rare Memory Enhancement for Computer Vision”, this embodiment has obtained the feature F’ enhanced by attention and rare memory. c , here this embodiment aligns and fuses it with the inertial feature.

[0115] 3) Optimal Transport Theory-Based Multimodal Feature Alignment (OT-FA):

[0116] This embodiment first maps the features of the inertial sensor and the optical sensor to a unified feature space through a self-attention mechanism:

[0117]

[0118] in, and Have the same feature dimension. Considering that they record the same set of motions, the two modal features should have potential consistency. However, due to the difference in their modalities, this consistency is difficult to mine and constrain. Therefore, this embodiment proposes a multimodal feature alignment based on optimal transmission theory (Optimal Transportbased FeatureAlign, OT-FA). Optimal transmission theory finds the optimal mapping between two different distributions to minimize the transmission cost, which is very suitable for mining the potential consistency of multimodal features.

[0119] Assume that the distribution of inertial sensor characteristics is μ inertial , the distribution of the optical sensor characteristics is μ vision The goal of the optimal transport problem is to find the optimal mapping γ∈Γ(μ inertial ,μ vision ), this mapping can minimize the transmission cost between the two distributions. The optimal transmission problem can be formalized as the following optimization problem:

[0120]

[0121] Among them, Γ(μ inertial ,μ vision ) represents the set of all coupled distributions, f i and represent the characteristics of inertial and optical sensors respectively, c(f i ,f j ) is the cost function, and this embodiment uses the Euclidean distance between features:

[0122] c(f i ,f j )=||f i -f j || 2 .

[0123] In order to solve this optimization problem, this embodiment uses the SInkhorn algorithm for approximate calculation. This algorithm can efficiently calculate the approximate optimal transmission mapping P * . Furthermore, this embodiment can map both the inertial signal features and the visual signal features to an aligned feature space:

[0124]

[0125] Subsequently, this embodiment may constrain the consistency of the multimodal signal in the aligned feature space:

[0126]

[0127] In summary, the OT-FA method can effectively align features of different modalities and maximize the supervision information obtained from unpaired data by utilizing the optimal transfer theory, thereby ensuring more stable model training and more accurate feature alignment.

[0128] 4) Time correlation constraint based on KANIAdakIs entropy regularization:

[0129] The multimodal feature interaction based on optimal transmission theory can utilize the potential correlation between modalities, thereby providing more usable information for motion capture of each modality. Based on these high-quality features, this embodiment can achieve 3D motion trajectory reconstruction with the help of VQ-VAE (Vector Quantized Variational Autoencoder) and Transformer modules. Assume that the motion tracking and trajectory reconstruction results based on inertial sensors are The result of human key point trajectory reconstruction based on vision is: Taking into account the insufficient use of temporal correlation information in the reconstruction process, this embodiment designs a KANIAdakIs entropy regularized temporal correlation constraint, which utilizes the continuity and local smoothness of motion to constrain trajectory generation results, thereby forcing the model to optimize feature extraction and inference prediction processes.

[0130] KANIadakIs entropy is an asymmetric entropy form that can control the nonlinear effects and heterogeneity of the system, and is particularly suitable for dealing with non-equilibrium problems in complex systems. The calculation formula of KANIadakIs entropy is as follows:

[0131]

[0132] Where κ is the deformation parameter of KANIAdakIs entropy. When , KANIAdakIs entropy degenerates into the classic Shannon entropy. i is the probability distribution of the trajectory points. This embodiment uses kernel density estimation (KDE, Kernel DensIty EstImatIon) to estimate the trajectory points ( or )’s probability density function p i :

[0133]

[0134] in, is a kernel function (e.g., Gaussian kernel) used to measure the similarity between trajectory points and . n represents the total number of trajectory points.

[0135] By Adding it as a regularization term to the total loss function can penalize overly complex or non-smooth trajectory outputs, forcing the model to relearn the training parameters.

[0136] In this embodiment, the motion analysis module is used to analyze the motion pattern and posture change of the motion object based on the motion trajectory of the motion object, as follows:

[0137] like Figure 2 As shown, the motion analysis module can be applied to golf motion analysis.

[0138] In this application scenario, this embodiment applies the aforementioned multi-sensor fusion and trajectory reconstruction method to golf motion analysis, especially to calculate the X-factor in golf swing. The X-factor is an important parameter in golf kinematics. It reflects the rotation angle difference between the shoulder and pelvis during the swing and is usually used to evaluate the athlete's trunk rotation ability and efficiency when hitting the ball. The specific process is as follows:

[0139] First, data collection:

[0140] (1) Objective: To accurately capture the physical motion data of a golfer during the golf swing, including the 3D position information of key joints and limbs.

[0141] (2) Sensor settings:

[0142] Inertial Measurement Units (IMUs):

[0143] Location: Installed at key parts of the golfer and the club, including: Torso: sternum, lumbar spine; Limbs: upper arms, forearms, thighs, calves; Head: back of the head; Club: club head.

[0144] Sampling frequency: Set above 100Hz to capture fast motion details.

[0145] Measurement data: acceleration, angular velocity, magnetic field strength.

[0146] Optical sensor (camera system): either monocular or multi-camera.

[0147] (3) Collection process:

[0148] Preparation phase: Install and calibrate all sensors. Have golfers do warm-up activities to get used to the presence of sensors.

[0149] Data recording: Have the golfer perform multiple standardized swings, covering different forces and speeds. During each swing, all sensors record data simultaneously.

[0150] Data storage: The collected raw data is stored securely, ready for subsequent processing and analysis.

[0151] Then perform inertial-visual signal enhancement, fusion and motion capture:

[0152] The previously designed multimodal data fusion model is used to fuse the data from inertial sensors and optical sensors to obtain a high-precision three-dimensional trajectory.

[0153] Then the X-factor is calculated as follows:

[0154] X-factor refers to the difference in rotation between the shoulders and hips during the golf swing. A larger X-factor is generally associated with more swing power and longer hitting distance. The calculation steps are as follows:

[0155] (1) Key point extraction:

[0156] Shoulder key points: Left shoulder point: S L =[x SL ,y SL ,z SL ]. Right shoulder point: S L =[x SR ,y SR ,z SR ] Hip key points: Left hip point: H L =[x HL ,y HL ,z HL ]. Right hip point: H R =[x HR ,y HR ,z HR ].

[0157] (2) Define the shoulder and hip vectors:

[0158] Shoulder Vector: V shoulder =S R -S L ; Hip vector: V hip =H R -H L .

[0159] (3) Projection onto the horizontal plane:

[0160] In order to calculate the horizontal rotation angle, this embodiment projects the above vector onto a horizontal plane (usually the XY plane): shoulder vector projection: Among them, vsx =x SR -x SL , v sy =y SR -y SL .

[0161] Hip vector projection: Among them, v hx =x HR -x HL , v hy =y HR -y HL .

[0162] (4) Calculate the angle of the vector:

[0163] Shoulder angle: θ shoulder =arctan2(v sy ,v sx ); Hip angle: θ hip =arctan2(v hy ,v hx ).

[0164] (5) Calculate the X-factor:

[0165] X-Factor Angle: X factor =θ shoulder -θ hip ; Positive values ​​indicate backward rotation of the shoulder relative to the hip, negative values ​​indicate forward rotation.

[0166] (6) Time series analysis:

[0167] For each time frame t, calculate the corresponding X factor , get the X-factor change curve during the whole swing process. At the same time, calculate the X-factor at key moments: Top of Backswing: usually the moment when the X-factor reaches its maximum value. Impact: observe the regression degree of the X-factor and evaluate the energy release efficiency.

[0168] Finally, motion analysis is performed as follows:

[0169] Through the foregoing, this embodiment can obtain accurate motion capture results, and calculate indicators such as X-factor based on the motion capture results. Based on these indicators, this embodiment can carry out a more detailed analysis of the motion process. For example, a curve of the X-factor changing over time is drawn to intuitively show the rotation difference during the swing process; X-factor curves of different swing attempts or different golfers are compared horizontally; the training model predicts the optimal X-factor range to improve the hitting effect; abnormal patterns are identified, and real-time feedback and correction suggestions are provided. Practical applications include but are not limited to:

[0170] (1) Technical evaluation and improvement:

[0171] Power output: Evaluate the player's power output ability by analyzing the maximum value and change rate of the X-factor. Movement coordination: Observe the rotation synchronization of the shoulder and hip to find potential movement incoordination problems. Swing efficiency: Evaluate the efficiency of energy transfer and release through the changes of the X-factor at different stages.

[0172] (2) Injury prevention:

[0173] Risk of over-rotation: Excessive X-factor may increase the risk of lower back injury, and timely adjustment can prevent injury. Asymmetry detection: Find the difference in left and right rotation and guide players to carry out targeted training and balance.

[0174] (3) Development of personalized training program:

[0175] Targeted training: Develop a training plan to improve core strength and flexibility based on the results of the X-factor analysis. Progress tracking: Evaluate the training effect and progress by continuously monitoring the changes in the X-factor.

[0176] By applying multi-sensor fusion and trajectory reconstruction technology to golf motion analysis, this embodiment can capture and analyze the player's motion details with high precision, especially the key X-factor indicators. This provides a scientific basis and effective means for golf technical training, performance evaluation and injury prevention, and promotes the intelligent and data-driven nature of golf motion analysis.

[0177] In summary, this application has the following technical effects:

[0178] 1) Over-range inertial signal reconstruction based on I-KAN: This application introduces the Kolmogorov-Arnold network (KAN) for inertial signal processing, especially for the signal saturation problem of inertial sensors in high dynamic environments. The KAN network has a built-in learnable spline function, which approximates complex nonlinear functions layer by layer through a multi-layer structure, thereby achieving high-precision reconstruction of the signal. Compared with traditional methods, the network can regard the signal as a continuous curve rather than a sequence of discrete points. Therefore, it can effectively capture the continuous characteristics of the signal and reduce the information loss caused by discretization. It is suitable for over-range reconstruction of sensor signals and significantly expands the working range of low-cost inertial sensors. In fact, this application is the first to apply the KAN architecture to sensor signal processing.

[0179] 2) Generative Hallucination Entropy (GHE) is used to reduce the phenomenon of generated hallucinations in deep learning models: In response to the "hallucination" phenomenon that may occur in the signal reconstruction process of deep learning models (i.e., generating chaotic outputs that do not conform to the actual situation), this application proposes generated hallucination entropy (GHE) as a quantitative evaluation indicator. GHE quantifies the reconstruction stability of the model at different resolutions through multi-resolution downsampling, reconstructed signal consistency evaluation, and Pearson correlation coefficient calculation. Introducing GHE into the loss function and minimizing GHE can effectively reduce the model's hallucination phenomenon and ensure the reliability of signal reconstruction, which is especially important in high-risk fields such as aerospace, medical equipment, and autonomous driving.

[0180] 3) Sharma-MIttal entropy is used for rare memory enhancement of deep learning models: In human key point detection, in order to cope with the problem of sparse feature detection in images, this application proposes a rare memory enhancement (RME) module based on Sharma-MIttal entropy. By introducing the regularization processing of Sharma-MIttal entropy on the feature layer, this module can significantly enhance the model's memory ability for low-probability and sparse features. Combining the multi-channel feature extraction of the ResNet50 network and the feature weighting of the attention mechanism, the SM-Detector of this application has shown high accuracy in 2D key point detection and 3D reconstruction, especially in the extraction and memory of sparse human features.

[0181] 4) Multimodal feature alignment based on optimal transport theory: To solve the problem of multimodal feature alignment between inertial sensors and visual sensors, this application adopts the optimal transport theory and uses the SInkhorn algorithm for efficient calculation to achieve precise alignment of feature space. The OT-FA method can maximize the supervisory information of unpaired data by minimizing the transmission cost between modalities while retaining the potential consistency of inertial and visual signals. This alignment method improves the stability and consistency of the fusion of inertial and visual data, ensuring that the signal fusion in the motion capture process is more accurate and reliable.

[0182] 5) Time correlation constraint based on KANiadakIs entropy regularization: Taking into account the problem of insufficient utilization of time correlation information in the process of motion trajectory reconstruction, this application designs a time correlation constraint based on KANiadakIs entropy regularization. KANiadakIs entropy controls the nonlinear effects and heterogeneity of the system, and is particularly suitable for dealing with non-equilibrium problems. The probability density function of the trajectory points is estimated by combining the kernel density estimation method, and it is included in the loss function as a regularization term, forcing the model to optimize the feature extraction and reasoning process, thereby generating a smooth and time-correlated 3D motion trajectory.

[0183] 6) This application proposes an innovative multimodal signal processing architecture that deeply integrates inertial signal processing and visual signal processing to form an integrated motion capture and intelligent analysis system. By integrating Kolmogorov-Arnold network (KAN) for signal reconstruction, generative hallucination entropy (GHE) for quality control, and rare memory enhancement (RME) and multimodal feature alignment (OT-FA) modules in the same architecture, this application realizes all-round and multi-level processing of motion data. This modular architecture not only improves the robustness of the system, but also significantly enhances the ability of different modal data to work together, enabling the entire system to achieve accurate and stable motion capture and analysis in complex environments.

[0184] 7) The system architecture of this application adopts a modular design, and each functional module can be operated independently or in combination with other modules. This design greatly improves the scalability of the system, allowing the system to be flexibly adjusted according to specific application requirements. For example, for different sensor data types, the corresponding feature extraction and alignment modules can be flexibly introduced or replaced. This scalability not only extends the life cycle of the system, but also increases its applicability in different application scenarios, ensuring the system's continued competitiveness in future technological developments.

[0185] 8) From the application perspective, this application applies the fusion of inertial signal processing and visual signal processing to the precise and intelligent analysis of golf. The system architecture is specially optimized for the swing action in golf, combining multimodal data processing and motion trajectory reconstruction technology to achieve a comprehensive analysis of the golfer's swing trajectory and posture. The system can not only capture the three-dimensional motion trajectory of the club through inertial sensors, but also analyze the changes in the golfer's posture in combination with visual signals to generate key human body mechanical parameters such as X-Factor and tightening factor. Through this fusion and analysis of multimodal data, the system can provide accurate technical guidance for golf beginners and help coaches develop personalized training plans, greatly improving the training efficiency and scientificity of golf.

[0186] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0188] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A motion analysis system based on inertial-visual signal enhancement and fusion, characterized in that: include: Inertial sensor module, visual sensor module, signal processing module, inertial-visual information fusion module and motion analysis module; The inertial sensor module is used to collect inertial signals of a moving object; the inertial signals include acceleration, angular velocity and magnetic field strength of a set part of the moving object; The visual sensor module is a rare memory enhancement module based on Sharma-MIttal entropy, which is used to collect key point information of the human body of the moving object; The signal processing module is used to: use the improved deep learning model to enhance the inertial signal, and reconstruct the over-range signal in the inertial signal based on the Kolmogorov-Arnold network; the improved deep learning model is a deep learning model that introduces generative hallucination entropy into the loss function of the deep learning model; The inertial-visual information fusion module is used to fuse the key point information of the human body and the inertial signal after signal processing based on the multimodal feature alignment method of the optimal transport theory and the SInkhorn algorithm to obtain the motion trajectory of the moving object; The motion analysis module is used to analyze the motion pattern and posture change of the motion object based on the motion trajectory of the motion object.

2. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 1 is characterized in that: The signal processing module is used to reconstruct the over-range signal in the inertial signal based on the Kolmogorov-Arnold network. The signal processing module is used to extract the features of the input inertial signal using 1D-ResNet to obtain a feature vector; Each feature in the feature vector is transformed by a learnable first spline parameterized function to generate the first layer of intermediate nodes; the first layer of intermediate nodes are processed by a second spline parameterized function to generate the second layer of intermediate nodes; the process of generating intermediate nodes is repeated, and the reconstructed signal is generated through multi-layer spline function transformation.

3. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 2 is characterized in that: In the signal processing module, each feature in the feature vector is transformed by a learnable first spline parameterized function, and the formula expression for generating the first layer intermediate node is: Among them, g ij is the first learnable parameterized function, is the intermediate node of the first layer; m is the number of feature vectors in the first layer, F j is the jth feature vector after feature extraction of the input inertial signal through 1D-ResNet.

4. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 3 is characterized in that: In the signal processing module, the second spline parameterized function is used to process the first layer intermediate nodes, and the formula expression for generating the second layer intermediate nodes is: Among them, f ki is a learnable second spline parameterized function, is the middle node of the second layer; I is the Ith eigenvector of the second layer, and p is the number of eigenvectors of the second layer.

5. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 1 is characterized in that: The calculation formula for the generative hallucination entropy introduced by the deep learning model in the signal processing module is: Where, σ is the SIgmoId function used to normalize the Pearson correlation coefficient, H GHE By weighted penalty on the consistency of output signals under similar inputs, P ij is the Pearson correlation coefficient, len (j) is the data of the jth eigenvector, len (0) is the data of the initial feature vector, len (i) is the data of the I-th eigenvector.

6. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 1 is characterized in that: The visual sensor module includes a feature extraction submodule, an attention enhancement submodule, a rare memory enhancement submodule based on Sharma-MIttal entropy, a two-dimensional key point detection submodule and a three-dimensional visual reconstruction submodule; The feature extraction submodule is used to extract multi-channel features in the input image; The attention enhancement submodule is used to enhance the attention to sparse features in multi-channel features through an attention mechanism; The Sharma-MIttal entropy-based rare memory enhancement submodule is used to regularize the feature layer and enhance the memory of the visual detection model for the sparse features; The two-dimensional key point detection submodule is used to output the positions of the two-dimensional key points of the human body in the input image; The three-dimensional visual reconstruction submodule is used to output the positions of three-dimensional key points of the human body in the input image.

7. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 6 is characterized in that: The feature extraction submodule uses the ResNet50 network to extract multi-channel features from the input image; wherein the extraction formula is: F = ResNet50 (I); Where I is the input image, and its dimension is H0 and W0 are the height and width of the input image respectively, and F is the extracted feature with dimension H and W are the height and width of the feature map, and c is the number of feature channels.

8. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 6 is characterized in that: The attention enhancement submodule is used to enhance the sparse features in the multi-channel features through the attention mechanism. In terms of attention, the attention enhancement submodule is used to: According to the formula Calculate the importance of each position on the feature map and determine the position of the sparse features; according to the formula F'=A·V, V=W v F, weighting the sparse features; Where Q = W q F, Q is the query matrix, W q is the first linear transformation matrix that can be learned, with dimension K=W k F, K are key value matrices, W k is the learnable second linear transformation matrix with dimension d k is the dimension of the key value matrix, A is the attention weight matrix, which is used to represent the importance of different positions on the feature map, and its dimension is V is the value matrix, W v is the learnable third linear transformation matrix with dimension F' is the weighted feature, and its dimension is 9. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 6, characterized in that: In terms of regularizing the feature layer by the Sharma-MIttal entropy-based rare memory enhancement submodule to enhance the memory of the sparse features by the visual detection model, the formula expression of the Sharma-MIttal entropy regularization term of the Sharma-MIttal entropy-based rare memory enhancement submodule is: Among them, λ is the regularization coefficient, S r,q (F' c ) is the Sharma-MIttal entropy, c represents the cth channel, and c represents the number of channels.

10. The motion analysis system based on inertial-visual signal enhancement and fusion according to claim 6, characterized in that: In terms of the two-dimensional key point detection submodule being used to output the positions of the two-dimensional key points of the human body in the input image, the two-dimensional key point detection submodule is used to output the positions of the two-dimensional key points of the human body in the input image according to the formula Predict the positions of 2D key points of the human body; Among them, W y and b y are learnable parameters of the linear transformation used to map features to the coordinate space of key points, is the predicted position of the 2D key points of the human body, F vision These are the characteristics after attention enhancement and rare memory enhancement.

Images