Dynamic calibration method and device for motion sensor used in intelligent terminal equipment

Abstract

The present invention relates to a dynamic calibration method and device for a motion sensor for an intelligent terminal device, comprising the following steps: real-time acquisition of multi-source sensors on a target intelligent terminal device is performed through a multi-sensor fusion technology to obtain a multi-modal sensor data set; feature extraction is performed on the multi-modal sensor data set to obtain a feature vector set; error pattern recognition is performed on the feature vector set to obtain a sensor error pattern; error compensation is performed on the multi-modal sensor data set based on the sensor error pattern to obtain a compensated multi-modal sensor data set; data is reconstructed based on the compensated multi-modal sensor data set to obtain a reconstructed motion trajectory, thereby solving the technical problem that traditional sensor data is often affected by external interference and self-drift, resulting in reduced measurement accuracy.

Description

Technical Field

[0001] The present invention relates to the field of sensor technology, and in particular to a dynamic calibration method and device for a motion sensor for an intelligent terminal device. Background Art

[0002] As smart terminal devices become increasingly popular, motion sensors, as key components, are essential for achieving functions such as accurate positioning, gesture recognition, and motion tracking. However, due to the limitations of manufacturing processes and the influence of environmental factors during use, traditional sensor data is often affected by external interference and self-drift, resulting in reduced measurement accuracy. These problems seriously affect the performance and user experience of smart terminal devices. Especially in smartphones, wearable devices, and virtual reality devices, the lack of sensor accuracy will directly lead to problems such as navigation deviation and inaccurate motion capture, which greatly restricts the development and application of these technologies.

[0003] Traditional calibration methods usually rely on static calibration or simple linear compensation under laboratory conditions, which are difficult to meet the needs of complex and changing application scenarios. In addition, the data of a single sensor often has limitations. For example, accelerometers are easily affected by gravity and produce errors, and gyroscopes may cause angle calculation errors due to integral drift. Faced with the above challenges, how to integrate information from different sensors through effective methods and perform accurate dynamic calibration on this basis has become the key to improving the performance of smart terminal devices. The core of research in this field is to explore how to use multi-sensor fusion technology to overcome the limitations of a single sensor, thereby improving the stability and accuracy of the overall system.

[0004] In response to the problems existing in the existing technology, researchers proposed a dynamic calibration method based on multi-sensor fusion technology, which aims to correct sensor errors in real time through comprehensive analysis and processing of multiple sensor data to ensure that smart terminal devices can provide more accurate and reliable motion information. This method requires not only a deep understanding of the characteristics of the sensor itself, but also the development of efficient algorithms to process large amounts of complex sensor data to achieve fast and accurate error compensation and data reconstruction. With the development of artificial intelligence and machine learning technologies, new ideas and technical means have been provided to solve these problems, making more efficient and intelligent dynamic calibration possible. This is not only a major innovation in traditional calibration methods, but also lays a solid foundation for the widespread application of smart terminal devices. Summary of the invention

[0005] The main purpose of the present invention is to provide a dynamic calibration method and device for a motion sensor for an intelligent terminal device, which solves the technical problem that traditional sensor data is often affected by external interference and its own drift, resulting in a decrease in measurement accuracy.

[0006] To achieve the above object, the present invention provides a dynamic calibration method for a motion sensor for an intelligent terminal device, comprising the following steps:

[0007] The multi-source sensors on the target intelligent terminal device are collected in real time through multi-sensor fusion technology to obtain a multi-modal sensor data set; wherein the multi-source sensors include accelerometers, gyroscopes, and magnetometers;

[0008] Performing feature extraction on the multimodal sensor data set to obtain a feature vector set;

[0009] Performing error pattern recognition on the feature vector set to obtain a sensor error pattern;

[0010] Performing error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set;

[0011] Data reconstruction is performed based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory.

[0012] Furthermore, the multi-source sensors on the target intelligent terminal device are collected in real time by multi-sensor fusion technology to obtain a multi-modal sensor data set, including:

[0013] Using multi-sensor fusion technology to collect data from multiple sensors on the target intelligent terminal device in real time to obtain preliminary collected data;

[0014] Using a preset three-dimensional spatial timing synchronization network, timing aligning the preliminary collected data to obtain an aligned sensor data stream, and performing coordinate transformation on the aligned sensor data stream to obtain a global coordinate system data stream;

[0015] Based on a deep convolutional neural network, multimodal fusion is performed on the global coordinate system data stream to obtain a multimodal sensor data set; wherein the deep convolutional neural network uses 3 convolutional layers, the convolution kernel sizes of the 3 convolutional layers are 5×5, 3×3, and 3×3 respectively, the number of convolution kernels in each of the convolutional layers is 64, 128, and 256 respectively, and the step size of the convolution kernel is 1.

[0016] Furthermore, the step of extracting features from the multimodal sensor data set to obtain a feature vector set includes:

[0017] Decomposing the multimodal sensor data set by multi-resolution wavelet transform to obtain sub-band signals at different scales, and performing time-frequency domain analysis on each sub-band signal to obtain a multi-scale feature vector set; wherein the multi-resolution wavelet transform adopts Daubechies wavelet function, the sampling frequency is 256 Hz, and the number of decomposition layers is 5;

[0018] Based on the Markov random field model, spatial dependency modeling is performed on the multi-scale feature vector set to obtain a spatial dependency matrix, and Laplace matrix decomposition is performed on the spatial dependency matrix to obtain a feature subspace feature vector set;

[0019] Performing local structure analysis on the feature subspace feature vector set to obtain a local structure feature vector set, and performing high-dimensional feature mapping on the local structure feature vector set based on a deep belief network to obtain a feature vector set.

[0020] Furthermore, performing error pattern recognition on the feature vector set to obtain the sensor error pattern includes:

[0021] Processing the feature vector set by a bidirectional long short-term memory network to obtain a time series feature sequence;

[0022] Based on the variational autoencoder, latent variables are extracted from the time series feature sequence to obtain a latent feature space, and Gaussian mixture clustering is performed on the latent feature space to obtain an error pattern candidate set;

[0023] Performing importance weight calculation on the error pattern candidate set to obtain weighted error features, and performing spectral clustering analysis on the weighted error features to obtain error subclass features;

[0024] Based on a preset dynamic Bayesian network, probability reasoning is performed on the error subclass characteristics to obtain an error transfer probability matrix, and maximum likelihood estimation is performed on the error transfer probability matrix to obtain an error state sequence; wherein the error state sequence includes a steady-state error, a transient error, and a periodic error;

[0025] The time-frequency features of the error state sequence are extracted to obtain an error feature spectrum, and error pattern recognition is performed based on the error feature spectrum to obtain a sensor error pattern; wherein the sensor error pattern includes temperature drift error, hysteresis error, nonlinear error and inter-axis crosstalk error.

[0026] Furthermore, performing error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set includes:

[0027] Decomposing the multimodal sensor data set into error components based on the sensor error pattern to obtain a multimodal error component set;

[0028] Performing state estimation on the multimodal error component set through a preset Kalman filter to obtain an error state vector, and performing curve fitting on the error state vector to obtain an error fitting curve; wherein the error fitting curve is used to reflect the change trend of the error state vector over time;

[0029] Calculating the error accumulation amount of the multimodal error component set based on the error fitting curve to obtain the error accumulation component;

[0030] An error compensation matrix is ​​constructed based on the error accumulation component to obtain a preliminary error compensation matrix; and a nonlinear regression calculation is performed on the preliminary error compensation matrix by using a local weighted regression technique to obtain an error compensation coefficient;

[0031] A point-by-point compensation operation is performed on the multimodal sensor data set based on the error compensation coefficient to obtain a compensated multimodal sensor data set.

[0032] Furthermore, performing point-by-point compensation operation on the multimodal sensor data set based on the error compensation coefficient to obtain a compensated multimodal sensor data set includes:

[0033] Performing Gaussian filtering on the error compensation coefficient to obtain a smoothed error compensation coefficient;

[0034] Preliminarily compensating the multimodal sensor data set by using a smoothing error compensation coefficient to obtain a preliminary compensated data set;

[0035] Performing wavelet transform processing on the preliminary compensation data set to obtain a multi-resolution compensation data set, and performing weighted calculation and fusion on the multi-resolution compensation data set through a preset attention mechanism to obtain a fused compensation data set; wherein the multi-resolution compensation data set includes an approximate component and a detail component;

[0036] A point-by-point compensation operation is performed on the multimodal sensor data set based on the fused compensation data set to obtain a compensated multimodal sensor data set.

[0037] Furthermore, the data reconstruction based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory includes:

[0038] Performing attitude calculation on the compensated multimodal sensor data set to obtain an attitude quaternion sequence; and performing position integration on the compensated multimodal sensor data set based on the attitude quaternion sequence to obtain a preliminary motion trajectory;

[0039] Segmenting the preliminary motion trajectory to obtain motion trajectory segments, and performing Hausdorff distance calculation on the motion trajectory segments to obtain a trajectory segment similarity matrix;

[0040] Performing cluster analysis on the trajectory segment similarity matrix by using a spectral clustering algorithm to obtain trajectory clustering clusters, and performing trajectory fusion on the trajectory clustering clusters to obtain fused motion trajectory segments;

[0041] performing map matching based on the fused motion trajectory segments to obtain a motion trajectory after map matching, and performing topological correction on the motion trajectory after map matching to obtain a motion trajectory after topological correction;

[0042] The topology-corrected motion trajectory is subjected to cubic spline interpolation to obtain a reconstructed motion trajectory.

[0043] The present invention also provides a dynamic calibration device for a motion sensor for an intelligent terminal device, comprising:

[0044] The acquisition module is used to acquire multi-source sensors on the target intelligent terminal device in real time through multi-sensor fusion technology to obtain a multi-modal sensor data set; wherein the multi-source sensors include accelerometers, gyroscopes, and magnetometers;

[0045] An extraction module, used for performing feature extraction on the multimodal sensor data set to obtain a feature vector set;

[0046] An identification module, used for performing error pattern identification on the feature vector set to obtain a sensor error pattern;

[0047] A compensation module, configured to perform error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set;

[0048] The reconstruction module is used to reconstruct data based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory.

[0049] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0050] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0051] The dynamic calibration method of motion sensors for smart terminal devices provided by the present invention comprises the following steps: real-time acquisition of multi-source sensors on the target smart terminal device by multi-sensor fusion technology to obtain a multi-modal sensor data set; feature extraction of the multi-modal sensor data set to obtain a feature vector set; error pattern recognition of the feature vector set to obtain a sensor error pattern; error compensation of the multi-modal sensor data set based on the sensor error pattern to obtain a compensated multi-modal sensor data set; data reconstruction based on the compensated multi-modal sensor data set to obtain a reconstructed motion trajectory, which solves the technical problem that traditional sensor data is often affected by external interference and self-drift, resulting in a decrease in measurement accuracy, realizes data reconstruction based on the compensated multi-modal sensor data set, and realizes high-precision reconstruction of the user's motion trajectory. This process not only improves the quality of applications such as fitness tracking, navigation services, and virtual reality experience, but also improves the speed and smoothness of the interactive interface response, providing users with a more intuitive and smooth operation experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a schematic diagram of the steps of a dynamic calibration method for a motion sensor for a smart terminal device in one embodiment of the present invention;

[0053] Figure 2 is a structural block diagram of a dynamic calibration device for a motion sensor used in an intelligent terminal device in one embodiment of the present invention;

[0054] Figure 3 It is a schematic block diagram of the structure of a computer device according to an embodiment of the present invention.

[0055] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] like Figure 1 As shown, Figure 1 It is a schematic diagram of the steps of a dynamic calibration method for a motion sensor for a smart terminal device in one embodiment of the present invention;

[0058] In one embodiment of the present invention, a dynamic calibration method for a motion sensor for a smart terminal device is provided, comprising the following steps:

[0059] Step S1, real-time acquisition of multi-source sensors on a target intelligent terminal device is performed through multi-sensor fusion technology to obtain a multi-modal sensor data set; wherein the multi-source sensors include an accelerometer, a gyroscope, and a magnetometer.

[0060] Specifically, in smart terminal devices, in order to achieve accurate motion perception and gesture recognition, it is usually necessary to rely on data from multiple sensors. Real-time acquisition of multi-source sensors on target smart terminal devices through multi-sensor fusion technology first means using three key sensors, accelerometer, gyroscope and magnetometer, to work simultaneously to capture the dynamic changes of the device in three-dimensional space. Accelerometers are used to measure linear acceleration, gyroscopes detect changes in angular velocity, and magnetometers can determine the direction of the magnetic field and infer the azimuth angle of the device. When these sensors work together, they can provide complementary information to form a multimodal sensor data set. For example, in smartphones or wearable devices, this multi-sensor fusion technology can help the device accurately determine the user's mobile state, whether walking, running or standing still. Specifically, this process involves integrating data streams from different sensors and processing them synchronously to ensure time consistency. This means that the data from all sensors needs to be collected and recorded at the same time for subsequent analysis and processing. For example, in an application scenario where a user uses a smartwatch to track his or her daily activities, the accelerometer records the speed and direction of the user's arm swing, the gyroscope monitors the change in the angle of the wrist rotation, and the magnetometer helps determine the user's orientation relative to the earth's magnetic field. This information is combined to not only accurately reconstruct the user's movement trajectory, but also identify different types of activities. In this way, even in complex environments such as city streets or indoor gyms, the device's positioning and posture estimation can be ensured to be accurate enough to provide users with reliable health monitoring and location services. This technical solution based on multi-sensor fusion not only improves the performance limit of a single sensor, but also enhances the robustness and adaptability of the entire system, allowing smart terminal devices to perform well in a variety of application scenarios. In addition, since this method allows real-time data acquisition and processing, it also supports an instant feedback mechanism, which is particularly important for occasions that require fast response. In short, by combining data from accelerometers, gyroscopes, and magnetometers, smart terminal devices can achieve more accurate motion perception and posture recognition, meeting the needs of modern users for efficient and reliable applications.

[0061] Step S2: extracting features from the multimodal sensor data set to obtain a feature vector set.

[0062] Specifically, in smart terminal devices, after obtaining multimodal sensor data sets of accelerometers, gyroscopes, and magnetometers through multi-sensor fusion technology, the next step is to extract features from these data to obtain feature vector sets. This process aims to extract the most representative and discriminative information fragments from the raw sensor data for subsequent error pattern recognition and compensation. Specifically, feature extraction involves the use of mathematical algorithms and statistical methods to analyze the time series characteristics, frequency domain characteristics, and other potential patterns of multimodal data, thereby converting complex raw data into more concise and highly informative feature vectors. For example, when processing accelerometer data, the mean, variance, peak value, and frequency component of the signal can be calculated; for gyroscope data, features such as the rate of change of angular velocity and moment of inertia can be extracted; and magnetometer data can help determine the trend of changes in magnetic field strength. To better understand this process, we can consider a specific scenario: a user is using a smart watch to track daily activities. In this application scenario, the accelerometer records the amplitude and frequency of the user's arm swing, the gyroscope captures the speed changes of the wrist rotation, and the magnetometer provides information about the user's azimuth angle. By extracting features from these data, the system can generate a set of feature vectors, which may include statistical properties of each sensor data such as maximum, minimum, average, and standard deviation, as well as high-level features such as short-time Fourier transform coefficients or wavelet transform coefficients based on time windows. These feature vectors not only reflect the user's motion state, but also reveal the characteristics of the sensor itself and its potential error patterns. For example, if the accelerometer readings show an abnormally high variance over a period of time, this may be due to errors caused by external vibration interference; similarly, if the angular velocity change rate of the gyroscope fluctuates irregularly, it may indicate an integral drift problem. In this way, the feature extraction step lays the foundation for subsequent error pattern recognition, allowing the system to more accurately identify and compensate for sensor errors, thereby improving the accuracy and reliability of the entire motion tracking system. Ultimately, the feature-extracted data set can provide clearer and more useful input, supporting smart terminal devices to achieve higher quality user experience and services.

[0063] Step S3, performing error pattern recognition on the feature vector set to obtain a sensor error pattern.

[0064] Specifically, in the smart terminal device, after feature extraction of the multimodal sensor data set, a feature vector set is obtained. The next key step is to perform error pattern recognition on these feature vector sets to determine the sensor error pattern. This process involves the use of advanced algorithms and techniques to analyze abnormal patterns and systematic deviations in feature vectors to identify specific error types of sensors. Specifically, error pattern recognition can be achieved through machine learning models or statistical methods, which can automatically find features that match known error patterns from feature vector sets. For example, using supervised learning algorithms, a classifier can be trained to identify different types of error patterns, such as gravity offset of accelerometers, integral drift of gyroscopes, and magnetic field interference of magnetometers. Continuing with the smart watch application scenario mentioned above, assume that the user is running, and his motion data is captured by the accelerometer, gyroscope, and magnetometer and extracted to obtain a feature vector set. At this stage, the system will further analyze these feature vectors to find any possible error patterns. For example, if the accelerometer data shows continuous high-frequency fluctuations over a period of time, this may be due to errors caused by external vibrations; while if the gyroscope data shows a gradually increasing rate of change of angular velocity, it may indicate an integral drift problem. Similarly, the magnetometer data may be biased by metal objects in the surrounding environment. By applying appropriate machine learning models, the system can identify these error patterns and classify them as specific types of sensor errors. Once the error pattern is identified, the system can more accurately understand the problems in the current sensor data and provide a basis for subsequent error compensation. This error pattern recognition based on feature vector sets not only improves the reliability of sensor data, but also enhances the robustness and adaptability of the entire system, enabling smart terminal devices to maintain a high level of performance in various complex environments. Ultimately, by accurately identifying error patterns, the system can make targeted compensations to ensure that the user's motion trajectory and posture estimation are more accurate, improving user experience and service quality. For example, in health monitoring applications, accurate motion tracking can help doctors better assess the patient's rehabilitation progress, while in navigation applications, accurate posture recognition helps improve positioning accuracy and path planning efficiency.

[0065] Step S4: performing error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set.

[0066] Specifically, in the smart terminal device, after the error pattern recognition of the feature vector set, the error pattern of the sensor has been determined. The next key step is to perform error compensation on the multimodal sensor data set based on these error patterns to obtain a compensated multimodal sensor data set. This process aims to correct the errors in the original sensor data by applying specific algorithms and techniques to improve the accuracy and reliability of the data. Specifically, error compensation involves adjusting the sensor readings according to the identified error pattern so that the compensated data can more realistically reflect the actual motion state of the device. For example, if the accelerometer has a gravity offset error, the accelerometer data can be corrected by subtracting the offset value; if the gyroscope detects an integral drift problem, the Kalman filter or other filtering methods can be used to smooth the angular velocity data and reduce the cumulative error. Continuing with the smart watch application scenario mentioned above, assume that the user is running, and his motion data is captured by the accelerometer, gyroscope and magnetometer and undergoes feature extraction and error pattern recognition. At this point, the system has identified high-frequency fluctuations in the accelerometer data during certain time periods, which may be due to errors caused by external vibrations; while the gyroscope data shows a gradually increasing rate of change of angular velocity, indicating an integral drift problem. In order to compensate for the errors, the system applies corresponding compensation algorithms for these specific error patterns. For example, when processing accelerometer data, the system can calculate and apply a compensation factor to eliminate the deviation caused by external vibrations; for gyroscope data, a Kalman filter can be used to smooth the angular velocity data to reduce the impact of integral drift. In addition, if the magnetometer data shows magnetic field interference, the system can correct it by referring to historical data or combining other sensor information to ensure the accuracy of the azimuth angle. Throughout the error compensation process, the system not only relies on a single compensation technology, but may also use a combination of multiple methods to deal with different types of errors. For example, combining adaptive filters with machine learning models can dynamically adjust the compensation strategy to adapt to changing environmental conditions. In this way, the system can effectively correct the errors in the sensor data and ensure that the resulting multimodal sensor data set is more accurate and reliable. This accurate data set provides a solid foundation for subsequent data reconstruction, allowing smart terminal devices to more accurately track the user's movement trajectory and posture changes. For example, in health monitoring applications, accurate motion data can help doctors better evaluate the patient's recovery progress; in navigation applications, accurate posture recognition helps improve positioning accuracy and path planning efficiency, thereby improving the overall user experience and service quality. In short, the process of error compensation based on sensor error patterns greatly improves the performance of smart terminal devices in various application scenarios.

[0067] Step S5, performing data reconstruction based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory.

[0068] Specifically, in smart terminal devices, the multimodal sensor data set after error compensation has been significantly optimized. The next key step is to reconstruct the data based on these compensated data to obtain an accurate reconstructed motion trajectory. This process involves integrating the compensated accelerometer, gyroscope and magnetometer data, and reconstructing the device's motion path and posture changes through a series of complex algorithms. Specifically, data reconstruction first needs to use the compensated accelerometer data to calculate the linear acceleration of the device and convert it into position information through two integrations; at the same time, the angular velocity data provided by the gyroscope is used to calculate the device's posture changes, including pitch, roll and yaw angles; and the magnetometer data helps correct the direction deviation to ensure that the device's direction information is accurate. These processed data are fused together to form a complete three-dimensional motion model. Continuing with the smart watch application scenario mentioned above, assuming that the user is running, his motion data has been feature extracted, error pattern recognized and error compensated. The system now needs to reconstruct the data based on the compensated multimodal sensor data set. At this stage, the system uses compensated accelerometer data to calculate the user's stride length and cadence, combined with gyroscope data to determine the angle change of wrist rotation, and magnetometer data to correct the direction deviation caused by environmental magnetic field interference. For example, in a specific running scenario, the accelerometer may show the frequency and amplitude of the user's arm swing, the gyroscope records the angle change of wrist rotation, and the magnetometer provides information about the user's direction relative to the earth's magnetic field. By inputting these compensated data into the kinematic model, the system can accurately reconstruct the user's motion trajectory and posture changes. To achieve this goal, the system usually uses advanced algorithms such as Kalman filters or extended Kalman filters for data fusion and state estimation. These algorithms can not only smooth the noise in the sensor data, but also effectively handle the synchronization problem between different sensors, ensuring that the final generated motion trajectory has a high degree of continuity and accuracy. In addition, the system can also combine historical data and other auxiliary information (such as GPS data) to further optimize the accuracy of the motion trajectory. For example, when running in an urban environment, the GPS signal may be blocked by buildings, but by combining multi-sensor data, the system can still provide relatively accurate positioning and motion trajectory. This method of data reconstruction based on compensated multimodal sensor data sets enables smart terminal devices to maintain a high performance level in various complex environments, and can provide more reliable and accurate services for both health monitoring and navigation applications. Ultimately, through accurate data reconstruction, the system can not only restore the user's actual motion trajectory, but also analyze more motion details, such as gait analysis, activity type recognition, etc., thereby improving user experience and service quality.

[0069] In a specific embodiment, the multi-source sensors on the target intelligent terminal device are collected in real time by using multi-sensor fusion technology to obtain a multi-modal sensor data set, including:

[0070] Using multi-sensor fusion technology to collect data from multiple sensors on the target intelligent terminal device in real time to obtain preliminary collected data;

[0071] Using a preset three-dimensional spatial timing synchronization network, timing aligning the preliminary collected data to obtain an aligned sensor data stream, and performing coordinate transformation on the aligned sensor data stream to obtain a global coordinate system data stream;

[0072] Based on a deep convolutional neural network, multimodal fusion is performed on the global coordinate system data stream to obtain a multimodal sensor data set; wherein the deep convolutional neural network uses 3 convolutional layers, the convolution kernel sizes of the 3 convolutional layers are 5×5, 3×3, and 3×3 respectively, the number of convolution kernels in each of the convolutional layers is 64, 128, and 256 respectively, and the step size of the convolution kernel is 1.

[0073] Specifically, in smart terminal devices, in order to achieve accurate motion perception and gesture recognition, it is usually necessary to rely on data from multiple sensors. Real-time acquisition of multi-source sensors on the target smart terminal device through multi-sensor fusion technology first means using three key sensors, accelerometer, gyroscope and magnetometer, to work simultaneously to capture the dynamic changes of the device in three-dimensional space. This process includes multiple steps: preliminary data acquisition, time alignment, coordinate transformation, and multimodal fusion based on deep convolutional neural network. Specifically, first, multi-source sensors on the target smart terminal device are collected in real time through multi-sensor fusion technology to obtain preliminary data. At this stage, the accelerometer records linear acceleration, the gyroscope detects changes in angular velocity, and the magnetometer provides magnetic field direction information. These sensors work independently and generate a large amount of raw data streams. For example, in an application scenario where a user uses a smart watch to track his daily activities, the accelerometer records the speed and direction of the user's arm swing, the gyroscope monitors the angle change of the wrist rotation, and the magnetometer helps determine the user's direction relative to the earth's magnetic field. However, since the data acquisition frequency and timestamp of each sensor may not be exactly the same, further processing is required to ensure data consistency and synchronization. Next, the preset three-dimensional spatial timing synchronization network is used to perform time alignment on the preliminary collected data to obtain the aligned sensor data stream, and the aligned sensor data stream is subjected to coordinate transformation to obtain the global coordinate system data stream. This process is crucial because the data of different sensors may have slight deviations in time. If this deviation is not corrected, it will seriously affect the subsequent analysis results. The three-dimensional spatial timing synchronization network can automatically detect and correct these time differences so that the data of all sensors can be aligned at the same time point. For example, during running, the accelerometer and gyroscope may have time differences due to different sampling rates. These differences can be eliminated through the timing synchronization network to ensure data consistency. In addition, the coordinate transformation step converts the local coordinate system of each sensor into a global coordinate system for subsequent fusion processing. This means that the relative position information of the sensor is converted into a unified reference frame, thereby simplifying the integration process of multi-sensor data. After completing the timing alignment and coordinate transformation, the system performs multimodal fusion on the global coordinate system data stream based on a deep convolutional neural network to obtain a multimodal sensor data set. As a powerful machine learning tool, deep convolutional neural network (CNN) is particularly suitable for processing high-dimensional data such as images and videos, but it is also used here for sensor data fusion. Specifically, the deep convolutional neural network contains 3 convolutional layers, each with a convolution kernel size of 5×5, 3×3, and 3×3, respectively. The number of convolution kernels in each convolutional layer is 64, 128, and 256, respectively, and the stride of the convolution kernel is 1. This structural design is designed to gradually extract high-level features from the data, from low-level spatial features to more complex pattern recognition.First, the first convolutional layer uses a 5×5 convolution kernel to extract preliminary feature maps. These feature maps represent basic patterns in the input data, such as edges or simple textures. Then, the second convolutional layer uses a 3×3 convolution kernel to further refine these features and identify more complex structures, such as specific shapes or patterns. The third convolutional layer also uses a 3×3 convolution kernel, but the number of convolution kernels is increased to 256. This step is designed to capture higher-level features, such as specific action patterns or posture changes. After each convolution operation, the model's expressiveness is enhanced through nonlinear activation functions (such as ReLU), and the size of the feature map is reduced through the pooling layer to reduce the computational complexity. Throughout the multimodal fusion process, the deep convolutional neural network can not only effectively combine data from different sensors, but also automatically learn and adapt to data patterns under various environmental conditions. For example, in the application scenario of smart watches, the large amount of sensor data generated by users during running can accurately reflect the user's actual motion state after the above processing. The data from the accelerometer, gyroscope, and magnetometer are deeply fused to form a complete three-dimensional motion model. This model not only includes the user's motion trajectory, but also can identify specific action types, such as walking, running, and standing still. Through this multi-level convolutional neural network processing, the multimodal sensor data set generated in the end has a high information content and accuracy. This not only improves the performance limit of a single sensor, but also enhances the robustness and adaptability of the entire system, enabling smart terminal devices to perform well in various application scenarios. For example, in health monitoring applications, accurate motion data can help doctors better evaluate the patient's rehabilitation progress; while in navigation applications, accurate posture recognition helps improve positioning accuracy and path planning efficiency. In addition, since this method allows real-time data processing, it also supports an instant feedback mechanism, which is particularly important for occasions that require rapid response. In short, through multi-sensor fusion technology, multi-source sensors on the target smart terminal device are collected in real time, and the three-dimensional spatial time synchronization network is used for time alignment and coordinate transformation, and then multi-modal fusion is performed based on the deep convolutional neural network. The final multimodal sensor data set can significantly improve the performance of smart terminal devices. Whether it is health monitoring or navigation applications, it can provide more reliable and accurate services, greatly improving user experience and service quality. This advanced technical solution not only solves many problems in traditional single-sensor data processing, but also provides a solid foundation for the development of future smart terminal devices. For example, in virtual reality and augmented reality applications, this multimodal fusion technology can provide a more immersive experience, allowing users to experience a more realistic interactive environment.

[0074] In a specific embodiment, extracting features from the multimodal sensor data set to obtain a feature vector set includes:

[0075] Decomposing the multimodal sensor data set by multi-resolution wavelet transform to obtain sub-band signals at different scales, and performing time-frequency domain analysis on each sub-band signal to obtain a multi-scale feature vector set; wherein the multi-resolution wavelet transform adopts Daubechies wavelet function, the sampling frequency is 256 Hz, and the number of decomposition layers is 5;

[0076] Based on the Markov random field model, spatial dependency modeling is performed on the multi-scale feature vector set to obtain a spatial dependency matrix, and Laplace matrix decomposition is performed on the spatial dependency matrix to obtain a feature subspace feature vector set;

[0077] Performing local structure analysis on the feature subspace feature vector set to obtain a local structure feature vector set, and performing high-dimensional feature mapping on the local structure feature vector set based on a deep belief network to obtain a feature vector set.

[0078] Specifically, in smart terminal devices, in order to extract the most representative and discriminative feature information from multimodal sensor data sets, a series of complex processing steps are usually required. Specifically, the multimodal sensor data set is first decomposed by multi-resolution wavelet transform to obtain sub-band signals at different scales, and these sub-band signals are analyzed in the time and frequency domain to obtain a multi-scale feature vector set. This process uses the Daubechies wavelet function, sets the sampling frequency to 256 Hz, and the number of decomposition layers to 5 layers to ensure that the various time scales and frequency characteristics in the sensor data can be captured. Taking a smart watch as an example, assuming that the user is tracking daily activities, his motion data is collected by the accelerometer, gyroscope, and magnetometer and is preliminarily processed. At this stage, the system first applies a multi-resolution wavelet transform to decompose the data. The Daubechies wavelet function is widely used in the field of signal processing due to its good time-frequency localization performance. By setting the sampling frequency to 256 Hz, it can be ensured that the data has sufficient resolution to capture subtle motion changes. The number of decomposition layers is set to 5, which means that the system can analyze data at multiple scales, from high-frequency details to low-frequency trends, gradually revealing potential patterns in the data. For example, during running, the arm swing recorded by the accelerometer may contain high-frequency vibrations and low-frequency periodic changes. Through multi-resolution wavelet transform, the system can decompose these signals into sub-band signals of different scales and further analyze the time-frequency characteristics of each sub-band. This can not only identify the specific action type of the user, but also discover potential abnormal patterns or sources of errors. Next, the multi-scale feature vector set is spatially modeled based on the Markov random field model to obtain the spatial dependency matrix, and the matrix is ​​Laplace matrix decomposed to obtain the feature subspace feature vector set. The Markov random field model is a probabilistic graphical model used to describe the mutual dependence between variables, which is particularly suitable for processing the spatial structure of high-dimensional data. In this case, the model is used to capture the spatial correlation between feature vectors at different scales. Specifically, the system constructs a spatial dependency matrix that represents the degree of association between each feature vector. Then, by performing Laplace matrix decomposition on the matrix, it can be projected into a lower-dimensional feature subspace, thereby extracting a more representative set of feature vectors. For example, during a user's running, the data from different sensors may have certain spatial correlations, such as accelerometer and gyroscope data showing similar change trends in certain time periods. Through the Markov random field model and Laplace matrix decomposition, the system can better understand and utilize these spatial dependencies and improve the accuracy and robustness of feature extraction. Finally, the feature subspace feature vector set is subjected to local structure analysis to obtain the local structure feature vector set, and the local structure feature vector set is subjected to high-dimensional feature mapping based on the deep belief network to obtain the final feature vector set.Local structure analysis aims to identify local patterns and structures in feature vector sets in order to better understand the intrinsic characteristics of the data. For example, in the application scenario of smart watches, the user's motion trajectory may contain some local repetitive patterns, such as stride cycles or specific action sequences. By analyzing these local structures, the system can extract more refined feature information. Subsequently, a deep belief network (DBN) is used for high-dimensional feature mapping. As a deep learning model, DBN can automatically learn complex data distributions and map input data into a higher-dimensional feature space. In this case, DBN takes the local structure feature vector set as input and generates a more abstract and high-level feature representation through the learning process of a multi-layer neural network. This high-dimensional feature mapping not only improves the expressiveness of the features, but also enhances the generalization and robustness of the system. In the entire feature extraction process, each step is closely connected and works together on the raw sensor data to gradually extract the most representative and discriminative feature vector set. For example, in health monitoring applications, accurate feature extraction can help doctors better evaluate the patient's recovery progress; while in navigation applications, accurate posture recognition helps improve positioning accuracy and path planning efficiency. In addition, since this method allows real-time data processing, it also supports an immediate feedback mechanism, which is particularly important for situations that require a quick response. By combining a variety of advanced technologies and methods such as multi-resolution wavelet transform, Markov random field model and deep belief network, the system can effectively extract useful feature information from complex multimodal sensor data, greatly improving the performance of smart terminal devices. In short, through the above series of steps, the system can not only extract rich feature information from multimodal sensor data, but also effectively cope with challenges under various environmental conditions. Whether it is health monitoring or navigation applications, it can provide more reliable and accurate services, greatly improving user experience and service quality. This advanced technical solution not only solves many problems in traditional single sensor data processing, but also provides a solid foundation for the development of future smart terminal devices. For example, in virtual reality and augmented reality applications, this feature extraction technology can provide a more immersive experience, allowing users to experience a more realistic interactive environment. By continuously optimizing and improving these technologies, future smart terminal devices will be able to better serve people's daily life and work needs.

[0079] In a specific embodiment, performing error pattern recognition on the feature vector set to obtain the sensor error pattern includes:

[0080] Processing the feature vector set by a bidirectional long short-term memory network to obtain a time series feature sequence;

[0081] Based on the variational autoencoder, latent variables are extracted from the time series feature sequence to obtain a latent feature space, and Gaussian mixture clustering is performed on the latent feature space to obtain an error pattern candidate set;

[0082] Performing importance weight calculation on the error pattern candidate set to obtain weighted error features, and performing spectral clustering analysis on the weighted error features to obtain error subclass features;

[0083] Based on a preset dynamic Bayesian network, probability reasoning is performed on the error subclass characteristics to obtain an error transfer probability matrix, and maximum likelihood estimation is performed on the error transfer probability matrix to obtain an error state sequence; wherein the error state sequence includes a steady-state error, a transient error, and a periodic error;

[0084] The time-frequency features of the error state sequence are extracted to obtain an error feature spectrum, and error pattern recognition is performed based on the error feature spectrum to obtain a sensor error pattern; wherein the sensor error pattern includes temperature drift error, hysteresis error, nonlinear error and inter-axis crosstalk error.

[0085] Specifically, in order to accurately identify the error pattern in the sensor data in the intelligent terminal device, it is usually necessary to perform a series of complex processing steps on the feature vector set. Specifically, firstly, the feature vector set is processed by the bidirectional long short-term memory network (Bi-LSTM) to obtain the time series feature sequence; then, the time series feature sequence is extracted based on the variational autoencoder (VAE) to obtain the latent feature space, and Gaussian mixture clustering is performed on it to obtain the error pattern candidate set; then, the importance weight is calculated on the error pattern candidate set to obtain the weighted error feature, and the spectral clustering analysis is performed on it to obtain the error subclass feature; next, the error subclass feature is probabilistically inferred based on the preset dynamic Bayesian network (DBN) to obtain the error transfer probability matrix, and the maximum likelihood estimation is performed on it to obtain the error state sequence; finally, the error state sequence is subjected to time-frequency feature extraction to obtain the error feature spectrum, and the error pattern recognition is performed based on the error feature spectrum, and finally the sensor error pattern is obtained. First, the feature vector set is processed by the bidirectional long short-term memory network (Bi-LSTM) to obtain the time series feature sequence. Bi-LSTM is a neural network structure that can effectively capture the long-term dependencies of time series data, and is particularly suitable for processing sensor data with time series characteristics. For example, in the application scenario of smart watches, the accelerometer, gyroscope, and magnetometer data generated by users during running have significant time dependencies. By applying the Bi-LSTM model, the system can extract time series feature sequences containing time information from these feature vector sets. Specifically, Bi-LSTM processes time series data from both forward and backward directions to better capture the long-term dependencies and short-term fluctuation characteristics in the data. This can not only identify the user's movement pattern, but also discover potential sources of error, such as sensor drift or noise interference. Next, based on the variational autoencoder (VAE), the time series feature sequence is extracted with latent variables to obtain a latent feature space, and the space is clustered with Gaussian mixtures to obtain a candidate set of error patterns. As a generative model, VAE can maintain the probability distribution characteristics of the data while extracting the latent representation of the data. At this stage, the system inputs the time series feature sequence into the VAE, and through the latent variable extraction process, the original high-dimensional feature vector is mapped to a low-dimensional latent feature space. This latent feature space contains the core features and patterns of the data, making subsequent clustering analysis more efficient and accurate. For example, during a user's running, the latent feature space may reveal different types of motion patterns, such as stride cycle, arm swing frequency, etc. By performing Gaussian mixture clustering on the latent feature space, the system can cluster similar features together to form multiple error pattern candidate sets. These candidate sets represent possible error patterns and provide a basis for further error identification.Subsequently, the importance weights of the error pattern candidate sets are calculated to obtain weighted error features, which are then subjected to spectral clustering analysis to obtain error subclass features. Importance weight calculation aims to evaluate the importance of each error pattern candidate set in order to more effectively identify key error patterns. For example, in the application scenario of smart watches, some error patterns may have a greater impact on the user's health monitoring results, so they need to be given higher weights. By calculating weighted error features, the system can better distinguish different error patterns and further classify them into error subclass features. Spectral clustering is a clustering method based on graph theory that can effectively handle complex data structures. At this stage, the system performs spectral clustering analysis on the weighted error features, classifies similar error patterns into one category, and forms a more refined set of error subclass features. These error subclass features not only contain specific error patterns, but also reflect the intrinsic connections and interactions between them. Next, based on the preset dynamic Bayesian network (DBN), the error subclass features are probabilistically inferred to obtain the error transition probability matrix, which is then subjected to maximum likelihood estimation to obtain the error state sequence. DBN is a probabilistic graphical model used to model time series data and uncertainty relationships. By applying DBN, the system can derive the error transition probability matrix from the error subclass features, which describes the probability of transition between different error modes. For example, when the user is running, the sensor will experience multiple error states such as steady-state error, transient error and periodic error. Through maximum likelihood estimation, the system can extract the most likely error state sequence from the error transition probability matrix. This error state sequence not only includes various types of errors, but also reflects their evolution process in time, providing an important reference for subsequent error pattern recognition. Finally, the error state sequence is extracted by time-frequency features to obtain the error feature map, and error pattern recognition is performed based on the error feature map to finally obtain the sensor error pattern. Time-frequency feature extraction aims to extract feature maps containing time and frequency information from the error state sequence in order to better understand the dynamic characteristics of the error. For example, in the application scenario of smart watches, the error feature map may show temperature drift error, hysteresis error, nonlinear error and inter-axis crosstalk error in a specific time period. By analyzing the error feature map, the system can identify specific error patterns and take corresponding compensation measures. For example, if the accelerometer is detected to have temperature drift error, the sensor reading can be corrected by temperature correction algorithm; if the gyroscope is found to have nonlinear error, nonlinear compensation technology can be used to improve data accuracy. In the entire error pattern recognition process, each step is closely linked and works together on the feature vector set to gradually extract the most representative and discriminative error pattern.By combining a variety of advanced technologies and methods such as Bi-LSTM, VAE, spectral clustering and DBN, the system can effectively identify potential error patterns from complex multimodal sensor data, greatly improving the performance of smart terminal devices. Whether it is health monitoring or navigation applications, it can provide more reliable and accurate services, greatly improving user experience and service quality. This advanced technical solution not only solves many problems in traditional single sensor data processing, but also provides a solid foundation for the development of future smart terminal devices. For example, in virtual reality and augmented reality applications, this error pattern recognition technology can provide a more immersive experience, allowing users to experience a more realistic interactive environment. By continuously optimizing and improving these technologies, future smart terminal devices will be able to better serve people's daily life and work needs.

[0086] In a specific embodiment, performing error compensation on the multimodal sensor dataset based on the sensor error pattern to obtain a compensated multimodal sensor dataset includes:

[0087] Decomposing the multimodal sensor data set into error components based on the sensor error pattern to obtain a multimodal error component set;

[0088] Performing state estimation on the multimodal error component set through a preset Kalman filter to obtain an error state vector, and performing curve fitting on the error state vector to obtain an error fitting curve; wherein the error fitting curve is used to reflect the change trend of the error state vector over time;

[0089] Calculating the error accumulation amount of the multimodal error component set based on the error fitting curve to obtain the error accumulation component;

[0090] An error compensation matrix is ​​constructed based on the error accumulation component to obtain a preliminary error compensation matrix; and a nonlinear regression calculation is performed on the preliminary error compensation matrix by using a local weighted regression technique to obtain an error compensation coefficient;

[0091] A point-by-point compensation operation is performed on the multimodal sensor data set based on the error compensation coefficient to obtain a compensated multimodal sensor data set.

[0092] Specifically, in smart terminal devices, in order to accurately compensate for the errors in sensor data, it is usually necessary to perform a series of complex processing steps on the multimodal sensor data set based on the identified sensor error pattern. Specifically, first, the error component of the multimodal sensor data set is decomposed based on the sensor error pattern to obtain a multimodal error component set; then, the state of the multimodal error component set is estimated by a preset Kalman filter to obtain an error state vector, and the error state vector is curve fitted to obtain an error fitting curve; then, the error accumulation amount of the multimodal error component set is calculated based on the error fitting curve to obtain the error accumulation component; next, the error compensation matrix is ​​constructed based on the error accumulation component to obtain a preliminary error compensation matrix, and the preliminary error compensation matrix is ​​nonlinearly calculated by the local weighted regression technology to obtain the error compensation coefficient; finally, the multimodal sensor data set is compensated point by point based on the error compensation coefficient to obtain the compensated multimodal sensor data set. First, the error component of the multimodal sensor data set is decomposed based on the sensor error pattern to obtain a multimodal error component set. This process aims to subdivide the original multimodal sensor data set according to the identified error pattern and extract the error components of each sensor. For example, in the application scenario of smart watches, the accelerometer, gyroscope and magnetometer data generated by users during running may have various types of errors, such as temperature drift error, hysteresis error, nonlinear error and inter-axis crosstalk error. By decomposing these data into error components, the system can clarify the specific manifestation of each error and its scope of influence. This step not only helps with subsequent error compensation, but also provides users with more accurate data feedback. Next, the state of the multimodal error component set is estimated by the preset Kalman filter to obtain the error state vector, and the error state vector is curve fitted to obtain the error fitting curve. The Kalman filter is a recursive algorithm that can optimally estimate the state of the system in the presence of noise. At this stage, the system inputs the multimodal error component set into the Kalman filter and estimates the error state vector at each time point by iterative updating. For example, during the user's running, the accelerometer may generate high-frequency noise due to external vibration. The Kalman filter can effectively smooth these noises and obtain a more stable error state vector. Subsequently, the system will curve fit the error state vector to generate an error fitting curve that reflects the trend of error change over time. This curve not only describes the variation law of the error, but also provides a basis for the subsequent calculation of the error accumulation. Then, the error accumulation of the multimodal error component set is calculated based on the error fitting curve to obtain the error accumulation component. The error accumulation refers to the cumulative effect of the error over a period of time, which reflects the gradual accumulation of errors over time.For example, when a user runs for a long time, the accelerometer and gyroscope may have accumulated errors, resulting in inaccurate position and attitude estimation. Through the error fitting curve, the system can calculate the cumulative effect of these errors and obtain the error accumulation components. These accumulated components not only contain the size of the error, but also reflect its trend over time, which provides an important basis for subsequent error compensation. Next, the error compensation matrix is ​​constructed based on the error accumulation components to obtain a preliminary error compensation matrix, and the preliminary error compensation matrix is ​​nonlinearly regressed by the local weighted regression technology to obtain the error compensation coefficient. The error compensation matrix is ​​a key tool for correcting errors in sensor data. At this stage, the system constructs a preliminary error compensation matrix based on the error accumulation components, which contains the error information of each sensor at different time points. However, since the errors in practical applications are often nonlinear, the compensation matrix needs to be further optimized. Local weighted regression (LWR) is a regression method that can handle nonlinear relationships. It fits the model by weighted regression in the local area near a given sample point. By applying LWR technology, the system can perform nonlinear regression calculations on the preliminary error compensation matrix to obtain more accurate error compensation coefficients. These coefficients can better reflect the actual change of the error, thereby improving the compensation effect. Finally, the multimodal sensor data set is compensated point by point based on the error compensation coefficient to obtain the compensated multimodal sensor data set. Point-by-point compensation operation means that at each time point, the system will use the corresponding error compensation coefficient to correct the sensor data. For example, in the application scenario of smart watches, assuming that the user has certain specific error patterns during running, the system can adjust the data of the accelerometer, gyroscope and magnetometer point by point according to the identified error patterns and calculated error compensation coefficients. This can not only eliminate the known sources of error, but also significantly improve the accuracy and reliability of the data. After the point-by-point compensation operation, the multimodal sensor data set finally obtained can more realistically reflect the actual movement state of the user. In the entire error compensation process, each step is closely linked and acts on the feature vector set together to gradually extract the most representative and discriminative error compensation scheme. By combining a variety of advanced technologies and methods such as Kalman filter, error fitting curve, and local weighted regression, the system can effectively identify and compensate errors from complex multimodal sensor data, greatly improving the performance of smart terminal devices. Whether it is health monitoring or navigation applications, it can provide more reliable and accurate services, greatly improving user experience and service quality. This advanced technical solution not only solves many problems in traditional single sensor data processing, but also provides a solid foundation for the development of future smart terminal devices. For example, in virtual reality and augmented reality applications, this error compensation technology can provide a more immersive experience, allowing users to experience a more realistic interactive environment.By continuously optimizing and improving these technologies, future smart terminal devices will be able to better serve people's daily life and work needs.

[0093] In a specific embodiment, performing point-by-point compensation operation on the multimodal sensor data set based on the error compensation coefficient to obtain a compensated multimodal sensor data set includes:

[0094] Performing Gaussian filtering on the error compensation coefficient to obtain a smoothed error compensation coefficient;

[0095] Preliminarily compensating the multimodal sensor data set by using a smoothing error compensation coefficient to obtain a preliminary compensated data set;

[0096] Performing wavelet transform processing on the preliminary compensation data set to obtain a multi-resolution compensation data set, and performing weighted calculation and fusion on the multi-resolution compensation data set through a preset attention mechanism to obtain a fused compensation data set; wherein the multi-resolution compensation data set includes an approximate component and a detail component;

[0097] A point-by-point compensation operation is performed on the multimodal sensor data set based on the fused compensation data set to obtain a compensated multimodal sensor data set.

[0098] Specifically, in smart terminal devices, in order to further improve the accuracy and reliability of sensor data, the process of performing point-by-point compensation operations on multimodal sensor data sets based on error compensation coefficients is crucial. Specifically, this process includes Gaussian filtering the error compensation coefficients to obtain smoothed error compensation coefficients; performing preliminary compensation on the multimodal sensor data set by smoothing the error compensation coefficients to obtain a preliminary compensation data set; performing wavelet transform on the preliminary compensation data set to obtain a multi-resolution compensation data set, and performing weighted calculation and fusion on the multi-resolution compensation data set through a preset attention mechanism to obtain a fused compensation data set; finally, performing point-by-point compensation operations on the multimodal sensor data set based on the fused compensation data set to obtain a compensated multimodal sensor data set. First, the error compensation coefficients are Gaussian filtered to obtain smoothed error compensation coefficients. Gaussian filtering is a commonly used image processing technique for removing noise and smoothing data. At this stage, the system inputs the error compensation coefficients into the Gaussian filter and smoothes them through convolution operations. For example, in the application scenario of smart watches, the error compensation coefficients generated by users during running may contain some high-frequency noise or local fluctuations. Through Gaussian filtering, the system can effectively remove these noises, making the error compensation coefficient smoother and more stable. This can not only reduce the error amplification effect in the compensation process, but also improve the accuracy of subsequent compensation steps. Next, the multimodal sensor data set is preliminarily compensated by smoothing the error compensation coefficient to obtain a preliminary compensation data set. At this stage, the system uses the smoothed error compensation coefficient after Gaussian filtering to correct the original multimodal sensor data set point by point. For example, during the user's running, assuming that the accelerometer data has deviations due to external vibrations, the system can correct these deviations by applying the smoothed error compensation coefficient to generate a preliminary compensation data set. This preliminary compensation data set not only contains the corrected sensor readings, but also reflects the preliminary correction effect of the system on the original data. This step provides basic data for subsequent wavelet transform processing. Then, the preliminary compensation data set is processed by wavelet transform to obtain a multi-resolution compensation data set, and the multi-resolution compensation data set is weighted and fused through the preset attention mechanism to obtain a fused compensation data set. Wavelet transform is a signal processing method that can decompose data into approximate components and detail components of different scales, thereby realizing multi-resolution analysis. At this stage, the system will perform wavelet transform on the preliminary compensation data set to extract information in different frequency bands. For example, when a user is running, there may be low-frequency trends and high-frequency details in the preliminary compensation data set. Through wavelet transform, the system can decompose it into approximate components (reflecting low-frequency trends) and detail components (reflecting high-frequency changes). Subsequently, the system will use the preset attention mechanism to perform weighted calculations and fusion on these multi-resolution compensation data sets.The attention mechanism can dynamically adjust the importance weights of data in different frequency bands to ensure that the final fusion result can better reflect the actual motion state. For example, if the high-frequency details in a certain time period are very important for the user's stride cycle recognition, the system will give it a higher weight, thereby increasing the contribution of this part of the data in the final fusion result. Next, the multimodal sensor dataset is compensated point by point based on the fused compensation dataset to obtain the compensated multimodal sensor dataset. In this step, the system uses the fused compensation dataset processed by wavelet transform and attention mechanism to correct the original multimodal sensor dataset point by point. For example, during the user's running process, it is assumed that after preliminary compensation and wavelet transform processing, the system obtains a fused compensation dataset containing optimized approximate components and detail components. The system will apply these optimized data point by point to the original accelerometer, gyroscope and magnetometer data to generate the final compensated multimodal sensor dataset. This not only eliminates known sources of error, but also significantly improves the accuracy and reliability of the data. In the entire error compensation process, each step is closely linked and acts on the feature vector set together to gradually extract the most representative and discriminative error compensation scheme. By combining a variety of advanced technologies and methods such as Gaussian filtering, wavelet transform and attention mechanism, the system can effectively identify and compensate errors from complex multimodal sensor data, greatly improving the performance of smart terminal devices. Whether it is health monitoring or navigation applications, it can provide more reliable and accurate services, greatly improving user experience and service quality. This advanced technical solution not only solves many problems in traditional single sensor data processing, but also provides a solid foundation for the development of future smart terminal devices. For example, in virtual reality and augmented reality applications, this error compensation technology can provide a more immersive experience, allowing users to experience a more realistic interactive environment. By continuously optimizing and improving these technologies, future smart terminal devices will be able to better serve people's daily life and work needs. Especially in wearable devices such as smart watches, accurate motion tracking can not only help users better understand their health status, but also provide more accurate navigation services and improve the overall user experience. For example, when running outdoors, the sensor data after error compensation can more accurately record the user's motion trajectory, helping them better plan routes and evaluate exercise effects. In short, through this series of complex error compensation steps, smart terminal devices can perform well in various application scenarios and meet users' needs for efficient and reliable applications.

[0099] In a specific embodiment, the data reconstruction based on the compensated multimodal sensor data set to obtain the reconstructed motion trajectory includes:

[0100] Performing attitude calculation on the compensated multimodal sensor data set to obtain an attitude quaternion sequence; and performing position integration on the compensated multimodal sensor data set based on the attitude quaternion sequence to obtain a preliminary motion trajectory;

[0101] Segmenting the preliminary motion trajectory to obtain motion trajectory segments, and performing Hausdorff distance calculation on the motion trajectory segments to obtain a trajectory segment similarity matrix;

[0102] Performing cluster analysis on the trajectory segment similarity matrix by using a spectral clustering algorithm to obtain trajectory clustering clusters, and performing trajectory fusion on the trajectory clustering clusters to obtain fused motion trajectory segments;

[0103] performing map matching based on the fused motion trajectory segments to obtain a motion trajectory after map matching, and performing topological correction on the motion trajectory after map matching to obtain a motion trajectory after topological correction;

[0104] The topology-corrected motion trajectory is subjected to cubic spline interpolation to obtain a reconstructed motion trajectory.

[0105] Specifically, in smart terminal devices, in order to reconstruct an accurate motion trajectory from a compensated multimodal sensor data set, a series of complex processing steps are usually required. Specifically, firstly, the compensated multimodal sensor data set is subjected to attitude solution to obtain an attitude quaternion sequence, and the position of the compensated multimodal sensor data set is integrated based on the attitude quaternion sequence to obtain a preliminary motion trajectory; then, the preliminary motion trajectory is segmented to obtain motion trajectory fragments, and the Hausdorff distance calculation is performed on the motion trajectory fragments to obtain a trajectory fragment similarity matrix; then, the trajectory fragment similarity matrix is ​​clustered by a spectral clustering algorithm to obtain trajectory clustering clusters, and the trajectory clustering clusters are subjected to trajectory fusion to obtain fused motion trajectory fragments; next, map matching is performed based on the fused motion trajectory fragments to obtain a motion trajectory after map matching, and topological correction is performed on the motion trajectory after map matching to obtain a motion trajectory after topological correction; finally, cubic spline interpolation is performed on the topologically corrected motion trajectory to obtain a reconstructed motion trajectory. First, the compensated multimodal sensor data set is posture solved to obtain a posture quaternion sequence, and the compensated multimodal sensor data set is position integrated based on the posture quaternion sequence to obtain a preliminary motion trajectory. Posture solution refers to the use of gyroscope, accelerometer and magnetometer data to calculate the device's posture change. In this stage, the system uses the compensated sensor data to calculate the device's posture quaternion sequence through a posture solution algorithm (such as a Madgwick filter or an extended Kalman filter). For example, in the application scenario of a smart watch, the accelerometer, gyroscope and magnetometer data generated by the user during running are error compensated, and the system can accurately calculate the user's posture change. These posture quaternions describe the rotation state of the device in three-dimensional space. Based on these posture quaternions, the system can calculate a preliminary motion trajectory through position integration. For example, by converting accelerometer data into linear velocity and further integrating to obtain position information, the system can generate a preliminary motion trajectory that reflects the user's approximate movement path. Next, the preliminary motion trajectory is segmented to obtain motion trajectory segments, and the Hausdorff distance calculation is performed on the motion trajectory segments to obtain a trajectory segment similarity matrix. Trajectory segmentation refers to dividing a continuous motion trajectory into multiple shorter segments for subsequent analysis and processing. For example, when a user is running, the system can divide the entire motion trajectory into several stride cycles or specific action segments. By calculating the Hausdorff distance of these trajectory segments, the system can quantify the similarity between different trajectory segments. The Hausdorff distance is a metric used to measure the distance between two sets, which can capture the overall shape differences of the trajectory segments.For example, during running, some stride cycles may have high similarity, while other stride cycles may be different due to external interference or changes in user behavior. By calculating the Hausdorff distance between trajectory segments, the system can construct a trajectory segment similarity matrix, which reflects the similarity between each trajectory segment. Then, the trajectory segment similarity matrix is ​​clustered by the spectral clustering algorithm to obtain trajectory clusters, and the trajectory clusters are fused to obtain the fused motion trajectory segments. Spectral clustering is a clustering method based on graph theory that can effectively handle complex data structures. At this stage, the system inputs the trajectory segment similarity matrix into the spectral clustering algorithm, and classifies similar trajectory segments into one category through feature decomposition and clustering analysis. For example, during the user's running, the system may find that some stride cycles have highly similar trajectory patterns, and these trajectory segments will be clustered together to form a trajectory cluster. Subsequently, the system will perform trajectory fusion on these trajectory clusters to generate smoother and more consistent motion trajectory segments. This trajectory fusion can not only reduce the impact of noise, but also improve the overall coherence and accuracy of the trajectory. Next, map matching is performed based on the fused motion trajectory fragments to obtain the map-matched motion trajectory, and the map-matched motion trajectory is topologically corrected to obtain the topologically corrected motion trajectory. Map matching refers to comparing the motion trajectory of the device with known map data to improve positioning accuracy. At this stage, the system matches the fused motion trajectory fragments with the map data in the geographic information system (GIS). For example, while the user is running, the system can compare the trajectory recorded by the device with the actual road network to ensure that the trajectory points fall on a reasonable road. Map matching can not only correct the deviation in the trajectory, but also fill in the missing data points. Subsequently, the system will perform topological correction on the motion trajectory after map matching. Topological correction refers to adjusting the topological structure in the trajectory to make it conform to the actual geographical environment. For example, if the user's trajectory shows that it passes through a building, the system will adjust the trajectory according to the map data so that it walks along the road around the building. Finally, cubic spline interpolation is performed on the topologically corrected motion trajectory to obtain a reconstructed motion trajectory. Cubic spline interpolation is a commonly used curve fitting method that can generate smooth and continuous curves. At this stage, the system will perform cubic spline interpolation on the topologically corrected motion trajectory to generate the final reconstructed motion trajectory. For example, when the user is running, after error compensation, posture solution, trajectory segmentation, cluster analysis, map matching and topological correction, the system obtains a relatively accurate motion trajectory but may have some discontinuities. Through cubic spline interpolation, the system can generate a smooth and continuous motion trajectory that better reflects the user's actual motion path.This reconstructed motion trajectory not only contains the user's specific motion trajectory, but also provides more detailed information, such as stride cycle, cadence, etc., thus providing more reliable data support for health monitoring and navigation applications. In the entire data reconstruction process, each step is closely linked and works together on the compensated multimodal sensor data set to gradually extract the most representative and discriminative reconstructed motion trajectory. By combining a variety of advanced technologies and methods such as posture solution, Hausdorff distance calculation, spectral clustering, map matching and cubic spline interpolation, the system can effectively reconstruct accurate motion trajectories from complex multimodal sensor data, greatly improving the performance of smart terminal devices. Whether it is health monitoring or navigation applications, it can provide more reliable and accurate services, greatly improving user experience and service quality. This advanced technical solution not only solves many problems in traditional single sensor data processing, but also provides a solid foundation for the development of future smart terminal devices. For example, when running outdoors, the reconstructed motion trajectory can more accurately record the user's motion path, helping them better plan routes and evaluate exercise effects. In short, through this series of complex steps, smart terminal devices can perform well in various application scenarios and meet users' needs for efficient and reliable applications.

[0106] The above describes the dynamic calibration method for a motion sensor for a smart terminal device in an embodiment of the present invention. The following describes the dynamic calibration device for a motion sensor for a smart terminal device in an embodiment of the present invention. Figure 2 In one embodiment of the present invention, a dynamic calibration device for a motion sensor for a smart terminal device includes:

[0107] The acquisition module 21 is used to acquire multi-source sensors on the target intelligent terminal device in real time through multi-sensor fusion technology to obtain a multi-modal sensor data set; wherein the multi-source sensors include accelerometers, gyroscopes, and magnetometers;

[0108] An extraction module 22, configured to perform feature extraction on the multimodal sensor data set to obtain a feature vector set;

[0109] An identification module 23, used for performing error pattern identification on the feature vector set to obtain a sensor error pattern;

[0110] A compensation module 24, configured to perform error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set;

[0111] The reconstruction module 25 is used to perform data reconstruction based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory.

[0112] In this embodiment, for the specific implementation of each unit in the above device embodiment, please refer to the above method embodiment, which will not be repeated here.

[0113] Reference Figure 3 The present invention also provides a computer device in an embodiment, wherein the internal structure of the computer device can be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a display screen, an input device, a network interface and a database connected through a system bus. Among them, the processor designed by the computer is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the corresponding data in this embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0114] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied.

[0115] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It can be understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0116] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided by the present invention and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM.

[0117] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0118] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A dynamic calibration method for a motion sensor for an intelligent terminal device, characterized in that: The following steps are involved: The multi-source sensors on the target intelligent terminal device are collected in real time through multi-sensor fusion technology to obtain a multi-modal sensor data set; wherein the multi-source sensors include accelerometers, gyroscopes, and magnetometers; Performing feature extraction on the multimodal sensor data set to obtain a feature vector set; Performing error pattern recognition on the feature vector set to obtain a sensor error pattern; Performing error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set; Reconstructing data based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory; The performing error pattern recognition on the feature vector set to obtain the sensor error pattern includes: Processing the feature vector set by a bidirectional long short-term memory network to obtain a time series feature sequence; Based on the variational autoencoder, latent variables are extracted from the time series feature sequence to obtain a latent feature space, and Gaussian mixture clustering is performed on the latent feature space to obtain an error pattern candidate set; Performing importance weight calculation on the error pattern candidate set to obtain weighted error features, and performing spectral clustering analysis on the weighted error features to obtain error subclass features; Based on a preset dynamic Bayesian network, probability reasoning is performed on the error subclass characteristics to obtain an error transfer probability matrix, and maximum likelihood estimation is performed on the error transfer probability matrix to obtain an error state sequence; wherein the error state sequence includes a steady-state error, a transient error, and a periodic error; The time-frequency features of the error state sequence are extracted to obtain an error feature spectrum, and error pattern recognition is performed based on the error feature spectrum to obtain a sensor error pattern; wherein the sensor error pattern includes temperature drift error, hysteresis error, nonlinear error and inter-axis crosstalk error.

2. The dynamic calibration method for a motion sensor for an intelligent terminal device according to claim 1, characterized in that: The multi-source sensors on the target intelligent terminal device are collected in real time by multi-sensor fusion technology to obtain a multi-modal sensor data set, including: Using multi-sensor fusion technology to collect data from multiple sensors on the target intelligent terminal device in real time to obtain preliminary collected data; Using a preset three-dimensional spatial timing synchronization network, timing aligning the preliminary collected data to obtain an aligned sensor data stream, and performing coordinate transformation on the aligned sensor data stream to obtain a global coordinate system data stream; Based on a deep convolutional neural network, multimodal fusion is performed on the global coordinate system data stream to obtain a multimodal sensor data set; wherein the deep convolutional neural network uses 3 convolutional layers, the convolution kernel sizes of the 3 convolutional layers are 5×5, 3×3, and 3×3 respectively, the number of convolution kernels in each of the convolutional layers is 64, 128, and 256 respectively, and the step size of the convolution kernel is 1.

3. The dynamic calibration method for a motion sensor for an intelligent terminal device according to claim 1, characterized in that: The extracting features from the multimodal sensor data set to obtain a feature vector set includes: Decomposing the multimodal sensor data set by multi-resolution wavelet transform to obtain sub-band signals at different scales, and performing time-frequency domain analysis on each sub-band signal to obtain a multi-scale feature vector set; wherein the multi-resolution wavelet transform adopts Daubechies wavelet function, the sampling frequency is 256 Hz, and the number of decomposition layers is 5; Based on the Markov random field model, spatial dependency modeling is performed on the multi-scale feature vector set to obtain a spatial dependency matrix, and Laplace matrix decomposition is performed on the spatial dependency matrix to obtain a feature subspace feature vector set; Performing local structure analysis on the feature subspace feature vector set to obtain a local structure feature vector set, and performing high-dimensional feature mapping on the local structure feature vector set based on a deep belief network to obtain a feature vector set.

4. The dynamic calibration method for a motion sensor for an intelligent terminal device according to claim 1, characterized in that: The performing error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set includes: Decomposing the multimodal sensor data set into error components based on the sensor error pattern to obtain a multimodal error component set; Performing state estimation on the multimodal error component set through a preset Kalman filter to obtain an error state vector, and performing curve fitting on the error state vector to obtain an error fitting curve; wherein the error fitting curve is used to reflect the change trend of the error state vector over time; Calculating the error accumulation amount of the multimodal error component set based on the error fitting curve to obtain the error accumulation component; An error compensation matrix is ​​constructed based on the error accumulation component to obtain a preliminary error compensation matrix; and a nonlinear regression calculation is performed on the preliminary error compensation matrix by using a local weighted regression technique to obtain an error compensation coefficient; A point-by-point compensation operation is performed on the multimodal sensor data set based on the error compensation coefficient to obtain a compensated multimodal sensor data set.

5. The dynamic calibration method for a motion sensor for an intelligent terminal device according to claim 4, characterized in that: The performing point-by-point compensation operation on the multimodal sensor data set based on the error compensation coefficient to obtain a compensated multimodal sensor data set includes: Performing Gaussian filtering on the error compensation coefficient to obtain a smoothed error compensation coefficient; Preliminarily compensating the multimodal sensor data set by using a smoothing error compensation coefficient to obtain a preliminary compensated data set; Performing wavelet transform processing on the preliminary compensation data set to obtain a multi-resolution compensation data set, and performing weighted calculation and fusion on the multi-resolution compensation data set through a preset attention mechanism to obtain a fused compensation data set; wherein the multi-resolution compensation data set includes an approximate component and a detail component; A point-by-point compensation operation is performed on the multimodal sensor data set based on the fused compensation data set to obtain a compensated multimodal sensor data set.

6. The dynamic calibration method for a motion sensor for an intelligent terminal device according to claim 1, characterized in that: The data reconstruction based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory includes: Performing attitude calculation on the compensated multimodal sensor data set to obtain an attitude quaternion sequence; and performing position integration on the compensated multimodal sensor data set based on the attitude quaternion sequence to obtain a preliminary motion trajectory; Segmenting the preliminary motion trajectory to obtain motion trajectory segments, and performing Hausdorff distance calculation on the motion trajectory segments to obtain a trajectory segment similarity matrix; Performing cluster analysis on the trajectory segment similarity matrix by using a spectral clustering algorithm to obtain trajectory clustering clusters, and performing trajectory fusion on the trajectory clustering clusters to obtain fused motion trajectory segments; performing map matching based on the fused motion trajectory segments to obtain a motion trajectory after map matching, and performing topological correction on the motion trajectory after map matching to obtain a motion trajectory after topological correction; The topology-corrected motion trajectory is subjected to cubic spline interpolation to obtain a reconstructed motion trajectory.

7. A dynamic calibration device for a motion sensor for an intelligent terminal device, characterized in that: include: The acquisition module is used to acquire multi-source sensors on the target intelligent terminal device in real time through multi-sensor fusion technology to obtain a multi-modal sensor data set; wherein the multi-source sensors include accelerometers, gyroscopes, and magnetometers; An extraction module, used for performing feature extraction on the multimodal sensor data set to obtain a feature vector set; An identification module, used for performing error pattern identification on the feature vector set to obtain a sensor error pattern; A compensation module, configured to perform error compensation on the multimodal sensor data set based on the sensor error pattern to obtain a compensated multimodal sensor data set; A reconstruction module, used for performing data reconstruction based on the compensated multimodal sensor data set to obtain a reconstructed motion trajectory; The performing error pattern recognition on the feature vector set to obtain the sensor error pattern includes: Processing the feature vector set by a bidirectional long short-term memory network to obtain a time series feature sequence; Based on the variational autoencoder, latent variables are extracted from the time series feature sequence to obtain a latent feature space, and Gaussian mixture clustering is performed on the latent feature space to obtain an error pattern candidate set; Performing importance weight calculation on the error pattern candidate set to obtain weighted error features, and performing spectral clustering analysis on the weighted error features to obtain error subclass features; Based on a preset dynamic Bayesian network, probability reasoning is performed on the error subclass characteristics to obtain an error transfer probability matrix, and maximum likelihood estimation is performed on the error transfer probability matrix to obtain an error state sequence; wherein the error state sequence includes a steady-state error, a transient error, and a periodic error; The time-frequency features of the error state sequence are extracted to obtain an error feature spectrum, and error pattern recognition is performed based on the error feature spectrum to obtain a sensor error pattern; wherein the sensor error pattern includes temperature drift error, hysteresis error, nonlinear error and inter-axis crosstalk error.

8. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images