Multi-modal adaptive step counting method and system based on deep learning

Abstract

The invention relates to the technical field of multi-modal self-adaptive step counting methods, in particular to a multi-modal self-adaptive step counting method and system based on deep learning, and the method comprises the following steps: obtaining multi-source sensor data; based on multi-source sensor data, multi-modal features are extracted through a multi-resolution feature extraction network; weighting the multi-modal features by using a quantum inspired dynamic attention mechanism to obtain weighted features; inputting the weighted features into an adaptive multi-scale fusion network to obtain fusion features; on the basis of the fusion features, performing self-supervised pre-training, transfer learning adaptation and online learning in sequence to obtain optimized features; performing multi-scale consistency check on the optimized features to obtain a final step counting result; through multi-modal data fusion and an innovative deep learning architecture, extremely high step counting accuracy (98.7%) can be kept in various complex environments; different wearing positions and use scenes can be quickly adapted, and the adaptability score is as high as 96.2%.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal adaptive pedometer methods, and in particular to a multimodal adaptive pedometer method and system based on deep learning. Background Art

[0002] With the rapid development of wearable device technology, pedometer function has become a basic configuration of smart watches, smart bracelets and other products. Accurate pedometer is not only the basis for monitoring users' daily activities, but also an important basis for evaluating health status and formulating exercise plans. However, existing pedometer technology faces many challenges and is difficult to meet the growing needs of users.

[0003] Early pedometers mainly relied on a single acceleration sensor and used a fixed threshold algorithm to identify the number of steps. This method is simple and direct, but it is prone to misjudgment in complex environments, especially in atypical walking patterns such as walking up and down stairs and jogging, where the accuracy of step counting drops significantly. For example, the accuracy of the traditional threshold-based algorithm can reach 90% when walking on flat ground, but the accuracy drops to less than 70% when climbing stairs.

[0004] With the advancement of technology, researchers have begun to try to fuse multiple sensor data to improve the accuracy of step counting. Some scholars have proposed a method that combines accelerometers and gyroscopes to improve the stability of step counting to a certain extent. However, this simple data fusion method is still difficult to adapt to the gait characteristics of different users and the diverse wearing positions. Experiments show that when the device moves from the wrist to the pocket, the accuracy of such algorithms drops by an average of 15%.

[0005] In recent years, deep learning technology has been widely used in the field of pedometers. Some scholars have proposed a hybrid model based on convolutional neural networks (CNN) and long short-term memory networks (LSTM), which has achieved good results under laboratory conditions. However, such models often require a large amount of labeled data for training and are difficult to adapt to various complex situations in actual use. Studies have pointed out that when facing new users or new wearing positions, existing deep learning models need to re-collect a large amount of data and perform fine-tuning, which is extremely inconvenient in practical applications.

[0006] In summary, the existing pedometer technology mainly has the following problems:

[0007] 1. Single sensors and simple algorithms are difficult to cope with complex motion patterns and environmental changes.

[0008] 2. The multi-sensor fusion method lacks an effective adaptive mechanism and is difficult to handle different wearing positions and user habits.

[0009] 3. Existing deep learning models have insufficient generalization capabilities and are unable to quickly adapt to new users and new scenarios.

[0010] 4. Lack of effective personalization mechanism, unable to optimize for the unique gait characteristics of different users.

[0011] 5. High-precision algorithms often bring a large computational burden, affecting the battery life of the device.

[0012] These problems seriously restrict the practicality and user experience of the pedometer function, and an innovative method that can comprehensively solve the above problems is urgently needed. Summary of the invention

[0013] The multimodal adaptive pedometer method based on deep learning proposed in this invention is designed to address the shortcomings of the existing technology. This method integrates multi-source sensor data, adopts an innovative multi-resolution feature extraction network and an adaptive multi-scale fusion network, and introduces a quantum-inspired dynamic attention mechanism to achieve accurate recognition of complex motion patterns. At the same time, by comparing multi-task learning pre-training, progressive domain adaptation, and meta-learning enhanced online learning, the present invention greatly improves the generalization ability and personalization level of the model.

[0014] The present invention provides a multi-modal adaptive step counting method based on deep learning, comprising the following steps:

[0015] Obtain multi-source sensor data;

[0016] Based on the multi-source sensor data, extracting multimodal features through a multi-resolution feature extraction network;

[0017] Using a quantum-inspired dynamic attention mechanism to weight the multimodal features to obtain weighted features;

[0018] Inputting the weighted features into an adaptive multi-scale fusion network to obtain fusion features;

[0019] Based on the fusion features, self-supervised pre-training, transfer learning adaptation and online learning are sequentially performed to obtain optimized features;

[0020] A multi-scale consistency check is performed on the optimized features to obtain the final step counting result.

[0021] Specifically, the multi-source sensor data includes accelerometer data, gyroscope data and barometer data, wherein:

[0022] The accelerometer data and gyroscope data are collected by the BMI160 sensor with a sampling rate of 100 Hz;

[0023] The barometer data is collected by a BMP388 sensor with a sampling rate of 10 Hz.

[0024] Specifically, the multi-resolution feature extraction network includes:

[0025] Temporal convolution module, whose output feature F_t is calculated by the following formula:

[0026] F t =ReLU(BN(Conv1D(X,k t ,s t )))

[0027] Where X is the input sequence, k t is the convolution kernel size, s t is the step size, BN is batch normalization, and ReLU is the activation function;

[0028] Frequency convolution module, whose output feature F f Calculated by the following formula:

[0029] F f =ReLU(BN(Conv1D(FFT(X),k f ,s f )))

[0030] Among them, FFT is fast Fourier transform; t and F f Connect to get multi-resolution features F mr .

[0031] Specifically, the quantum-inspired dynamic attention mechanism includes the following steps:

[0032] Initialize quantum state |ψ>=∑α i |i|, where α i is the complex amplitude;

[0033] Apply the quantum gate operation U|ψ〉=exp(-iHt)|ψ〉, where H is the Hamiltonian operator;

[0034] Measure the attention weight w i =|<i|U|ψ> | 2 ;

[0035] Weighting the features: F att =∑w i ·F i .

[0036] Specifically, the adaptive multi-scale fusion network includes:

[0037] The fractal dimension feature pyramid is constructed in the following process:

[0038] F0=F att

[0039] F i =FractalConv(Pool(F i-1 ,2),d i )

[0040] Among them, FractalConv is a fractal convolution operation, defined as:

[0041] FractalConv(X,d)=∑w j ·X (d / 2) ·|XS j (X)| ((2-d) / 2)

[0042] Among them, S j is the scaling operator, w j is the weight, d is the fractal dimension;

[0043] Adaptive feature fusion module, its algorithm is:

[0044] Calculate importance score: s i =SoftPlus(MLP(GlobalAvgPool(F i )))

[0045] Normalized score:

[0046] Fusion features: F fused =∑α i ·F i ;

[0047] Topological adversarial feature learning, the algorithm is:

[0048] Generate topological perturbations:

[0049] Constructing adversarial examples: F a dv=F fused +δ

[0050] Adversarial Training: L a dv=E[log(D(F fused ))]+E[log(1-D(F adv ))].

[0051] Specifically, the self-supervised pre-training adopts contrastive multi-task learning, and its algorithm includes:

[0052] Define the auxiliary task set T = {motion type recognition, step frequency estimation, energy consumption prediction} and construct the contrast loss for each task:

[0053]

[0054] Among them, sin is the cosine similarity, τ is the temperature parameter;

[0055] Multi-task joint optimization:

[0056] L cmtl =∑λ i ·(L ti +γ·L ci )

[0057] Among them, L ti is the supervision loss for task i, λ i and γ are weight coefficients.

[0058] Specifically, the transfer learning adaptation adopts progressive domain adaptation, and its algorithm includes:

[0059] Build domain classifier: D(F) = Softmax(MLP(F));

[0060] Domain adversarial loss: L d a=-∑y d log(D(F)), where y d is the domain label;

[0061] Introducing a progressive factor: Where t is the current iteration number and γ is the adjustment factor;

[0062] Update the feature extractor parameters: Where L task is the main task loss, and η is the learning rate.

[0063] Specifically, the online learning adopts meta-learning enhanced online learning, and its algorithm includes:

[0064] Define the meta-learning objective: Where D i and D j For different small batches of data;

[0065] Online update rules: Where β is the meta-learning rate;

[0066] Personalized adaptation: For each user u, Where D u For user-specific data.

[0067] Specifically, the algorithm of the multi-scale consistency check includes:

[0068] Count steps at different time scales:

[0069]

[0070] Calculate the consistency score:

[0071] To apply adaptive smoothing:

[0072] S s mooth=α·S+(1-α)·S prev , where α = sigmoid(C), S prev Estimated number of steps at the previous moment;

[0073] Output the final number of steps: number of steps = round(S smooth ), confidence level = C.

[0074] A multi-modal adaptive pedometer system based on deep learning for implementing the method described above comprises:

[0075] A multi-source sensor module, used to obtain multi-source sensor data;

[0076] A multi-resolution feature extraction module, used for extracting multi-modal features based on the multi-source sensor data;

[0077] A quantum-inspired dynamic attention module for weighting the multimodal features;

[0078] Adaptive multi-scale fusion module, used to process weighted features and obtain fused features;

[0079] A learning optimization module, used for performing self-supervised pre-training, transfer learning adaptation and online learning based on the fusion features;

[0080] The multi-scale consistency verification module is used to verify the optimized features and obtain the final step counting results.

[0081] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0082] 1. High accuracy: Through multimodal data fusion and innovative deep learning architecture, the present invention can maintain extremely high step counting accuracy (98.7%) in various complex environments, far exceeding the existing technical level.

[0083] 2. Strong adaptability: Thanks to the quantum-inspired dynamic attention mechanism and progressive domain adaptation algorithm, the present invention can quickly adapt to different wearing positions and usage scenarios, with an adaptability score of up to 96.2%.

[0084] 3. Personalized optimization: Using a meta-learning enhanced online learning method, the present invention can continuously optimize the unique gait characteristics of each user, and the degree of personalization is improved by 12.3%.

[0085] 4. Energy efficiency balance: Despite the introduction of a complex algorithm, the present invention controls the additional battery consumption within an acceptable range (only 5.2% more than the simple algorithm) through optimized design, achieving a good balance between performance and energy efficiency.

[0086] 5. Strong scalability: The modular design of the present invention makes it easy to integrate new sensors or expand to other motion recognition tasks, providing convenience for future functional upgrades.

[0087] In summary, the present invention not only solves the main problems faced by existing pedometer technology, but also achieves significant breakthroughs in accuracy, adaptability and personalization, providing a comprehensive and high-performance pedometer solution for wearable devices. This innovation is expected to significantly improve user experience and promote the further development of smart wearable devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is the overall block diagram of the system of the present invention.

[0089] Figure 2 It is a logic block diagram of the multi-source sensor module of the present invention.

[0090] Figure 3 This is a logic block diagram of the multi-resolution feature extraction module of the present invention.

[0091] Figure 4 This is the logical block diagram of the quantum-inspired dynamic attention module of the present invention.

[0092] Figure 5 This is a logic block diagram of the adaptive multi-scale fusion module of the present invention.

[0093] Figure 6 This is a logic block diagram of the learning optimization module of the present invention.

[0094] Figure 7 This is a logic block diagram of the multi-scale consistency verification module of the present invention.

[0095] Figure 8 This is a logic block diagram of the adaptive feature fusion module of the present invention.

[0096] Fig. 9 This is a logic diagram of topological adversarial feature learning of the present invention.

[0097] Fig.10 This is a logic block diagram of the comparative multi-task learning pre-training of the present invention.

[0098] Fig.11 This is a logic block diagram of the progressive domain adaptation of the present invention.

[0099] Fig.12 It is a performance radar chart of the present invention and comparative examples 1-3.

[0100] Fig.13 It is a stacked bar chart of the performance of the present invention and comparative examples 1-3.

[0101] Fig.14 This is a schematic diagram of the framework of the adaptive multi-scale fusion module of the present invention.

[0102] Fig.15 This is a schematic diagram of the framework of the learning optimization module of the present invention. DETAILED DESCRIPTION

[0103] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0104] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0105] The present invention relates to a multimodal adaptive pedometer method and system based on deep learning, which is particularly suitable for the pedometer function in wearable devices such as smart watches and smart bracelets. It can adapt to the gait characteristics of different users, diverse wearing positions, and complex and changeable sports environments, thereby significantly improving the accuracy and reliability of pedometer counting.

[0106] The technical solution of the present invention is described in detail below in conjunction with specific embodiments.

[0107] Please refer to Figure 1-15 , a multimodal adaptive step counting method based on deep learning, comprising the following steps:

[0108] 1. Obtain multi-source sensor data. In daily use scenarios, users wear smart watches or smart bracelets to perform various activities, such as walking, running, or daily work. The system of the present invention collects the user's motion data in real time through multiple sensors on the device. Specifically, the system uses the following sensors:

[0109] a) Accelerometer (model: BMI160): collects three-axis acceleration data (ax, ay, az) with a sampling rate of 100 Hz. These data reflect the acceleration changes of the device in three dimensions and can capture the movement characteristics of the user's arm or body.

[0110] b) Gyroscope (model: BMI160): collects three-axis angular velocity data (ωx, ωy, ωz) with a sampling rate of 100Hz. Angular velocity data can reflect the rotational movement of the device and help identify complex movement patterns such as wrist flips or arm swings.

[0111] c) Barometer (model: BMP388): collects atmospheric pressure data (p) at a sampling rate of 10Hz. The barometric pressure data can help the system identify whether the user is climbing stairs or has vertical displacement, and improve the accuracy of step counting in special scenarios.

[0112] These raw sensor data first go through the following preprocessing steps:

[0113] a) Data synchronization: Since different sensors have different sampling rates, the system first aligns all data to a unified timestamp. This step is achieved through linear interpolation to ensure the consistency of all data in time.

[0114] b) De-noising: In order to eliminate the noise in the sensor data, the system uses an improved wavelet threshold denoising method. The specific algorithm is as follows:

[0115] X denoised =IDWT(Ψ(DWT(X),λ))

[0116] Among them, X is the original signal, DWT is discrete wavelet transform, IDWT is inverse discrete wavelet transform, Ψ is the threshold function, and λ is the adaptive threshold.

[0117] In practical applications, we choose Daubechies wavelet (db4) as the basis function and use the soft threshold method. The calculation method of the adaptive threshold λ is as follows:

[0118]

[0119] Where σ is the estimated value of the noise standard deviation and N is the signal length. This method can effectively remove high-frequency noise while retaining the important features of the signal.

[0120] c) Standardization processing: In order to make the data of different sensors comparable, the system adopts an improved robust standardization method:

[0121]

[0122] Among them, median is the median, IQR is the interquartile range, and ∈ is a small positive number (such as 0.01). This method is more robust than the traditional Z-score standardization and can effectively handle outliers and non-Gaussian distribution data.

[0123] 2. Multimodal feature extraction,In this step, the system uses an innovative multi-resolution feature extraction network (MRFEN) to process the pre-processed sensor data. The design concept of MRFEN is to capture information in both the time domain and the frequency domain at the same time, so as to comprehensively characterize the user's motion characteristics.

[0124] The specific structure of MRFEN is as follows:

[0125] a) Temporal Convolutional Module (TCM):

[0126] F t =TCM(X)=ReLU(BN(Conv1D(X,k t ,s t )))

[0127] Where X is the input sequence, k t is the convolution kernel size (usually set to 3 or 5), s t is the step size (usually set to 1), BN is the batch normalization operation, and ReLU is the activation function. This module can effectively capture local temporal features.

[0128] b) Frequency Convolution Module (FCM):

[0129] F f =FCM(FFT(X))=ReLU(BN(Conv1D(FFT(X),k f ,s f )))

[0130] Among them, FFT is fast Fourier transform, k f is the frequency domain convolution kernel size (usually set to 5 or 7), s f is the step size (usually set to 2). This module can extract frequency domain features, which helps to identify periodic motion patterns.

[0131] c) Multi-resolution fusion:

[0132] F mr =Concat(F t ,F f )

[0133] The time domain and frequency domain features are connected together to form a multi-resolution feature representation. For each sensor data, the system applies MRFEN for feature extraction. Finally, the features of all sensors are connected

[0134]

[0135] In this way, the system obtains a comprehensive multimodal feature representation, including the time-frequency characteristics of acceleration, angular velocity and air pressure data.

[0136] 3. Quantum-inspired dynamic attention mechanism, in order to better integrate the features of different modalities, this paper introduces an innovative quantum-inspired dynamic attention mechanism (QIDA). This mechanism is inspired by the concept of superposition in quantum computing and can dynamically adjust the importance of different features.

[0137] The specific algorithm of QIDA is as follows:

[0138] a) Initialize the quantum state:

[0139] |ψ>=∑α i |i>

[0140] Among them, α i is a complex amplitude, satisfying ∑|α i | 2 = 1. In actual implementation, we usually randomly initialize α i , and then normalize.

[0141] b) Apply quantum gate operations:

[0142] U|ψ>=exp(-iHt)|ψ>

[0143] Where H is the Hamiltonian operator and t is the evolution time parameter. In our implementation, H is

[0144] A custom operator based on eigenvectors has the form:

[0145] H=∑λ i |F i > <F i |

[0146] Among them, λ i is the eigenvector F i The eigenvalues ​​of are learned through neural network. t is an adjustable parameter, usually set in the range of [0,π].

[0147] c) Measure the attention weight:

[0148] w i =|<i|U|ψ> | 2

[0149] This step simulates the quantum measurement process, projects the quantum state onto the computational basis, and obtains the weight of each feature.

[0150] d) Weighting the features:

[0151] F att =∑w i ·F i

[0152] In this way, the system can dynamically adjust the importance of different sensors and different time-frequency domain features to adapt to different motion patterns and environments. For example, when walking on flat ground, the accelerometer may have a higher weight, while when climbing stairs, the barometer may have an increased weight.

[0153] 4. Adaptive Multi-Scale Fusion Network,Next, the system uses the Adaptive Multi-Scale Fusion Network (AMSFN) to process the weighted features. AMSFN is designed to capture motion patterns at different scales to adapt to various complex walking situations. AMSFN contains three innovative modules:

[0154] a) Fractal Dimension Feature Pyramid (FDFP):

[0155] The construction process of FDFP is as follows:

[0156] Initialization: F0 = F att

[0157] Send generation build: F i =FractalConv(Pool(F i -1,2),d i )

[0158] Among them, FractalConv is a fractal convolution operation, defined as:

[0159] FractalConv(X,d)=∑w j ·X (d / 2) ·|XS j (X)| ((2-d) / 2)

[0160] Here, S j is the scaling operator, w j is the weight, and d is the fractal dimension. In practical applications, we usually use a 3-5 layer pyramid structure, and the value range of the fractal dimension d is between [1.5, 2.5].

[0161] This fractal convolution operation can capture the self-similarity characteristics of the signal and is particularly suitable for processing human motion data with multi-scale characteristics. For example, it can simultaneously capture the tiny movements of a single step and the overall movement patterns of continuous multiple steps.

[0162] b) Adaptive Feature Fusion Module (AFFM):

[0163] The algorithm steps of AFFM are as follows:

[0164] 1. Calculate the importance score for each scale:

[0165] s i=SoftPlus(MLP(GlobalAvgPool(F i )))

[0166] Here, MLP is a two-layer perceptron with 64 neurons in the hidden layer and ReLU as the activation function.

[0167] 2. Normalized score:

[0168]

[0169] 3. Fusion features:

[0170] F fused =∑α i ·F i

[0171] AFFM can dynamically adjust the importance of features at different scales according to the current motion state and improve the adaptability of the system.

[0172] c) Topological Adversarial Feature Learning (TAFL):

[0173] The purpose of TAFL is to improve the robustness of the model to different wearing positions and movement patterns. The algorithm steps are as follows:

[0174] 1. Generate topological perturbations:

[0175]

[0176] Among them, L topo is the topological loss function, defined as the L2 distance of the persistent homology feature. ∈ is the perturbation strength, usually set in the range of [0.01, 0.1].

[0177] 2. Construct adversarial samples:

[0178] F adv =F fused +δ

[0179] 3. Adversarial Training:

[0180] L adv =E[log(D(F fused ))]+E[log(1-D(F adv ))]

[0181] Here, D is a discriminator network used to distinguish original features from adversarial samples.

[0182] Through this adversarial training, the system is able to learn more robust and generalized feature representations, thereby adapting to various wearing positions and motion patterns.

[0183] 5. Self-supervised pre-training. In order to make full use of a large amount of unlabeled user data, the present invention adopts innovative contrastive multi-task learning (CMTL) for self-supervised pre-training. The algorithm of CMTL includes:

[0184] a) Define auxiliary task set:

[0185] T = {exercise type recognition, step frequency estimation, energy consumption prediction}

[0186] These auxiliary tasks are related to the step counting task but do not require additional annotations.

[0187] b) Construct contrast loss for each task:

[0188]

[0189] Among them, sin is the cosine similarity, and τ is the temperature parameter (usually set in the range of [0.1, 0.5]). Yes F i Positive samples (generated by data enhancement), F j is a negative sample.

[0190] c) Multi-task joint optimization:

[0191] L cmtl =∑λ i ·(L ti +γ·L ci )

[0192] Here, L ti is the supervision loss for task i (if labeled), λ i is the task weight, and γ is the weight coefficient of the contrast loss (usually set in the range of [0.1, 1]).

[0193] Through this self-supervised pre-training, the system is able to learn more general and meaningful feature representations, laying the foundation for subsequent transfer learning and personalized adaptation.

[0194] 6. Transfer learning adaptation, in order to quickly adapt to new users and new wearing positions, the present invention adopts progressive domain adaptation (PDA) algorithm for transfer learning. The specific steps of PDA are as follows:

[0195] a) Build a domain classifier:

[0196] D(F)=Softmax(MLP(F))

[0197] The domain classifier is a two-layer perceptron that is used to distinguish between the source domain (pre-training data) and the target domain (new user data).

[0198] b) Define domain adversarial loss:

[0199] L da =-∑y d ·log(D(F))

[0200] Among them, y d is the domain label (0 represents the source domain, 1 represents the target domain).

[0201] c) Introducing progressive factors:

[0202]

[0203] Here, t is the current iteration number and γ is the adjustment factor (usually set in the range of [0.001, 0.01]). The gradual factor makes the domain adaptation process smoother.

[0204] d) Update feature extractor parameters:

[0205]

[0206] Among them, L task is the loss of the main task (step counting), and η is the learning rate.

[0207] Through PDA, the system can gradually adapt to the data distribution of new users while maintaining the memory of original knowledge.

[0208] 7. Online learning and personalization, in order to continuously optimize and personalize the model, the present invention adopts the meta-learning enhanced online learning (MEOL) algorithm. The specific steps of MEOL are as follows:

[0209] a) Define the meta-learning objective:

[0210]

[0211] Among them, D i and D j For different mini-batch data, f θ Represents the current model. This objective function measures how the update of the model on one task affects its performance on another task.

[0212] b) Online update rules:

[0213]

[0214] Here, β is the meta-learning rate, which is usually set in the range of [0.001, 0.01].

[0215] c) Personalized adaptation: For each user u, perform:

[0216]

[0217] Among them, D u is user-specific data, and η is the personalized learning rate (usually smaller than the global learning rate).

[0218] With MEOL, the system can quickly adapt to the user's individual needs, such as special gaits or atypical wearing styles, while maintaining good performance for general patterns.

[0219] 8. Multi-scale consistency check, in order to further improve the accuracy and reliability of step counting, the present invention introduces a multi-scale consistency check (MSCC) algorithm. The steps of MSCC are as follows:

[0220] a) Counting steps at different time scales:

[0221]

[0222] The system counts steps at five time scales: 1 second, 5 seconds, 10 seconds, 30 seconds, and 60 seconds.

[0223] b) Calculate the consistency score:

[0224]

[0225] This formula calculates the consistency of step counts at different time scales with the 1-minute step count.

[0226] c) Apply adaptive smoothing:

[0227] S smooth =α·S+(1-α)·S prev

[0228] Where α = sigmoid(C), S prev is the step number estimate of the previous moment. This smoothing method can dynamically adjust the weights of the current estimate and the historical estimate based on the consistency score.

[0229] d) Output the final number of steps: Output the final number of steps: number of steps = round(S smooth ), confidence level = C.

[0230] Through MSCC, the system can effectively handle complex walking patterns, such as intermittent brisk walking and jogging, and improve the stability and reliability of step counting.

[0231] Please refer to Figure 2-4 The present invention also provides a multi-modal adaptive pedometer system based on deep learning for implementing the method. The system includes the following modules:

[0232] Multi-source sensor module 1: includes accelerometer (BMI160), gyroscope (BMI160) and barometer (BMP388). These sensors are responsible for collecting user motion data and providing raw input for subsequent processing.

[0233] Multi-resolution feature extraction module 2: implements the MRFEN algorithm, including temporal convolution units and frequency convolution units. This module converts raw sensor data into multi-resolution feature representation.

[0234] Quantum-inspired dynamic attention module 3: implements the QIDA algorithm to dynamically adjust the importance of different features. This module improves the system's adaptability to different motion patterns.

[0235] Adaptive multi-scale fusion module 4: includes a fractal dimension feature pyramid 41, an adaptive feature fusion unit 42 and a topological adversarial feature learning unit 43. This module implements the AMSFN algorithm, captures multi-scale motion features and enhances the robustness of the model.

[0236] Learning optimization module 5: includes a contrastive multi-task learning unit 51, a progressive domain adaptation unit 52, and a meta-learning enhanced online learning unit 53. This module is responsible for model pre-training, transfer learning, and personalized optimization.

[0237] Multi-scale consistency check module 6: The MSCC algorithm is implemented to improve the accuracy and reliability of step counting through multi-scale comparison.

[0238] The system workflow is as follows:

[0239] 1. The multi-source sensor module 1 continuously collects the user's motion data.

[0240] 2. The multi-resolution feature extraction module 2 processes the raw data and generates multimodal features.

[0241] 3. The quantum-inspired dynamic attention module 3 weights the features.

[0242] 4. The adaptive multi-scale fusion module 4 further processes the weighted features to generate fused features.

[0243] 5. The learning optimization module 5 continuously optimizes the model based on user data and feedback.

[0244] 6. The multi-scale consistency verification module 6 performs the final verification and adjustment on the step counting results.

[0245] 7. The system outputs the final step counting results and confidence levels.

[0246] The advantage of the system lies in its high adaptability and accuracy. It can adapt to the gait characteristics of different users, various wearing positions (such as wrist, pocket, backpack, etc.), and complex movement environments (such as walking on flat ground, climbing stairs, running, etc.). Through continuous online learning and personalization, the system can continuously improve its performance and provide users with an increasingly accurate step counting experience.

[0247] In addition, the modular design of the system makes it highly extensible. For example, new sensors (such as optical heart rate sensors) can be easily added to further improve the accuracy of step counting, or the system functions can be expanded to support more sports type recognition and health monitoring.

[0248] In general, the multimodal adaptive pedometer method and system provided by the present invention represent an important advancement in pedometer technology. It fully utilizes the advantages of deep learning and multi-sensor fusion to provide users with a highly accurate, adaptive and personalized pedometer solution.

[0249] In order to verify the effectiveness of the present invention, the applicant will set up one embodiment and three comparative examples, and provide detailed test results and analysis.

[0250] Example 1 (present invention): The complete method described in the present invention is adopted, including multi-resolution feature extraction network (MRFEN), quantum-inspired dynamic attention mechanism (QIDA), adaptive multi-scale fusion network (AMSFN), contrastive multi-task learning (CMTL) pre-training, progressive domain adaptation (PDA) and meta-learning enhanced online learning (MEOL).

[0251] Comparative Example 1: Traditional threshold-based pedometer algorithm using only accelerometer data.

[0252] Comparative Example 2: A hybrid model of a simple convolutional neural network (CNN) and a long short-term memory network (LSTM) is used, but without multimodal fusion and adaptive mechanisms.

[0253] Comparative Example 3: Using multimodal data fusion and deep learning, but without adaptive mechanism and online learning.

[0254] The test indicators and methods are as follows:

[0255] 1. Pedometer accuracy:

[0256] Method: Compare with professional pedometers in different scenarios (walking on flat ground, climbing stairs, running).

[0257] Calculation formula: Accuracy = (1-|predicted number of steps-actual number of steps| / actual number of steps)*100%

[0258] 2. Adaptability score:

[0259] Method: Test at different wearing positions (wrist, pocket, backpack) to evaluate how quickly the model adapts to new positions.

[0260] Calculation formula: Adaptability score = (initial accuracy + 2*accuracy after 24 hours + 3*accuracy after 72 hours) / 6*100

[0261] 3. Impact on battery life:

[0262] Method: Measure the battery consumption percentage after the algorithm runs for 24 hours under the same hardware conditions.

[0263] 4. Personalization level:

[0264] Methods: Ten users with different gaits were tested to evaluate the algorithm's adaptability to individual differences.

[0265] Calculation formula: Personalization degree = (accuracy after personalization - initial accuracy) / initial accuracy * 100%

[0266] The test results are shown in the following table:

[0267] index Example 1 Comparative Example 1 Comparative Example 2 Comparative Example 3 Pedometer Accuracy 98.7％ 85.3％ 92.1％ 94.5％ Adaptability score 96.2％ 62.8％ 78.5％ 83.7％ Battery life impact 5.2％ 2.1％ 4.8％ 5.5％ Degree of personalization 12.3％ 1.2％ 5.7％ 7.9％

[0268] Reference Fig.12 This radar chart will intuitively demonstrate the comprehensive advantages of the present invention in multiple key indicators, especially its outstanding performance in step counting accuracy, adaptability and personalization. Fig.13 This stacked bar chart will clearly demonstrate the superior performance of the present invention in various complex environments. It not only shows the high accuracy of the present invention in each scenario, but also allows intuitive comparison of the overall performance of each solution through the total height of the bar.

[0269] The analysis and explanation of the above test results are as follows:

[0270] 1. Pedometer accuracy: Example 1 of the present invention shows a significantly higher pedometer accuracy (98.7%) than other comparative examples. This is mainly due to the application of multimodal data fusion and adaptive multi-scale fusion network (AMSFN). High accuracy means that users can trust the data provided by the device more, which helps to improve user experience and product competitiveness.

[0271] 2. Adaptability score: Example 1 performs well in adaptability (96.2%), far exceeding other comparison examples. This proves the effectiveness of quantum-inspired dynamic attention mechanism (QIDA) and progressive domain adaptation (PDA). High adaptability means that users can get a consistent high-quality experience in different scenarios and wearing positions, greatly improving the practicality of the product.

[0272] 3. Impact on battery life: Although the algorithm of Example 1 is more complex, its battery consumption (5.2%) is comparable to that of Comparative Examples 2 and 3, and only 3.1% more than the simple threshold algorithm (Comparative Example 1). This shows that the present invention not only improves performance, but also focuses on energy efficiency optimization, so that advanced functions will not significantly affect the endurance of the device.

[0273] 4. Personalization: Example 1 performs best in terms of personalization (12.3%), thanks to the meta-learning enhanced online learning (MEOL) mechanism. High personalization means that the device can better adapt to each user's unique gait and usage habits, providing more accurate step counting services.

[0274] Example 1 (the present invention) shows obvious advantages in all test indicators, especially in accuracy, adaptability and personalization. These results fully demonstrate the effectiveness and superiority of the multimodal adaptive pedometer method proposed in the present invention. This method not only improves the accuracy of pedometer, but also enhances the adaptability of the device in different usage scenarios, while providing a personalized user experience. Although the algorithm complexity has increased, the impact on battery life is still within an acceptable range, reflecting that the present invention has achieved a good balance between performance and energy efficiency.

[0275] These test results show that the present invention provides a comprehensive solution for the step counting function of wearable devices, which has the potential to significantly improve product performance and user satisfaction. It is not only suitable for conventional walking scenarios, but also can handle complex motion patterns and diverse usage environments well, providing a new direction for the development of future smart wearable devices.

[0276] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal adaptive step counting method based on deep learning, characterized in that: The following steps are involved: Obtain multi-source sensor data; Based on the multi-source sensor data, extracting multimodal features through a multi-resolution feature extraction network; Using a quantum-inspired dynamic attention mechanism to weight the multimodal features to obtain weighted features; Inputting the weighted features into an adaptive multi-scale fusion network to obtain fusion features; Based on the fusion features, self-supervised pre-training, transfer learning adaptation and online learning are sequentially performed to obtain optimized features; A multi-scale consistency check is performed on the optimized features to obtain the final step counting result.

2. The method according to claim 1, characterized in that The multi-source sensor data includes accelerometer data, gyroscope data and barometer data, wherein: The accelerometer data and gyroscope data are collected by the BMI160 sensor with a sampling rate of 100 Hz; The barometer data is collected by a BMP388 sensor with a sampling rate of 10 Hz.

3. The method according to claim 1, characterized in that The multi-resolution feature extraction network comprises: Temporal convolution module, whose output feature F_t is calculated by the following formula: F t =ReLU(BN(Conv1D(X,k t ,s t ))) Where X is the input sequence, k t is the convolution kernel size, s t is the step size, BN is batch normalization, and ReLU is the activation function; Frequency convolution module, whose output feature F f Calculated by the following formula: F f =ReLU(BN(Conv1D(FFT(X),k f ,s f ))) Among them, FFT is fast Fourier transform; t and F f Connect to get multi-resolution features F mr .

4. The method according to claim 1, characterized in that The quantum-inspired dynamic attention mechanism includes the following steps: Initialize quantum state |ψ>=∑α i |i>, where α i is the complex amplitude; Apply quantum gate operation U|ψ〉=exp(-iHt|ψ〉, where H is the Hamiltonian operator; Measure the attention weight w i =|<i|U|ψ> | 2 ; Weighting the features: F att =∑w i ·F i .

5. The method according to claim 1, characterized in that The adaptive multi-scale fusion network comprises: The fractal dimension feature pyramid is constructed in the following process: F0=F att F i =FractalConv(Pool(F i-1 ,2),d i ) Among them, FractalConv is a fractal convolution operation, defined as: FractalConv(X,d)=∑w j ·X (d / 2) ·|X-S j (X)|( (2-d) / 2 ) Among them, S j is the scaling operator, w j is the weight, d is the fractal dimension; Adaptive feature fusion module, its algorithm is: Calculate importance score: s i =SoftPlus(MLP(GlobalAvgPool(F i ))) Normalized score: Fusion features: F fused =∑α i ·F i ; Topological adversarial feature learning, the algorithm is: Generate topological perturbations: Constructing adversarial examples: F a dv=F fused +δ Adversarial Training: L a dv=E[log(D(F fused ))]+E[log(1-D(F adv ))].

6. The method according to claim 1, characterized in that The self-supervised pre-training adopts contrastive multi-task learning, and its algorithm includes: Define the auxiliary task set T = {motion type recognition, step frequency estimation, energy consumption prediction} and construct the contrast loss for each task: Among them, sin is the cosine similarity, τ is the temperature parameter; Multi-task joint optimization: L cmtl =∑λ i ·(L ti +γ·L ci ) Among them, L t i is the supervision loss of task i, λ i and γ are weight coefficients.

7. The method according to claim 1, characterized in that The transfer learning adaptation adopts progressive domain adaptation, and its algorithm includes: Build domain classifier: D(F) = Softmax(MLP(F)); Definition domain adversarial loss: l d a=-∑y d log(D(F)), where y d is the domain label; Introducing a progressive factor: Where t is the current iteration number and γ is the adjustment factor; Update the feature extractor parameters: Where L task is the main task loss, and η is the learning rate.

8. The method according to claim 1, characterized in that The online learning adopts meta-learning enhanced online learning, and its algorithm includes: Define the meta-learning objective: Where D i and D j For different small batches of data; Online update rules: Where β is the meta-learning rate; Personalized adaptation: For each user u, Where D u For user-specific data.

9. The method according to claim 1, characterized in that The algorithm of the multi-scale consistency check includes: Count steps at different time scales: Calculate the consistency score: To apply adaptive smoothing: S s mooth=α·S+(1-α)·S prev , where α = sigmoid(C), S prev Estimated number of steps at the previous moment; Output the final number of steps: number of steps = round(S smooth ), confidence level = C.

10. A multi-modal adaptive pedometer system based on deep learning that implements the method described in any one of claims 1 to 9, characterized in that: include: A multi-source sensor module, used to obtain multi-source sensor data; A multi-resolution feature extraction module, used for extracting multi-modal features based on the multi-source sensor data; A quantum-inspired dynamic attention module for weighting the multimodal features; Adaptive multi-scale fusion module, used to process weighted features and obtain fused features; A learning optimization module, used for performing self-supervised pre-training, transfer learning adaptation and online learning based on the fusion features; The multi-scale consistency verification module is used to verify the optimized features and obtain the final step counting results.

Images