A smart clothing perception and recognition method based on multi-modal sensor signal fusion

Abstract

The application provides a smart clothing perception and recognition method based on multi-modal sensing signal fusion, and relates to the technical field of smart clothing recognition, and comprises the following steps: collecting multi-channel original signals of human body actions through a nine-axis sensing module integrated in smart clothing; preprocessing the original signals to obtain preprocessed signals; extracting time domain features, frequency domain features and large model semantic features; adopting a dynamic weight fusion mechanism to weight and fuse the time domain features, the frequency domain features and the large model semantic features based on feature contribution and category correlation to obtain fused features; and inputting the fused features into a multi-layer adaptive classification network, performing regularization processing and probability mapping, and outputting human body action recognition results. The application can solve the problems of insufficient signal preprocessing adaptability, insufficient use of single feature domain information and limited generalization ability of the classification model of the existing method in a complex wearing scene.

Description

Technical Field

[0001] This invention relates to the field of smart clothing recognition technology, and in particular to a smart clothing perception and recognition method based on multimodal sensor signal fusion. Background Technology

[0002] With the synergistic development of microelectronics, flexible materials, and wireless communication technologies, smart wearable devices have become an important carrier in the field of human motion sensing. Smart clothing, as a typical example, integrates sensing modules into the garment structure, enabling continuous collection of human motion data without affecting the user's daily activities. It has broad application prospects in scenarios such as daily activity monitoring, motion posture analysis, and rehabilitation training assessment. However, human motion signals inherently possess characteristics such as strong non-stationarity, significant individual differences, and high inter-class similarity. Coupled with random motion noise and sensor drift in the wearable environment, accurately and robustly extracting motion features from raw signals and achieving reliable classification remains a core technological challenge in this field.

[0003] Chinese invention patent CN117084671A discloses a motion evaluation system based on gyroscope signals. This system collects motion information, including position, angle, and acceleration, using a wearable gyroscope. After noise suppression through median filtering and Butterworth filtering, the data is segmented using a sliding window method, and statistical quantities such as mean, standard deviation, kurtosis, and skewness are extracted as features in the time domain. Finally, an SVM or Multilayer Perceptron (MLP) classifier is used to output the motion category and quality evaluation results. This patent demonstrates the feasibility of a technical path for motion recognition and evaluation based on inertial sensing signals. However, its signal preprocessing scheme uses fixed-parameter filtering, which has limited adaptive detrending capabilities for non-stationary signals; feature extraction is limited to time-domain statistics, failing to effectively utilize the frequency-domain structure information of the signal; and the classifier used also has insufficient ability to express high-dimensional complex features, limiting its generalization performance in fine-grained recognition tasks of complex movements. Summary of the Invention

[0004] In view of this, the present invention provides a smart clothing perception and recognition method based on multimodal sensor signal fusion. By constructing an adaptive signal preprocessing mechanism, a multi-domain collaborative feature extraction framework and a dynamic weight fusion classification system, it solves the problems of insufficient adaptiveness of signal preprocessing, insufficient utilization of single feature domain information and limited generalization ability of classification models in complex wearable scenarios of existing methods, thereby improving the accuracy and robustness of human action recognition.

[0005] The technical solution of this invention is implemented as follows: This invention provides a smart clothing sensing and recognition method based on multimodal sensor signal fusion, comprising: S1. By integrating a nine-axis sensing module into smart clothing, multi-channel raw signals of human body movements are collected. The raw signals include three-axis acceleration signals, three-axis angular velocity signals, and three-axis magnetic field signals. S2. The original signal is denoised using an adaptive dynamic noise suppression algorithm. Then, the denoised signal is subjected to dynamic window detrending and local normalization in sequence to obtain a preprocessed signal. S3. A dual-branch convolutional network is used to extract time-domain features from the preprocessed signal to obtain time-domain features; an adaptive discrete transform is used to extract frequency-domain features from the preprocessed signal to obtain frequency-domain features; and a pre-trained large model is used to extract semantic features from the preprocessed signal to obtain large-model semantic features. S4. Employ a dynamic weighted fusion mechanism to weight and fuse the time-domain features, frequency-domain features, and large-model semantic features based on feature contribution and category correlation to obtain fused features; S5. Input the fused features into a multi-layer adaptive classification network, and after regularization and probability mapping, output the human action recognition result.

[0006] Preferably, the adaptive dynamic noise suppression algorithm in step S2 specifically includes: A fixed-length sliding window is used to perform local noise statistics on the original signal, and the normalized noise level is calculated. The formula is: ; in, This represents the variance of amplitude fluctuation; This is the cross-channel consistency coefficient. , This represents the pairwise combination number of the nine-channel signals. The nine channels are a 9-dimensional data vector composed of three sets of sensor data: three-axis acceleration, three-axis angular velocity, and three-axis magnetic field. Let be the covariance of the signals in channels i and j. These are the standard deviations of the corresponding channel signals; As a time-series stationarity indicator, , To fix the length of the sliding window, Let c be the signal value of the c-th channel at time t; These are nonlinear mapping coefficients; Based on the noise level The dynamic filter kernel parameters are adaptively determined, and the normalization formula for the filter kernel is as follows: ; in, The coefficients of the filter kernel at position p are... The filter kernel length is adaptively adjusted according to the noise level. The original weight distribution function of the filter kernel is divided into three forms: Gaussian, Gaussian-uniform mixture, and uniform. The global attenuation coefficient of the filter kernel is adaptively adjusted according to the noise level; , This is the internal position index of the filter kernel; With the filter kernel Multidimensional convolution is performed on the nine-channel signal, and second-order differential regularization and inter-channel difference compensation are introduced to complete cross-channel collaborative filtering, resulting in the filtered signal. , Let be the amplitude of the filtered signal at time t.

[0007] Preferably, the dynamic window detrending process in step S2 specifically includes: Calculate the rate of change of the signal , To characterize the comprehensive quantitative index of the fluctuation intensity of the filtered signal, based on the filtered signal The mean of the time-by-time difference amplitude is obtained by averaging it over the entire time series, and the calculation formula is as follows: ; in, The length of the time series corresponding to the total sampling duration of the filtered signal; Let be the amplitude of the filtered signal at time t; , These represent the signal amplitudes of the filtered signal at time points t-1 and t-2, respectively. Then, dynamic window length adaptation is performed, and the window length is adjusted accordingly. With rate of change Adaptive adjustment, calculated using the following formula: ; in, The length of the dynamically sliding window. This is a rounding function to ensure that the window length is a positive integer. The rate of change of the signal; The preset baseline rate of change; Generate nonlinear weighting coefficients, and assign the weights of each historical time point within the window using a nonlinear mapping function, as shown in the following formula: ; in, Here, m represents the dynamic weighting coefficient, and m is the time offset index within the window. For summation index; Perform precise trend stripping to remove trend signals. The calculation formula is as follows: ; in, The amplitude of the signal after detrending. Let be the amplitude of the filtered signal at time t. The trend term of the dynamic weighted fitting is obtained by summing the products of the signals at each time point within the window and the corresponding weighting coefficients.

[0008] Preferably, the local standardization process in step S2 specifically includes: based on Calculate the dynamically weighted local sliding standard deviation The formula is as follows: ; Where m is the time offset within the window, and its value ranges from 1 to 2. ; The weighting coefficients are dynamic, based on the rate of change of the signal. The independent variable is adaptively adjusted according to the window position m; Detrending signals at historical moments The instantaneous value at that point; The dynamic sliding window length is the rate of change of the signal at time t. Adaptive adjustment; This is the dynamically weighted local moving average of the detrended signal; Based on the calculated normalized noise level For the dynamically weighted local sliding standard deviation Perform adaptive noise calibration to obtain the calibrated standard deviation. ; Then perform a uniform scaling mapping on the detrended signal, as shown in the formula: ; in, This is a preprocessed signal after local standardization. To prevent numerical stability terms from being divided by zero.

[0009] Preferably, in step S3, the dual-branch convolutional network extracts temporal features from the preprocessed signal, specifically including: The first branch uses variable dilation rate dilated convolution to extract long-term dependent features, with the following formula: ; in, The long-term dependency features output by the first branch. For variable dilation rate dilated convolution operations, The kernel size is [size]. The dynamic expansion rate is adaptively adjusted by the signal timing span. For batch normalization operations, It is a non-linear activation function; This refers to the preprocessed signal at time point t. The second branch uses depthwise separable convolution to extract short-term local features, as shown in the formula: ; in, The short-time local features output by the second branch. For depthwise separation convolution operations, To separate the kernel size; Then, the features from the two branches are dynamically weighted and fused using a channel attention mechanism, as shown in the formula: ; ; in, This is the final output of temporal feature extraction. For adaptive attention weights, This is the time complexity evaluation function. This is the weighting adjustment coefficient. This is a bias term.

[0010] Preferably, in step S3, the adaptive discrete transform extracts frequency domain features from the preprocessed signal, specifically through an adaptive basis function. The preprocessed signal is mapped to the frequency domain using the following formula: ; in, For the transform domain The frequency domain signal corresponding to each frequency component. For transform domain frequency index, each The amplitude and phase information corresponding to a frequency point For preprocessing signals Total number of sampling points This represents the sampling point index of the time-domain signal, with values ​​ranging from 1 to T. The preprocessed signal amplitude corresponding to the nth sampling point. For frequency-dependent components Adaptively adjusted basis function parameters; The specific expression is: ; in The imaginary unit satisfies ; The formula for calculating the frequency-adaptive window width parameter is as follows: ; in As the base window width, This is the window width adjustment factor, with a default value of 0.5 and a range of values ​​of [missing value]. This enables adaptive time-frequency resolution matching with a wide window in the low-frequency band and a narrow window in the high-frequency band. It is a natural exponential function; These are the basis function normalization coefficients, ensuring the conservation of signal energy before and after the transformation; Extracting frequency domain feature vectors from frequency domain signals .

[0011] Preferably, in step S3, the pre-trained large model extracts semantic features from the preprocessed signal, specifically including: After the preprocessed signal is reshaped into a sequence, it is then processed by the channel adaptation matrix. Mapping to the feature space yields the adapted features. The formula is: ; in, The reconstructed sequence signal, To adapt the weight matrix to the channel, For bias terms; Will Inputting the pre-trained temporal feature encoding network yields high-dimensional semantic features. The formula is: ; in, This is a pre-trained temporal feature encoding network with a multi-layer Transformer encoder structure, where each layer contains a multi-head self-attention module and a feedforward neural network module; These are the pre-trained parameters.

[0012] Preferably, the dynamic weight fusion mechanism in step S4 specifically includes: For time domain features and frequency domain features Perform linear transformations separately to match the semantic features of the large model. Dimensional alignment yields alignment features. ; Based on feature contribution and category relevance, an attention mechanism is used to dynamically calculate the fusion weights of temporal features, frequency features, and semantic features of the large model. ,satisfy And it adjusts dynamically according to the input features; The final weighted fusion formula is: ; in, This is a feature of fusion.

[0013] Preferably, the processing procedure of the multi-layer adaptive classification network in step S5 specifically includes: Fusion features The input is a multi-layer adaptive classification network, which is then mapped through a fully connected layer to obtain high-level semantic features. ; Finally, the recognition probability of each action category is output through Softmax probability mapping: ; in, In order to integrate features Under these conditions, the sample belongs to the action category. The probability, The total number of action categories. The output features of the last fully connected layer in the multi-layer adaptive classification network. , For category The corresponding output layer weight matrix, , For category The corresponding output layer bias term, , For category calibration items, () is the natural exponential function.

[0014] Preferably, in step S5, the training of the multi-layer adaptive classification network employs a cross-entropy loss function combined with weight decay regularization, and iteratively optimizes the network parameters through backpropagation until the loss function converges; during training, the weight matrix of the multi-layer adaptive classification network is subjected to... Regularization constraints are used to suppress overfitting and improve the model's ability to generalize to unseen action samples.

[0015] The present invention has the following advantages over the prior art: (1) This invention dynamically determines the filter kernel parameters by comprehensively evaluating the amplitude fluctuation variance, cross-channel consistency coefficient, and time series stationarity index of the signal, thereby achieving adaptive filtering under different noise intensity conditions. Simultaneously, it adaptively adjusts the sliding window length and weighting coefficients according to the signal's local rate of change, accurately separating non-stationary trend terms in the signal, and eliminating amplitude differences between channels through local standardization. Compared to fixed-parameter preprocessing schemes, the above mechanism can better preserve effective motion-related features while suppressing noise.

[0016] (2) This invention employs a dual-branch convolutional network to extract temporal features with long-term dependence and short-term local features respectively. Simultaneously, it utilizes an improved adaptive discrete transform to extract multi-dimensional frequency domain features such as frequency domain energy distribution and phase consistency. Furthermore, it introduces a pre-trained temporal coding model to obtain high-level semantic features. These three types of features complement each other from different dimensions to describe the structural information of the action signal. Compared with methods that rely solely on a single feature domain, the synergistic utilization of multi-domain features helps reduce the information bottleneck caused by a single feature dimension and improves the ability to distinguish similar actions.

[0017] (3) This invention uses an attention mechanism to dynamically calculate the fusion weights of features in each domain based on feature contribution and category relevance, replacing fixed weights or simple concatenation fusion strategies. This allows the fusion process to adaptively emphasize more discriminative feature sources according to different action categories. While preserving the independent information of features in each domain, this mechanism effectively reduces the interference of redundant information and improves the inter-class separability of the comprehensive features.

[0018] (4) This invention introduces a feature perturbation term and an adaptive Dropout regularization mechanism into the classification network to suppress overfitting, and alleviates the impact of imbalanced training samples on the classification boundary through a class calibration term. Finally, the recognition result is output through Softmax probability mapping. The above design enables the model to maintain relatively stable classification performance when facing individual differences, action variants and small sample classes, and adapts to the uneven data distribution in real application scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the preprocessing process of the present invention; Figure 3 This is a diagram of the multi-domain feature extraction structure of the present invention; Figure 4 This is a schematic diagram of the weight fusion mechanism of the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, this invention provides a smart clothing sensing and recognition method based on multimodal sensor signal fusion, comprising: S1. By integrating a nine-axis sensing module into smart clothing, multi-channel raw signals of human body movements are collected. The raw signals include three-axis acceleration signals, three-axis angular velocity signals, and three-axis magnetic field signals. S2. The original signal is denoised using an adaptive dynamic noise suppression algorithm. Then, the denoised signal is subjected to dynamic window detrending and local normalization in sequence to obtain a preprocessed signal. S3. A dual-branch convolutional network is used to extract time-domain features from the preprocessed signal to obtain time-domain features; an adaptive discrete transform is used to extract frequency-domain features from the preprocessed signal to obtain frequency-domain features; and a pre-trained large model is used to extract semantic features from the preprocessed signal to obtain large-model semantic features. S4. Employ a dynamic weighted fusion mechanism to weight and fuse the time-domain features, frequency-domain features, and large-model semantic features based on feature contribution and category correlation to obtain fused features; S5. Input the fused features into a multi-layer adaptive classification network, and after regularization and probability mapping, output the human action recognition result.

[0023] This invention achieves comprehensive acquisition of human motion signals through a nine-axis sensing module, coupled with multi-stage adaptive preprocessing to optimize data quality, and relies on multi-domain collaborative feature fusion to deeply mine the essential features of the motion. It then combines an adaptive classification network to complete the entire process design for accurate recognition. This not only significantly improves the accuracy of motion recognition, but also maintains stable recognition performance in high-noise scenarios. Overall, it has strong anti-interference capabilities and wide applicability to various scenarios, and can flexibly adapt to various practical application needs such as daily activity monitoring, motion posture analysis, and rehabilitation training assessment.

[0024] In one embodiment of the present invention, step S1 includes: The key areas of smart clothing (such as the upper arm, waist, and thigh) integrate a nine-axis sensing module. This module simultaneously collects three-axis acceleration signals, three-axis angular velocity signals, and three-axis magnetic field signals, forming raw time-series data across nine signal channels. The three-axis acceleration signal is used to capture linear motion characteristics of the human body, the three-axis angular velocity signal is used to sense changes in rotational posture, and the three-axis magnetic field signal is used to obtain directional positioning information. These three types of signals complement each other, fully reflecting the spatial dynamic characteristics of human movements and effectively distinguishing between similar actions such as walking and jogging, bending over and squatting. This overcomes the limitation of traditional single sensors that can only acquire partial motion data.

[0025] like Figure 2 As shown, in one embodiment of the present invention, step S2 specifically includes: S21: An adaptive dynamic noise suppression algorithm is used to remove motion interference and environmental noise from the signal. Accurate noise reduction is achieved through multi-dimensional noise assessment and dynamic kernel optimization. S22: Perform dynamic window detrending processing on the filtered signal. To address the non-stationary characteristics of human motion signals, the trend term is accurately removed through adaptive window adjustment and weighted fitting.

[0026] S23 performs local statistics calculation, channel noise calibration, and standardized processing on the detrended signal, and finally outputs a preprocessed signal with uniform scale and prominent features, providing consistent input for subsequent feature extraction.

[0027] In this embodiment, step S21 specifically includes: S211. Perform local noise statistics on the original signal using a fixed-length sliding window, wherein the sliding window length is fixed at 30 sampling points, and calculate the normalized noise level. The formula is: ; in, This represents the variance of amplitude fluctuation; Cross-channel consistency coefficient: ; This represents the pairwise combination number of the nine-channel signals. The nine channels are a 9-dimensional data vector composed of three sets of sensor data: three-axis acceleration, three-axis angular velocity, and three-axis magnetic field. Let be the covariance of the signals in channels i and j. These are the standard deviations of the corresponding channel signals; For time-series stationarity indicators: ; To fix the length of the sliding window, Let c be the signal value of the c-th channel at time t; These are fixed coefficients for the nonlinear mapping, used to map the evaluation vector to noise levels. Their values ​​range from [0,2], with a default value of [value missing]. The optimal value was determined through a grid search optimization using 1000 sets of labeled noise samples. It is a natural exponential function. The larger the value, the stronger the noise interference.

[0028] S212, Based on the noise level Adaptively determine dynamic filter kernel parameters.

[0029] Specifically, low noise range Use small core ( Default value is 3), concentrated weight distribution (Gaussian form), weak decay coefficient ( (Default value is 0.1); Medium noise range Using a medium-sized core ( Default value is 5), mixed balanced weight distribution (mixture Gaussian-uniform form), medium decay coefficient ( (Default value is 0.3); High noise range Using a large core ( Default value is 7), uniform weight distribution (uniform form), strong attenuation coefficient ( (Default value is 0.5). The filter kernel length is increased sequentially for each of the three intervals. (range of values) Adjusting the weight distribution pattern and increasing the attenuation coefficient (range of values) This allows for precise matching with noise intensity.

[0030] The normalization formula for the filter kernel is: ; in, The coefficients of the filter kernel at position p are... The filter kernel length is adaptively adjusted according to the noise level. The original weight distribution function of the filter kernel is divided into three forms: Gaussian, Gaussian-uniform mixture, and uniform. The global attenuation coefficient of the filter kernel is adaptively adjusted according to the noise level; This is the internal position index of the filter kernel, and its value range is the same as the corresponding... Consistent.

[0031] S213, with the aforementioned filter core Multidimensional convolution is performed on the nine-channel signal, and second-order differential regularization and inter-channel difference compensation are introduced to complete cross-channel collaborative filtering. The filtering formula is as follows: ; in, The filtered signal at the current time t. This is the original nine-channel timing signal. This represents a cross-channel convolution operation with dimension matching. For the second-order gradient term of the signal, This is the gradient balancing coefficient, with a default value of 0.08 and a range of values ​​of [value missing]. The dynamic range of the signal is adaptively adjusted; δ is the noise suppression coefficient, with a default value of 0.04 and a range of [0.01, 0.08], which is related to the noise level. They show a linear positive correlation. This is the signal confidence function. The final filtered signal is obtained. , Let be the amplitude of the filtered signal at time t.

[0032] The aforementioned adaptive dynamic noise suppression algorithm quantifies the noise state of the current signal through multi-dimensional noise evaluation, and then dynamically matches the filter kernel shape and intensity based on the quantization results. This solves the problem that traditional fixed filter kernels cannot adapt to dynamic random noise. While effectively suppressing motion interference and environmental electromagnetic noise, it preserves motion detail signals and avoids information loss caused by excessive smoothing.

[0033] In this embodiment, step S22 specifically includes: S221. First, calculate the rate of change of the signal. , To characterize the comprehensive quantitative index of the fluctuation intensity of the filtered signal, based on the filtered signal The mean of the time-by-time difference amplitude is obtained by averaging it over the entire time series, and the calculation formula is as follows: ; in, The length of the time series corresponding to the total sampling duration of the filtered signal; Let be the amplitude of the filtered signal at time t; , These represent the signal amplitudes of the filtered signal at time points t-1 and t-2, respectively.

[0034] S222, Next, perform dynamic window length adaptation, window length... With rate of change Adaptive adjustment, calculated using the following formula: ; in, The length of the dynamically sliding window. This is a rounding function to ensure that the window length is a positive integer. The rate of change of the signal; The preset baseline rate of change is set to 0.5 by default, with a range of [value missing]. It is determined based on the statistical characteristics of fluctuations in human daily movement signals.

[0035] when When the change is large (strong abrupt change in action), As the value approaches the minimum of 6, the window is shortened to preserve details of the mutation; when When the movement is smaller (smooth), Approaching the maximum value of 35, the window is extended to fully strip away the slow trend term.

[0036] S223. Subsequently, nonlinear weighting coefficients are generated, and the weights for each historical moment within the window are allocated using a nonlinear mapping function. The larger the value, the more concentrated the weight is on recent data. The smaller the weight, the more Gaussian the distribution, as shown in the formula below: ; in, Here, m represents the dynamic weighting coefficient, and m is the time offset index within the window. For the summation index, For the rate of change of the signal, The preset baseline rate of change, This is the length of the dynamically sliding window.

[0037] S224. Based on the above window length and weighting coefficients, perform precise stripping of the trend term to remove the trend signal. The calculation formula is as follows: ; in, The amplitude of the signal after detrending. Let be the amplitude of the filtered signal at time t. The trend term of the dynamic weighted fitting is obtained by summing the products of the signals at each time point within the window and the corresponding weighting coefficients.

[0038] Specifically, the dynamic window length adaptively adjusts with the intensity of motion fluctuations. Combined with nonlinear weighting coefficients, it enables precise stripping of trend terms from rapidly changing motions while preserving the slow change patterns of stable motions. This avoids the loss of details or residual trends caused by traditional fixed-window detrending, making the signal more closely match the essential characteristics of the motion.

[0039] In this embodiment, step S23 specifically includes: S231, based on Further calculation of the dynamically weighted local sliding standard deviation The formula is as follows: ; Where m is the time offset within the window, and its value ranges from 1 to 2. ; The weighting coefficients are dynamic, based on the rate of change of the signal. The independent variable is adaptively adjusted according to the window position m; Detrending signals at historical moments The instantaneous value at that point; The dynamic sliding window length is the rate of change of the signal at time t. Adaptive adjustment; This is the dynamically weighted local moving average of the detrended signal.

[0040] S232. Based on the calculated normalized noise level For the dynamically weighted local sliding standard deviation Adaptive noise calibration is performed using the following formula: ; in, This is the local sliding standard deviation after noise calibration. The local sliding standard deviation before calibration. This is for normalizing the noise level. The higher the noise level, the larger the standard deviation after calibration, which can avoid standardization distortion caused by underestimation of the standard deviation in high-noise scenarios.

[0041] S233. Next, perform a uniform scaling mapping on the detrended signal, using the following formula: ; in, This is a preprocessed signal after local standardization. To prevent the numerical stability term from being divided by zero, a fixed value is taken. .

[0042] Local standardization maps the signals of each channel to a uniform scale. Combined with the standard deviation after noise calibration, it eliminates the amplitude differences between channels and avoids standardization distortion caused by noise, providing a consistent input with uniform scale and prominent features for subsequent feature extraction.

[0043] like Figure 3 As shown, in one embodiment of the present invention, step S3 includes: S31: Temporal feature extraction, which extracts local temporal features and long-term dependent features through a dual-branch convolutional structure, and assigns dynamic weights by combining a channel attention mechanism; S32: Frequency domain feature extraction. An improved discrete transform algorithm is used to convert the time domain signal to the frequency domain to extract the frequency distribution and phase correlation features of the motion signal. S33: Extract semantic features of a large model using a pre-trained large model.

[0044] In this embodiment, step S31 uses a dual-branch convolutional network to extract temporal features from the preprocessed signal, specifically including: S311. The first branch uses variable dilation rate dilated convolution to extract long-term dependent features. Considering the temporal continuity of human motion signals, the dilation rate is dynamically adjusted to adapt to the temporal span of different actions. The formula is as follows: ; in, The long-term dependency features output by the first branch. For variable dilation rate dilated convolution operations, This represents the kernel size, with a default value of 5 and a range of [3, 9]. The dynamic expansion rate is adaptively adjusted by the signal timing span, and its value ranges from [1, 8]. For batch normalization operations, It is a non-linear activation function. This is the preprocessed signal at time t. Dilated convolution expands the receptive field without increasing the number of parameters by inserting gaps into the convolution kernel. The dynamic dilation rate enables the network to adaptively adjust the feature capture range for periodic actions with different temporal spans, such as walking and running.

[0045] S312. The second branch uses depthwise separable convolution to extract short-term local features, targeting the instantaneous changes in human movements. It accurately preserves local dynamic information while reducing computational complexity. The formula is: ; in, The short-time local features output by the second branch. For depthwise separation convolution operations, To separate the convolution kernel size, the default value is 3, and the range is [1,4]. Depthwise separable convolution decomposes standard convolution into channel-wise convolution and point-wise convolution, effectively reducing the amount of computation while focusing on capturing instantaneous local details such as the start and end of actions.

[0046] S313. Introduce a channel attention dynamic weight allocation mechanism to adaptively adjust the fusion weights of the two-branch features based on the temporal complexity of the action signal, thereby achieving dynamic feature enhancement. The formula is as follows: ; ; in, This is the final output of temporal feature extraction. For adaptive attention weights, This is the time complexity evaluation function. This is the weighting adjustment coefficient, with a default value of 2.0 and a range of [0.5, 5.0]. This is a bias term, with a fixed value of 0. When the timing complexity of the actions is high... Approaching 1, long-term dependency characteristics dominate; when the action is stable, As the complexity of the action decreases, the weight of short-term local features increases accordingly. The bi-branch convolution focuses on the long-term correlation and short-term details of the action, respectively. The channel attention mechanism dynamically allocates weights according to the complexity of the action, realizing the adaptive complementarity of "long-term-short-term" features. It captures the long-term patterns of periodic actions such as walking and running, while not missing the instantaneous details such as the start and end of the action, thus improving the comprehensiveness of temporal features.

[0047] The specific expression is: ; in, The rate of change of the signal within the sliding window at the current moment is calculated using the same formula as step S221. This is the statistical minimum of the signal change rate, determined based on statistics of all samples in the training set, with a default value of 0.01. This is the statistical maximum value of the signal change rate, determined based on statistics of all samples in the training set, with a default value of 2.0. This function normalizes the signal change rate to the [0,1] interval, and the larger the value, the higher the temporal complexity of the action.

[0048] In this embodiment, step S32 uses an adaptive discrete transform to extract frequency domain features from the preprocessed signal, specifically through an adaptive basis function. The preprocessed signal is mapped to the frequency domain using the following formula: ; in, For the transform domain The frequency domain signal corresponding to each frequency component. For transform domain frequency index, each The amplitude and phase information corresponding to a frequency point For preprocessing signals Total number of sampling points This represents the sampling point index of the time-domain signal, with values ​​ranging from 1 to T. The preprocessed signal amplitude corresponding to the nth sampling point. For frequency-dependent components Adaptively adjusted basis function parameters.

[0049] Compared to the standard discrete Fourier transform which uses fixed sinusoidal basis functions, adaptive basis functions... It can adaptively adjust the shape of the basis function according to the signal characteristics of different frequency bands, and more effectively mine the frequency-phase features in human motion signals.

[0050] The specific expression is: ; in The imaginary unit satisfies ; The formula for calculating the frequency-adaptive window width parameter is as follows: ; in As the base window width, This is the window width adjustment factor, with a default value of 0.5 and a range of values ​​of [missing value]. This enables adaptive time-frequency resolution matching with a wide window in the low-frequency band and a narrow window in the high-frequency band. It is a natural exponential function; These are the basis function normalization coefficients, ensuring the conservation of signal energy before and after the transformation; Extracting frequency domain feature vectors from frequency domain signals The formulas for calculating each feature are as follows: ; ; ; ; in, For frequency domain energy entropy, Dominant frequency bandwidth, The phase consistency coefficient, For the attenuation rate of high-frequency components, For subsequent expansion of other frequency domain features; For the first The energy percentage at each frequency point It is the natural logarithm function; This is the highest frequency at which the cumulative energy accounts for 90%. The lowest frequency at which the cumulative energy accounts for 90%; For the first Phase at each frequency point It is a cosine function; This refers to the amplitude at the boundary between the mid-frequency and high-frequency ranges. The amplitude at the highest frequency. This represents the total number of frequency points. The four types of frequency domain features described above characterize the frequency domain structure of a signal from four dimensions: energy distribution, effective bandwidth, phase continuity, and high-frequency attenuation characteristics, respectively, providing complementary frequency domain discrimination information for different action categories.

[0051] Specifically, the adaptive basis function of the improved discrete transform is better suited to the non-stationary characteristics of human motion signals. The extracted multi-dimensional frequency domain features characterize the frequency domain essence of the motion from the perspectives of energy distribution, frequency range, and phase correlation, making up for the problem that time domain features are insufficient in capturing periodicity and frequency specificity.

[0052] In this embodiment, step S33 uses a pre-trained large model to extract semantic features from the preprocessed signal, specifically including: S331, preprocess the signal Through dimensional reshaping operation Reconstructing it into a sequence form yields a dimension of sequence signal Where 4 is the number of subsequences, This is the time step, with a default value of 250, and a range of values ​​of [value missing]. It is compatible with sensing sampling frequencies from 10Hz to 100Hz (sampling frequency is hour, Pick arrive ), where 9 represents the number of channels. The formula is expressed as follows: ; Then via channel adaptation matrix Mapping to the feature space yields the adapted features. The formula is: ; in, The reconstructed sequence signal, To adapt the weight matrix for the channel, the dimension is... , The bias term has the following dimensions: , To adapt the input features of the network, the dimension is... .

[0053] Will Inputting the pre-trained temporal feature encoding network yields high-dimensional semantic features. The formula is: ; in, This is a pre-trained temporal feature encoding network with a 4-layer Transformer encoder structure. Each layer contains a multi-head self-attention module (8 heads) and a feedforward neural network module. These are pre-trained parameters; The output of the network is a high-dimensional semantic feature with a dimension of 512.

[0054] The detailed structure and training paradigm of the pre-trained temporal feature encoding network are as follows: The network adopts a 4-layer Transformer encoder architecture. Each encoder layer includes an 8-head self-attention module, a layer normalization module, a feedforward neural network (hidden layer dimension 256), and a residual connection module. The network input dimension is [4,L,64], the output dimension is 512, and the total number of parameters is 2.3M, which is suitable for the lightweight deployment requirements of embedded devices. Pre-training adopts an unsupervised mask temporal modeling task, randomly masking 15% of the time steps in the input sequence, and reconstructing the signal features of the mask position through the network to learn the general temporal semantics of human motion signals. The pre-training dataset uses the publicly available human motion sensing datasets PAMAP2 and Opportunity, as well as a self-built smart clothing motion dataset, which includes 120 subjects, 18 categories of daily actions, and over 200 hours of nine-axis sensing data, covering subjects of different ages, genders, and body types, and actions in various scenarios such as daily activities, sports, and rehabilitation training. When fine-tuning for the target scenario, the parameters of the first two encoder layers of the network are frozen, and only the parameters of the last two encoder layers and the output layer are fine-tuned. The fine-tuning batch size is 32, the initial learning rate is 1e-4, a cosine annealing learning rate decay strategy is adopted, the training epochs are 50, and the weight decay coefficient is 1e-4 to avoid overfitting.

[0055] Pre-trained large models provide highly discriminative semantic priors for subsequent cross-domain fusion by establishing general semantic representations on large-scale multi-scene action data, thus making up for the shortcomings of pure signal-driven features at the action semantic level.

[0056] like Figure 4 As shown, in one embodiment of the present invention, the dynamic weight fusion mechanism in step S4 specifically includes: For time domain features and frequency domain features Perform linear transformations separately to match the semantic features of the large model. The dimensional (512-dimensional) alignment is calculated using the following formula: ; ; in, , These are the alignment features obtained after linear transformation of time-domain features and frequency-domain features, respectively, both with a dimension of 512. The linear transformation weight matrix for time-domain features has dimension 1. ; The linear transformation weight matrix for frequency domain features has dimension 1. ; These are the original dimensions of the time-domain and frequency-domain features, respectively; These are the bias terms for the linear transformations in the time and frequency domains, respectively. These are the fine-tuning coefficients, with default values ​​of 0.03 and 0.02 respectively. This is the calibration vector for the characteristic distribution.

[0057] Based on feature contribution and category relevance, an attention mechanism is used to dynamically calculate the fusion weights of temporal features, frequency features, and semantic features of the large model. ,satisfy Furthermore, it dynamically adjusts with the input features. The fusion weights are obtained by calculating the aligned three types of features through an attention scoring network and then normalizing them using Softmax, reflecting the actual contribution of each feature domain in the current action sample to the recognition result.

[0058] The final weighted fusion formula is: ; in, The fusion feature has 512 dimensions. Compared to fusion methods with fixed weights or simple concatenation, dynamic weight fusion is based on a dual evaluation of feature contribution and class relevance, adaptively learning the optimal combination ratio of the three types of features. It can maintain high fusion efficiency under different action categories and collection scenarios, effectively improving the discriminative ability of the fused features.

[0059] In one embodiment of the present invention, the processing procedure of the multi-layer adaptive classification network in step S5 specifically includes: The first fully connected layer fuses features. To enhance the expressive power of features, a non-linear mapping is performed, as shown in the following formula: ; in, The output features of the first fully connected layer have a dimension of 256. This is the weight matrix of the first fully connected layer, with dimension 1. , This is the bias term for the first fully connected layer, with a dimension of 256. The characteristic perturbation term takes the value of , To generate random numbers that follow a standard normal distribution (mean 0, variance 1), It is a non-linear activation function. This is a batch normalization operation. The feature perturbation term introduces small random fluctuations, allowing the model to adapt to signal variations in real-world scenarios during training, thus avoiding overfitting to clean samples. The combination of activation and batch normalization enhances the nonlinear expression of features and stabilizes their distribution.

[0060] The second fully connected layer introduces adaptive Dropout regularization to further optimize feature representation and suppress overfitting, as shown in the following formula: ; in, The output features of the second fully connected layer have 128 dimensions. This is the weight matrix of the second fully connected layer, with dimension 1. , This is the bias term for the second fully connected layer, with a dimension of 128. This is a quadratic disturbance term, with a value of ; To adapt to Dropout operations, To adapt the Dropout probability. This represents the current training iteration number. This is the maximum number of iterations and is fixed at 10000. Adaptive Dropout suppresses overfitting in the early stages of training by using a higher probability of inactivation, and gradually reduces the probability of inactivation as the number of iterations increases, so that the model retains more complete feature information during the training convergence phase.

[0061] The output layer uses a Softmax probability map with a class calibration term to transform the final features into a probability distribution of action classes, as shown in the following formula: ; in, In order to integrate features Under these conditions, the sample belongs to the action category. The probability, The total number of action categories. , For category The corresponding output layer weight matrix, , For category The corresponding output layer bias term, , For category calibration items, () represents the natural exponential function. The category calibration term compensates for the sample imbalance problem by introducing sample frequency priors for each category, ensuring that the recognition probability of the rare category is not systematically underestimated due to the small number of training samples, thereby improving the recognition fairness of the model on datasets with uneven class distribution.

[0062] The training of the multilayer adaptive classification network employs a cross-entropy loss function combined with weight decay regularization. The network parameters are iteratively optimized through backpropagation until the loss function converges. During training, a weight matrix of the multilayer adaptive classification network is applied... Regularization constraints are used to suppress overfitting and improve the model's ability to generalize to unseen action samples.

[0063] Multilayer adaptive classification networks enhance the model's adaptability to real-world signal variations through feature perturbation, while adaptive Dropout dynamically adjusts the regularization strength based on the training process. The class calibration term mitigates the impact of sample imbalance. These mechanisms work synergistically to improve the model's robustness and generalization ability while maintaining high recognition accuracy.

[0064] This invention undergoes specific experimental verification to demonstrate the effectiveness of its method. The experimental details are as follows: A self-built smart clothing motion dataset was used. The data was collected from 10 participants aged 20 to 30 wearing smart clothing equipped with a nine-axis inertial sensor in both indoor and outdoor scenarios. The dataset contains 40,000 labeled samples. The data was divided into a training set at an 8:2 ratio for model training and performance evaluation. Regarding training parameters and optimization strategies, the model training uses cross-entropy loss as the loss function, with the formula: ,in One-hot encoding for real action categories, For the class probability distribution predicted by the model, the optimizer uses Adam with an initial learning rate of The weight decay coefficient is Momentum parameters The training rounds are 100, and the accuracy is evaluated on the validation set after each round of training.

[0065] Table 1: Overall performance results.

[0066] Table 2: Classification performance results.

[0067] The results are shown in Tables 1 and 2. The classification performance of the validation set is as follows: standing action: 99.21% (loss value 0.0220), bowing action: 99.12% (loss value 0.0306), sitting action: 99.41% (loss value 0.0144), squatting action: 99.25% (loss value 0.0346), climbing stairs action: 99.53% (loss value 0.0250), descending stairs action: 99.13% (loss value 0.0257), walking action: 99.31% (loss value 0.0451), running action: 99.53% (loss value 0.0266). The overall accuracy of the validation set is 99.31%, precision is 99.32%, recall is 99.31%, F1 score is 99.31%, and validation loss is 0.0280. All indicators are at a high level, verifying that the multimodal sensing signal fusion perception and recognition method proposed in this invention has good practicality and generalization ability.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A smart clothing sensing and recognition method based on multimodal sensor signal fusion, characterized in that, include: S1. By integrating a nine-axis sensing module into smart clothing, multi-channel raw signals of human body movements are collected. The raw signals include three-axis acceleration signals, three-axis angular velocity signals, and three-axis magnetic field signals. S2. The original signal is denoised using an adaptive dynamic noise suppression algorithm. Then, the denoised signal is subjected to dynamic window detrending and local normalization in sequence to obtain a preprocessed signal. S3. A dual-branch convolutional network is used to extract temporal features from the preprocessed signal to obtain temporal features; The preprocessed signal is subjected to frequency domain feature extraction using adaptive discrete transformation to obtain frequency domain features; the preprocessed signal is subjected to semantic feature extraction using a pre-trained large model to obtain large model semantic features. S4. Employ a dynamic weighted fusion mechanism to weight and fuse the time-domain features, frequency-domain features, and large-model semantic features based on feature contribution and category correlation to obtain fused features. S5. Input the fused features into a multi-layer adaptive classification network, and after regularization and probability mapping, output the human action recognition result.

2. The method according to claim 1, characterized in that, The adaptive dynamic noise suppression algorithm in step S2 specifically includes: A fixed-length sliding window is used to perform local noise statistics on the original signal, and the normalized noise level is calculated. The formula is: ； in, This represents the variance of amplitude fluctuation; This is the cross-channel consistency coefficient. , This represents the pairwise combination number of the nine-channel signals. The nine channels are a 9-dimensional data vector composed of three sets of sensor data: three-axis acceleration, three-axis angular velocity, and three-axis magnetic field. Let be the covariance of the signals in channels i and j. These are the standard deviations of the corresponding channel signals; As a time-series stationarity indicator, , To fix the length of the sliding window, Let c be the signal value of the c-th channel at time t; These are nonlinear mapping coefficients; Based on the noise level The dynamic filter kernel parameters are adaptively determined, and the normalization formula for the filter kernel is as follows: ； in, The coefficients of the filter kernel at position p are... The filter kernel length is adaptively adjusted according to the noise level. The original weight distribution function of the filter kernel is divided into three forms: Gaussian, Gaussian-uniform mixture, and uniform. The global attenuation coefficient of the filter kernel is adaptively adjusted according to the noise level; , This is the internal position index of the filter kernel; With the filter kernel Multidimensional convolution operation is performed on the nine-channel signal, and second-order differential regularization and inter-channel difference compensation are introduced to complete cross-channel collaborative filtering and obtain the filtered signal.

3. The method according to claim 1, characterized in that, The dynamic window detrending process in step S2 specifically includes: Calculate the rate of change of the signal , To characterize the comprehensive quantitative index of the fluctuation intensity of the filtered signal, based on the filtered signal The mean of the time-by-time difference amplitude is obtained by averaging it over the entire time series, and the calculation formula is as follows: ； in, The length of the time series corresponding to the total sampling duration of the filtered signal; Let be the amplitude of the filtered signal at time t; , These represent the signal amplitudes of the filtered signal at time points t-1 and t-2, respectively. Then, dynamic window length adaptation is performed, and the window length is adjusted accordingly. With rate of change Adaptive adjustment, calculated using the following formula: ； in, The length of the dynamically sliding window. This is a rounding function to ensure that the window length is a positive integer. The rate of change of the signal; The preset baseline rate of change; Generate nonlinear weighting coefficients, and assign the weights of each historical time point within the window using a nonlinear mapping function, as shown in the following formula: ； in, Here, m represents the dynamic weighting coefficient, and m is the time offset index within the window. For summation index; Perform precise trend stripping to remove trend signals. The calculation formula is as follows: ； in, The signal amplitude after detrending. Let be the amplitude of the filtered signal at time t. The trend term of the dynamic weighted fitting is obtained by summing the products of the signals at each time point within the window and the corresponding weighting coefficients.

4. The method according to claim 3, characterized in that, The local standardization process in step S2 specifically includes: based on Calculate the dynamically weighted local sliding standard deviation The formula is as follows: ； Where m is the time offset within the window, and its value ranges from 1 to 2. ; The weighting coefficients are dynamic, based on the rate of change of the signal. The independent variable is adaptively adjusted according to the window position m; Detrending signals at historical moments The instantaneous value at that point; The dynamic sliding window length is the rate of change of the signal at time t. Adaptive adjustment; This is the dynamically weighted local moving average of the detrended signal; Based on the calculated normalized noise level For the dynamically weighted local sliding standard deviation Perform adaptive noise calibration to obtain the calibrated standard deviation. ; Then perform a uniform scaling mapping on the detrended signal, as shown in the formula: ； in, This is a preprocessed signal after local standardization. To prevent numerical stability terms from being divided by zero.

5. The method according to claim 1, characterized in that, In step S3, the dual-branch convolutional network extracts temporal features from the preprocessed signal, specifically including: The first branch uses variable dilation rate dilated convolution to extract long-term dependent features, with the following formula: ； in, The long-term dependency features output by the first branch. For variable dilation rate dilated convolution operation, The kernel size is [size]. The dynamic expansion rate is adaptively adjusted by the signal timing span. For batch normalization operations, It is a non-linear activation function; This refers to the preprocessed signal at time point t; The second branch uses depthwise separable convolution to extract short-term local features, as shown in the formula: ； in, The short-time local features output by the second branch. For depthwise separation convolution operations, To separate the kernel size; Then, the features from the two branches are dynamically weighted and fused using a channel attention mechanism, as shown in the formula: ； ； in, This is the final output of temporal feature extraction. For adaptive attention weights, This is the time complexity evaluation function. This is the weighting adjustment coefficient. This is a bias term.

6. The method according to claim 5, characterized in that, In step S3, the adaptive discrete transform extracts frequency domain features from the preprocessed signal, specifically through an adaptive basis function. The preprocessed signal is mapped to the frequency domain using the following formula: ； in, For the transform domain The frequency domain signal corresponding to each frequency component. For transform domain frequency index, each The amplitude and phase information corresponding to a frequency point For preprocessing signals Total number of sampling points This represents the sampling point index of the time-domain signal, with values ​​ranging from 1 to T. The preprocessed signal amplitude corresponding to the nth sampling point. For frequency-dependent components Adaptively adjusted basis function parameters; The specific expression is: ； in The imaginary unit satisfies ; The formula for calculating the frequency-adaptive window width parameter is as follows: ； in The base window width, This is the window width adjustment factor, with a default value of 0.5 and a range of values ​​of [missing value]. This enables adaptive time-frequency resolution matching with a wide window in the low-frequency band and a narrow window in the high-frequency band. It is a natural exponential function; The normalization coefficient is used to ensure energy conservation; Extracting frequency domain feature vectors from frequency domain signals , Represents the frequency domain energy entropy. Indicates the dominant frequency bandwidth. Represents the phase consistency coefficient. Indicates the attenuation rate of high-frequency components. This indicates other frequency domain features that can be subsequently expanded.

7. The method according to claim 1, characterized in that, In step S3, the pre-trained large model extracts semantic features from the preprocessed signal, specifically including: After the preprocessed signal is reshaped into a sequence, it is then processed by the channel adaptation matrix. Mapping to the feature space yields the adapted features. The formula is: ； in, The reconstructed sequence signal, To adapt the weight matrix to the channel, For bias terms; Will Inputting the pre-trained temporal feature encoding network yields high-dimensional semantic features. The formula is: ； in, This is a pre-trained temporal feature encoding network with a multi-layer Transformer encoder structure, where each layer contains a multi-head self-attention module and a feedforward neural network module; These are the pre-trained parameters.

8. The method according to claim 1, characterized in that, The dynamic weight fusion mechanism in step S4 specifically includes: For time domain features and frequency domain features Perform linear transformations separately to match the semantic features of the large model. Dimensional alignment yields alignment features. ; Based on feature contribution and category relevance, an attention mechanism is used to dynamically calculate the fusion weights of temporal features, frequency features, and semantic features of the large model. ,satisfy And it adjusts dynamically according to the input features; The final weighted fusion formula is: ； in, This is a feature of fusion.

9. The method according to claim 1, characterized in that, The processing procedure of the multi-layer adaptive classification network in step S5 specifically includes: Fusion features The input is a multi-layer adaptive classification network, which is then mapped through a fully connected layer to obtain high-level semantic features. ; Finally, the recognition probability of each action category is output through Softmax probability mapping: ； in, In order to integrate features Under these conditions, the sample belongs to the action category. The probability, The total number of action categories. The output features of the last fully connected layer in the multi-layer adaptive classification network. , For category The corresponding output layer weight matrix, , For category The corresponding output layer bias term, , For category calibration items, () is the natural exponential function.

10. The method according to claim 1, characterized in that, In step S5, the training of the multi-layer adaptive classification network employs a cross-entropy loss function combined with weight decay regularization. The network parameters are iteratively optimized through backpropagation until the loss function converges. During training, the weight matrix of the multi-layer adaptive classification network is subjected to... Regularization constraints are used to suppress overfitting and improve the model's ability to generalize to unseen action samples.

Images