A Method and System for Fusion of Somatosensory Temporal Data Based on Cross-Modal Attention Mechanism

Abstract

This invention relates to the field of data fusion analysis, and more particularly to a method and system for fusion of somatosensory temporal data based on a cross-modal attention mechanism. This invention can dynamically learn the correlation and weights of different modal data at different time steps and fuse them to generate a unified somatosensory state representation. Specifically, it acquires comprehensive somatosensory data from a combination of heterogeneous sensors for several training actions corresponding to a monitoring target in real time, extracting the output synchronization deviation characteristics of each category of somatosensory data corresponding to each training action; evaluates the synchronization deviation representation parameters of the comprehensive somatosensory data to classify the degree of synchronization deviation of the comprehensive somatosensory data; and adaptively evaluates and performs fusion analysis on the comprehensive somatosensory data. This invention enables dynamic alignment and information fusion of asynchronous temporal data from different modalities, improving the robustness and accuracy of dynamic perception of human physiological signals.

Description

Technical Field

[0001] This invention relates to the field of data fusion analysis, and in particular to a method and system for fusion of somatosensory temporal data based on a cross-modal attention mechanism. Background Technology

[0002] With the iteration of sensor technology, the application of heterogeneous sensor combinations is becoming increasingly widespread. By using sensors of different modalities, such as inertial measurement units (IMUs), electromyography (EMG) sensors, and pressure sensors, precise acquisition of human inertial motion data, EMG force data, and contact force data can be achieved. This type of somatosensory data has significant temporal characteristics; its output is highly synchronized with the human movement execution process and physiological state changes, serving as the core basis for dynamic analysis of human condition. However, differences in sampling frequency and data transmission delay among different sensor modalities make it difficult to accurately align the temporal data on the time axis. Furthermore, human somatosensory data is affected by individual movement habits, physiological differences, and environmental interference, resulting in a highly dynamic and uncertain data distribution, further increasing the difficulty of cross-modal data fusion.

[0003] The core requirement of cross-modal data fusion is to break down the heterogeneity barriers between different modalities and achieve effective information complementarity. Cross-modal data fusion technology has become a core means of capturing the state characteristics of complex objects, especially in the scenario of human somatic state monitoring. By integrating multi-dimensional sensor data, it can more comprehensively reflect the overall state of human movement, physiological response, etc., and provide support for applications such as training optimization, rehabilitation assessment, and health management.

[0004] Furthermore, the attention mechanism can adaptively learn the importance weights of information across different modalities and time steps, enhancing the contribution of key information and suppressing the interference of redundant information and noise. Based on this, combining the attention mechanism with cross-modal time-series data fusion aims to solve the alignment problem of heterogeneous data and the dynamic weight allocation problem, and promote the development of somatic state monitoring technology towards higher accuracy and stronger robustness.

[0005] Chinese Patent Application Publication No. CN118340607A discloses an adaptive below-knee prosthesis socket based on somatosensory feedback and its control method, including the following control steps: (A) Real-time acquisition of EEG, sEMG, and interface pressure signals of the prosthesis wearer using an EEG-EMG-interface force multimodal somatosensory signal acquisition system, and denoising of the acquired EEG and sEMG signals using wavelet transform, adaptive filtering, and independent component analysis; (B) Extraction of feature parameters of sEMG signals under muscle fatigue state, EEG signals under skin pain state, and interface pressure signals; (C) Based on information fusion technology, combined with a deep fusion method of EEG, sEMG, and interface pressure signal feature parameters, obtaining fused feature parameters of sEMG signals under muscle fatigue state, EEG signals under skin pain state, and interface pressure signals; (D) Construction of a classification and recognition feature matrix of EEG, sEMG, and interface pressure signals using the fused feature parameters in step (C), obtaining a real-time collaborative control method for the prosthesis socket based on multimodal somatosensory feedback signals of EEG, sEMG, and interface force.

[0006] However, the following problems still exist in the existing technology.

[0007] In the process of fusing multi-source data, the focus is often on feature extraction and simple splicing in a single dimension, without fully considering the differences in sampling frequency, data transmission delay, etc. of different modal sensors. This makes it difficult to accurately align and effectively fuse time-series data on the time axis. Summary of the Invention

[0008] To address this, the present invention provides a somatosensory temporal data fusion method and system based on a cross-modal attention mechanism, which overcomes the problem in the prior art that, during the fusion of multi-source data, the focus is often on single-dimensional feature extraction and simple splicing, failing to fully consider the differences in sampling frequency, data transmission delay, etc., of different modal sensors, resulting in the difficulty of accurately aligning and effectively fusing temporal data on the time axis.

[0009] To achieve the above objectives, this invention provides a method for fusion of somatosensory temporal data based on a cross-modal attention mechanism, comprising:

[0010] Real-time acquisition of comprehensive somatosensory data of several training actions corresponding to the monitoring target collected by a combination of heterogeneous sensors, in order to extract the output synchronization deviation characteristics of each category of somatosensory data corresponding to each training action. The output synchronization deviation characteristics include a first ratio of the number of overlapping timestamps to the total number of output timestamps and a second ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum single category of somatosensory data.

[0011] The synchronization deviation characterization parameters of the integrated somatosensory data are evaluated based on the output synchronization deviation characteristics to classify the degree of synchronization deviation of the integrated somatosensory data.

[0012] Based on the aforementioned synchronization deviation level category, the integrated somatosensory data is evaluated and fused for analysis, including:

[0013] Based on the time interval between the overlapping timestamps corresponding to the integrated somatosensory data, the temporal deviation characteristics of completing a single training action are determined, so as to calculate the temporal deviation degree characterization value for the integrated somatosensory data and determine whether it meets the temporal output deviation benchmark.

[0014] Identify the tightness of the connection between the comprehensive somatosensory data in the temporal sequence, adjust the weight coefficients of each category of somatosensory data to adjust the interpolation granularity corresponding to each category of somatosensory data, put the adjusted somatosensory data of each category into the somatosensory state representation model, and fuse them to generate a somatosensory state representation.

[0015] Alternatively, maintain the initial weight coefficients of each category of somatic sensory data, input them into the somatic sensory state representation model, and fuse them to generate a somatic sensory state representation;

[0016] The time-series deviation characteristics include the uniformity of the time interval and the maximum difference in the output data volume of each category of somatosensory data within the shortest time interval.

[0017] Furthermore, the process of evaluating the synchronization deviation characterization parameters of the integrated somatosensory data includes:

[0018] The ratio of the first ratio of the number of overlapping timestamps to the total number of output timestamps to the first ratio threshold is used as the first synchronization deviation feature;

[0019] The ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum number of single-type somatosensory data and the second ratio threshold are used as the second synchronization deviation feature;

[0020] The sum of the characteristics of the first synchronization deviation and the second synchronization deviation is used as the synchronization deviation characterization parameter.

[0021] Furthermore, the degree of synchronization deviation of the integrated somatosensory data is categorized, including:

[0022] If the synchronization deviation characterization parameter of the integrated somatosensory data is less than the synchronization deviation characterization parameter threshold, then the synchronization deviation level of the integrated somatosensory data is classified into the high deviation level category.

[0023] If the synchronization deviation characterization parameter of the integrated somatosensory data is greater than or equal to the synchronization deviation characterization parameter threshold, then the synchronization deviation level of the integrated somatosensory data is classified into the low deviation level category.

[0024] Furthermore, based on the category of synchronization deviation, the integrated somatosensory data is evaluated and fused, including,

[0025] If the comprehensive somatosensory data is classified as high-biased, the temporal deviation characteristics of completing a single training action are determined based on the time interval between the overlapping timestamps corresponding to the comprehensive somatosensory data, so as to calculate the temporal deviation characterization value for the comprehensive somatosensory data and determine whether it meets the temporal output deviation benchmark.

[0026] Identify the tightness of the connection between the comprehensive somatosensory data in the temporal sequence, adjust the weight coefficients of each category of somatosensory data to adjust the interpolation granularity corresponding to each category of somatosensory data, put the adjusted somatosensory data of each category into the somatosensory state representation model, and fuse them to generate a somatosensory state representation.

[0027] If the overall somatic sensation data belongs to the low-bias category, then the initial weight coefficients of each category of somatic sensation data are maintained, and the data are incorporated into the somatic sensation state representation model to generate a somatic sensation state representation.

[0028] Further, the process of calculating the temporal deviation characterization value for the integrated somatosensory data includes:

[0029] The ratio of the uniformity of the time interval to the uniformity threshold is used as the first deviation characteristic.

[0030] The ratio of the maximum difference in output data volume of each category of somatosensory data within the shortest time interval to the threshold of the maximum difference in output data volume is used as the second deviation feature;

[0031] The sum of the first deviation degree feature and the second deviation degree feature is used as the time series deviation degree characterization value.

[0032] Further, it is determined whether the timing output deviation benchmark is met, including:

[0033] If the temporal deviation characterization value of the somatosensory data is less than the temporal deviation characterization threshold, it is determined that it meets the temporal output deviation benchmark.

[0034] Furthermore, the process of identifying the tightness of the temporal relationship between the integrated somatosensory data includes:

[0035] Call the timestamp data corresponding to the output of the integrated somatosensory data;

[0036] Based on the timestamp data, determine the average output interval and the magnitude of change in the output interval between any two categories of somatosensory data in the time sequence output;

[0037] The sum of the ratio of the average output interval to the average output interval threshold and the ratio of the output interval variation magnitude to the output interval variation magnitude threshold is taken as the connection tightness.

[0038] Furthermore, the process of adjusting the weighting coefficients for each category of somatosensory data includes:

[0039] Based on the tightness of the connection between any two categories of somatic data, the weight coefficients of the two categories of somatic data are reduced, and the weight coefficient of the other category of somatic data is increased;

[0040] The adjustment amount of the weighting coefficient is positively correlated with the tightness of connection;

[0041] The adjustment gradient of the weight coefficients is preset.

[0042] Furthermore, the interpolation granularity corresponding to each category of somatosensory data is adjusted, including:

[0043] Increase the interpolation granularity corresponding to the two categories of somatosensory data, and the increase in interpolation granularity is positively correlated with the corresponding weight coefficient;

[0044] Maintain the initial interpolation granularity of the other category of somatosensory data.

[0045] Furthermore, a system is also provided that applies a somatosensory temporal data fusion method based on a cross-modal attention mechanism, including,

[0046] The somatosensory acquisition module is used to acquire in real time the comprehensive somatosensory data of several training actions corresponding to the monitoring target collected by the combination of heterogeneous sensors, so as to extract the output synchronization deviation characteristics of each category of somatosensory data corresponding to each training action. The output synchronization deviation characteristics include a first ratio of the number of overlapping timestamps to the total number of output timestamps and a second ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum single category of somatosensory data.

[0047] A synchronization analysis module, which is connected to the somatosensory acquisition module, is used to evaluate the synchronization deviation characterization parameters of the integrated somatosensory data based on the output synchronization deviation characteristics, so as to classify the degree of synchronization deviation of the integrated somatosensory data.

[0048] An evaluation fusion module, connected to the synchronization analysis module, is used to perform evaluation fusion analysis on the integrated somatosensory data based on the category of synchronization deviation, including:

[0049] Based on the time interval between the overlapping timestamps corresponding to the integrated somatosensory data, the temporal deviation characteristics of completing a single training action are determined, so as to calculate the temporal deviation degree characterization value for the integrated somatosensory data and determine whether it meets the temporal output deviation benchmark.

[0050] Identify the tightness of the connection between the comprehensive somatosensory data in the temporal sequence, adjust the weight coefficients of each category of somatosensory data to adjust the interpolation granularity corresponding to each category of somatosensory data, put the adjusted somatosensory data of each category into the somatosensory state representation model, and fuse them to generate a somatosensory state representation.

[0051] Alternatively, maintain the initial weight coefficients of each category of somatic sensory data, input them into the somatic sensory state representation model, and fuse them to generate a somatic sensory state representation;

[0052] The time-series deviation characteristics include the uniformity of the time interval and the maximum difference in the output data volume of each category of somatosensory data within the shortest time interval.

[0053] Compared with existing technologies, this invention includes a somatosensory acquisition module, which acquires in real time comprehensive somatosensory data of several training actions corresponding to a monitoring target from a combination of heterogeneous sensors, to extract the output synchronization deviation characteristics of each category of somatosensory data corresponding to each training action; a synchronization analysis module, connected to the somatosensory acquisition module, evaluates the synchronization deviation characterization parameters of the comprehensive somatosensory data based on the output synchronization deviation characteristics, to classify the degree of synchronization deviation of the comprehensive somatosensory data; and an evaluation and fusion module, connected to the synchronization analysis module, performs evaluation and fusion analysis on the comprehensive somatosensory data according to the degree of synchronization deviation. This invention can achieve dynamic alignment and information fusion of asynchronous time-series data from different modalities, improving the robustness and accuracy of dynamic perception of human physiological signals.

[0054] In particular, this invention considers the potential differences in sampling frequency and output rhythm among heterogeneous sensor combinations in real-world scenarios. It flexibly adapts to these differences through a dual ratio system, accurately capturing data synchronization issues between different sensor combinations. While covering the overall synchronization range, it focuses on key data, namely the synchronization completeness of the minimum number of single-type somatosensory data. Specifically, the first ratio quantifies the overall coverage of cross-modal data synchronization, reflecting the proportion of data with "completely aligned timestamps" among the three types of somatosensory data in all output data. The second ratio quantifies the relative completeness of synchronized data, focusing on the single type of somatosensory data with the least output data volume, avoiding synchronization misjudgments caused by missing data of a certain type, and reflecting the proportion of this type of data with timestamp alignment with the other two types. Therefore, this invention combines the aforementioned two features to evaluate the synchronization deviation characterization parameters of comprehensive somatosensory data, assessing the alignment degree of cross-modal somatosensory data on the time axis and the severity of data synchronization deviation, characterizing the synchronization quality level of cross-modal somatosensory data, providing data support for classifying corresponding synchronization deviation categories, and thus providing a reliable basis for subsequent differentiated fusion strategies. This invention can achieve multi-dimensional and accurate characterization of synchronization deviation.

[0055] In particular, this invention focuses on the time span of a single training movement completed by the monitoring target, specifically analyzing the interval characteristics of overlapping timestamps. This aligns with the continuity and phased nature of movements in sports training, accurately identifying temporal deviations during movement execution. The uniformity of time intervals reflects the stability of the cross-modal data output rhythm, quantifying the stability of time intervals between overlapping timestamps. The maximum difference in output data volume for each category of somatosensory data within the shortest time interval quantifies the output density differences of different categories of somatosensory data corresponding to key movement phases, reflecting the data matching degree of key movement phases. Based on this, this invention further refines the evaluation of the temporal consistency of cross-modal data generated for a single training movement. Dual features comprehensively cover the core scenarios of temporal deviations, reflecting both the stability of data temporal sequence throughout the overall movement and focusing on the data synchronization quality of key movement phases, comprehensively evaluating the temporal deviation status of cross-modal data within a single training movement. Furthermore, it calculates a characterization value for the degree of temporal deviation of comprehensive somatosensory data to represent the overall degree of temporal deviation in the generated cross-modal data, providing data support for subsequent determination of whether it meets the temporal output deviation benchmark, and improving the accuracy and reliability of cross-modal data fusion.

[0056] In particular, this invention focuses on the temporal continuity of cross-modal data output, calculating the continuity tightness using both average output interval and output interval variation amplitude as indicators, reflecting the overall synchronization rhythm of the two types of somatosensory data output. Simultaneously, it captures the stability of interval fluctuations based on the output interval variation amplitude. This comprehensive evaluation based on these two features avoids adjustment biases caused by subjective judgment, ensuring clear data support for adjustments to weights and interpolation granularity. Furthermore, it indirectly reflects the temporal reliability and synchronization adaptability of the other type of somatosensory data. Therefore, this invention avoids the problem of synchronization failure of multi-type somatosensory data through a "reducing two types and improving one type" strategy, ensuring that the fusion result is based on high-quality data and reducing interference from low-quality data. By accurately characterizing the temporal continuity quality and adjusting fusion parameters in a coordinated manner, this invention significantly improves the accuracy and reliability of the somatosensory state representation generated in high-bias scenarios, achieving dynamic adaptive fusion of cross-modal somatosensory data. Attached Figure Description

[0057] Figure 1 This is a schematic diagram illustrating the steps of a somatosensory temporal data fusion method based on a cross-modal attention mechanism, as described in an embodiment of the invention.

[0058] Figure 2 A logic decision diagram for classifying the degree of synchronization deviation of somatosensory data in an embodiment of the invention;

[0059] Figure 3 This is a logic diagram for determining whether a timing output deviation benchmark is met in an embodiment of the invention.

[0060] Figure 4This is a functional block diagram of a system for a somatosensory temporal data fusion method based on a cross-modal attention mechanism, as described in an embodiment of the invention. Detailed Implementation

[0061] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0062] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0063] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0064] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0065] Please see Figure 1 The diagram illustrates the steps of a somatosensory temporal data fusion method based on a cross-modal attention mechanism according to an embodiment of the present invention. The somatosensory temporal data fusion method based on a cross-modal attention mechanism according to an embodiment of the present invention includes:

[0066] Step S1: In real time, acquire comprehensive somatosensory data of several training actions corresponding to the monitoring target collected by the combination of heterogeneous sensors, so as to extract the output synchronization deviation characteristics of each category of somatosensory data corresponding to each training action. The output synchronization deviation characteristics include a first ratio of the number of overlapping timestamps to the total number of output timestamps and a second ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum single category of somatosensory data.

[0067] Step S2: Evaluate the synchronization deviation characterization parameters of the integrated somatosensory data based on the output synchronization deviation characteristics, so as to classify the synchronization deviation degree category of the integrated somatosensory data;

[0068] Step S3, based on the category of synchronization deviation, performs an evaluation and fusion analysis on the integrated somatosensory data, including,

[0069] Based on the time interval between the overlapping timestamps corresponding to the integrated somatosensory data, the temporal deviation characteristics of completing a single training action are determined, so as to calculate the temporal deviation degree characterization value for the integrated somatosensory data and determine whether it meets the temporal output deviation benchmark.

[0070] Identify the tightness of the connection between the comprehensive somatosensory data in the temporal sequence, adjust the weight coefficients of each category of somatosensory data to adjust the interpolation granularity corresponding to each category of somatosensory data, put the adjusted somatosensory data of each category into the somatosensory state representation model, and fuse them to generate a somatosensory state representation.

[0071] Alternatively, maintain the initial weight coefficients of each category of somatic sensory data, input them into the somatic sensory state representation model, and fuse them to generate a somatic sensory state representation;

[0072] The time-series deviation characteristics include the uniformity of the time interval and the maximum difference in the output data volume of each category of somatosensory data within the shortest time interval.

[0073] Specifically, the comprehensive somatosensory data includes three categories of somatosensory data: inertial motion data, electromyographic force data, and contact force data.

[0074] Among them, three types of somatosensory data are collected by combining heterogeneous sensors: IMU sensor to collect inertial motion data, EMG sensor to collect electromyographic force data, and pressure sensor to collect contact force data.

[0075] Specifically, the somatic state representation refers to a unified digital description that accurately reflects the monitoring target, namely the real-time physiological and motor state of the human body, formed by cross-modal time-series data fusion.

[0076] Specifically, cross-modal data integration comprehensively captures the somatic state of the monitored target, including movement posture, muscle exertion, and contact force, avoiding the information limitations of single data types and making the assessment of the training recovery degree of the monitored target more comprehensive.

[0077] Specifically, this invention effectively filters out synchronization errors and temporal noise from heterogeneous sensor data through multi-dimensional deviation correction and dynamic fusion, resulting in higher quality data input to the somatic state representation model. The final fused somatic state representation is more closely aligned with the actual physical state of the monitored target, providing a reliable basis for subsequent training guidance or rehabilitation program adjustments.

[0078] Specifically, the process of evaluating the synchronization deviation characterization parameters of the integrated somatosensory data includes:

[0079] The ratio of the first ratio of the number of overlapping timestamps to the total number of output timestamps to the first ratio threshold is used as the first synchronization deviation feature;

[0080] The ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum number of single-type somatosensory data and the second ratio threshold are used as the second synchronization deviation feature;

[0081] The sum of the characteristics of the first synchronization deviation and the second synchronization deviation is used as the synchronization deviation characterization parameter.

[0082] In this embodiment, the purpose of setting the first ratio threshold and the second ratio threshold is to characterize the situation where the alignment of cross-modal somatosensory data on the time axis is low, the severity of data synchronization deviation is high, and the synchronization quality level of cross-modal somatosensory data is poor. By acquiring historical comprehensive somatosensory data for the same training action corresponding to several monitoring targets that have completed training, the historical data of the first ratio of the number of overlapping timestamps to the total number of output timestamps, and the historical data of the second ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum single-type somatosensory data are called to solve the mean of the first ratio and the mean of the second ratio, and the corresponding values ​​are used as the benchmark values ​​under normal circumstances. Based on the purpose of setting the above two thresholds, the first ratio threshold is determined as the product of the mean of the first ratio and the first deviation coefficient, and the second ratio threshold is determined as the product of the mean of the second ratio and the second deviation coefficient. The first deviation coefficient is selected in the interval [0.9, 0.95], preferably 0.9 in the implementation, and the second deviation coefficient is selected in the interval [0.9, 0.95], preferably 0.9 in the implementation.

[0083] Specifically, this invention considers the potential differences in sampling frequency and output rhythm among heterogeneous sensor combinations in real-world scenarios. It flexibly adapts to these differences using a dual ratio system to accurately capture data synchronization issues between different sensor combinations. While covering the overall synchronization range, it focuses on key data, namely the synchronization completeness of the minimum number of single-type somatosensory data. The first ratio quantifies the overall coverage of cross-modal data synchronization, reflecting the proportion of data with "completely aligned timestamps" among the three types of somatosensory data in all output data; a higher proportion indicates stronger overall synchronization. The second ratio quantifies the relative completeness of synchronized data, focusing on the single type of somatosensory data with the least output data volume. This avoids synchronization misjudgments caused by missing data of a certain type, reflecting the proportion of timestamp alignment between this type of data and the other two types; a higher proportion indicates more reliable synchronization quality of the core data. Therefore, this invention combines the aforementioned two features to evaluate the synchronization deviation characterization parameters of comprehensive somatosensory data, assessing the alignment degree of cross-modal somatosensory data on the time axis and the severity of data synchronization deviation, characterizing the synchronization quality level of cross-modal somatosensory data, providing data support for classifying corresponding synchronization deviation categories, and thus providing a reliable basis for subsequent differentiated fusion strategies. This invention enables precise multi-dimensional characterization of synchronization deviation.

[0084] Specifically, please refer to Figure 2 As shown, this is a logic determination diagram for classifying the degree of synchronization deviation of the comprehensive somatic data according to an embodiment of the present invention. Classifying the degree of synchronization deviation of the comprehensive somatic data includes:

[0085] If the synchronization deviation characterization parameter of the integrated somatosensory data is less than the synchronization deviation characterization parameter threshold, then the synchronization deviation level of the integrated somatosensory data is classified into the high deviation level category.

[0086] If the synchronization deviation characterization parameter of the integrated somatosensory data is greater than or equal to the synchronization deviation characterization parameter threshold, then the synchronization deviation level of the integrated somatosensory data is classified into the low deviation level category.

[0087] The synchronization deviation characterization parameter threshold is predetermined. It is determined by calculating the synchronization deviation characterization parameter threshold when the first ratio of the number of overlapping timestamps to the total number of output timestamps and the first ratio threshold are equal, and the second ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum single-type somatosensory data is equal to the second ratio threshold.

[0088] Specifically, based on the category of synchronization deviation, the integrated somatosensory data is evaluated and fused, including,

[0089] If the comprehensive somatosensory data is classified as high-biased, the temporal deviation characteristics of completing a single training action are determined based on the time interval between the overlapping timestamps corresponding to the comprehensive somatosensory data, so as to calculate the temporal deviation characterization value for the comprehensive somatosensory data and determine whether it meets the temporal output deviation benchmark.

[0090] Identify the tightness of the connection between the comprehensive somatosensory data in the temporal sequence, adjust the weight coefficients of each category of somatosensory data to adjust the interpolation granularity corresponding to each category of somatosensory data, put the adjusted somatosensory data of each category into the somatosensory state representation model, and fuse them to generate a somatosensory state representation.

[0091] If the overall somatic sensation data belongs to the low-bias category, then the initial weight coefficients of each category of somatic sensation data are maintained, and the data are incorporated into the somatic sensation state representation model to generate a somatic sensation state representation.

[0092] Specifically, this invention sets up a differentiated analysis strategy. In high-bias scenarios, high-quality data dominates the fusion result through weight adjustment, avoiding the error amplification caused by fusion of data with high synchronization deviations using fixed weights. In low-bias scenarios, efficient fusion is maintained, improving the overall accuracy of the output result.

[0093] Specifically, the process of calculating the temporal deviation characterization value for the integrated somatosensory data includes:

[0094] The ratio of the uniformity of the time interval to the uniformity threshold is used as the first deviation characteristic.

[0095] The ratio of the maximum difference in output data volume of each category of somatosensory data within the shortest time interval to the threshold of the maximum difference in output data volume is used as the second deviation feature;

[0096] The sum of the first deviation degree feature and the second deviation degree feature is used as the time series deviation degree characterization value.

[0097] In this embodiment, the purpose of setting a uniformity threshold is to characterize the poor stability of data timing during the overall action process, and the purpose of setting a maximum difference threshold for output data volume is to characterize the low data synchronization quality during the core action phase. By acquiring historical comprehensive somatosensory data for the same training action corresponding to several monitoring targets that have completed training, calling historical uniformity data of time intervals, and historical data of the maximum difference in output data volume of each category of somatosensory data within the shortest time interval, the mean of uniformity and the mean of the maximum difference in output data volume are solved, and these are used as the benchmark values ​​under normal conditions. Based on the purpose of setting the above two thresholds, the uniformity threshold is determined as the product of the mean of uniformity and the uniformity deviation coefficient, and the maximum difference threshold for output data volume is determined as the product of the mean of the maximum difference in output data volume and the output deviation coefficient. The uniformity deviation coefficient is selected in the interval [1.1, 1.2], preferably 1.1 in the implementation, and the output deviation coefficient is selected in the interval [1.2, 1.4], preferably 1.2 in the implementation.

[0098] The uniformity is calculated by taking the standard deviation of the time interval. The smaller the standard deviation, the more uniform the time interval. This will not be elaborated further.

[0099] Specifically, this invention focuses on monitoring the time span of a single training movement completed by the target, specifically analyzing the interval characteristics of overlapping timestamps, and aligning with the continuity and phased characteristics of movements in sports training to accurately identify temporal deviations during movement execution. The uniformity of time intervals reflects the stability of the cross-modal data output rhythm and quantifies the stability of time intervals between overlapping timestamps. A smaller uniformity indicates that various sensor data are output synchronously at a stable rhythm, with more regular temporal connections; conversely, a larger uniformity indicates that the data output is inconsistent, with significant temporal fluctuations. The maximum difference in the output data volume of each type of somatosensory data within the shortest time interval quantifies the output density differences of different types of somatosensory data corresponding to key movement phases, reflecting the data matching degree of key movement phases. A smaller maximum difference indicates consistent output density of each type of somatosensory data in key stages, with stronger information synchronization; conversely, a larger maximum difference indicates a lower degree of data acquisition synchronization in key movement phases, with information gaps or redundancy. Based on this, the present invention further refines the evaluation of the temporal consistency of cross-modal data generated for a single training action. Dual features comprehensively cover the core scenarios of temporal deviation, reflecting both the stability of data temporal sequence throughout the overall action and focusing on the data synchronization quality during key action phases, thus comprehensively evaluating the temporal deviation status of cross-modal data within a single training action. Furthermore, a characterization value for the degree of temporal deviation of the integrated somatosensory data is calculated to represent the overall degree of temporal deviation in the generated cross-modal data. This provides data support for subsequent determination of whether the data meets the temporal output deviation benchmark, improving the accuracy and reliability of cross-modal data fusion.

[0100] Specifically, please refer to Figure 3 As shown, this is a logic diagram for determining whether the timing output deviation benchmark is met according to an embodiment of the present invention. The determination of whether the timing output deviation benchmark is met includes:

[0101] If the temporal deviation characterization value of the somatosensory data is less than the temporal deviation characterization threshold, it is determined that it meets the temporal output deviation benchmark.

[0102] If the temporal deviation characterization value of the somatosensory data is greater than or equal to the temporal deviation characterization threshold, it is determined that it does not meet the temporal output deviation benchmark.

[0103] The threshold for characterizing the degree of temporal deviation is predetermined. The time deviation characterization value calculated is determined as the time deviation characterization threshold when the uniformity of the time interval is equal to the uniformity threshold and the maximum difference in the output data volume of each category of somatosensory data within the shortest time interval is equal to the maximum difference in the output data volume threshold.

[0104] Specifically, the process of identifying the temporal coherence of the integrated somatosensory data includes:

[0105] Call the timestamp data corresponding to the output of the integrated somatosensory data;

[0106] Based on the timestamp data, determine the average output interval and the magnitude of change in the output interval between any two categories of somatosensory data in the time sequence output;

[0107] The sum of the ratio of the average output interval to the average output interval threshold and the ratio of the output interval variation magnitude to the output interval variation magnitude threshold is taken as the connection tightness.

[0108] In this embodiment, the purpose of setting the average output interval threshold and the output interval variation amplitude threshold is to characterize the situation where the overall synchronization rhythm of the output of any two types of somatosensory data is low, the output time lag is serious, and the degree of inconsistency in the synchronization response rhythm is high, while the temporal reliability of the other type of somatosensory data is high and the synchronization adaptability is high. By acquiring the historical comprehensive somatosensory data of the same training action corresponding to several monitoring targets that have completed training, calling the historical data of the average output interval and the historical data of the output interval variation amplitude of the temporal output of any two types of somatosensory data before and after, the mean of the average output interval and the mean of the output interval variation amplitude are calculated, and the corresponding values ​​are used as the benchmark values ​​under normal conditions. Based on the purpose of setting the above two thresholds, the average output interval threshold is determined as the product of the mean of the average output interval and the first offset coefficient, and the output interval variation amplitude threshold is determined as the product of the mean of the output interval variation amplitude and the second offset coefficient. The first offset coefficient is selected in the interval [1.1, 1.2], preferably 1.1 in the implementation, and the second offset coefficient is selected in the interval [1.2, 1.3], preferably 1.2 in the implementation.

[0109] Specifically, the process of adjusting the weighting coefficients for each category of somatosensory data includes:

[0110] Based on the tightness of the connection between any two categories of somatic data, the weight coefficients of the two categories of somatic data are reduced, and the weight coefficient of the other category of somatic data is increased;

[0111] The adjustment amount of the weighting coefficient is positively correlated with the tightness of connection;

[0112] The adjustment gradient of the weight coefficients is preset.

[0113] In this embodiment, optionally,

[0114] The connection tightness is compared with the preset first connection tightness comparison threshold and the second connection tightness comparison threshold.

[0115] When the tightness of the connection is greater than the second tightness comparison threshold, the adjustment amount of the weight coefficient is determined as the first adjustment amount. The first adjustment amount is set to increase or decrease the adjustment gradient by 3 times based on the corresponding initial weight coefficient.

[0116] When the connection tightness is greater than or equal to the first connection tightness comparison threshold and less than or equal to the second connection tightness comparison threshold, the adjustment amount of the weight coefficient is determined to be the second adjustment amount. The second adjustment amount is set to increase or decrease the adjustment gradient by 2 times based on the corresponding initial weight coefficient.

[0117] When the tightness of the connection is less than the first tightness comparison threshold, the adjustment amount of the weight coefficient is determined to be the third adjustment amount. The third adjustment amount is set to increase or decrease the adjustment gradient by 1 times based on the corresponding initial weight coefficient.

[0118] The first connection tightness comparison threshold is 1.1 times the connection tightness threshold, and the second connection tightness comparison threshold is 1.2 times the connection tightness threshold. The adjustment amount includes a decrease and an increase.

[0119] The connection tightness threshold is predetermined. The connection tightness is calculated when the average output interval is equal to the average output interval threshold and the output interval change magnitude is equal to the output interval change magnitude threshold.

[0120] Understandably, the purpose of adjusting the weighting coefficients is to dynamically adapt to the temporal synchronization quality differences of cross-modal somatosensory data. By quantifying the adjustment logic, the contribution of high-quality data is strengthened, while the interference of low-quality data is suppressed, ultimately improving the accuracy and reliability of the fusion results. Furthermore, the adjustment amount of the weighting coefficients can also be adjusted by those skilled in the art under different circumstances.

[0121] Specifically, there are no specific limitations on how the initial weight coefficients for each category of somatosensory data are determined. They can be determined based on the inherent reliability of the sensor, assigning weights based on the sensor's technological maturity, sampling accuracy, and anti-interference capability; that is, sensors with higher accuracy and stronger stability correspond to higher weights for somatosensory data. For example, EMG sensors have high sampling accuracy in muscle state monitoring, so their initial weight can be higher than that of pressure sensors; IMU sensors have strong anti-environment interference capabilities, so their weights can be appropriately tilted. Alternatively, they can be determined based on the applicable scenario. For example, this invention aims to generate a fused somatosensory state representation as the output goal, prioritizing somatosensory data strongly correlated with the evaluation indicators and assigning higher weights. For example, in rehabilitation training scenarios, muscle strength recovery is a core indicator, so EMG data has the highest initial weight; in motion posture correction scenarios, IMU inertial motion data has priority. Of course, those skilled in the art can also use other methods of determination, which will not be elaborated here.

[0122] It is understandable that since the weight coefficients of each category of somatosensory data are based on reasonable benchmarks determined by sensor reliability and scenario requirements, the setting of the adjustment gradient of the weight coefficients should take into account both effectively suppressing low-quality data and preventing the weight distribution from becoming unbalanced due to excessive adjustment gradients, such as a sudden drop in the weight of a certain type of somatosensory data to 0 or a sudden increase to too high, to ensure a smooth transition of the fusion result. Therefore, in this implementation, the adjustment gradient is set to 0.1.

[0123] Specifically, this invention focuses on the temporal continuity of cross-modal data output. It calculates the continuity tightness using both average output interval and the magnitude of output interval variation as indicators, reflecting the overall synchronization rhythm of the two types of somatosensory data output. A smaller average output interval indicates a shorter time difference between the two types of data outputs, resulting in stronger immediacy of synchronization triggering; conversely, a larger average output interval indicates a more severe time lag between the two types of data outputs, leading to less consistent synchronization response rhythms. Simultaneously, it captures the stability of interval fluctuations. A smaller magnitude of output interval variation indicates a stable output time difference between the two types of data, with minimal fluctuations in synchronization; while a larger magnitude of output interval variation indicates inconsistent synchronization between the two types of data, resulting in poorer temporal continuity stability. This comprehensive evaluation based on these two characteristics avoids adjustment biases caused by subjective judgment, providing clear data support for the adjustment of weights and interpolation granularity. Furthermore, it can indirectly reflect the temporal reliability and synchronization adaptability of another type of somatosensory data. If the average output interval and variation of the two types of data are small, indicating strong synchronization, but the overall fused data still has deviations, it can be inferred that the other type of data may have output delays, sampling anomalies, or other issues, resulting in low temporal reliability. Conversely, if the average output interval and variation of the two types of data are large, indicating poor synchronization, but the other type of data can maintain a relatively good temporal connection with either of the other types, it indicates that the other type of data has stronger synchronization adaptability and can be used as high-reliability data to participate in the fusion weight enhancement. Therefore, this invention avoids the problem of synchronization failure of multi-class somatosensory data through a strategy of "reducing two types and enhancing one type," ensuring that the fusion result is based on high-quality data and reducing interference from low-quality data. This invention significantly improves the accuracy and reliability of the somatosensory state representation generated by fusion in high-bias scenarios by accurately characterizing the temporal connection quality and adjusting the fusion parameters in a coordinated manner, achieving dynamic adaptive fusion of cross-modal somatosensory data.

[0124] Specifically, adjusting the interpolation granularity for each category of somatosensory data includes:

[0125] Increase the interpolation granularity corresponding to the two categories of somatosensory data, and the increase in interpolation granularity is positively correlated with the corresponding weight coefficient;

[0126] Maintain the initial interpolation granularity of the other category of somatosensory data.

[0127] Specifically, the purpose of adjusting the interpolation granularity for each category of somatosensory data is to use denser interpolation to compensate for the insufficient information reliability of low-weight somatosensory data, while avoiding excessive interpolation interference with high-quality somatosensory data. Based on this, the increase in interpolation granularity can be determined by increasing the unit interpolation granularity by the same factor, based on the adjustment gradient factor of the adjusted weight coefficients.

[0128] Specifically, please refer to Figure 4 The diagram shown is a functional block diagram of a system for a somatosensory temporal data fusion method based on a cross-modal attention mechanism according to an embodiment of the present invention. A system for applying the somatosensory temporal data fusion method based on a cross-modal attention mechanism is also provided, including:

[0129] The somatosensory acquisition module is used to acquire in real time the comprehensive somatosensory data of several training actions corresponding to the monitoring target collected by the combination of heterogeneous sensors, so as to extract the output synchronization deviation characteristics of each category of somatosensory data corresponding to each training action. The output synchronization deviation characteristics include a first ratio of the number of overlapping timestamps to the total number of output timestamps and a second ratio of the number of overlapping timestamps to the number of timestamps corresponding to the minimum single category of somatosensory data.

[0130] A synchronization analysis module, which is connected to the somatosensory acquisition module, is used to evaluate the synchronization deviation characterization parameters of the integrated somatosensory data based on the output synchronization deviation characteristics, so as to classify the degree of synchronization deviation of the integrated somatosensory data.

[0131] An evaluation fusion module, connected to the synchronization analysis module, is used to perform evaluation fusion analysis on the integrated somatosensory data based on the category of synchronization deviation, including:

[0132] Based on the time interval between the overlapping timestamps corresponding to the integrated somatosensory data, the temporal deviation characteristics of completing a single training action are determined, so as to calculate the temporal deviation degree characterization value for the integrated somatosensory data and determine whether it meets the temporal output deviation benchmark.

[0133] Identify the tightness of the connection between the comprehensive somatosensory data in the temporal sequence, adjust the weight coefficients of each category of somatosensory data to adjust the interpolation granularity corresponding to each category of somatosensory data, put the adjusted somatosensory data of each category into the somatosensory state representation model, and fuse them to generate a somatosensory state representation.

[0134] Alternatively, maintain the initial weight coefficients of each category of somatic sensory data, input them into the somatic sensory state representation model, and fuse them to generate a somatic sensory state representation;

[0135] The time-series deviation characteristics include the uniformity of the time interval and the maximum difference in the output data volume of each category of somatosensory data within the shortest time interval.

[0136] Specifically, there are no restrictions on the specific structure of the somatosensory acquisition module, the synchronous analysis module, and the evaluation fusion module. Each module or its units can be composed of logical components or combinations of logical components. Logical components include field-programmable processors, computers, or microprocessors in computers.

[0137] Specifically, this invention simultaneously covers three types of heterogeneous somatosensory data: inertial motion, electromyographic force, and contact force, rather than fusing a single type of data. Furthermore, it designs deviation assessment and fusion rules specifically for the characteristics of cross-modal data, balancing data diversity and fusion adaptability. At the same time, the linkage mechanism can adapt to cross-modal data of varying temporal quality. Whether in ideal scenarios with good sensor synchronization or complex scenarios with large synchronization deviations and loose temporal connections, it can stably output effective results, adapting to the actual data characteristics of various scenarios such as sports training and rehabilitation monitoring.

[0138] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A somatosensory time series data fusion method based on a cross-modal attention mechanism, characterized in that, The method comprises the following steps: real-time acquisition of comprehensive somatosensory data corresponding to a plurality of training actions of a monitoring target collected by a heterogeneous sensor combination, to extract output synchronization deviation features of each category of somatosensory data corresponding to each training action, wherein the output synchronization deviation features include a first ratio of the number of coincident time stamps to the total number of output time stamps and a second ratio of the number of coincident time stamps to the number of time stamps corresponding to the least single category of somatosensory data; evaluation of a synchronization deviation representation parameter of the comprehensive somatosensory data based on the output synchronization deviation features, to divide the synchronization deviation degree category of the comprehensive somatosensory data; evaluation and fusion analysis of the comprehensive somatosensory data according to the synchronization deviation degree category, The method comprises the following steps: determination of a time sequence deviation feature of a single training action according to the time interval between the coincident time stamps corresponding to the comprehensive somatosensory data, to calculate a time sequence deviation degree representation value of the comprehensive somatosensory data, and to determine whether it meets the time sequence output deviation benchmark; identification of the connection tightness of the comprehensive somatosensory data in the time sequence relationship, adjustment of the weight coefficient of each category of somatosensory data, adjustment of the interpolation granularity corresponding to each category of somatosensory data, and placement of the adjusted each category of somatosensory data into a somatosensory state representation model to generate a somatosensory state representation by fusion; or, maintaining the initial weight coefficient of each category of somatosensory data, placing it into a somatosensory state representation model, and generating a somatosensory state representation by fusion; wherein the time sequence deviation feature includes the uniformity of the time interval and the maximum difference value of the output data amount of each category of somatosensory data in the shortest time interval; The process of evaluating the synchronization deviation representation parameter of the comprehensive somatosensory data comprises: taking the ratio of the first ratio of the number of coincident time stamps to the total number of output time stamps and the first ratio threshold value as the first synchronization deviation feature; taking the ratio of the second ratio of the number of coincident time stamps to the number of time stamps corresponding to the least single category of somatosensory data and the second ratio threshold value as the second synchronization deviation feature; taking the sum of the first synchronization deviation and the second synchronization deviation feature as the synchronization deviation representation parameter; The process of calculating the time sequence deviation degree representation value of the comprehensive somatosensory data comprises: taking the ratio of the uniformity of the time interval and the uniformity threshold value as the first deviation degree feature; taking the ratio of the maximum difference value of the output data amount of each category of somatosensory data in the shortest time interval and the maximum difference value threshold value of the output data amount as the second deviation degree feature; taking the sum of the first deviation degree feature and the second deviation degree feature as the time sequence deviation degree representation value.

2. The somatosensory time series data fusion method based on cross-modal attention mechanism according to claim 1, characterized in that, The division of the synchronization deviation degree category of the comprehensive somatosensory data comprises: if the synchronization deviation representation parameter of the comprehensive somatosensory data is less than the synchronization deviation representation parameter threshold value, the synchronization deviation degree category of the comprehensive somatosensory data is divided into a high deviation degree category; if the synchronization deviation representation parameter of the comprehensive somatosensory data is greater than or equal to the synchronization deviation representation parameter threshold value, the synchronization deviation degree category of the comprehensive somatosensory data is divided into a low deviation degree category.

3. The somatosensory time series data fusion method based on cross-modal attention mechanism according to claim 2, characterized in that, The evaluation and fusion analysis of the comprehensive somatosensory data according to the synchronization deviation degree category, includes, If the comprehensive somatosensory data belongs to the high deviation degree category, a time deviation feature of completing a single training action is determined according to a time interval between time stamps corresponding to the comprehensive somatosensory data, so as to calculate a time deviation degree representation value of the comprehensive somatosensory data, and it is determined whether the time deviation degree representation value meets a time output deviation criterion; The connection tightness of the comprehensive somatosensory data in the time sequence relationship is identified, the weight coefficients of the somatosensory data of each category are adjusted, the interpolation granularity corresponding to the somatosensory data of each category is adjusted, and the adjusted somatosensory data of each category is put into a somatosensory state representation model to generate a somatosensory state representation by fusion. If the comprehensive somatosensory data belongs to the low deviation degree category, the initial weight coefficients of the somatosensory data of each category are maintained, the somatosensory data of each category is put into the somatosensory state representation model, and a somatosensory state representation is generated by fusion.

4. The somatosensory temporal data fusion method based on cross-modal attention mechanism according to claim 1, characterized in that, It is determined whether the time deviation degree representation value of the somatosensory data meets the time output deviation criterion. If the time deviation degree representation value of the somatosensory data is less than a time deviation degree representation threshold value, it is determined that the time deviation degree representation value meets the time output deviation criterion.

5. The somatosensory temporal data fusion method based on cross-modal attention mechanism according to claim 1, characterized in that, The process of identifying the connection tightness of the comprehensive somatosensory data in the time sequence relationship includes: The time stamp data corresponding to the output of the comprehensive somatosensory data is called; Based on the time stamp data, the average output interval and the output interval change amplitude of the time sequence output of any two categories of somatosensory data are determined; The sum of the ratio of the average output interval to an average output interval threshold value and the ratio of the output interval change amplitude to an output interval change amplitude threshold value is taken as the connection tightness.

6. The somatosensory temporal data fusion method based on cross-modal attention mechanism according to claim 5, characterized in that, The process of adjusting the weight coefficients of the somatosensory data of each category includes: Based on the connection tightness of any two categories of somatosensory data, the weight coefficients of the two categories of somatosensory data are reduced, and the weight coefficients of another category of somatosensory data are increased; The adjustment amount of the weight coefficient is positively correlated with the connection tightness; The adjustment gradient of the weight coefficient is pre-set.

7. The somatosensory temporal data fusion method based on cross-modal attention mechanism according to claim 6, characterized in that, The process of adjusting the interpolation granularity corresponding to the somatosensory data of each category includes: The interpolation granularity corresponding to the two categories of somatosensory data is increased, and the increase amount of the interpolation granularity is positively correlated with the corresponding weight coefficient; The initial interpolation granularity of the another category of somatosensory data is maintained.

8. A system using the somatosensory time series data fusion method based on cross-modal attention mechanism according to any one of claims 1-7, characterized in that, It includes: A somatosensory collection module is used to acquire comprehensive somatosensory data corresponding to a plurality of training actions of a monitoring target collected by a heterogeneous sensor combination in real time, to extract output synchronization deviation features of each category of somatosensory data corresponding to each training action, and the output synchronization deviation features include a first ratio of the number of coincident time stamps to the total number of output time stamps and a second ratio of the number of coincident time stamps to the number of time stamps corresponding to the least single category of somatosensory data; A synchronous analysis module connected with the somatosensory collection module is used to evaluate a synchronization deviation representation parameter of the comprehensive somatosensory data based on the output synchronization deviation features, so as to divide a synchronization deviation degree category of the comprehensive somatosensory data; An evaluation fusion module connected with the synchronous analysis module is used to perform evaluation fusion analysis on the comprehensive somatosensory data according to the synchronization deviation degree category, It includes: According to a time interval between the coincident time stamps corresponding to the comprehensive somatosensory data, a time sequence deviation feature of completing a single training action is determined to calculate a time sequence deviation degree representation value for the comprehensive somatosensory data, and it is determined whether the time sequence output deviation criterion is met; The connection tightness of the comprehensive somatosensory data in the time sequence relationship is identified, the weight coefficients of various types of somatosensory data are adjusted, the interpolation granularity corresponding to the various types of somatosensory data is adjusted, and the adjusted various types of somatosensory data are put into a somatosensory state representation model to generate a somatosensory state representation by fusion; Or, the initial weight coefficients of various types of somatosensory data are maintained, put into a somatosensory state representation model, and a somatosensory state representation is generated by fusion; The time sequence deviation feature includes the uniformity of the time interval and the maximum difference value of the output data quantity of various types of somatosensory data in the shortest time interval.

Images