Osteopathic Manipulation Movement Recognition Method and System Based on Multimodal Data

Abstract

The present invention provides a recognition method and system for osteopathic manipulation movements based on multi-modal data, relating to the field of machine learning. The recognition method includes: S1, data preprocessing; S2, data fusion; S3, active segment division; S4, multi-modal interval feature extraction; S5, interval support vector machine modeling and optimization. The system includes a database, a data preprocessing module, a data fusion module, an active segment division module, an interval feature extraction module, and an interval support vector machine module. The present invention makes multi-dimensional judgments through multi-modal data, and automatically extracts thresholds using an adaptive state detection conversion algorithm to improve the accuracy of action recognition; by extracting interval features, the uncertainty and fluctuation information of the signal are retained, improving the model's perception of data. An interval support vector machine model is adopted, and the robustness is improved and overfitting is reduced by referring to the entire interval.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning of computers, and particularly relates to a method and a system for recognizing bone-setting manipulation actions based on multi-modal data. Background Art

[0002] Bone setting is a traditional Chinese medical diagnosis and treatment technique centered on manual reduction of actions, mainly targeting bone and joint and soft tissue injuries caused by external forces. By adjusting the abnormal states of bones, joints, and muscles, the normal anatomical structure and function of the human body can be restored. However, with the changes of the times, bone-setting science is also facing unprecedented challenges. Bone-setting manipulation actions highly rely on expert experience. At present, traditional orthopedic teaching adopts a one-on-one teaching method between master and apprentice. Usually, expert-level physicians subjectively teach and explain. During the learning process of each orthopedic doctor, there is a certain degree of subjectivity and passivity. Moreover, the specific implementation of each manipulation action also requires a large number of clinical trials to obtain accumulation and improvement, resulting in a long learning cycle, poor effects, and difficulty in quantitative repetition. At the same time, grass-roots hospitals and remote areas with underdeveloped medical levels are restricted by conditions and have no opportunity to study outside. Their technical levels are limited and it is difficult to meet the needs of patients. These problems have led to an increasing shortage of expert-level physicians in bone-setting science.

[0003] Facing this severe situation, we must actively explore the modern inheritance and innovation path of bone-setting manipulation actions, and strive to integrate classic manipulation actions with modern technologies to promote the wide popularization of high-quality resources in a platform-based manner. Human movement is achieved through muscle contraction and the coordinated cooperation of multiple muscles, thereby generating force. Bone-setting manipulation actions are that expert doctors rely on the skillful exertion of fingers, wrists, and arms, combined with rich experience and skills, to ensure the coherence of actions and forces within an appropriate time. Through the detailed analysis of bone-setting manipulation actions, it is found that the essence lies in the selection of manipulation actions and the precise control of force. If the bone-setting manipulation actions can be accurately recognized by equipment, automatic error correction can be achieved during the practice of bone-setting manipulation actions, or a robot can be used to simulate bone-setting manipulation actions to achieve robot bone setting. Therefore, there is a need for a method and a system that can accurately recognize bone-setting manipulation actions at present. Summary of the Invention

[0004] In order to solve the above-mentioned deficiencies of the prior art, the purpose of the present invention is to provide a method and a system for recognizing bone-setting manipulation actions based on multi-modal data. Through in-depth quantitative analysis of the bilateral electromyography signals and six-dimensional force signals of existing bone-setting manipulation actions, and through data fusion between different modal data, interval features are extracted, and the method of interval support vector machine is used to recognize bone-setting manipulation actions, thereby improving the accuracy and stability of action recognition.

[0005] Specifically, on the one hand, the present invention provides a method for recognizing bone-setting manipulation actions based on multi-modal data, which includes the following steps:

[0006] S1: Data preprocessing: Perform data denoising and data normalization operations on the acquired multi-modal data of bone-setting manipulation actions;

[0007] S2: Data fusion: For the normalized data of different modalities, first align the time stamp units of the data, and then use linear interpolation and nearest neighbor matching algorithms to fuse the multi-modal data to obtain normalized data fusion multi-modal data;

[0008] S3: Active segment division: For the normalized data fusion multi-modal data, use the adaptive state detection conversion algorithm to perform active segment division;

[0009] S31: Obtain feature quantities: Obtain the feature quantities of the bilateral electromyography signals at time t according to the bilateral electromyography signals of C channels. The feature quantities include the root mean square of the bilateral electromyography signals of C channels within the sliding window at time t , local maximum and minimum energy difference and zero crossing rate ;

[0010] S32: Obtain adaptive thresholds: The adaptive thresholds include the high state threshold in the state thresholds , low state threshold , energy threshold and zero crossing rate threshold ;

[0011] S33: Obtain the time interval of the active segment: Divide the time interval of the active segment according to the feature quantities of the bilateral electromyography signals and the adaptive thresholds;

[0012] S34: Set labels: Set labels for the normalized data fusion multi-modal data according to the time interval of the active segment;

[0013] S4: Multi-modal interval feature extraction: For the normalized data fusion multi-modal data, use the sliding window mechanism to determine the action category and extract the multi-modal interval feature vector for the multi-modal interval feature signals;

[0014] S5: Interval support vector machine modeling and optimization: Train the interval support vector machine model, optimize the hyperparameters of the interval support vector machine, and perform evaluation.

[0015] Preferably, in S1, for the original bilateral electromyography signals of the i-th channel use a band-pass filter combined with a notch filter for denoising; for the j-th component of the six-axis force signal , use a low-pass filter for denoising;

[0016] For the osteopathic manipulation movements of the same type, the filtered bilateral EMG signals of the i-th channel at time t are normalized by the min-max normalization method ; the j-th component data of the six-dimensional force signal at time t is normalized by the z-score standardization method .

[0017] Preferably, in S2, for the data of different modalities after normalization, first align the time stamp units of the data, and then use the linear interpolation and nearest neighbor matching algorithms to fuse the multi-modal data to obtain the normalized data fusion multi-modal data; where the bilateral EMG signal of the i-th channel at time t is , and the j-th component of the six-dimensional force signal is .

[0018] Preferably, S32 is specifically as follows:

[0019] (1) Calculate the state threshold for judging the signal state. The calculation expression of the state threshold is:

[0020] ;

[0021] ;

[0022] Among them, and are constants respectively, and >[[]]END]] ; N is the total number of samples in the sliding window; k is the time index of the i-th channel sliding window at time t, and the corresponding bilateral EMG signal value is ; represents the minimum moment in the sliding window when the sliding window size is W1, represents the maximum moment in the sliding window when the sliding window size is W1;

[0023] (2) Obtain the energy threshold,

[0024] ;

[0025] Among them, is the threshold energy ratio coefficient, is the energy mean value at time t, is the smoothing coefficient;

[0026] (3) Obtain the zero-crossing rate threshold,

[0027] ;

[0028] Among them, is the zero-crossing rate threshold ratio coefficient, is the mean value of the number of zero-crossing points at time t, is the smoothing coefficient, and I is the indicator function.

[0029] Preferably, S33 is specifically:

[0030] If both are satisfied, the current time t is recorded as the activity start time ; if both are satisfied, the current time t is recorded as the activity end time ; where is the set minimum duration of the activity segment; the effective time interval of the activity segment is .

[0031] Preferably, S34 is specifically:

[0032] Based on the effective time interval of the activity segment, the normalized data fusion multi-modal data is divided into activity segments and static segments; in the activity segments, corresponding action category labels are marked according to the specific osteopathic manipulation actions, and static labels are marked for the data in the static segments; finally, each time point of the normalized data fusion multi-modal data is assigned an action category label, and the action category label includes the name labels of all actions in the osteopathic manipulation action dataset and the static label.

[0033] Preferably, S4 is specifically:

[0034] The multi-modal interval feature signal is , where , , and are respectively the interval vector, the minimum value within the window, and the maximum value within the window of the bilateral electromyogram signal of the i-th channel under the p-th window; , where, , and are respectively the interval vector, the minimum value within the window, and the maximum value within the window of the j-th six-dimensional force signal component under the p-th window;

[0035] The action category is represented as , where is the static label, is the q-th action category label, is the number of static labels within this window, is the number of the q-th action category labels within this window;

[0036] Extracting features from the multi-modal interval feature signal includes the interval mean and the interval variance , interval energy , and the interval entropy of the sEMG signals of both arms for the p-th window , and the multi-modal interval feature vector for the p-th window is obtained as , with the label being the action category .

[0037] Preferably, S5 is specifically as follows:

[0038] S51: Construct a data set: Use the osteopathic manipulation action data set to establish an action category data set, and use the five-fold cross-validation method to divide the action category data set into a training data set and a validation data set;

[0039] S52: Interval support vector machine modeling and optimization: Use the interval support vector machine method for interval action recognition; the kernel function of the interval support vector machine is the interval RBF kernel function;

[0040] S53: Model verification: Integrate the verification results obtained from the validation data set in each round, evaluate the trained interval support vector machine model, and output the accuracy rate and the confusion matrix.

[0041] On the other hand, the present invention provides an identification system for an osteopathic manipulation action recognition method based on multi-modal data, which includes a database, a data preprocessing module, a data fusion module, an activity segment division module, an interval feature extraction module, and an interval support vector machine module, and are connected in sequence; specifically:

[0042] The database is used to store the osteopathic manipulation action data set, which at least includes the serial number, name, sEMG signals of both arms, and the force signal data segment. The sEMG signals of both arms and the force signal together form multi-modal data;

[0043] The data preprocessing module is used to obtain the osteopathic manipulation action data from the database, and perform data denoising and data normalization operations on the multi-modal data according to step S1 to obtain normalized multi-modal data;

[0044] The data fusion module is used to align the time stamp units of different modal data of the normalized multi-modal data obtained by the data preprocessing module using the method in S2, and then use the linear interpolation and nearest neighbor matching algorithms to fuse the multi-modal data to obtain normalized data fusion multi-modal data;

[0045] The activity segment division module is used to use the method in S3 for the normalized data fusion multi-modal data obtained by the data fusion module, use the adaptive state detection conversion algorithm to perform activity segment division, and label each moment of the normalized data fusion multi-modal data;

[0046] The interval feature extraction module is used to determine the action category and extract the multi-modal interval feature vector for the multi-modal interval feature signal by using the method in S4 for the normalized data fusion of the multi-modal data with the labeled tags.

[0047] The interval support vector machine module first models and optimizes the multi-modal interval feature vector obtained by the interval feature extraction and recognition module by using the method in S5, and then is used to recognize the actions of the newly input osteopathic manipulation action data.

[0048] Preferably, the interval support vector machine module specifically includes: a data set construction sub-module, an action recognition sub-module, and a result evaluation sub-module;

[0049] Among them, the data set construction sub-module uses the osteopathic manipulation action data set to establish an action category data set, and divides the action category data set by using the five-fold cross-validation method into a training set and a validation set;

[0050] The action recognition sub-module performs interval action recognition on the normalized data fusion of the multi-modal data by using the method of the interval support vector machine, and constructs an interval support vector machine model based on the osteopathic manipulation action recognition;

[0051] The result evaluation sub-module calculates the accuracy rate and the confusion matrix for the constructed interval support vector machine model based on the osteopathic manipulation action recognition, and conducts evaluation and verification.

[0052] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0053] 1. The present invention can describe the same action from multiple angles by performing data-level fusion on data of different modalities, alleviate the problem of insufficient amount of single-modal data. At the same time, the multi-modal data information fusion can make more dimensional judgments, which helps to improve the accuracy rate of action recognition.

[0054] 2. The present invention uses the adaptive state detection conversion algorithm for the osteopathic manipulation action data, automatically extracts the threshold, and then divides the active segment, reducing the limitations of manually setting the threshold, having strong adaptability and flexibility, can segment the entire osteopathic manipulation action process, and define the starting point and ending point of the action with higher precision, thereby improving the accuracy rate of action recognition.

[0055] 3. The present invention can retain the uncertainty and fluctuation information of the signal by extracting interval features, improve the perception ability of the model for data, and has good compatibility when the data has noise interference and outliers. Combined with other features to form a multi-dimensional feature vector, it has strong generalization ability.

[0056] 4. The present invention uses an interval support vector machine, which can more realistically describe fuzzy data and similar actions. For problems such as sampling errors and environmental noise, by referring to the entire interval, the robustness can be improved and overfitting can be reduced. The experimental results show that this method has high stability and generalization ability in the task of identifying orthopedic manipulation actions, providing reliable technical support for the digital analysis of orthopedic manipulation data. Brief Description of the Drawings

[0057] Figure 1 It is a flowchart of a method for identifying orthopedic manipulation actions based on multi-modal data provided by the present invention;

[0058] Figure 2 It is a schematic diagram of a system for identifying orthopedic manipulation actions based on multi-modal data provided by the present invention;

[0059] Figure 3 It is an example of a confusion matrix generated by an embodiment of the present invention. Detailed Description of the Embodiments

[0060] The present invention provides a method and a system for identifying orthopedic manipulation actions based on multi-modal data. By performing data preprocessing on the relevant data in the orthopedic manipulation action process obtained from the database, and then fusing the multi-modal data, an adaptive state detection conversion algorithm is used to extract the active segments to obtain interval multi-modal data features; the five-fold cross-validation method is used to divide the orthopedic manipulation action data set, and the interval support vector machine is used to identify the orthopedic manipulation actions. The present invention will be described in detail below with reference to the drawings.

[0061] A method for identifying orthopedic manipulation actions based on multi-modal data proposed by the present invention, as Figure 1 shown, the present invention includes the following steps:

[0062] S1: Data preprocessing: In a specific embodiment, first, obtain the orthopedic manipulation action data from the database; the orthopedic manipulation action data set includes data segments such as serial number, name, bilateral electromyography signals, and force signals. Among them, the serial number is the identity code of each piece of data, used to distinguish each piece of data in the orthopedic manipulation action data set. The name is the name of the orthopedic manipulation action, representing various orthopedic manipulation actions existing in orthopedics. The bilateral electromyography signals are multi-channel bilateral electromyography signals collected by multiple sensors during the orthopedic manipulation action process. In this embodiment, C channels of bilateral electromyography signals are used. Therefore, the original bilateral electromyography signal of the i-th channel is represented as , , preferably, C is taken as 10. The force signal is the six-dimensional signal at the end of the arm obtained by a six-dimensional force sensor during the orthopedic manipulation action process, including three-dimensional force signals and three-dimensional torque signals. The j-th original component of the six-dimensional force signal is represented as , The bilateral EMG signals and force signals together constitute the multi-modal data of the bone-setting manipulation actions.

[0063] Perform data denoising and data normalization operations on the obtained multi-modal data of the bone-setting manipulation actions.

[0064] Regarding the situation where the data in the database may have noise interference, data anomalies, data missing, etc., assuming that there are C channels in the bilateral EMG signals, for the original bilateral EMG signal of the i-th channel at time t Use a fourth-order Butterworth band-pass filter with a frequency range of 20 - 450 Hz combined with a notch filter of 50 Hz to denoise the data, and obtain the bilateral EMG signal of the i-th channel at time t after filtering , and the specific calculation process is as follows:

[0065] ;

[0066] Among them, is the impulse response of the notch filter of 50H Z , * represents the convolution operation, s is the complex frequency variable in the Laplace transform, is the impulse response of the 20 - 450 Hz Butterworth band-pass filter.

[0067] For the j-th component of the six-dimensional force signal obtained at time t , use a low-pass filter for filtering to obtain the j-th component of the six-dimensional force signal at time t after filtering , and the specific process is as follows:

[0068] ;

[0069] Among them, is the impulse response of the low-pass filter.

[0070] Entries with different serial numbers but the same name in the database are called bone-setting manipulation actions of the same type. However, there are significant differences in the bilateral EMG signals and force signals corresponding to bone-setting manipulation actions of the same type, especially in the signal amplitude fluctuation range. Therefore, use the min-max normalization method to normalize the filtered bilateral EMG signal of the i-th channel at time t of the bone-setting manipulation actions of the same type to obtain the normalized bilateral EMG signal , and the specific process is as follows:

[0071] ;

[0072] Among them, is the minimum value of the filtered bilateral EMG signal of the i-th channel, is the maximum value of the filtered bipolar EMG signal for the i-th channel.

[0073] Under the same type of bone-setting manipulation movements, the six-axis force data may also show significant differences. Therefore, the z-score normalization method can be used to normalize the j-th component data of the six-axis force signal at time t after filtering for the same type of bone-setting manipulation movements to obtain the j-th component data of the six-axis force signal at time t after filtering The specific process is as follows:

[0074] ;

[0075] where, is the mean value of the filtered six-axis force component data for the j-th channel, is the standard deviation of the filtered six-axis force data for the j-th channel.

[0076] S2: Data fusion: For the normalized data of different modalities, first align their timestamp units, and then use linear interpolation and nearest neighbor matching algorithms to fuse the multi-modal data.

[0077] Since the sampling frequencies of the bipolar EMG signal and the six-axis force signal are different, problems such as inconsistent timestamp start points, asynchronous sampling intervals, and deviations in signal durations may occur in the time alignment of the original data. To improve the action recognition accuracy, data fusion of the multi-modal data is performed so that the corresponding bipolar EMG signal and six-axis force data information can be obtained at the same time point. The method for data fusion of the multi-modal data composed of the bipolar EMG signal and the six-axis force signal is as follows:

[0078] (1) Align the timestamp units of the bipolar EMG signal and the six-axis force signal, and express them in the same time unit.

[0079] (2) Sort the bipolar EMG signal data and the six-axis force data respectively in ascending order of timestamps.

[0080] (3) Taking the bipolar EMG signal as the reference, if the EMG timestamp is found in the six-axis force timestamp and the nearest neighbor point that meets the time difference requirement, the data is directly merged; otherwise, linear interpolation is used to interpolate the data of each component of the six-axis force until there is a nearest neighbor point that meets the time difference requirement. The specific implementation method in this embodiment is: find the EMG timestamp in the six-axis force timestamp and the nearest neighbor point that meets the time difference , if If it exists, directly merge the data; otherwise, interpolate the data of each component of the six-dimensional force using linear interpolation; assume that the two six-dimensional force timestamps of the nearest neighbor are respectively and , where , the six-dimensional force data corresponding to the moment is , the six-dimensional force data corresponding to the moment is , the six-dimensional force data at the moment , then the specific calculation process is as follows:

[0081] .

[0082] Using the above formula, the normalized multi-channel bilateral electromyogram signals and six-dimensional force signals with timestamp units aligned are obtained. This signal is called the normalized data fusion multi-modal data. In the normalized data fusion multi-modal data, the bilateral electromyogram signal of the i-th channel at time t is represented as and the j-th component of the six-dimensional force signal at time t is represented as . The data of the C channels of the bilateral electromyogram signal and the 6 components of the six-dimensional force signal are all aligned in time.

[0083] S3: Activity segment division: For the normalized data fusion multi-modal data, use the adaptive state detection conversion algorithm to perform activity segment division.

[0084] S31: Obtain the characteristic quantities of the bilateral electromyogram signal at time t according to the C channels of the bilateral electromyogram signal. The characteristic quantities include root mean square, local maximum and minimum energy difference, and zero crossing rate.

[0085] (1) Obtain the root mean square of the C channels of the bilateral electromyogram signal at time t within the sliding window.

[0086] Set the size of the sliding window to W1, where W1 is an integer greater than 1. Let the time index of the sliding window at time t for the i-th channel be k, and the corresponding bilateral electromyogram signal value be set as , where , , represents rounding down, represents the smallest moment within the sliding window when the sliding window size is W1, represents the largest moment within the sliding window when the sliding window size is W1. The root mean square of the C channels of the bilateral electromyogram signal at time t is:

[0087] .

[0088] (2) Obtain the local maximum and minimum energy difference of the bilateral sEMG signals of C channels at time t within the sliding window:

[0089] .

[0090] (3) Obtain the zero-crossing rate of the bilateral sEMG signals of C channels within the sliding window at time t to reflect the frequency change of the bilateral sEMG signals:

[0091] ;

[0092] where I is the indicator function, and when and have different signs, that is, when the signal crosses the zero point, the function value is 1, otherwise it is 0.

[0093] S32: Obtain the adaptive threshold, and the adaptive threshold includes the state threshold, energy threshold, and zero-crossing rate threshold.

[0094] (1) Obtain the state threshold for judging the signal state.

[0095] Each osteopathic manipulation action includes two states: the active segment and the static segment. For example, the stretching osteopathic manipulation action includes stretching and the holding after stretching in place, and then relaxing, stretching again, and holding again. Among them, stretching and the holding after stretching in place belong to the active segment, and relaxation belongs to the static segment. The state threshold includes the high state threshold and the low state threshold. In the present invention, the state is judged by calculating the double threshold. Among them, the high state threshold is set to judge whether the signal is in the active segment, and the low state threshold is set to judge whether the signal is in the static segment; the calculation expression of the state threshold:

[0096] ;

[0097] ;

[0098] where and are constants respectively and > , The preferred value of is taken as 0.5, and the preferred value of .

[0099] The low state threshold includes a mean term representing the energy expectation and a standard deviation term representing the energy fluctuation. Among them, the mean term represents the sliding window at time to Between them is the energy expectation of the i-th channel EMG signal. In the specific calculation process, first square the signal value at each moment k within this time period to obtain the local energy at each moment; subsequently, sum up the squared values for all channels C and all time points, and finally divide by the total number of samples N within the sliding window to obtain the overall energy mean . The standard deviation term is used to measure the degree of energy fluctuation within the sliding window. Specifically, first calculate the squared value of each in the same way as the mean term; then, calculate the squared difference between these squared values and the overall energy mean, sum up the squared difference terms for all channels and all time points, and further divide by the total number of samples N; finally, take the square root of the result to obtain the standard deviation term .

[0100] High state threshold is different from the low state threshold in that the high state threshold is amplified by a larger constant so that this threshold is applicable to identifying stronger EMG activities.

[0101] By designing dynamic upper and lower bound energy thresholds for state determination, a tolerance band is formed, enabling the data to dynamically adjust the threshold according to different signal strengths, independent of absolute values, preventing signal edges from jumping back and forth, improving robustness. After adding the standard deviation term, signals with large fluctuations can be correctly identified, thereby improving the robustness.

[0102] (2) Obtain the energy threshold.

[0103] Calculate the energy threshold so as to be able to adaptively adjust the threshold and adapt to changes in different individuals and different environments.

[0104] ;

[0105] Among them, is the threshold energy ratio coefficient, is the energy mean at time t, where

[0106] ;

[0107] Among them, is the smoothing coefficient; thus, the energy threshold can be expanded as:

[0108] .

[0109] The energy threshold is composed of a weighted combination of the historical energy expectation and the current window energy dynamic range. Among them, the historical expectation term Represents the average value of the overall energy calculated in the previous sliding window, which is used to provide a stable reference. The current energy range term is used to reflect the instantaneous fluctuation degree of the EMG signal energy within the current window. The specific calculation process is as follows. At the time of the sliding window to For each channel i, calculate the difference between the maximum and minimum values of the squared EMG signal in this window That is , then divide by the total number of channels C, and average the differences of all channels to obtain the dynamic range of the overall channel energy. Finally, the historical expectation term and the current energy range term are weighted and fused according to the weight coefficients and , and multiplied by the adjustment factor to obtain the energy threshold.

[0110] By introducing the historical sliding window threshold, the problem of signal energy drift can be solved. By calculating the current energy fluctuation amplitude, it can be reflected whether the current signal changes violently, and then quickly respond to mutations;

[0111] (3)Obtain the zero-crossing rate threshold.

[0112] Calculate the zero-crossing rate threshold , which is used to judge the degree of frequency change of the bilateral EMG signal:

[0113] ;

[0114] Among them, is the zero-crossing rate threshold proportionality coefficient, is the average number of zero-crossing points at time t.

[0115] ;

[0116] Among them, is the smoothing coefficient; Therefore, the zero-crossing rate threshold can be expanded as:

[0117] .

[0118] The zero-crossing rate threshold is composed of the weighted combination of the historical zero-crossing rate mean and the zero-crossing frequency within the current window. Among them, the historical zero-crossing rate mean term represents the expectation of the signal symbol change frequency in the previous sliding window, and the current zero-crossing rate term depicts the frequency of the signal symbol change in the current window. The specific calculation method is as follows. At the time of the sliding window to For each channel i and each adjacent moment, judge whether the product of the signal values at two adjacent moments is less than 0, that is, whether a zero-crossing occurs , and then, accumulate the zero-crossing events of all channels and time points, and divide by the normalization factor , the average zero-crossing rate per unit length is obtained. Finally, the historical mean and the current value are weighted and fused, and multiplied by an adjustment factor , to obtain the zero-crossing rate threshold.

[0119] By introducing historical zero-crossing rate information, certain robustness is maintained when dealing with signal noise, mutations, or drifts. By summing and averaging the zero-crossing events of all channels, the situation of excessive interference in a single channel is avoided, and the detection accuracy is improved. Complementary to the energy threshold, it can greatly enhance the accuracy and robustness of the active segment detection.

[0120] S33: Obtain the time interval of the active segment, and determine whether a state transition occurs according to the characteristic quantity of the surface electromyogram signal obtained in S31 and the threshold obtained in S32.

[0121] If both are satisfied, the current time t is recorded as the start time of the activity .

[0122] If both are satisfied, the current time t is recorded as the end time of the activity .

[0123] Among them, is the set minimum duration of the active segment.

[0124] According to the above judgment, the effective time interval of the active segment is .

[0125] S34: Set labels for the multi-modal data of normalized data fusion.

[0126] The label is an action category label, which consists of the names of all actions and stillness in the osteopathic manipulation action dataset. For example, there are 8 action names in the osteopathic manipulation action dataset in this embodiment, so the action category label consists of 8 names and stillness, a total of 9 labels. For the sake of simplicity of description, the names of the osteopathic manipulation actions are numbered from 1 to 8, and 0 is used when still.

[0127] According to the effective time interval of the active segment obtained in S33, two states, namely the active segment and the still segment, are distinguished from the multi-modal data of normalized data fusion. The data in the active segment are labeled with the corresponding action category labels according to the names of the osteopathic manipulation actions. The label names in the active segments of different osteopathic manipulation actions are different. The data in the still segment are labeled with the stillness label. Finally, each time point in the multi-modal data of normalized data fusion is assigned a clear action category label.

[0128] S4: Multi-modal Interval Feature Extraction: For the multi-modal data after normalized data fusion, using the sliding window mechanism, determine the action category for the multi-modal interval feature signals and extract the multi-modal interval feature vectors.

[0129] For the multi-modal data after normalized data fusion at time t, use the sliding window to extract the interval features. The sliding window here can be the same as the one in S3 or different, but the size W2 of the sliding window here must be an odd number greater than 1. Since both the sEMG signals and the six-axis force signals in the multi-modal data after normalized data fusion obtained according to S2 are time series, in the p-th window, the interval vector of the sEMG signals of C channels can be expressed as:

[0130] ;

[0131] where is the interval vector of the sEMG signal of the i-th channel under the p-th window; is the minimum value of the sEMG signal of the i-th channel within the p-th window, is the maximum value of the sEMG signal of the i-th channel within the p-th window, that is , , k still represents the time index of the sliding window at time t of the i-th channel. However, since the size of the sliding window is W2 at this time, so , where , represents the minimum time within the sliding window when the sliding window size is W2, represents the maximum time within the sliding window when the sliding window size is W2, is the value of the k-th sEMG signal of the i-th channel under the p-th window.

[0132] Similarly, in the p-th window, all six-axis force signal components can be expressed by intervals as:

[0133] ;

[0134] where is the interval vector of the j-th six-axis force signal component under the p-th window; is the minimum value of the j-th component of the six-axis force signal under the p-th window, , is the maximum value of the j-th component of the six-axis force signal under the p-th window, , where is the value of the j-th component of the six-axis force signal under the p-th window.

[0135] For the sEMG signals of the p-th window and six - dimensional force signals , for which the multi - modal interval feature signal is , the action category is represented as , where is the static label, is a certain action category label, is the number of static labels in this window, is the number of the q - th action category label in this window.

[0136] A multi - modal interval feature signal and the action category constitute a sample with labels.

[0137] Extract features from the multi - modal interval feature signal as follows:

[0138] (1) Interval mean of the bilateral sEMG signals and six - dimensional force signals for the p - th window:

[0139] ;

[0140] where , .

[0141] The interval mean is used to describe the central tendency of the multi - modal signal in the p - th window, and is jointly composed of the interval mean of the bilateral sEMG signals and the interval mean of the six - dimensional force signals; for the sEMG channel i = 1, 2,..., C, its interval mean is the arithmetic mean of the maximum and minimum values of the i - th sEMG channel in the current window; for the six - dimensional force signal j = 1, 2,..., 6, its interval mean is the arithmetic mean of the maximum and minimum values of the j - th six - dimensional force channel in the current window; through the above construction, can effectively reflect the average level of the multi - modal signal in the current window.

[0142] (2) Interval variance of the bilateral sEMG signals and six - dimensional force signals for the p - th window:

[0143] ;

[0144] where ,

[0145] .

[0146] The interval variance is used to characterize the fluctuation degree and amplitude discreteness of the multi-modal signal within the p-th window, and is jointly composed of the interval variance of the bilateral sEMG signal and the interval variance of the six-dimensional force signal; for the sEMG channel i = 1, 2,..., C, its interval variance is defined as the mean square of the deviation degrees of the maximum and minimum values of the signal of this channel within the current window relative to the interval mean ; for the six-dimensional force signal j = 1, 2,..., 6, its interval variance is defined as the mean square of the deviation degrees of the maximum and minimum values of the signal of this channel within the current window relative to the interval mean ; through the above construction, the interval variance can effectively reflect the amplitude fluctuation and uncertainty of the multi-modal signal within the current window, providing information support for the subsequent model on the degree of change.

[0147] (3) Interval energy of the bilateral sEMG signal and six-dimensional force signal for the p-th window:

[0148] ;

[0149] wherein, , .

[0150] The interval energy is used to measure the activity degree and overall strength of the multi-modal signal within the p-th window, and is jointly composed of the interval energy of the bilateral sEMG signal and the interval energy of the six-dimensional force signal; for the sEMG channel i = 1, 2,..., C, its interval energy is defined as the average value of the squares of the original signals of this channel within the current window, and for the six-dimensional force signal j = 1, 2,..., 6, its interval energy is the average value of the squares of the original signals of this channel within the current window; through the above construction, the interval energy can reflect the overall strength of the signal within the current window and the activity levels of the muscles and interaction forces, and is an important feature for identifying actions.

[0151] (4) Interval entropy of the bilateral sEMG signal for the p-th window:

[0152] ;

[0153] wherein, is the minimum value. The interval entropy is used to evaluate the amplitude fluctuation complexity and uncertainty of the bilateral sEMG signal within the p-th sliding window, reflecting the discrete degree of the signal change within this window.

[0154] Therefore, for the p-th window, the multi-modal interval feature vector is , and the label is the action category .

[0155] S5: Interval Support Vector Machine Modeling and Optimization: Train an interval support vector machine model, optimize the hyperparameters of the interval support vector machine, and conduct an evaluation.

[0156] S51: Construct a dataset: Use the osteopathic manipulation action dataset to establish an action category dataset, and adopt the method of five-fold cross-validation to divide the action category dataset into a training dataset and a validation dataset.

[0157] Based on S1 - S4, one piece of data in the osteopathic manipulation action dataset can obtain multiple samples of the same action category. By performing the same processing on each piece of data in the osteopathic manipulation action dataset, an action category dataset containing samples of different action categories can be obtained. The method of five-fold cross-validation is used to divide the action category dataset. Specifically: Randomly divide the action category dataset containing samples of different action categories into five parts, select four of them as the training dataset, and one as the validation dataset.

[0158] S52: Interval Support Vector Machine Modeling and Optimization: Use the method of interval support vector machine for interval action recognition; the kernel function of the interval support vector machine is the interval RBF kernel function.

[0159] Determine that the kernel function of the interval support vector machine is the interval RBF kernel function. For two interval vectors and , the interval RBF kernel function is:

[0160] ;

[0161] Where: are two interval vector samples in the dataset respectively, is the action category, is the action category, are the interval means of the two interval vectors respectively, are the interval variances of the two interval vectors respectively, is the scale parameter of the RBF kernel function, is the square of the norm.

[0162] The interval RBF kernel function is an exponential function with the Euclidean distance as the input. First, calculate the differences between the two samples in the interval means and the differences in the interval variances respectively, so as to calculate the squared norm of the differences in the Euclidean distance and the interval variances, and sum the two to obtain ; The interval RBF kernel takes into account both the interval center and the interval fluctuation, and can more comprehensively reflect the statistical distribution characteristics of time-varying signals within a time window.

[0163] Assume the current training set , where n is the number of samples in the current training set, and the optimization objective of the interval support vector machine is determined as:

[0164] .

[0165] The constraint conditions are:

[0166] ;

[0167] Among them, is the weight vector of the interval support vector machine, b is the bias term, A is the penalty factor, is the slack variable of the m1-th sample, is the Lagrange multiplier of the m1-th sample. In the constraint conditions, each sample must be correctly classified or the distance from the classification boundary does not exceed . Using the kernel function , the interval samples are mapped to a high-dimensional space.

[0168] S53: Model verification and action recognition: During training, loop five times. According to the output results of each validation set, the comprehensive model results are finally obtained.

[0169] Integrate the verification results of each round, evaluate the trained interval support vector machine model, realize the recognition and verification of the orthopedic manipulation actions, and output the accuracy rate and the confusion matrix.

[0170] In this embodiment, there are a total of 8 orthopedic manipulation actions. Through the above method, the calculated accuracy rate is 92.13%, indicating that when performing interval feature extraction on the bilateral EMG signals and six-dimensional force signals and using the interval support vector machine method for action recognition, it has a high accuracy rate; the generated confusion matrix is as Figure 3 shown. Each action is represented by "action_1" to "action_8" respectively. The abscissa Predicted is the predicted result, and the ordinate True is the true result. It can be seen that the present invention can have a high recognition accuracy rate for each orthopedic manipulation action, indicating that this method is effective.

[0171] In a preferred embodiment of the present invention, after the interval support vector machine modeling and optimization are performed in the interval where step S5 of the present invention is executed, it may further include a step of identifying the categories of osteopathic manipulation actions, inputting new osteopathic manipulation action data into the interval support vector machine model for action recognition. Specifically, input the electromyogram signals and force signals of both arms of the new osteopathic manipulation action; obtain the multi-modal data of normalized data fusion according to S1-S3; use a sliding window according to S4 to obtain the multi-modal interval feature signal at time t , and extract the multi-modal interval feature vector, but at this time the multi-modal interval feature signal does not have an action category; input the multi-modal interval feature vector into the interval support vector machine model trained in S5 to obtain the action category of the multi-modal interval feature signal at time t.

[0172] The present invention also provides an osteopathic manipulation action recognition system based on multi-modal data, including a database, a data preprocessing module, a data fusion module, an activity segment division module, an interval feature extraction module, and an interval support vector machine module, which are connected in sequence, as Figure 2 shown, specifically:

[0173] The database is used to store the osteopathic manipulation action data set, including at least data segments such as serial number, name, electromyogram signals of both arms, and force signals. The electromyogram signals of both arms and the force signals together form multi-modal data.

[0174] The data preprocessing module is used to obtain and store the osteopathic manipulation action data from the database, and perform operations such as data noise reduction and data normalization on the multi-modal data using the method in S1 to obtain the normalized multi-modal data.

[0175] The data fusion module is used for the normalized multi-modal data obtained by the data preprocessing module. For data of different modalities, use the method in S2 to align their time stamp units first, and then use the linear interpolation and nearest neighbor matching algorithms to fuse the multi-modal data to obtain the normalized data fusion multi-modal data.

[0176] The activity segment division module is used for the normalized data fusion multi-modal data obtained by the data fusion module, and uses the method in S3 to perform activity segment division using the adaptive state detection conversion algorithm, and label each moment of the normalized data fusion multi-modal data.

[0177] The interval feature extraction module is used for the normalized data fusion multi-modal data with labeled tags, and uses the method in S4 to determine the action category of the multi-modal interval feature signal and extract the multi-modal interval feature vector.

[0178] The interval support vector machine module first extracts the multi-modal interval feature vectors obtained by the interval feature extraction and recognition module, and after modeling and optimization using the method in S5, it is used to perform action recognition on the newly input osteopathic manipulation action data. The interval support vector machine module also includes a data set construction sub-module, an action recognition sub-module, and a result evaluation sub-module.

[0179] Among them, the data set construction sub-module uses the osteopathic manipulation action data set to establish an action category data set, and divides the action category data set by the method of five-fold cross-validation into a training data set and a validation data set.

[0180] The action recognition sub-module performs interval action recognition on the multi-modal data of normalized data fusion by using the method of interval support vector machine, and constructs an interval support vector machine model based on osteopathic manipulation action recognition.

[0181] The result evaluation sub-module calculates the accuracy rate and confusion matrix for the constructed interval support vector machine model based on osteopathic manipulation action recognition, and conducts evaluation and verification.

[0182] The embodiments described above are only descriptions of the preferred embodiments of the present invention, and do not limit the scope of the present invention. Without departing from the design spirit of the present invention, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present invention shall fall within the protection scope determined by the claims of the present invention.

Claims

1. A method for recognizing osteopathic manipulation movements based on multi-modal data, characterized in that: It includes the following steps: S1: Data preprocessing: Perform data denoising and data normalization operations on the obtained multi-modal data of bone-setting manipulation actions. S2: Data fusion: For the normalized data of different modalities, first align the time stamp units of the data, and then use linear interpolation and nearest neighbor matching algorithms to fuse the multi-modal data to obtain normalized data fusion multi-modal data. S3: Active segment division: For the normalized data fusion multi-modal data, use the adaptive state detection conversion algorithm to perform active segment division. S31: Obtain feature quantities: Obtain the feature quantities of the bilateral sEMG signals at time t based on the bilateral sEMG signals of C channels. The feature quantities include the root mean square of the bilateral sEMG signals of C channels within the sliding window at time t , the local maximum-minimum energy difference and the zero crossing rate ; S32: Obtain the adaptive threshold: The adaptive threshold includes the high state threshold in the state thresholds , the low state threshold , the energy threshold and the zero-crossing rate threshold ; S33: Obtain the time interval of the active segment: Divide the time interval of the active segment according to the characteristic quantity and adaptive threshold of the bilateral electromyogram signals. S34: Set labels: Set labels for the normalized data fusion multi-modal data according to the time interval of the active segment; specifically: Based on the effective time interval of the active segment, divide the normalized data fusion multi-modal data into active segments and static segments; in the active segments, label the corresponding action category labels according to the specific bone-setting manipulation actions, and label the static labels for the data in the static segments. Finally, each time point of the normalized data fusion multi-modal data is assigned an action category label, and the action category labels include the name labels of all actions in the bone-setting manipulation action dataset and the static label. S4: Multi-modal interval feature extraction: For the normalized data fusion multi-modal data, use the sliding window mechanism to determine the action category of the multi-modal interval feature signals and extract multi-modal interval feature vectors. Specifically: for the sEMG signals of the p-th window and the six-axis force signals , the multi-modal interval feature signal is , the action category is represented as , where is the static label, is a certain action category label, is the number of static labels in this window, is the number of the q-th action category labels in this window. A multi-modal interval feature signal and the action category form a labeled sample, and features are extracted from the multi-modal interval feature signal . S5: Interval support vector machine modeling and optimization: Train the interval support vector machine model, optimize the hyperparameters of the interval support vector machine, and perform evaluation, including constructing a dataset: Use the bone-setting manipulation action dataset to establish an action category dataset, and use the five-fold cross-validation method to divide the action category dataset into a training dataset and a validation dataset. After performing the interval support vector machine modeling and optimization in step S5, it also includes the step of recognizing bone-setting manipulation action categories, inputting new bone-setting manipulation action data into the interval support vector machine model for action recognition.

2. The bone-setting manipulation action recognition method based on multi-modal data according to claim 1, wherein: In S1, the original bilateral sEMG signal for the i-th channel Noise reduction is performed using a band-pass filter combined with a notch filter; for the j-th component of the six-axis force signal , noise reduction is performed using a low-pass filter; For the same type of bone-setting manipulation movements, the filtered bilateral EMG signals of the i-th channel at time t are normalized using the min-max normalization method ; the j-th component data of the six-dimensional force signal at time t is normalized using the z-score standardization method .

3. The bone-setting manipulation action recognition method based on multi-modal data according to claim 1, wherein: In S2, for the normalized data of different modalities, first align the time stamp units of the data, and then use the linear interpolation and nearest neighbor matching algorithms to fuse the multi-modal data to obtain the normalized data fusion multi-modal data; where the bilateral electromyogram signal of the i-th channel at time t is , and the j-th component of the six-dimensional force signal is .

4. The osteopathic manipulation motion recognition method based on multi-modal data according to claim 1, characterized in that: S32 is specifically: (1) Calculate the state threshold for judging the signal state, and the calculation expression of the state threshold is: ； ； Among them, and are constants respectively, and > ; N is the total number of samples in the sliding window; k is the time index of the sliding window of the i-th channel at time t, and the corresponding surface electromyogram signal value is ; represents the minimum moment in the sliding window when the sliding window size is W1, represents the maximum moment in the sliding window when the sliding window size is W1; (2) Obtain the energy threshold. ； Among them, is the threshold energy ratio coefficient, is the average energy at time t, is the smoothing coefficient; (3) Obtain the zero-crossing rate threshold. ； Among them, is the zero-crossing rate threshold proportionality coefficient, is the average number of zero-crossings at time t, is the smoothing coefficient, and I is the indicator function.

5. The method for identifying the orthopedic manipulation actions based on multi-modal data according to claim 4, characterized in that: S33 is specifically: If the following conditions are satisfied simultaneously , the current time t is recorded as the start time of the activity ; if the following conditions are satisfied simultaneously , the current time t is recorded as the end time of the activity ; where is the minimum duration of the activity segment set; the valid time interval of the activity segment is .

6. The osteopathic manipulation motion recognition method based on multi-modal data according to claim 1, characterized in that: S4 is specifically: The multi-modal interval feature signal is , where , , and are respectively the interval vector, the minimum value within the window, and the maximum value within the window of the bilateral electromyography signal of the i-th channel under the p-th window; , where , and are respectively the interval vector, the minimum value within the window, and the maximum value within the window of the j-th six-dimensional force signal component under the p-th window; For multi-modal interval feature signals The extracted features include the interval mean of the bilateral EMG signals and the six-dimensional force signals for the p-th window , the interval variance , the interval energy , and the interval entropy of the bilateral EMG signals for the p-th window . The multi-modal interval feature vector for the p-th window is obtained as , with the label being the action category .

7. The method for identifying the bone-setting manipulation actions based on multi-modal data according to claim 1, wherein: S5 also includes: S52: Interval support vector machine modeling and optimization: Use the method of interval support vector machine to perform interval action recognition; the kernel function of the interval support vector machine is the interval RBF kernel function. S53: Model verification: Integrate the verification results obtained from each round according to the validation dataset, evaluate the trained interval support vector machine model, and output the accuracy rate and confusion matrix.

8. An identification system for the osteopathic manipulation motion identification method based on multi-modal data according to claim 1, characterized in that: It includes a database, a data preprocessing module, a data fusion module, an active segment division module, an interval feature extraction module, and an interval support vector machine module, and they are connected in sequence. The database is used to store the osteopathic manipulation action dataset, including the serial number, name, bilateral electromyography signals, and force signal data segments. The bilateral electromyography signals and force signals together constitute multimodal data; The data preprocessing module is used to obtain the osteopathic manipulation action data from the database, and perform data denoising and data normalization operations on the multimodal data according to step S1 to obtain normalized multimodal data; The data fusion module is used to align the timestamp units of different modalities of the normalized multimodal data obtained by the data preprocessing module using the method in S2, and then fuse the multimodal data using the linear interpolation and nearest neighbor matching algorithms to obtain normalized data fusion multimodal data; The activity segment division module is used to divide the normalized data fusion multimodal data obtained by the data fusion module using the method in S3, and label each moment of the normalized data fusion multimodal data using the adaptive state detection conversion algorithm; The interval feature extraction module is used to determine the action category and extract the multimodal interval feature vectors for the multimodal interval feature signals of the normalized data fusion multimodal data with labeled tags using the method in S4; The interval support vector machine module first models and optimizes according to the multimodal interval feature vectors obtained by the interval feature extraction recognition module using the method in S5, and then is used to perform action recognition on the input new osteopathic manipulation action data.

9. The recognition system for the osteopathic manipulation motion recognition method based on multi-modal data according to claim 8, characterized in that: The interval support vector machine module specifically includes: a dataset construction sub-module, an action recognition sub-module, and a result evaluation sub-module; Among them, the dataset construction sub-module uses the osteopathic manipulation action dataset to establish an action category dataset, and divides the action category dataset using the five-fold cross-validation method into a training set and a validation set; The action recognition sub-module performs interval action recognition on the normalized data fusion multimodal data using the interval support vector machine method, and constructs an interval support vector machine model based on osteopathic manipulation action recognition; The result evaluation sub-module calculates the accuracy rate and confusion matrix for the constructed interval support vector machine model based on osteopathic manipulation action recognition, and conducts evaluation and verification.

Images