Intelligent rehabilitation training evaluation method based on multi-modal information fusion

Abstract

The invention discloses an intelligent rehabilitation training evaluation method, device and equipment based on multi-modal information fusion and a computer readable storage medium, and the method comprises the steps: synchronously collecting user action video data and electromyographic signal data for a user performing rehabilitation training; aligning the user action video data and the electromyographic signal data based on the first timestamp and the second timestamp, and respectively generating a synchronous user posture feature sequence and a synchronous user electromyographic feature sequence; generating a comprehensive rehabilitation evaluation report by applying a feature layer fusion dynamic time warping algorithm; updating the personalized evaluation benchmark when the benchmark updating condition is met; when the stage promotion condition is met, determining that the user enters a new rehabilitation stage; on the basis of the comprehensive rehabilitation evaluation report and the new rehabilitation stage, targeted rehabilitation training guidance is generated and output. The method has the advantages of accurately identifying and deeply diagnosing compensatory actions and providing a personalized and intelligent adaptive rehabilitation process.

Description

Technical Field

[0001] The present invention relates to the field of rehabilitation training technology, and in particular to an intelligent rehabilitation training evaluation method, device, equipment and computer-readable storage medium based on multimodal information fusion. Background Art

[0002] Rehabilitation training is an important medical approach to help patients recover from or improve functional impairments caused by injuries or illnesses. Traditional rehabilitation training assessment relies primarily on the therapist's observational experience, which is highly subjective and time-consuming. Technological advancements have led to the emergence of various objective, sensor-based assessment methods. Visual-based movement assessment and surface electromyography (EMG)-based assessment are two of the main approaches.

[0003] However, existing technical solutions have several inherent defects in achieving accurate rehabilitation assessment. On the one hand, although visual assessment methods based solely on posture estimation can use algorithms such as dynamic time warping (DTW) to evaluate the standardization of movements, they mainly focus on the spatial trajectory and temporal rhythm of movements, lack consideration of muscle force, and cannot effectively identify compensatory movements that are "similar in form but not in spirit." On the other hand, assessment methods based solely on electromyographic signals, although they can reflect the functional state of muscles, are difficult to independently evaluate the overall coordination and spatial accuracy of movements. Although some studies have attempted to combine these two modalities, their data collection and analysis processes are often discrete, especially the lack of an effective timestamp synchronization mechanism, which makes it difficult to accurately associate the instantaneous relationship between specific movement postures and muscle force, greatly reducing the effectiveness of fusion assessment.

[0004] In summary, existing rehabilitation training assessment technologies generally suffer from a single assessment dimension or weak correlation between multimodal data. The root cause lies in the lack of precise synchronization between visual motion capture data and electromyographic signal data. This deficiency results in a lack of comprehensiveness in assessment reports, incapable of providing comprehensive feedback on multiple dimensions such as time, space, and force. This, in turn, results in a lack of intelligent and personalized adjustment of assessment benchmarks and rehabilitation stages. Therefore, this field urgently needs a new intelligent assessment solution that can dynamically adjust assessment benchmarks and intelligently divide rehabilitation stages. Summary of the Invention

[0005] The embodiments of the present application provide an intelligent rehabilitation training evaluation method based on multimodal information fusion, aiming to achieve dynamic adjustment of evaluation criteria and intelligent division of rehabilitation stages.

[0006] To achieve the above objectives, the present invention provides an intelligent rehabilitation training evaluation method based on multimodal information fusion, comprising:

[0007] synchronously collecting, for a user undergoing rehabilitation training, user motion video data generated by a vision sensor and containing first timestamps, and electromyography signal data generated by an electromyography sensor and containing second timestamps;

[0008] aligning the user motion video data and the electromyography signal data based on the first timestamps and the second timestamps, and performing feature extraction on the aligned data to respectively generate a synchronized user posture feature sequence and a user electromyography feature sequence;

[0009] applying a feature layer fusion dynamic time warping algorithm to compare the synchronized user posture feature sequence and the user electromyography feature sequence with corresponding standard posture feature sequence and standard electromyography feature sequence respectively, to generate a comprehensive rehabilitation evaluation report containing evaluation results of spatial accuracy, force adaptability, and posture-muscle activation synchronization;

[0010] comparing the evaluation results of the comprehensive rehabilitation evaluation report with pre-stored personalized evaluation criteria, and updating the personalized evaluation criteria when a criterion updating condition is met;

[0011] based on the comprehensive rehabilitation evaluation report and historical evaluation data, determining that the user enters a new rehabilitation stage when a stage promotion condition is met;

[0012] based on the comprehensive rehabilitation evaluation report and the new rehabilitation stage, generating and outputting targeted rehabilitation training guidance.

[0013] In an embodiment, aligning the user motion video data and the electromyography signal data based on the first timestamps and the second timestamps comprises:

[0014] aligning system clocks of the vision sensor and the electromyography sensor with a unified network time protocol server;

[0015] for each frame of the user motion video data, finding and matching a corresponding electromyography signal window in the electromyography signal data according to the first timestamp of the frame.

[0016] In an embodiment, performing feature extraction on the aligned data to respectively generate a synchronized user posture feature sequence and a user electromyography feature sequence comprises:

[0017] performing a posture estimation algorithm on each frame of the user motion video data to extract joint three-dimensional coordinates to constitute the user posture feature sequence;

[0018] performing band-pass filtering processing on electromyography signal data in each of the electromyography signal windows, and calculating at least one of a root mean square value, an integrated electromyography value, and an average power frequency of the filtered signal to constitute the user electromyography feature sequence.

[0019] In an embodiment, the feature layer fusion dynamic time warping algorithm calculates the local cost d_fused between corresponding frames in two sets of synchronized feature sequences by the following formula:

[0020] d_fused=w_pose·d(P_u,P_s)+w_emg·d(E_u,E_s),

[0021] wherein P_u and E_u are user posture features and electromyographic features extracted from the user posture feature sequence and the user electromyographic feature sequence respectively, P_s and E_s are corresponding standard posture features and electromyographic features, d is a distance function, and w_pose and w_emg are preset modal weights.

[0022] In an embodiment, the evaluation result of the comprehensive rehabilitation evaluation report is compared with a pre-stored personalized evaluation benchmark, and the personalized evaluation benchmark is updated when a benchmark update condition is met, including:

[0023] When the comprehensive score of the comprehensive rehabilitation evaluation report is higher than the corresponding score of the personalized evaluation benchmark, a confirmation request is sent to a rehabilitation therapist;

[0024] After receiving the confirmation instruction sent by the rehabilitation therapist, the current user posture feature sequence and the user electromyographic feature sequence are replaced with the standard posture and electromyographic feature sequences in the pre-stored personalized evaluation benchmark.

[0025] In an embodiment, based on the comprehensive rehabilitation evaluation report and historical evaluation data, when a stage promotion condition is met, it is determined that the user enters a new rehabilitation stage, including:

[0026] From the historical evaluation data, the number of continuous evaluations of the user in the current rehabilitation stage is retrieved;

[0027] It is determined whether the number of times that the comprehensive score continuously meets the standard exceeds a preset promotion number threshold in the number of continuous evaluations;

[0028] When the number of continuous meeting the standard exceeds the promotion number threshold, the rehabilitation stage identifier is updated in a user profile associated with the user identity.

[0029] In an embodiment, based on the comprehensive rehabilitation evaluation report and the new rehabilitation stage, a targeted rehabilitation training guidance is generated and output, including:

[0030] From the comprehensive rehabilitation evaluation report, structured error feature data is extracted, the error feature data including a deviation value of a spatial trajectory and an electromyographic activation level of a target muscle;

[0031] The structured error feature data is used as input prompts and fed into a language generation model to generate natural language feedback including specific action guidance and muscle force prompts.

[0032] To achieve the above objectives, the present application also proposes an intelligent rehabilitation training evaluation device based on multimodal information fusion, comprising:

[0033] A data acquisition module is used to synchronously collect user action video data containing a first time stamp generated by a visual sensor and myoelectric signal data containing a second time stamp generated by an electromyographic sensor for a user undergoing rehabilitation training;

[0034] a synchronization and extraction module, configured to align the user motion video data and the electromyographic signal data based on the first timestamp and the second timestamp, and perform feature extraction on the aligned user motion video data and the electromyographic signal data to generate a synchronized user posture feature sequence and a user electromyographic feature sequence, respectively;

[0035] a comprehensive evaluation module, configured to apply a feature-layer fusion dynamic time warping algorithm to comprehensively compare the synchronized user posture feature sequence and the user's electromyographic feature sequence with corresponding standard posture feature sequence and standard electromyographic feature sequence, so as to generate a comprehensive rehabilitation evaluation report including evaluation results of spatial accuracy, force adaptability, and posture-muscle activation synchronization;

[0036] A benchmark management module, configured to compare the assessment results of the comprehensive rehabilitation assessment report with a pre-stored personalized assessment benchmark, and update the personalized assessment benchmark when a benchmark update condition is met;

[0037] a stage division module for determining, based on the comprehensive rehabilitation assessment report and historical assessment data, that the user enters a new rehabilitation stage when stage advancement conditions are met;

[0038] The guidance generation module is used to generate and output targeted rehabilitation training guidance based on the comprehensive rehabilitation assessment report and the new rehabilitation stage.

[0039] To achieve the above-mentioned purpose, an embodiment of the present application also proposes an intelligent rehabilitation training evaluation device based on multimodal information fusion, including a memory, a processor, and an intelligent rehabilitation training evaluation program based on multimodal information fusion stored in the memory and runnable on the processor. When the processor executes the intelligent rehabilitation training evaluation program based on multimodal information fusion, it implements the intelligent rehabilitation training evaluation method based on multimodal information fusion as described in any one of the above items.

[0040] To achieve the above object, the embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores an intelligent rehabilitation training evaluation program based on multi-modal information fusion, and the intelligent rehabilitation training evaluation program based on multi-modal information fusion is executed by a processor to realize the intelligent rehabilitation training evaluation method based on multi-modal information fusion.

[0041] The intelligent rehabilitation training evaluation method based on multi-modal information fusion has the following beneficial effects:

[0042] 1. Precise identification and deep diagnosis of compensatory actions are realized

[0043] The application adopts a feature layer fusion dynamic time warping algorithm, which performs weighted fusion of posture features reflecting external motion forms and electromyographic features reflecting internal muscle forces in the cost calculation link of action comparison. This "early fusion" strategy enables the method to deeply analyze the instantaneous relationship between posture and muscle activation, thereby accurately identifying the "similar in shape but not in spirit" compensatory actions that cannot be found by traditional visual methods, which are extremely critical in rehabilitation training, and greatly improving the clinical diagnostic value of the evaluation.

[0044] 2. An objective, comprehensive and multi-dimensional rehabilitation evaluation report is provided

[0045] The application can further decompose and generate independent evaluation indicators for multiple dimensions such as spatial accuracy, force adaptability and posture-muscle activation synchronicity from the comparison results of feature layer fusion. This makes the final comprehensive rehabilitation evaluation report extremely rich, objective and in-depth, and can provide rehabilitation therapists and patients with more comprehensive and in-depth quantitative insights than traditional methods, thereby guiding the rehabilitation process more scientifically.

[0046] 3. A highly personalized and intelligent adaptive rehabilitation process is realized

[0047] The application establishes and dynamically updates the personal evaluation benchmark for each user, and automatically divides the rehabilitation stage according to the continuous performance, thereby constructing a complete and intelligent adaptive rehabilitation process. The process ensures that the evaluation standard and training difficulty closely match the unique recovery curve of each user, avoiding the problem of low training efficiency or increased risk caused by the use of a "one-size-fits-all" universal standard, and truly realizing personalized rehabilitation. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in these drawings without creative labor.

[0049] Figure 1 A module structure diagram of an embodiment of the intelligent rehabilitation training evaluation device based on multi-modal information fusion of the present application.

[0050] Figure 2 A flowchart of an embodiment of the intelligent rehabilitation training evaluation method based on multi-modal information fusion of the present application.

[0051] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0053] In order to better understand the above technical solutions, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0054] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of other elements or steps than those listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, the several means can be embodied by one and the same item of hardware. The use of the terms "first", "second", and "third", etc. does not imply any order but rather are used as names.

[0055] As Figure 1 shown, Figure 1 is a structural schematic diagram of a server 1 (also called an intelligent rehabilitation training evaluation device based on multi-modal information fusion) of a hardware running environment involved in an embodiment of the present application.

[0056] The server of the embodiment of the present application, such as an "Internet of Things device", a smart air conditioner with networking function, a smart electric lamp, a smart power supply, an AR / VR device with networking function, a smart sound box, an automatic driving car, a PC, a smart phone, a tablet computer, an electronic book reader, a portable computer and other devices with display function.

[0057] As shown in Figure 1 The server 1 comprises a memory 11, a processor 12 and a network interface 13.

[0058] The memory 11 comprises at least one type of readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the server 1, such as a hard disk of the server 1. In other embodiments, the memory 11 can also be an external storage device of the server 1, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the server 1.

[0059] Further, the memory 11 can also comprise an internal storage unit and an external storage device of the server 1. The memory 11 can be used not only to store application software and various data installed on the server 1, such as the code of the intelligent rehabilitation training evaluation program 10 based on multi-modal information fusion, etc., but also to temporarily store data that has been output or will be output.

[0060] The processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip in some embodiments, used to run the program code or process data stored in the memory 11, such as to execute the intelligent rehabilitation training evaluation program 10 based on multi-modal information fusion, etc.

[0061] The network interface 13 can optionally comprise a standard wired interface, a wireless interface (such as a WI-FI interface), etc., usually used to establish a communication connection between the server 1 and other electronic devices.

[0062] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth communication protocol, or a combination thereof.

[0063] Optionally, the server may further include a user interface, which may include a display and an input unit such as a keyboard. The optional user interface may also include a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display, which may also be referred to as a display screen or display unit, is used to display information processed in the server 1 and to display a visual user interface.

[0064] Figure 1 Only the server 1 having components 11-13 and the intelligent rehabilitation training evaluation program 10 based on multimodal information fusion is shown. It can be understood by those skilled in the art that Figure 1 The structure shown does not constitute a limitation on the server 1 , and the server 1 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0065] In this embodiment, the processor 12 may be configured to call the intelligent rehabilitation training evaluation program based on multimodal information fusion stored in the memory 11 and perform the following operations:

[0066] For a user undergoing rehabilitation training, synchronously collecting user action video data containing a first timestamp generated by a visual sensor and myoelectric signal data containing a second timestamp generated by an electromyographic sensor;

[0067] Aligning the user motion video data and the electromyographic signal data based on the first timestamp and the second timestamp, and performing feature extraction on the aligned data to generate a synchronized user posture feature sequence and a user electromyographic feature sequence, respectively;

[0068] Applying a feature-layer fusion dynamic time warping algorithm, the synchronized user posture feature sequence and the user's electromyographic feature sequence are compared with corresponding standard posture feature sequences and standard electromyographic feature sequences, respectively, to generate a comprehensive rehabilitation assessment report including spatial accuracy, force adaptability, and posture-muscle activation synchronization assessment results;

[0069] Comparing the evaluation results of the comprehensive rehabilitation evaluation report with a pre-stored personalized evaluation benchmark, and updating the personalized evaluation benchmark when a benchmark update condition is met;

[0070] Based on the comprehensive rehabilitation assessment report and historical assessment data, determining that the user enters a new rehabilitation stage when stage advancement conditions are met;

[0071] Based on the comprehensive rehabilitation assessment report and the new rehabilitation stage, targeted rehabilitation training guidance is generated and output.

[0072] Based on the hardware architecture of the aforementioned intelligent rehabilitation training assessment device based on multimodal information fusion, an embodiment of the present invention's intelligent rehabilitation training assessment method based on multimodal information fusion is proposed. This intelligent rehabilitation training assessment method based on multimodal information fusion aims to dynamically adjust assessment benchmarks and intelligently divide rehabilitation stages.

[0073] Reference Figure 2 , Figure 2 This is an embodiment of the intelligent rehabilitation training evaluation method based on multimodal information fusion of the present invention, which includes the following steps:

[0074] S10 . For a user undergoing rehabilitation training, synchronously collect user motion video data containing a first timestamp generated by a visual sensor and myoelectric signal data containing a second timestamp generated by an electromyographic sensor.

[0075] Specifically, user motion video data is a continuous sequence of image frames captured by visual sensors, reflecting the user's body posture and motion trajectory during rehabilitation training. Myoelectric signal data is a continuous bioelectric signal captured by surface electromyography sensors, reflecting the electrical activity level of the target muscle group during the user's movements. Both data streams contain independent timestamps: the first timestamp and the second timestamp.

[0076] In a specific embodiment, the synchronous acquisition process is as follows: the system simultaneously starts one or more visual sensors (e.g., cameras) and one or more surface electromyography sensors. The visual sensor is responsible for capturing the user's overall movements from the outside and generating a video data stream. At the same time, the surface electromyography sensor (e.g., through Ag / AgCl electrodes attached to the skin surface of the target muscle group, such as the quadriceps femoris) detects the weak voltage changes generated during muscle contraction, and after amplification and filtering, generates an electromyography signal data stream. In order to achieve precise time synchronization between the two, a key implementation method is to align the system clocks of all devices deploying visual sensors and electromyography sensors with a unified network time protocol (NTP) server. In this way, the first timestamp assigned by the system to each frame of the video data stream and the second timestamp assigned to each sampling point of the electromyography signal data stream will come from the same high-precision time reference, thereby ensuring the accuracy of subsequent data alignment.

[0077] For example, a patient is undergoing rehabilitation training for the "sitting knee extension" exercise. A camera captures their leg movements from the side, while a pair of EMG electrodes are attached to the quadriceps muscles on the front of their thighs. At a certain time t, the camera captures an image frame and assigns it a first timestamp of 1656570000.125. Simultaneously, the EMG sensor detects a signal point with a voltage of 0.8mV and assigns it a second timestamp of 1656570000.126. This series of synchronized video frames and EMG sampling points is the output of this step.

[0078] As can be understood, by aligning the system clocks of all acquisition devices with a unified network time protocol server, it is possible to ensure that the timestamps of the two heterogeneous signal sources, visual data and electromyographic data, have a unified, high-precision time base, fundamentally solving the problem of multimodal data synchronization. Furthermore, by synchronously acquiring video data reflecting external motion morphology and electromyographic data reflecting internal muscle exertion, the necessary high-quality synchronized data input is provided for subsequent steps, using feature-layer fusion algorithms to deeply analyze the instantaneous relationship between posture and muscle activation, and accurately identify compensatory movements that are "similar in form but not in spirit."

[0079] S20. Align the user action video data and the electromyographic signal data based on the first timestamp and the second timestamp, and perform feature extraction on the aligned data to generate a synchronized user posture feature sequence and a user electromyographic feature sequence, respectively.

[0080] Specifically, the user posture feature sequence is a time series data that has undergone feature extraction, and each element of the sequence represents the user's body posture feature (such as the three-dimensional coordinates of a joint) at a specific synchronous time point. The user electromyography feature sequence is another time series data that strictly corresponds to the former in time, and each element of the sequence represents the user's muscle activation state feature (such as the root mean square value of the electromyography signal) at the same synchronous time point.

[0081] In step S20, this method performs the crucial data alignment and feature extraction steps in the entire multimodal assessment process. This step serves two key purposes: First, based on the two heterogeneous data streams (video and EMG) collected in S10 and each with its own time stamp, precise time alignment ensures that each data point in subsequent analysis contains the same posture and EMG information at the same moment. Second, preliminary processing and feature extraction are performed on the aligned raw data, converting them from their original signal form into a structured feature sequence that can be directly used by the subsequent fusion and comparison algorithms.

[0082] In a specific embodiment, the above-mentioned process of aligning the user action video data and the electromyographic signal data is achieved by the following steps S21 and S22:

[0083] First, in step S21, to fundamentally ensure the reliability and comparability of timestamps, the system periodically calibrates and aligns the system clocks of the hardware devices used for visual sensors (such as cameras) used to capture video and myoelectric sensors used to collect myoelectric signals with a unified Network Time Protocol (NTP) server. This ensures that the first timestamp of the video frame and the second timestamp of the myoelectric signal are derived from the same high-precision time reference, eliminating synchronization errors that may be caused by clock drift between the devices.

[0084] After ensuring the time reference unification, in step S22, the system performs data frame matching based on the video data stream. Specifically, the system traverses each frame in the user action video data. For each frame image, the system records its first timestamp. Then, the system takes this timestamp as the center, and searches for and extracts a corresponding electromyography signal window in the synchronously collected electromyography signal data stream. For example, if the timestamp of the video frame is t, and the sampling rate of the electromyography signal is 1000 Hz, the system can extract a time window from t-50 ms to t+50 ms, which contains 100 electromyography signal sampling points. The electromyography data in this window is considered to correspond to the muscle activity at the moment represented by the posture of the video frame. Subsequently, the system performs posture feature extraction on the video frame, and electromyography feature extraction on the data in the electromyography signal window, thereby generating a pair of strictly synchronized posture features and electromyography features, and storing them in corresponding feature sequences.

[0085] It can be understood that, by matching a time-closely corresponding electromyography signal window for each posture snapshot (video frame), a momentary one-to-one corresponding relationship between posture and muscle effort in different dimensions of information is successfully established, thereby providing high-quality, non-offset synchronous data input for subsequent feature layer fusion analysis.

[0086] In a specific embodiment, the process of feature extraction on the aligned data in step S20 is realized in parallel through steps S23 and S24. The core of this embodiment is that it designs a targeted feature extraction pipeline for each of the two different modal original data streams, to convert them into structured feature sequences that can be used for subsequent algorithm processing.

[0087] First, in step S23, the system performs a preset posture estimation algorithm on each video frame that is aligned with the electromyography signal window. This algorithm can identify and extract the three-dimensional space coordinates of all body joint nodes of the user from the frame image. The set of three-dimensional coordinates extracted from all consecutive frames, arranged in time sequence, constitutes the posture feature sequence of the user.

[0088] At the same time, in step S24, the system performs a series of signal processing and feature calculations on the raw electromyographic signal data in the electromyographic signal window corresponding to each video frame. The process first includes a bandpass filtering process on the raw electromyographic signal, for example, filtering out low-frequency noise below 5Hz caused by motion artifacts and high-frequency noise above 500Hz to obtain a purer electromyographic signal. After obtaining the filtered signal, the system calculates one or more key characteristic parameters, such as: root mean square value (RMS), which can reflect the overall activity intensity of the muscle over a period of time; integrated electromyographic value (iEMG), which is the accumulation of the absolute value of the signal over time, also reflecting the total activity of the muscle; or average power frequency (MPF), which can reflect the type and fatigue state of the muscle fiber. These electromyographic feature parameters (for example, a series of RMS values) calculated corresponding to all video frames are arranged in chronological order to form the user's electromyographic feature sequence.

[0089] For example, for the synchronized data point at t = 10.5 seconds, the system extracts a set of three-dimensional joint coordinates from the corresponding video frame in S23 as an element in the posture feature sequence. In S24, the system processes the data within the 100ms EMG signal window from t = 10.45 seconds to t = 10.55 seconds. First, it performs a bandpass filter and calculates the RMS value of the signal within this 100ms window to be 1.2mV. This 1.2mV value is then used as the element in the EMG feature sequence corresponding to the posture at t = 10.5 seconds.

[0090] As can be seen, by designing independent feature extraction processes for visual data and electromyographic data that are tailored to their respective signal characteristics, high-quality structured features representing "kinematic morphology" and "muscle dynamics" can be effectively extracted from these two distinct raw signals. This, in turn, provides the subsequent feature-layer fusion DTW algorithm with two parallel input sequences that are strictly synchronized in time, rich in content, and standardized in data format, enabling accurate compensatory movement recognition.

[0091] S30. Apply the feature layer fusion dynamic time warping algorithm to compare the synchronized user posture feature sequence and the user electromyography feature sequence with the corresponding standard posture feature sequence and standard electromyography feature sequence, respectively, to generate a comprehensive rehabilitation assessment report including spatial accuracy, force adaptability and posture-muscle activation synchronization assessment results.

[0092] Specifically, the core of the feature layer fusion dynamic time warping algorithm lies in the calculation of the cost function, which weightedly fuses the features from two different modalities: posture and electromyography.

[0093] The comprehensive rehabilitation assessment report is a final output that contains a structured report with detailed assessment results of the user's rehabilitation movements in multiple dimensions (such as space, strength, and timing coordination).

[0094] In step S30, this method performs a comprehensive comparison of the synchronized user posture and EMG feature sequences generated in step S20 against a pre-stored standard template. This step no longer evaluates posture and EMG independently, but instead integrates these two modalities at the lowest level of the algorithm, enabling in-depth analysis of the instantaneous relationship between posture and muscle activation.

[0095] In a specific embodiment, the comparison process of the above-mentioned application feature layer fusion dynamic time warping algorithm includes the following core steps:

[0096] First, the local cost is calculated and the cost matrix is ​​constructed. The system initializes a two-dimensional cost matrix D of size N x M, where N is the length of the user's synchronized feature sequence and M is the length of the standard template sequence. For the i-th time point in the user sequence and the j-th time point in the template sequence, the corresponding local cost d_{fused}(i,j) is calculated using the following fusion formula:

[0097] d fused (i, j) = w pose ·d(P u (i), P s (j))+w emg ·d(E u (i), E s (j)),

[0098] Where P_u(i) and E_u(i) are the user's posture and electromyographic features at time i, respectively. P_s(j) and E_s(j) are the standard features corresponding to time j. d(...) is a distance function, such as the Euclidean distance. w_{pose} and w_{emg} are preset modal weights. The system iterates over all (i, j) combinations, calculates all d_{fused} values, and uses them to populate the cost matrix D.

[0099] After constructing the cost matrix D, the system will then construct the cumulative cost matrix γ and find the optimal regularized path. The system will construct another cumulative cost matrix γ of the same size and fill it with the following dynamic programming recursive formula:

[0100] γ(i,j)=D(i,j)+min{γ(i-1,j), γ(i-1,j-1), γ(i,j-1)}.

[0101] After the filling is completed, the value γ(N, M) located at the terminal point (N, M) of the matrix is the minimum cumulative total cost. The system will start from this terminal point and find a path that minimizes the total cost through backtracking. This path is the optimal normalization path.

[0102] Finally, after obtaining the optimal normalization path and the minimum cumulative total cost, the system generates evaluation results in multiple dimensions such as spatial accuracy, force adaptation, and posture-muscle activation synchronization. The process is as follows:

[0103] 1. Generation of spatial accuracy evaluation results: The system will separately extract the posture component in the fusion cost formula, i.e., w_pose·d(P_u, P_s). By accumulating and normalizing these posture components along the optimal normalization path, a score reflecting the similarity between user actions and standard templates in spatial geometry is obtained.

[0104] 2. Generation of force adaptation evaluation results: Similarly, the system will separately extract the electromyography component, i.e., w_emg·d(E_u, E_s)). By accumulating and normalizing these electromyography components along the optimal normalization path, a score reflecting the difference between user target muscle activation levels and standard patterns is obtained. This score can be used to determine whether there are problems such as excessive muscle activation, insufficient muscle activation, etc.

[0105] 3. Generation of posture-muscle activation synchronization evaluation results: First, one or more "expected synchronization events" are defined in the standard template, such as "when the knee joint extension angle reaches its maximum value, the root mean square (RMS) value of the quadriceps muscle electromyography signal is expected to reach its peak value". After obtaining the optimal normalization path, the system will first locate the frame where the knee joint angle reaches its maximum value in the standard template. Then, through the optimal normalization path, the actual posture frame corresponding to this frame in the user sequence is found. Next, the system searches for the actual peak point of the user's quadriceps muscle RMS value within a small time window around the actual posture frame. Finally, the system calculates the "time difference" between the expected event occurrence time point in the standard template and the actual electromyography peak time point in the user sequence. This time difference (e.g., 50 milliseconds) or its normalized correlation score is the final posture-muscle activation synchronization evaluation result, which accurately quantifies whether the user's muscle activation is "ahead of" or "behind" their body posture.

[0106] Finally, the system will integrate the evaluation results of the above three dimensions (spatial accuracy, force adaptation, and synchronization) to form the final comprehensive rehabilitation evaluation report.

[0107] As can be understood, by weightedly fusing posture and electromyographic features at the feature level when calculating the local cost of DTW, it is possible to capture the subtle correlation between the user's posture performance and their muscle force pattern at each moment, making it possible to accurately identify compensatory movements. Furthermore, since the fused comparison results can be further decomposed and generated into independent evaluation indicators for multiple dimensions such as spatial, force, and temporal coordination, the final rehabilitation assessment report is extremely rich in content and highly diagnostic, providing rehabilitation therapists and patients with far more comprehensive and in-depth insights than traditional methods.

[0108] S40: Compare the evaluation result of the comprehensive rehabilitation evaluation report with a pre-stored personalized evaluation benchmark, and update the personalized evaluation benchmark when a benchmark update condition is met.

[0109] Specifically, the personalized evaluation benchmark is a reference template associated with the user as a record of his or her personal best performance. The template contains the standard posture feature sequence and standard electromyographic feature sequence corresponding to the user's high-quality completion of an action.

[0110] A baseline update condition is a logical condition used to determine whether an older personal best should be replaced with a current, better performance.

[0111] In step S40, the method performs a personalized, dynamic assessment baseline management step. The core purpose of this step is to enable the assessment system to "learn" and "adapt" to the user's individual progress. Rather than using a fixed, universally applicable "standard template," it maintains a personalized "best performance record" for each user. When the quality of a user's performance exceeds their historical best, the system updates this personal baseline, ensuring that subsequent assessments are based on the user's individual performance level, achieving true personalization.

[0112] In a specific embodiment, the process of updating the personalized evaluation benchmark is implemented through a process including "human-machine collaborative confirmation", namely steps S41 and S42:

[0113] First, in step S41, if the comprehensive score in the comprehensive rehabilitation assessment report generated in step S30 exceeds the score corresponding to the currently stored personalized assessment benchmark, the system automatically sends a confirmation request to the management terminal of the authorized rehabilitation therapist. This request displays the assessment results of the current action and prompts the rehabilitation therapist, "The patient's performance has exceeded the historical best. Do you want to set this as the new personal standard?"

[0114] After the therapist analyzes the patient's movement and makes a professional judgment, if he believes that this performance is indeed a high-quality, non-compensatory, and worthy of a new standard movement, he will send a confirmation instruction. In step S42, after receiving the confirmation instruction, the system will perform the actual update operation. It will replace the old standard sequence originally stored in the personalized evaluation benchmark with the complete user posture feature sequence and user electromyographic feature sequence corresponding to the current high-quality completed movement.

[0115] For example, assume that a patient's personalized evaluation benchmark (i.e., his historical best performance) corresponds to a comprehensive score of 85. In a certain training, the patient performs exceptionally well, and the comprehensive score generated by S30 is 92. In step S41, since 92 is higher than 85, the system will pop up a prompt box on the therapist's computer screen. The therapist may play back the patient's movement video and electromyographic curve, and after confirming that the movement is correct and the force is correct, clicks the "Confirm Update" button. In step S42, after the system receives the instruction, it will replace the old sequence corresponding to the score of 85 with the complete posture and electromyographic feature sequence corresponding to the score of 92 as the patient's new, higher personal evaluation benchmark.

[0116] It can be understood that by dynamically updating the user's personal evaluation benchmark according to the user's real-time performance, the evaluation standard can closely match the user's current actual ability level, achieving high personalization and avoiding inaccurate evaluation due to the use of a "one-size-fits-all" universal standard. In addition, the introduction of the therapist's confirmation link when updating the benchmark ensures that each benchmark upgrade is conducted under professional and safe guidance, effectively avoiding the situation where the patient's occasional, but potentially risky "exceptional performance" is mistakenly adopted as the new standard by the system, significantly improving the reliability and clinical practical value of the entire adaptive evaluation system.

[0117] S50, based on the comprehensive rehabilitation evaluation report and historical evaluation data, when the stage promotion condition is met, determining that the user enters a new rehabilitation stage.

[0118] Specifically, the historical evaluation data is the updated training record in S40, which includes the user's evaluation records of previous rehabilitation training.

[0119] The rehabilitation stage refers to different levels of rehabilitation training difficulty or content according to the patient's ability level, such as from the "early passive activity stage" to the "late active resistance training stage".

[0120] The stage promotion condition is a logical condition for determining whether the user has the ability to enter the next higher level of rehabilitation stage.

[0121] The core purpose of this step is to automatically determine and confirm that the user can enter a new rehabilitation stage based on the user's current and historical comprehensive rehabilitation evaluation performance when the preset stage promotion condition is met. This enables the rehabilitation plan to adaptively and stepwise evolve according to the user's actual recovery progress, avoiding long-term stagnation in excessively low or high difficulty training stages, thereby maximizing rehabilitation efficiency.

[0122] In a specific embodiment, the process of determining that the user enters a new rehabilitation stage is implemented through the following steps S51 to S53:

[0123] First, in step S51, after the user completes a new evaluation, the system retrieves from the historical evaluation data of the user how many times the user has consecutively performed evaluations in the current rehabilitation stage. Then, in step S52, the system determines whether the comprehensive score in the comprehensive rehabilitation evaluation report has continuously reached the preset "standard" score line in the number of consecutive evaluations, and counts the number of "continuous standard" times. Only when the number of continuous standard times exceeds a preset promotion number threshold, the system considers that the user's ability has been stably and reliably consolidated.

[0124] After meeting the above conditions, the system performs the final stage update operation in step S53. It automatically updates the rehabilitation stage identifier from the current stage to the next preset higher level rehabilitation stage in the user profile uniquely associated with the user's identity.

[0125] For example, assume that a rehabilitation plan stipulates that promotion from "Stage Two" to "Stage Three" requires a comprehensive score of 80 or more for 5 consecutive times (i.e., the promotion number threshold is 5 and the standard score is 80). A patient scored 75, 78, 81, and 79 in the early training in "Stage Two". At this time, although he reached 80 or more in the third time, he did not "sustain" and therefore did not meet the promotion condition. Subsequently, the patient made efforts and scored 82, 85, 88, 83, 86, and 90 in the next 6 evaluations. After completing the 6th evaluation (the 10th time in total), the system retrieves that the user has consecutively performed evaluations for 6 times in S51, and determines that all of the last 6 scores are higher than 80 in S52. The number of continuous standard times "6" has exceeded the preset promotion number threshold "5". Therefore, in step S53, the system automatically updates the rehabilitation stage identifier in the user profile of the patient from "Stage Two" to "Stage Three". In the next training, the system will load the training content and evaluation standards of "Stage Three" for the patient.

[0126] As you can understand, judging whether a user can "consistently and stably" achieve a certain performance level as the basis for their rehabilitation stage advancement avoids premature and unreasonable stage advancement due to a single or occasional exceptional performance, ensuring solid and safe rehabilitation progress. Furthermore, the ability to automatically update the rehabilitation stage identifier in the user's profile makes the entire rehabilitation plan management and evolution process highly automated and intelligent, reducing the burden on rehabilitation therapists to manually adjust plans and ensuring that every user receives the training challenges most appropriate to their current ability level at the most appropriate time.

[0127] S60: Generate and output targeted rehabilitation training guidance based on the comprehensive rehabilitation assessment report and the new rehabilitation stage.

[0128] Specifically, rehabilitation training guidance refers to the targeted suggestions or instructions ultimately generated by the system that aim to help users improve their rehabilitation movements and correct incorrect force patterns.

[0129] In step S60, the system generates and outputs highly targeted rehabilitation training guidance based on the comprehensive rehabilitation assessment report generated in S30 and the user's new rehabilitation stage determined in S50. The core purpose of this step is to transform the complex, multi-dimensional analysis and assessment results of all previous steps into clear, understandable, and actionable feedback that is practical and helpful to the user. This guidance not only takes into account the user's specific performance at that time, but also takes into account their overall rehabilitation stage, achieving a unified approach of tactical error correction and strategic guidance.

[0130] In a specific embodiment, the process of generating targeted rehabilitation training guidance is achieved through a process of "error feature extraction-language model generation", namely steps S61 and S62:

[0131] First, in step S61, the system extracts structured error feature data from the comprehensive rehabilitation assessment report generated in step S30. This structured data is a refined summary of the error, containing at least two key dimensions of information: the deviation value of the spatial trajectory reflecting the difference in movement form (for example, how many centimeters a joint deviates from the standard trajectory); and the electromyographic activation level of the target muscle, reflecting the correctness of the muscle force pattern (for example, whether the RMS value of the target muscle is higher or lower than the standard).

[0132] After extracting the structured error feature data containing both the "shape" and "force" aspects, in step S62, the system inputs the data as a prompt into a pre-trained language generation model. The model is trained to deeply understand the clinical rehabilitation meaning behind these structured error feature data. It will automatically organize language to generate a natural language feedback containing specific action guidance for errors and specific muscle activation prompts based on the input specific bias values and muscle activation levels. This text is the final rehabilitation training guidance.

[0133] For example, a patient after knee surgery is doing "straight leg raise" rehabilitation training. In S61, the system extracts the structured error features from the evaluation report as: {error type: 'compensation', spatial bias: 'hip flexor compensation 2 cm', muscle activation level: {quadriceps_RMS: '30% lower than standard', iliopsoas_RMS: '40% higher than standard'}}. In step S62, this structured data is input into the language generation model. After receiving the prompt, the model may generate the following rehabilitation training guidance: "In this leg raising exercise, we detect that you are not using enough muscle power on the front of your thigh (quadriceps) and relying too much on the hip muscles to lift your leg. Please note that the core of this training is to activate and strengthen the quadriceps. Next time, please focus your attention completely on the front of your thigh, feel its tightening and activation, and try to keep your hips relaxed and stable."

[0134] It can be understood that by extracting structured error features containing both spatial trajectory bias and muscle activation level from comprehensive rehabilitation evaluation reports, an extremely specific, deep, and directly targeted diagnostic input is provided for subsequent feedback generation. In addition, by further utilizing the language generation model to process these complex, multi-modal error feature data, the method can automatically convert professional, clinical-level rehabilitation diagnosis and analysis into humanized oral guidance that ordinary patients can easily understand and execute, greatly improving the effectiveness and accessibility of intelligent rehabilitation training.

[0135] From the above embodiments, it can be seen that the intelligent rehabilitation training evaluation method based on multi-modal information fusion has the following beneficial effects:

[0136] 1. Accurate identification and deep diagnosis of compensatory movements are achieved

[0137] This application uses a feature-layer fusion dynamic time warping algorithm that, during the cost calculation phase of motion comparison, weightedly fuses posture features reflecting external motion morphology with myoelectric features reflecting internal muscle exertion. This "early fusion" strategy enables the method to deeply analyze the instantaneous relationship between posture and muscle activation, thereby accurately identifying compensatory movements that are "similar in form but not in spirit" and are crucial in rehabilitation training, which are invisible to traditional purely visual methods. This greatly enhances the clinical diagnostic value of the assessment.

[0138] 2. Provides an objective, comprehensive and multi-dimensional rehabilitation assessment report

[0139] This application can further decompose and generate independent evaluation indicators for multiple dimensions, such as spatial accuracy, force adaptability, and posture-muscle activation synchronization, from the comparison results of feature-level fusion. This makes the final comprehensive rehabilitation assessment report extremely rich, objective, and in-depth, providing rehabilitation therapists and patients with far more comprehensive and in-depth quantitative insights than traditional methods, thereby more scientifically guiding the rehabilitation process.

[0140] 3. Achieved a highly personalized and intelligent adaptive rehabilitation process

[0141] This application builds a complete, intelligent, adaptive rehabilitation process by establishing and dynamically updating a personalized assessment benchmark for each user and automatically stratifying rehabilitation stages based on their ongoing performance. This process ensures that assessment criteria and training difficulty closely align with each user's unique recovery curve, avoiding the inefficient training or increased risk associated with a one-size-fits-all approach, truly achieving personalized rehabilitation tailored to each individual.

[0142] The embodiment of the present invention further proposes an intelligent rehabilitation training evaluation device based on multimodal information fusion, wherein the intelligent rehabilitation training evaluation device based on multimodal information fusion comprises:

[0143] A data acquisition module is used to synchronously collect user action video data containing a first time stamp generated by a visual sensor and myoelectric signal data containing a second time stamp generated by an electromyographic sensor for a user undergoing rehabilitation training;

[0144] a synchronization and extraction module, configured to align the user motion video data and the electromyographic signal data based on the first timestamp and the second timestamp, and perform feature extraction on the aligned user motion video data and the electromyographic signal data to generate a synchronized user posture feature sequence and a user electromyographic feature sequence, respectively;

[0145] a comprehensive evaluation module, configured to apply a feature-layer fusion dynamic time warping algorithm to comprehensively compare the synchronized user posture feature sequence and the user's electromyographic feature sequence with corresponding standard posture feature sequence and standard electromyographic feature sequence, so as to generate a comprehensive rehabilitation evaluation report including evaluation results of spatial accuracy, force adaptability, and posture-muscle activation synchronization;

[0146] A benchmark management module, configured to compare the assessment results of the comprehensive rehabilitation assessment report with a pre-stored personalized assessment benchmark, and update the personalized assessment benchmark when a benchmark update condition is met;

[0147] a stage division module for determining, based on the comprehensive rehabilitation assessment report and historical assessment data, that the user enters a new rehabilitation stage when stage advancement conditions are met;

[0148] The guidance generation module is used to generate and output targeted rehabilitation training guidance based on the comprehensive rehabilitation assessment report and the new rehabilitation stage.

[0149] Among them, the steps for implementing each functional module of the intelligent rehabilitation training evaluation device based on multimodal information fusion can refer to the various embodiments of the intelligent rehabilitation training evaluation method based on multimodal information fusion of the present invention, and will not be repeated here.

[0150] In addition, embodiments of the present invention further provide a computer-readable storage medium. The computer-readable storage medium can be any one of, or any combination of, a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), or a USB memory device. The computer-readable storage medium includes an intelligent rehabilitation training evaluation program 10 based on multimodal information fusion. The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the specific implementation of the intelligent rehabilitation training evaluation method based on multimodal information fusion and the server 1 described above, and will not be further described here.

[0151] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0153] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0155] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the attached claims are intended to embrace all such variations and modifications as fall within the scope of the present application.

[0156] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An intelligent rehabilitation training evaluation method based on multimodal information fusion, characterized in that: include: For a user undergoing rehabilitation training, synchronously collecting user action video data containing a first timestamp generated by a visual sensor and myoelectric signal data containing a second timestamp generated by an electromyographic sensor; Aligning the user motion video data and the electromyographic signal data based on the first timestamp and the second timestamp, and performing feature extraction on the aligned data to generate a synchronized user posture feature sequence and a user electromyographic feature sequence, respectively; Applying a feature-layer fusion dynamic time warping algorithm, the synchronized user posture feature sequence and the user's electromyographic feature sequence are compared with corresponding standard posture feature sequences and standard electromyographic feature sequences, respectively, to generate a comprehensive rehabilitation assessment report including spatial accuracy, force adaptability, and posture-muscle activation synchronization assessment results; Comparing the evaluation results of the comprehensive rehabilitation evaluation report with a pre-stored personalized evaluation benchmark, and updating the personalized evaluation benchmark when a benchmark update condition is met; Based on the comprehensive rehabilitation assessment report and historical assessment data, determining that the user enters a new rehabilitation stage when stage advancement conditions are met; Based on the comprehensive rehabilitation assessment report and the new rehabilitation stage, targeted rehabilitation training guidance is generated and output.

2. The intelligent rehabilitation training evaluation method based on multimodal information fusion according to claim 1, characterized in that: Aligning the user motion video data and the electromyographic signal data based on the first timestamp and the second timestamp includes: Aligning the system clocks of the visual sensor and the electromyographic sensor with a unified network time protocol server; For each frame in the user motion video data, a temporally corresponding electromyographic signal window is searched and matched in the electromyographic signal data according to the first timestamp of the frame.

3. The intelligent rehabilitation training evaluation method based on multimodal information fusion according to claim 2, characterized in that: Feature extraction is performed on the aligned data to generate synchronized user posture feature sequences and user electromyographic feature sequences, including: Executing a posture estimation algorithm on each frame of the user action video data to extract the three-dimensional coordinates of the joint points to form the user posture feature sequence; Bandpass filtering is performed on the electromyographic signal data in each electromyographic signal window, and at least one of the root mean square value, integrated electromyographic value and average power frequency of the filtered signal is calculated to form the user electromyographic feature sequence.

4. The intelligent rehabilitation training evaluation method based on multimodal information fusion according to claim 1, characterized in that: The feature layer fusion dynamic time warping algorithm calculates the local cost d_fused between corresponding frames in two sets of synchronized feature sequences using the following formula: d_fused=w_pose·d(P_u,P_s)+w_emg·d(E_u,E_s), Among them, P_u and E_u are the user posture features and electromyographic features extracted from the user posture feature sequence and user electromyographic feature sequence, respectively, P_s and E_s are the corresponding standard posture features and electromyographic features, respectively, d is the distance function, w_pose and w_emg are the preset modal weights.

5. The intelligent rehabilitation training evaluation method based on multimodal information fusion according to claim 1, characterized in that: Comparing the evaluation results of the comprehensive rehabilitation evaluation report with a pre-stored personalized evaluation benchmark, and updating the personalized evaluation benchmark when a benchmark update condition is met, including: When the comprehensive score of the comprehensive rehabilitation assessment report is higher than the score corresponding to the personalized assessment benchmark, sending a confirmation request to the rehabilitation therapist; After receiving the confirmation instruction sent by the rehabilitation therapist, the current user posture feature sequence and user electromyographic feature sequence replace the pre-stored standard posture and electromyographic feature sequence in the personalized evaluation benchmark.

6. The intelligent rehabilitation training evaluation method based on multimodal information fusion according to claim 1, characterized in that: Based on the comprehensive rehabilitation assessment report and historical assessment data, the user is determined to enter a new rehabilitation stage when the stage advancement conditions are met, including: Retrieving the number of consecutive assessments of the user in the current rehabilitation stage from the historical assessment data; Determining whether the number of times the comprehensive score continuously meets the standard during the consecutive evaluations exceeds a preset promotion threshold; When the number of times the target is continuously met exceeds the promotion number threshold, the rehabilitation stage identifier is updated in the user profile associated with the user identity.

7. The intelligent rehabilitation training evaluation method based on multimodal information fusion according to claim 1, characterized in that: Based on the comprehensive rehabilitation assessment report and the new rehabilitation stage, targeted rehabilitation training guidance is generated and output, including: extracting structured error feature data from the comprehensive rehabilitation assessment report, wherein the error feature data includes a deviation value of a spatial trajectory and an electromyographic activation level of a target muscle; The structured error feature data is used as input prompts and fed into a language generation model to generate natural language feedback including specific action guidance and muscle force prompts.

8. An intelligent rehabilitation training evaluation device based on multimodal information fusion, characterized in that: include: A data acquisition module is used to synchronously collect user action video data containing a first time stamp generated by a visual sensor and myoelectric signal data containing a second time stamp generated by an electromyographic sensor for a user undergoing rehabilitation training; a synchronization and extraction module, configured to align the user motion video data and the electromyographic signal data based on the first timestamp and the second timestamp, and perform feature extraction on the aligned user motion video data and the electromyographic signal data to generate a synchronized user posture feature sequence and a user electromyographic feature sequence, respectively; a comprehensive evaluation module, configured to apply a feature-layer fusion dynamic time warping algorithm to comprehensively compare the synchronized user posture feature sequence and the user's electromyographic feature sequence with corresponding standard posture feature sequence and standard electromyographic feature sequence, so as to generate a comprehensive rehabilitation evaluation report including evaluation results of spatial accuracy, force adaptability, and posture-muscle activation synchronization; A benchmark management module, configured to compare the assessment results of the comprehensive rehabilitation assessment report with a pre-stored personalized assessment benchmark, and update the personalized assessment benchmark when a benchmark update condition is met; a stage division module for determining, based on the comprehensive rehabilitation assessment report and historical assessment data, that the user enters a new rehabilitation stage when stage advancement conditions are met; The guidance generation module is used to generate and output targeted rehabilitation training guidance based on the comprehensive rehabilitation assessment report and the new rehabilitation stage.

9. An intelligent rehabilitation training assessment device based on multimodal information fusion, characterized in that: The invention comprises a memory, a processor, and an intelligent rehabilitation training evaluation program based on multimodal information fusion stored in the memory and executable on the processor, wherein when the processor executes the intelligent rehabilitation training evaluation program based on multimodal information fusion, the intelligent rehabilitation training evaluation method based on multimodal information fusion as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an intelligent rehabilitation training evaluation program based on multimodal information fusion. When the intelligent rehabilitation training evaluation program based on multimodal information fusion is executed by the processor, it implements the intelligent rehabilitation training evaluation method based on multimodal information fusion as described in any one of claims 1 to 7.

Images