Breast cancer postoperative body rehabilitation system based on multi-mode sensing and video analysis
By integrating multimodal sensing and video analytics, the problem of decoupling muscle exertion and movement trajectory in post-masturbation rehabilitation for breast cancer has been solved, enabling personalized rehabilitation assessment and real-time intervention, and improving the safety and efficiency of rehabilitation training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TONGJI HOSPITAL ATTACHED TO TONGJI MEDICAL COLLEGE HUAZHONG SCI TECH
- Filing Date
- 2026-01-12
- Publication Date
- 2026-04-21
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Current breast cancer postoperative rehabilitation techniques cannot accurately identify the decoupling problem between muscle exertion and movement trajectory, lack personalized guidance, leading to decreased patient compliance, and fail to assess balance status in real time, increasing the risk of falls.
Employing multimodal sensing and video analysis fusion technology, the system synchronously collects patient motion information and biomechanical signals through a high-definition RGB-D camera, flexible electromyography sensor, and inertial measurement unit. It combines SLAM algorithm and timestamp synchronization algorithm to achieve data alignment and uses joint-muscle-balance priority hierarchical error correction logic to generate personalized rehabilitation guidance.
It enables precise and personalized guidance for rehabilitation assessment, improves patient compliance, reduces the risk of falls, and enhances the safety and efficiency of rehabilitation training.
Smart Images
Figure CN121904464A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of rehabilitation medicine technology, and in particular to a post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis. Background Technology
[0002] Post-mastectomy patients often face problems such as limited shoulder joint mobility, lymphedema, and muscle weakness. Timely and effective rehabilitation training is crucial for restoring their physical function. Currently, commonly used clinical rehabilitation assessment and training aids fall into two main categories: one is a low-cost, single-wearable sensor solution, centered on an inertial measurement unit (IMU), which monitors upper limb movement trajectories to assess basic rehabilitation movements. This solution is cost-effective, easy to wear, and convenient for home use. The other is a high-precision optical motion capture solution, employing Kinect or optical marker technology for kinematic analysis. This solution can accurately reconstruct joint movement trajectories and calculate joint angles, providing precise kinematic data support for rehabilitation assessment. In addition, traditional rehabilitation assessments also include subjective observation by physicians, standardized scale assessments, and joint range of motion measurements, which are widely used in routine clinical rehabilitation examinations.
[0003] Existing technologies generally suffer from the problem of decoupling trajectory and force exertion. Single wearable sensors cannot acquire accurate three-dimensional spatial trajectories and are difficult to distinguish muscle exertion. While optical motion capture can accurately reconstruct trajectories, it lacks the perception of deep muscle electrical activity and cannot penetrate the surface of the trajectory to identify compensatory erroneous force exertion patterns. This results in ineffective activation of the target rehabilitation muscle groups and may even lead to secondary injuries. Furthermore, existing rehabilitation programs are mostly standardized "one-size-fits-all" content, failing to consider real-time individual obstacles such as joint stiffness and pain, and lacking tiered and personalized guidance, leading to decreased patient compliance and even exacerbating symptoms due to forced practice. In addition, existing technologies do not include trunk and lower limb postural stability in the assessment dimension, making it impossible to assess the patient's balance in real time. Long-term neglect of compensatory posture can disrupt the body's force alignment and increase the risk of falls. At the same time, existing technologies are mainly based on static assessment and lack real-time intervention mechanisms for erroneous movements, which can easily lead to the solidification of erroneous movement patterns and increase the difficulty of later correction. Therefore, this invention proposes a post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis to solve the problems existing in the prior art. Summary of the Invention
[0004] To address the aforementioned issues, this invention proposes a post-mastopathy rehabilitation system based on multimodal sensing and video analysis. This system achieves precise spatiotemporal alignment of visual motion trajectories with muscle electrical signals and inertial signals through multimodal sensor fusion design. It overcomes the limitations of existing technologies that decouple trajectory and force exertion, and can accurately pinpoint whether the root cause of rehabilitation obstacles is limited joint mobility or muscle exertion disorder, providing a comprehensive and accurate basis for rehabilitation assessment.
[0005] To achieve the objectives of this invention, the following technical solution is provided: a post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis, comprising a data acquisition module, a feature extraction and analysis module, and a decision-making and guidance module. The data acquisition module is used to simultaneously acquire the patient's visual-motor information and biomechanical signals. The feature extraction and analysis module is used to perform spatiotemporal alignment processing on the acquired multimodal data, and to realize motion trajectory reconstruction, muscle force correlation analysis, and balance state assessment based on the aligned data.
[0006] Based on feature extraction and analysis results, the decision-making and guidance module uses a priority-based hierarchical error correction logic of joints, muscles, and balance to generate and push targeted rehabilitation and correction guidance information.
[0007] Further improvements are made in that: the data acquisition module includes a video acquisition unit and a sensor acquisition unit; the video acquisition unit uses a high-definition RGB-D camera to capture a sequence of whole-body motion images of the patient to extract the motion trajectory; the sensor acquisition unit includes a flexible electromyography (EMG) sensor and an inertial measurement unit, which are attached to the key muscle groups of the scapula, pectoralis major, and deltoid on the affected side of the patient, and are used to acquire muscle electrical signals and limb angular velocity signals, respectively.
[0008] Further improvements include: the flexible electromyography sensor uses a breathable silicone base with a thickness of ≤2mm, and the attachment method uses medical hypoallergenic gel to meet the needs of breast cancer patients with sensitive skin after surgery.
[0009] A further improvement is made in that the spatiotemporal alignment processing of the feature extraction and analysis module adopts a timestamp synchronization algorithm, the specific formula of which is:
[0010] ,
[0011] Among them, t sync For the aligned unified timestamp, t v Here, f represents the original timestamp of the video frame, Δt represents the time interval between video frames, and f is the original timestamp of the video frame. e f is the sampling frequency of the electromyography signal. vThe algorithm is used to synchronize the low-frequency video frame rate with the high-frequency electromyographic signals, establishing a precise mapping relationship between visual motion trajectories and biomechanical signals.
[0012] Further improvements include: the motion trajectory reconstruction of the feature extraction and analysis module uses the SLAM algorithm to extract the spatial trajectory of the affected limb's end effector, and compares it with a standard rehabilitation movement library. The trajectory similarity evaluation formula is as follows:
[0013] ,
[0014] Where S represents the trajectory similarity. The three-dimensional coordinates of the i-th sampling point at the distal end of the patient's affected limb are given. Here are the 3D coordinates of the corresponding sampling points in the standard action library, where n is the number of sampling points, and L is the number of sampling points. max S represents the maximum length of the standard motion trajectory; when S < 0.8, the trajectory is deemed not to meet the standard.
[0015] A further improvement lies in the fact that the muscle exertion correlation analysis of the feature extraction and analysis module achieves compensatory exertion identification by calculating muscle activation. The formula for calculating muscle activation is:
[0016] ,
[0017] Where Am represents the activation level of the m-th muscle, and sEMG m (t) represents the electromyographic signal value of the m-th muscle at time t, where t1 and t2 are the start and end times of the movement, respectively. max,m The maximum electromyographic signal amplitude of the m-th muscle is given. When the activation level of the target muscle is <0.3 and the activation level of the non-target muscle is >0.6, it is determined that compensatory force is present.
[0018] A further improvement is made in that: the balance state assessment of the feature extraction and analysis module is achieved by calculating the center of gravity offset, and the formula for calculating the center of gravity offset is:
[0019] ,
[0020] Among them, COM offset For the center of gravity offset, COM x COM y For the patient's real-time centroid x and y coordinates, COM x0 COM y0 The coordinates of the center of gravity in a standard standing posture; when COM offset If the value is greater than 5cm, the balance is considered to be substandard.
[0021] Further improvements are made in that the priority-based hierarchical error correction logic of the decision-making and guidance module dynamically adjusts the error correction weights according to the patient's recovery stage. The recovery stage is divided based on the postoperative time T: when T≤4 weeks, it is considered the early recovery stage, and the balance correction weight w is used. b =0.6, Joint Correction Weight w j =0.3, muscle correction weight w m =0.1, prioritize maintaining balance to prevent falls, then adjust joint mobility, and finally focus on muscle exertion; when 4 < T ≤ 12 weeks, it is considered the mid-stage of rehabilitation, w b =0.3、w j =0.4、w m =0.3, pay balanced attention to joints, muscles, and balance; when T>12 weeks is the late stage of rehabilitation, w b =0.1、w j =0.3、w m =0.6, prioritize correcting muscle exertion patterns to make rehabilitation more precise.
[0022] Further improvements are made in that: the guidance information output method of the decision and guidance module includes voice commands and visual prompts, and the output terminal is AR glasses or tablet devices; the voice commands adopt segmented guidance scripts, and the visual prompts are displayed in the patient's motion image by superimposing dynamic lines.
[0023] Further improvements include a data storage and transmission module for storing multimodal acquisition data, feature analysis results, and rehabilitation guidance records, and for remote data interaction with the hospital's rehabilitation management platform.
[0024] The beneficial effects of this invention are as follows:
[0025] 1. This invention achieves precise spatiotemporal alignment of visual motion trajectory with muscle electrical signals and inertial signals through multimodal sensor fusion design, breaking through the limitations of existing technology in decoupling trajectory and force exertion. It can accurately locate whether the root cause of rehabilitation obstacles is limited joint movement or muscle exertion disorder, providing a comprehensive and accurate basis for rehabilitation assessment.
[0026] 2. This invention is based on a joint-muscle-balance priority hierarchical error correction model, which can dynamically adjust the error correction weight according to the patient's postoperative time. At the same time, it combines the real-time movement achievement status, muscle exertion status and balance status to generate dynamic guidance information adapted to the individual patient's impairment, replacing the traditional one-size-fits-all standardized solution and effectively improving patient rehabilitation compliance.
[0027] 3. By monitoring balance indicators such as center of gravity shift in real time, this invention can promptly identify the risk of postural imbalance and send adjustment instructions, effectively avoiding the risk of falls caused by instability of the center of gravity during rehabilitation. At the same time, it can intervene in compensatory errors in real time to reduce the occurrence of secondary injuries.
[0028] 4. This invention automates and enables the entire process of rehabilitation assessment and guidance in real time, eliminating the need for doctors to be on duty throughout the process, thus reducing the occupation of medical resources. At the same time, the data storage and transmission module enables remote interaction of rehabilitation data, facilitating doctors to dynamically adjust rehabilitation plans and forming a closed-loop management of assessment-guidance-adjustment, which significantly improves rehabilitation efficiency. Attached Figure Description
[0029] Figure 1 This is a diagram illustrating the composition of the present invention. Detailed Implementation
[0030] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.
[0031] Example 1
[0032] according to Figure 1 As shown in the figure, this embodiment proposes a post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis, including a data acquisition module, a feature extraction and analysis module, and a decision-making and guidance module. The data acquisition module is used to simultaneously acquire the patient's visual-motor information and biomechanical signals; the feature extraction and analysis module is used to perform spatiotemporal alignment processing on the acquired multimodal data, and realize motion trajectory reconstruction, muscle force correlation analysis, and balance state assessment based on the aligned data.
[0033] The decision-making and guidance module, based on feature extraction and analysis results, employs a priority-based hierarchical error correction logic (joint-muscle-balance) to generate and push targeted rehabilitation correction guidance information. This enables collaborative processing of multi-dimensional rehabilitation assessment data, avoiding the limitations of single-data assessments; it provides precise data support for personalized rehabilitation guidance, improving the effectiveness of rehabilitation interventions.
[0034] The data acquisition module includes a video acquisition unit and a sensor acquisition unit. The video acquisition unit uses a high-definition RGB-D camera to capture a sequence of whole-body motion images of the patient to extract the motion trajectory. The sensor acquisition unit includes a flexible electromyography (EMG) sensor and an inertial measurement unit. These sensors are attached to key muscle groups on the affected side of the patient, including the scapula, pectoralis major, and deltoid muscles, to acquire muscle electrical signals and limb angular velocity signals, respectively. This multi-unit collaborative acquisition ensures the comprehensiveness of the motion trajectory and biomechanical signals, providing rich data for subsequent analysis. The precise attachment design to key muscle groups enhances the targeting of signal acquisition and ensures data acquisition quality.
[0035] The flexible electromyography (EMG) sensor uses a breathable silicone substrate with a thickness of ≤2mm and is attached with a medical-grade hypoallergenic gel, catering to the sensitive skin needs of breast cancer post-operative patients. The hypoallergenic and breathable design reduces sensor irritation to sensitive post-operative skin, improving patient comfort; the lightweight material minimizes the feeling of a foreign object, helping patients adhere to rehabilitation training for extended periods.
[0036] The spatiotemporal alignment processing of the feature extraction and analysis module adopts a timestamp synchronization algorithm, the specific formula of which is:
[0037] ,
[0038] Among them, t sync For the aligned unified timestamp, t v Here, f represents the original timestamp of the video frame, Δt represents the time interval between video frames, and f is the original timestamp of the video frame. e f is the sampling frequency of the electromyography signal. v This algorithm synchronizes low-frequency video frame rates with high-frequency electromyographic signals, establishing a precise mapping between visual motion trajectories and biomechanical signals. It resolves the temporal mismatch issue in multimodal data, improving the accuracy of data fusion analysis; and provides a precise spatiotemporal basis for the correlation analysis between trajectory and force exertion, aiding in the accurate identification of erroneous force exertion patterns.
[0039] The motion trajectory reconstruction in the feature extraction and analysis module uses the SLAM algorithm to extract the spatial trajectory of the affected limb's distal end and compares it with a standard rehabilitation movement library. The trajectory similarity evaluation formula is as follows:
[0040] ,
[0041] Where S represents the trajectory similarity. The three-dimensional coordinates of the i-th sampling point at the distal end of the patient's affected limb are given. Here are the 3D coordinates of the corresponding sampling points in the standard action library, where n is the number of sampling points, and L is the number of sampling points. max The maximum length of the standard movement trajectory is defined as S; when S < 0.8, the trajectory is considered unqualified. The SLAM algorithm achieves accurate reconstruction of the affected limb trajectory, providing a reliable basis for movement achievement assessment; the quantitative similarity judgment standard avoids subjective assessment errors and improves the objectivity of rehabilitation assessment.
[0042] The muscle exertion correlation analysis in the feature extraction and analysis module achieves compensatory exertion identification by calculating muscle activation. The formula for calculating muscle activation is as follows:
[0043] ,
[0044] Where Am represents the activation level of the m-th muscle, and sEMG m(t) represents the electromyographic signal value of the m-th muscle at time t, where t1 and t2 are the start and end times of the movement, respectively. max,m The maximum electromyographic signal amplitude of the m-th muscle is defined as follows: when the activation level of the target muscle is <0.3 and the activation level of the non-target muscle is >0.6, compensatory force is considered to be present. Precise quantification of muscle activation status enables early identification of compensatory force, reducing the risk of secondary injury; it also provides targeted guidance for target muscle group activation training, improving the efficiency of rehabilitation training.
[0045] The balance state assessment of the feature extraction and analysis module is achieved by calculating the centroid offset, and the formula for calculating the centroid offset is:
[0046] ,
[0047] Among them, COM offset For the center of gravity offset, COM x COM y For the patient's real-time centroid x and y coordinates, COM x0 COM y0 The coordinates of the center of gravity in a standard standing posture; when COM offset A deviation of more than 5cm indicates a failure to maintain balance. Real-time monitoring of center of gravity shift provides early warnings of fall risk, enhancing the safety of rehabilitation training. Quantitative balance assessment indicators offer precise guidance for balance correction training, aiding in the restoration of overall body alignment.
[0048] The priority-based hierarchical error correction logic of the decision-making and guidance module dynamically adjusts the error correction weights according to the patient's recovery stage. The recovery stage is divided based on the postoperative time T: when T≤4 weeks, it is considered the early recovery stage, and the balance correction weight w is used. b =0.6, Joint Correction Weight w j =0.3, muscle correction weight w m =0.1, prioritize maintaining balance to prevent falls, then adjust joint mobility, and finally focus on muscle exertion; when 4 < T ≤ 12 weeks, it is considered the mid-stage of rehabilitation, w b =0.3、w j =0.4、w m =0.3, pay balanced attention to joints, muscles, and balance; when T>12 weeks is the late stage of rehabilitation, w b =0.1、w j =0.3、w m =0.6, prioritizing the correction of muscle exertion patterns for more precise rehabilitation. Dynamic weight adjustment adapts to the needs of different rehabilitation stages, enabling phased and precise intervention; priority division ensures that key rehabilitation goals are achieved first, improving the systematic nature and efficiency of rehabilitation training.
[0049] The decision-making and guidance module outputs guidance information through voice commands and visual cues, with the output terminal being AR glasses or a tablet device. Voice commands employ segmented guidance scripts, while visual cues are displayed as dynamic lines overlaid on the patient's motion image. This multi-format guidance information adapts to different patients' acceptance habits, improving the recognizability of guidance commands. The intuitive visual overlay and segmented voice cues help patients quickly understand and correct their movements, reducing the learning cost.
[0050] It also includes a data storage and transmission module, which stores multimodal acquisition data, feature analysis results and rehabilitation guidance records, and performs remote data interaction with the hospital rehabilitation management platform, providing multiple data interaction channels and offering diverse functions.
[0051] Example 2
[0052] according to Figure 1 As shown, this embodiment proposes a post-mastopathy rehabilitation system based on multimodal sensing and video analysis, targeting patients in the early rehabilitation stage within 4 weeks after breast cancer surgery. During this stage, patients have significantly limited shoulder joint mobility, weak balance, and a high risk of falling.
[0053] System configuration: An Intel RealSense D455 RGB-D camera is used, deployed 1.5m directly in front of the patient, with a sampling frame rate of 30fps; flexible electromyography sensors are attached to the anterior deltoid, upper trapezius, and pectoralis major muscles on the affected side; IMU sensors are attached to the upper arm and forearm on the affected side; the guidance terminal uses AR glasses.
[0054] Working Process: 1. Data Acquisition: During shoulder abduction training, the camera captures real-time images of the whole body, the electromyography sensor collects electrical signals from three target muscles, and the IMU collects the angular velocities of the upper arm and forearm; 2. Feature Extraction and Analysis: Multimodal data alignment is achieved through a timestamp synchronization algorithm (Formula 1), the SLAM algorithm is used to extract the three-dimensional trajectory of the fingertip, and the trajectory similarity algorithm (Formula 2) is used to compare it with the standard abduction trajectory to calculate muscle activation (Formula 3) to identify trapezius muscle overcompensation. The balance status is assessed through a center of gravity offset algorithm (Formula 4); 3. Decision-Making and Guidance: Due to the early stage of rehabilitation, the balance correction weight is 0.6, and the joint... With a correction weight of 0.3 and a muscle correction weight of 0.1, the system detected that the patient's shoulder abduction angle was insufficient (trajectory similarity S=0.65) and the center of gravity was shifted to the affected side (COMoffset=6.2cm). The system first pushed a voice command through the AR glasses: "Stand with your feet shoulder-width apart, imagine your center of gravity is between your legs." After the center of gravity shift decreased to 3.5cm, the system pushed a command: "Please retract your scapula to increase the range of motion of your upper arm." At the same time, dynamic lines were superimposed on the AR glasses to indicate the correct position of the scapula. 4. Data storage: The system automatically stores the trajectory data, electromyographic signals, balance indicators and guidance records of this training and uploads them to the hospital rehabilitation management platform.
[0055] Example 3
[0056] according to Figure 1 As shown, this embodiment proposes a post-breast cancer surgery physical rehabilitation system based on multimodal sensing and video analysis, targeting elderly patients over 65 years of age who have undergone breast cancer surgery. These patients have poor balance, slow reaction speed, and a higher acceptance of voice commands than visual cues.
[0057] System configuration: A Kinect V2 RGB-D camera is used, deployed 1.2m in front of the patient, with a sampling frame rate of 25fps; flexible electromyography sensors are attached to the deltoid, latissimus dorsi, and trapezius muscles on the affected side, and IMU sensors are attached to the upper arm, forearm, and waist on the affected side; the guidance terminal uses a tablet device, and voice commands use a slow speech and repetitive playback mode.
[0058] Working Process: During the patient's upper limb forward flexion training, the system, through multimodal data collection and analysis, found that the patient had latissimus dorsi overactivation (activation A=0.72) and anterior deltoid underactivation (A=0.25), with a center of gravity shift of 4.8cm. Since the patient was in the mid-rehabilitation stage (8 weeks post-surgery), the error correction weights were allocated as balance 0.3, joints 0.4, and muscles 0.3. The system first pushed a slow-speed voice command via tablet: "Feet slightly wider than shoulder-width apart, slowly adjust your center of gravity to the middle of your legs," repeating this twice. After the center of gravity shift decreased to 3.2cm, the system pushed: "Keep your head still, try to push forward using your chest muscles, feel the muscles on the front of your arms engaging," while simultaneously displaying a diagram of anterior deltoid activation on the tablet. During training, the system repeated the core command every 30 seconds to ensure accurate patient understanding. After training, a concise rehabilitation report was generated, including the movement attainment rate, number of force errors, and balance score, for easy review by the patient and their family.
[0059] Example 4
[0060] according to Figure 1 As shown, this embodiment proposes a post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis. For home rehabilitation scenarios, the sensor layout is optimized to improve ease of use and reduce equipment costs.
[0061] System configuration: A home-grade Intel RealSense D435 RGB-D camera is used, which is deployed in the corner of the living room and fixed with a bracket; the flexible electromyography (EMG) sensor is simplified to two, which are attached to the deltoid and trapezius muscles on the affected side respectively, and the IMU sensor is integrated into the EMG sensor; the guidance terminal uses the patient's smartphone to receive guidance information through a dedicated APP.
[0062] Working Process: Patients initiate rehabilitation training via the app. The system automatically calibrates the camera and loads a standard movement library adapted for home environments (movement range reduced by 10% compared to clinical standards, suitable for confined home spaces). During training, the system collects trajectory data of the patient's upper limb lifting movements and electromyographic signals of the two core muscles. Data fusion is achieved through a spatiotemporal alignment algorithm. When trapezius muscle activation of 0.68 and deltoid muscle activation of 0.22 (compensatory force) are detected, a voice command is pushed through the mobile app: "Relax your shoulders and try to lift using the outer side of your arm." Simultaneously, a comparison video of the real-time movement and the standard movement is displayed on the app interface, marking the areas of incorrect force exertion. After training, the app generates a rehabilitation log, recording training duration, number of times the movement was achieved, and types of incorrect force exertion, and pushes it to the family doctor's WeChat account for remote monitoring of the rehabilitation progress.
[0063] Example 5
[0064] according to Figure 1As shown, this embodiment proposes a post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis, targeting patients with mild lymphedema after breast cancer surgery. These patients have swelling in the affected upper limb and higher skin sensitivity, so it is necessary to avoid pressure on the swollen area caused by sensor attachment.
[0065] System configuration: An Intel RealSense L515 RGB-D camera is used, positioned directly above the patient to achieve omnidirectional trajectory capture; the flexible electromyography (EMG) sensor adopts a ring design and is fitted onto the proximal end of the affected upper arm (non-swollen area) and the distal end of the forearm, with the IMU sensor integrated into the ring EMG sensor, and is attached by a light fixation with an elastic bandage; the guidance terminal uses a tablet device, and the guided movements are mainly weightless and small-amplitude.
[0066] Working Process: The patient performs fist-clenching and relaxation training on the affected upper limb. The system captures the upper limb movement trajectory through a camera, collects electrical signals from the biceps and triceps brachii muscles using a circular electromyography (EMG) sensor, and acquires the upper limb angular velocity using an IMU. Through feature extraction and analysis, muscle activation is calculated to assess the adequacy of force exertion, while simultaneously monitoring the upper limb range of motion (to avoid excessive activity that could worsen edema). When the system detects a biceps activation level of 0.28 (insufficient force exertion) when the patient clenches their fist, it sends a voice command: "Slowly clench your fist, feel the contraction of the muscles on the inside of your arm, hold for 3 seconds, and then slowly relax." The tablet interface displays a diagram of muscle exertion during the fist-clenching action, marking the target muscle groups. During the training, the system monitors the upper limb range of motion in real time, and immediately sends a stop command when the range exceeds a safe threshold to ensure training safety.
[0067] Validation data:
[0068] Sixty post-operative breast cancer patients were selected and divided into an experimental group (n=30, using the present invention) and a control group (n=30, using the traditional standardized rehabilitation program). A 12-week comparative rehabilitation training experiment was conducted, and the results are shown in the table below:
[0069] Evaluation indicators Experimental group (this invention) Control group (traditional regimen) Increase shoulder abduction range of motion improvement value 35.2°±4.8° 22.6°±5.3° 55.7% Target muscle group activation rate 89.3% 62.7% 42.4% Rehabilitation training compliance 93.3% 70.0% 33.3% Fall risk during training 3.3% 20.0% 83.5% (Decrease) Incorrect action pattern correction time 2.8 ± 0.6 weeks 5.6 ± 0.8 weeks 50.0% (shortening)
[0070] The results showed that the experimental group using the present invention was significantly better than the control group in terms of core indicators such as improved shoulder joint range of motion, target muscle group activation rate, and rehabilitation compliance. At the same time, the incidence of falls was significantly reduced and the correction time for incorrect movement patterns was significantly shortened, which fully verified the effectiveness and superiority of the present invention.
[0071] This post-mastopathy rehabilitation system based on multimodal sensing and video analysis achieves precise spatiotemporal alignment of visual motion trajectories with muscle electrical and inertial signals through multimodal sensor fusion design. It overcomes the limitations of existing technologies that decouple trajectory and force exertion, accurately pinpointing whether rehabilitation obstacles stem from limited joint mobility or muscle exertion dysfunction, providing comprehensive and accurate diagnostic criteria for rehabilitation assessment. Furthermore, this invention utilizes a joint-muscle-balance priority hierarchical error correction model, dynamically adjusting error correction weights based on post-operative time. Simultaneously, it combines real-time movement achievement, muscle exertion status, and balance status to generate dynamic guidance information tailored to each patient's individual impairments, replacing traditional one-size-fits-all standardized solutions and effectively improving patient adherence to rehabilitation. Meanwhile, by monitoring balance indicators such as center of gravity shift in real time, this invention can promptly identify the risk of postural imbalance and push adjustment instructions, effectively avoiding the risk of falls caused by instability during rehabilitation. It also intervenes in compensatory errors in real time, reducing the occurrence of secondary injuries. In addition, this invention automates and real-time the entire process of rehabilitation assessment and guidance, eliminating the need for doctors to be on duty throughout the process, reducing the occupation of medical resources. At the same time, the data storage and transmission module enables remote interaction of rehabilitation data, facilitating doctors to dynamically adjust rehabilitation plans, forming a closed-loop management of assessment-guidance-adjustment, and significantly improving rehabilitation efficiency.
[0072] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.
Claims
1. A post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis, comprising a data acquisition module, a feature extraction and analysis module, and a decision-making and guidance module, characterized in that: The data acquisition module is used to simultaneously acquire the patient's visual-motor information and biomechanical signals; the feature extraction and analysis module is used to perform spatiotemporal alignment processing on the acquired multimodal data, and to realize motion trajectory reconstruction, muscle force correlation analysis and balance state assessment based on the aligned data. Based on feature extraction and analysis results, the decision-making and guidance module uses a priority-based hierarchical error correction logic of joints, muscles, and balance to generate and push targeted rehabilitation and correction guidance information.
2. The post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 1, characterized in that: The data acquisition module includes a video acquisition unit and a sensor acquisition unit. The video acquisition unit uses a high-definition RGB-D camera to capture a sequence of whole-body motion images of the patient to extract the motion trajectory. The sensor acquisition unit includes a flexible electromyography (EMG) sensor and an inertial measurement unit. The flexible EMG sensor and the inertial measurement unit are attached to the key muscle groups of the scapula, pectoralis major, and deltoid on the affected side of the patient to collect muscle electrical signals and limb angular velocity signals, respectively.
3. The post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 2, characterized in that: The flexible electromyography sensor uses a breathable silicone substrate with a thickness of ≤2mm and is attached with a medical-grade hypoallergenic gel to meet the needs of breast cancer patients with sensitive skin after surgery.
4. The post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 1, characterized in that: The spatiotemporal alignment processing of the feature extraction and analysis module adopts a timestamp synchronization algorithm, the specific formula of which is: ， Among them, t sync For the aligned unified timestamp, t v Here, f represents the original timestamp of the video frame, Δt represents the time interval between video frames, and f is the original timestamp of the video frame. e f is the sampling frequency of the electromyography signal. v The algorithm is used to synchronize the low-frequency video frame rate with the high-frequency electromyographic signals, establishing a precise mapping relationship between visual motion trajectories and biomechanical signals.
5. A post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 4, characterized in that: The motion trajectory reconstruction in the feature extraction and analysis module uses the SLAM algorithm to extract the spatial trajectory of the affected limb's distal end and compares it with a standard rehabilitation movement library. The trajectory similarity evaluation formula is as follows: ， Where S represents the trajectory similarity. The three-dimensional coordinates of the i-th sampling point at the distal end of the patient's affected limb are given. Here are the 3D coordinates of the corresponding sampling points in the standard action library, where n is the number of sampling points, and L is the number of sampling points. max S represents the maximum length of the standard motion trajectory; when S < 0.8, the trajectory is deemed not to meet the standard.
6. A post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 5, characterized in that: The muscle exertion correlation analysis in the feature extraction and analysis module achieves compensatory exertion identification by calculating muscle activation. The formula for calculating muscle activation is as follows: ， Where Am represents the activation level of the m-th muscle, and sEMG m (t) represents the electromyographic signal value of the m-th muscle at time t, where t1 and t2 are the start and end times of the movement, respectively. max,m The maximum electromyographic signal amplitude of the m-th muscle is given. When the activation level of the target muscle is <0.3 and the activation level of the non-target muscle is >0.6, it is determined that compensatory force is present.
7. A post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 6, characterized in that: The balance state assessment of the feature extraction and analysis module is achieved by calculating the centroid offset, and the formula for calculating the centroid offset is: ， Among them, COM offset For the center of gravity offset, COM x COM y For the real-time x and y coordinates of the patient's center of gravity, COM x0 COM y0 The coordinates of the center of gravity in a standard standing posture; when COM offset If the value is greater than 5cm, the balance is considered to be substandard.
8. A post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 1, characterized in that: The priority-based hierarchical error correction logic of the decision-making and guidance module dynamically adjusts the error correction weights according to the patient's recovery stage. The recovery stage is divided based on the postoperative time T: when T≤4 weeks, it is considered the early recovery stage, and the balance correction weight w is used. b =0.6, Joint Correction Weight w j =0.3, muscle correction weight w m =0.1, prioritize maintaining balance to prevent falls, then adjust joint mobility, and finally focus on muscle exertion; when 4 < T ≤ 12 weeks, it is considered the mid-stage of rehabilitation, w b =0.3、w j =0.4、w m =0.3, pay balanced attention to joints, muscles, and balance; when T>12 weeks is the late stage of rehabilitation, w b =0.1、w j =0.3、w m =0.6, prioritize correcting muscle exertion patterns to make rehabilitation more precise.
9. A post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 1, characterized in that: The decision-making and guidance module outputs guidance information through voice commands and visual prompts, with the output terminal being AR glasses or a tablet device. The voice commands use segmented guidance scripts, while the visual prompts are displayed as dynamic lines superimposed on the patient's motion image.
10. A post-mastectomy physical rehabilitation system based on multimodal sensing and video analysis according to claim 1, characterized in that: It also includes a data storage and transmission module for storing multimodal acquisition data, feature analysis results, and rehabilitation guidance records, and for remote data interaction with the hospital's rehabilitation management platform.