Shoulder joint disorder diagnosis and rehabilitation deduction system based on image and digital twinning

Abstract

The application provides a shoulder joint disorder diagnosis and rehabilitation deduction system based on images and digital twinning, which comprises a front-end visual intelligent physical examination module, a middle-end image-muscle electromyography digital twinning coupled diagnosis module and a rear-end personalized rehabilitation real-time deduction optimization module, realizes accurate quantitative evaluation of shoulder joint physical examination motion through the front-end visual intelligent physical examination module, breaks through the bottleneck of strong subjectivity and difficulty in standardization of traditional physical examination, provides objective, continuous and traceable evaluation data for clinical use, the middle-end image-muscle electromyography digital twinning coupled diagnosis module fuses multidimensional data to construct a physical information coupled calculation engine, significantly improves the accuracy of disease differential diagnosis, and the rear-end personalized rehabilitation real-time deduction optimization module realizes real-time monitoring and adaptive optimization of the rehabilitation process, reduces the risk of postoperative re-tearing, and the overall system is suitable for primary medical scenes with the characteristics of low cost and high precision, and protects the joint health of agricultural laborers.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence medical technology, and in particular to a shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin. Background Technology

[0002] Currently, the diagnosis and rehabilitation of shoulder joint disorders suffers from problems such as over-reliance on MRI, strong subjective diagnosis, and rigid rehabilitation plans. Traditional physical examinations rely on doctors' experience and lack unified standards, making it difficult to capture subtle changes in special movements such as the drop arm test and empty can test, resulting in insufficient consistency and traceability of assessment results. Clinical diagnoses are often based on single images, and there are often discrepancies between imaging findings and clinical symptoms, leading to prominent issues of misdiagnosis, missed diagnosis, and over-surgery. Rehabilitation training often adopts fixed patterns, lacking real-time load monitoring, restraint of contraindicated movements, and dynamic adjustment capabilities, making it difficult to adapt to individual patient differences and real-time changes in condition. At the same time, primary healthcare settings suffer from limitations such as data fragmentation, high equipment costs, and insufficient data privacy protection, making it impossible to provide patients with full-cycle, precise diagnosis and rehabilitation services. There is an urgent need for an integrated solution that combines multimodal data and digital twin technology. Summary of the Invention

[0003] In view of this, in order to solve the problems existing in the technical background, the present invention proposes a shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin. Specifically, it includes the following: A shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin includes a front-end visual intelligent physical examination module, a mid-end image-electromyography digital twin coupled diagnosis module, and a back-end personalized rehabilitation real-time simulation and optimization module. The three modules are connected in sequence. The front-end visual intelligent physical examination module uses a spatial grid as the positioning reference and realizes multi-view synchronous acquisition and precise positioning of shoulder joint physical examination movements through a bionic compound eye array. It quantifies and evaluates the physical examination movements and outputs objective physical examination data. The mid-range imaging-electromyography digital twin coupled diagnostic module receives front-end physical examination data, integrates medical images, sEMG electromyography signals, and clinical history data to construct a physical information coupling computing engine, and completes the identification of shoulder joint lesions and the determination of the necessity of surgery; The backend personalized rehabilitation real-time simulation and optimization module receives mid-end diagnostic results and constructs a three-level mechanism of monitoring-regulation-feedback based on multimodal perception and temporal modeling to realize real-time monitoring and adaptive optimization of rehabilitation exercise prescriptions, forming a closed-loop system for the entire process from physical examination and diagnosis to rehabilitation simulation.

[0004] In some embodiments of the present invention, the front-end visual intelligent physical examination module includes a three-dimensional perception unit, an action modeling unit, and a temporal decoupling unit. The three-dimensional perception unit completes multi-view data acquisition through a spatially gridded bionic compound eye array. The action modeling unit uses a Graph-LSTM-Attention model for action trajectory prediction. The temporal decoupling unit separates individual differences from action temporal features based on a spatiotemporal identity decoupling method, achieving data standardization and alignment. The output of the three-dimensional perception unit is connected to the input of the action modeling unit, the output of the action modeling unit is connected to the input of the temporal decoupling unit, and the output of the temporal decoupling unit is connected to the input of the mid-end image-electromyography digital twin coupled diagnostic module. The state modeling and observation process of the front-end visual intelligent physical examination module follows the formula: in, The vector represents the human motion state (including spatial keypoints, motion velocity, and posture structure), and s is an individual identity feature variable. The data is the observation data from the i-th viewpoint (obtained by multi-view acquisition from the compound eye array). This represents the state evolution function driven by the ganglion prediction network. This represents a multi-view observation mapping function. By decoupling spatiotemporal identity, it enables independent modeling of individual differences and action patterns, and completes data standardization and alignment under a unified spatial grid benchmark.

[0005] In some embodiments of the present invention, the mid-range image-electromyography digital twin coupled diagnostic module adopts a Transformer-based cross-modal attention mechanism. Spatial features are extracted from image data using a 3D CNN, textual features are encoded using BERT from clinical data, and temporal features are extracted from motion data using LSTM. Multimodal feature fusion is achieved through a cross-modal Transformer, and the fusion calculation follows the formula: Among them, F image F clinical F motion W represents the feature representations of imaging, clinical, and motion modalities, respectively. fusion The weight matrix is ​​dynamically learned through an attention mechanism.

[0006] Simultaneously, a federated averaging algorithm is used to achieve multi-center collaborative training, and parameter updates follow the formula: in, Here are the global model parameters for round t, and k is the number of participating organizations. Let k be the amount of data for the k-th institution. These are the local model parameters for the k-th institution. Simultaneously, the platform employs differential privacy technology to add noise to the uploaded gradients, further protecting data privacy; and uses blockchain technology to ensure the traceability and integrity of the model parameters.

[0007] In some embodiments of the present invention, the backend personalized rehabilitation real-time simulation and optimization module comprises a three-layer cascaded structure of a load monitoring layer, a behavior capture layer, and a rework guarantee layer. The load monitoring layer integrates surface electromyography and video information to quantify muscle activation levels and exercise load. The behavior capture layer completes motion key point detection and trajectory reconstruction based on posture estimation. The rework guarantee layer establishes a mapping relationship between load response and functional recovery to achieve effect evaluation and feedback correction. This module models exercise prescription optimization as a Markov decision process and uses a proximal strategy optimization algorithm for strategy learning. The objective function follows the formula: in, This represents the probability ratio between the old and new strategies. For the estimation of the advantage function, To tailor the parameters, a real-time feedback mechanism is introduced, allowing the model to adjust the prescription parameters based on the patient's dynamic response during training, thereby achieving adaptive optimization of the exercise prescription.

[0008] In some embodiments of the present invention, the system as a whole adopts a multimodal deep learning framework to achieve multi-source heterogeneous data fusion, completes MRI image volume segmentation based on a 3D CNN (nnU-Net) model, uses a BERT-based NLP model to parse unstructured medical record text, and combines computer vision technology to analyze patient movement patterns in real time and identify non-standard movements in rehabilitation training.

[0009] The above technical solution has the following beneficial effects: This system achieves precise quantitative assessment of shoulder joint movements through a front-end visual intelligent physical examination module, overcoming the bottlenecks of traditional physical examinations that are highly subjective and difficult to standardize. It provides objective, continuous, and traceable assessment data for clinical practice, reducing reliance on MRI images from the outset. The mid-end imaging-electromyography digital twin coupled diagnostic module integrates multi-dimensional data to construct a physical information coupling calculation engine, significantly improving the accuracy of disease differential diagnosis, accurately determining the necessity of surgery, and effectively avoiding misdiagnosis, missed diagnosis, and overtreatment. The back-end personalized rehabilitation real-time simulation and optimization module enables real-time monitoring and adaptive optimization of the rehabilitation process, reducing the risk of postoperative re-tears and improving rehabilitation efficiency and safety. The overall system is adapted to primary healthcare scenarios with its low cost and high precision, safeguarding the joint health of agricultural laborers. Detailed Implementation

[0010] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] Example 1: A shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin includes a front-end visual intelligent physical examination module, a mid-end image-electromyography digital twin coupled diagnosis module, and a back-end personalized rehabilitation real-time simulation and optimization module. The three modules are connected in sequence. The front-end visual intelligent physical examination module uses a spatial grid as the positioning reference and realizes multi-view synchronous acquisition and precise positioning of shoulder joint physical examination movements through a bionic compound eye array. It quantifies and evaluates the physical examination movements and outputs objective physical examination data. The mid-range imaging-electromyography digital twin coupled diagnostic module receives front-end physical examination data, integrates medical images, sEMG electromyography signals, and clinical history data to construct a physical information coupling computing engine, and completes the identification of shoulder joint lesions and the determination of the necessity of surgery; The backend personalized rehabilitation real-time simulation and optimization module receives mid-end diagnostic results and constructs a three-level mechanism of monitoring-regulation-feedback based on multimodal perception and temporal modeling to realize real-time monitoring and adaptive optimization of rehabilitation exercise prescriptions, forming a closed-loop system for the entire process from physical examination and diagnosis to rehabilitation simulation.

[0012] In some embodiments of the present invention, Embodiment 2, based on Embodiment 1, includes a front-end visual intelligent physical examination module comprising a three-dimensional perception unit. The three-dimensional perception unit completes multi-view data acquisition through a spatially gridded bionic compound eye array. The action modeling unit uses a Graph-LSTM-Attention model for action trajectory prediction. The temporal decoupling unit separates individual differences from action temporal features based on a spatiotemporal identity decoupling method, achieving data standardization and alignment. The output of the three-dimensional perception unit is connected to the input of the action modeling unit, the output of the action modeling unit is connected to the input of the temporal decoupling unit, and the output of the temporal decoupling unit is connected to the input of the mid-end image-electromyography digital twin coupled diagnostic module. The state modeling and observation process of the front-end visual intelligent physical examination module follows the formula: in, The vector represents the human motion state (including spatial keypoints, motion velocity, and posture structure), and s is an individual identity feature variable. The data is the observation data from the i-th viewpoint (obtained by multi-view acquisition from the compound eye array). This represents the state evolution function driven by the ganglion prediction network. This represents a multi-view observation mapping function. By decoupling spatiotemporal identity, it enables independent modeling of individual differences and action patterns, and completes data standardization and alignment under a unified spatial grid benchmark.

[0013] The mid-range image-electromyography digital twin coupled diagnostic module adopts a Transformer-based cross-modal attention mechanism. Image data is processed using 3D CNN to extract spatial features, clinical data is encoded with BERT for textual features, and motion data is processed using LSTM to extract temporal features. Multimodal feature fusion is achieved through a cross-modal Transformer, and the fusion calculation follows the formula: Among them, F image F clinical F motion W represents the feature representations of imaging, clinical, and motion modalities, respectively. fusion The weight matrix is ​​dynamically learned through an attention mechanism.

[0014] Simultaneously, a federated averaging algorithm is used to achieve multi-center collaborative training, and parameter updates follow the formula: in, Here are the global model parameters for round t, and k is the number of participating organizations. Let k be the amount of data for the k-th institution. These are the local model parameters for the k-th institution. Simultaneously, the platform employs differential privacy technology to add noise to the uploaded gradients, further protecting data privacy; and uses blockchain technology to ensure the traceability and integrity of the model parameters. These parameters are obtained through dynamic learning via an attention mechanism. On the internal test set, the fusion model achieved a diagnostic accuracy of 95.2%, significantly outperforming the single-image model (87.3%) and the dual-modal fusion model (91.5%).

[0015] This platform employs federated learning technology to achieve multi-center collaborative training where "the data remains stationary while the model moves." Each medical institution trains its model locally and only uploads the model parameters (gradients or weights) to the central server for aggregation, thus protecting data privacy while enabling continuous improvement in model performance.

[0016] The backend personalized rehabilitation real-time simulation and optimization module comprises a three-layer cascaded structure: a load monitoring layer, a behavior capture layer, and a rework guarantee layer. The load monitoring layer integrates surface electromyography and video information to quantify muscle activation levels and exercise load. The behavior capture layer performs motion keypoint detection and trajectory reconstruction based on posture estimation. The rework guarantee layer establishes a mapping relationship between load response and functional recovery to achieve effect evaluation and feedback correction. This module models exercise prescription optimization as a Markov decision process and employs a proximal policy optimization algorithm for policy learning. The objective function follows the formula: in, This represents the probability ratio between the old and new strategies. For the estimation of the advantage function, To tailor the parameters, a real-time feedback mechanism is introduced, allowing the model to adjust prescription parameters based on the patient's dynamic response during training, thereby achieving adaptive optimization of the exercise prescription. Preliminary application results show that this method can improve rehabilitation efficiency and reduce the risk of adverse reactions while ensuring training safety, providing continuous and quantifiable data support for individualized rehabilitation.

[0017] In other embodiments, the system as a whole adopts a multimodal deep learning framework to achieve multi-source heterogeneous data fusion, completes MRI image volume segmentation based on a 3D CNN (nnU-Net) model, uses a BERT-based NLP model to parse unstructured medical record text, and combines computer vision technology to analyze patients' movement patterns in real time and identify non-standard movements in rehabilitation training.

[0018] In practical implementation, the front end utilizes neuromorphic visual physical examination prediction technology, using a spatial grid as a benchmark to accurately quantify physical examination movements. This overcomes the bottlenecks of traditional physical examinations, which are highly subjective and difficult to standardize, providing doctors with objective, continuous, and traceable physical examination assessment data, reducing reliance on MRI images from the outset. The middle end uses image-electromyography digital twin coupling technology, integrating multi-dimensional data such as MRI / CT images, sEMG electromyography signals, and medical history semantics to construct a physical information coupling computing engine. This engine accurately determines the necessity of surgery, effectively avoiding misdiagnosis, missed diagnosis, and over-surgery, achieving "precise surgery when necessary and conservative treatment when unnecessary." The back end uses exercise prescription time-series analysis technology to construct a three-layer cascaded response network of "monitoring-regulation-feedback," enabling real-time monitoring and adaptive optimization of the rehabilitation process, preventing postoperative re-tears, ensuring patients' functional recovery, and directly protecting the joint health of millions of agricultural workers.

[0019] The front-end visual intelligent physical examination module uses a spatial grid as a benchmark and completes multi-view synchronous acquisition and precise positioning through bionic compound eye array three-dimensional perception technology. It adopts the Graph-LSTM-Attention model to dynamically model and predict the trajectory of the physical examination action. It separates individual differences and action temporal features through the spatiotemporal identity decoupling method to achieve data standardization and alignment. Its state modeling and observation follow the formula to output objective and quantitative physical examination data. The mid-range image-electromyography digital twin coupled diagnostic module receives the aforementioned data and extracts image, clinical, and motion modal features using 3D CNN, BERT, and LSTM respectively. These features are then fused using a cross-modal Transformer, with the following formula: Multi-center collaborative training is achieved using the federated averaging algorithm: Combining differential privacy and blockchain to ensure data security, the system completes lesion identification and surgical necessity determination. The backend personalized rehabilitation real-time simulation and optimization module constructs a three-tiered monitoring-regulation-feedback mechanism based on diagnostic results, integrates sEMG and video information to achieve load quantification, completes action parsing through pose estimation, models prescription optimization as a Markov decision process, and employs a proximal strategy optimization algorithm with the objective function as follows: The exercise prescription is adjusted in real time, forming a closed-loop operation of the entire process from physical examination and diagnosis to rehabilitation simulation.

[0020] This invention utilizes a biomimetic compound eye array-based 3D perception technology. Through spatial grid-based deployment, it achieves multi-view synchronous acquisition and precise positioning, significantly improving the ability to capture details of physical examination movements and spatial resolution, enabling "zero-radiation" non-invasive assessment during the screening process. Based on the Graph-LSTM-Attention model, it dynamically models and predicts the trajectory of physical examination movements, enhancing the foresight and sensitivity of abnormal movement recognition. Furthermore, it introduces a spatiotemporal identity decoupling modeling method to separate individual differences and temporal features of movements under a unified spatial grid benchmark, achieving data standardization and alignment across individuals and scenarios, and enhancing the comparability and stability of assessment results.

[0021] The system can identify and quantify abnormal physical examination movement patterns, analyze movement coordination and stability, and provide objective, continuous and quantifiable data support for clinical physical examination and functional assessment, promoting the transformation of physical examination patterns from experience-based judgment to intelligent and standardized assessment.

[0022] This invention employs a three-dimensional fusion diagnostic model integrating imaging, clinical findings, and movement, overcoming the limitations of traditional diagnoses that rely solely on single images or clinical symptoms. Traditional shoulder joint disease diagnosis primarily relies on MRI images, but inconsistencies between imaging findings and clinical symptoms often lead to a misdiagnosis rate as high as 30%. This platform innovatively integrates imaging features, clinical diagnostic information, and shoulder joint movement data to achieve multi-dimensional and comprehensive disease differential diagnosis.

[0023] Multimodal fusion employs a Transformer-based cross-modal attention mechanism. First, each modality extracts features through an independent encoder: image data uses a 3D CNN to extract spatial features, clinical data uses BERT to encode textual features, and motion data uses LSTM to extract temporal features. Then, a cross-modal Transformer learns the relationships between different modalities, achieving effective fusion of heterogeneous data.

[0024] This invention constructs a three-layer cascaded response mechanism: In the load monitoring layer, surface electromyography (sEMG) and video information are integrated to collect and quantify muscle activation levels and exercise load in real time during training; in the behavior capture layer, key point detection and trajectory reconstruction of human movement are performed based on posture estimation methods, and individualized musculoskeletal models are combined to realize action structure analysis and anomaly identification; in the rework guarantee layer, the training effect is quantitatively evaluated and feedback correction is performed by establishing a mapping relationship between load response and functional recovery.

[0025] Based on this, the exercise prescription optimization process is modeled as a Markov decision process (MDP). The state space S contains the patient's current functional state (such as pain level, joint range of motion, muscle strength, etc.), the action space A represents the type of training action and its combination of intensity, frequency, and duration, and the reward function R comprehensively considers the degree of functional improvement and training safety.

[0026] The basic principles and main features of the present invention have been described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are only illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from the spirit and scope of the present invention. All such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the invention is defined by the appended claims and their equivalents.

Claims

1. A shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin, characterized in that, It includes a front-end visual intelligent physical examination module, a mid-end image-electromyography digital twin coupled diagnosis module, and a back-end personalized rehabilitation real-time simulation and optimization module. The three modules are connected in sequence. The front-end visual intelligent physical examination module uses a spatial grid as the positioning reference and achieves multi-view synchronous acquisition and precise positioning of shoulder joint physical examination movements through a bionic compound eye array. It quantitatively evaluates the physical examination movements and outputs objective physical examination data. The mid-range imaging-electromyography digital twin coupled diagnostic module receives front-end physical examination data, integrates medical images, sEMG electromyography signals, and clinical history data to construct a physical information coupling computing engine, and completes the identification of shoulder joint lesions and the determination of the necessity of surgery; The backend personalized rehabilitation real-time simulation and optimization module receives mid-end diagnostic results and constructs a three-level mechanism of monitoring-regulation-feedback based on multimodal perception and temporal modeling to realize real-time monitoring and adaptive optimization of rehabilitation exercise prescriptions, forming a closed-loop system for the entire process from physical examination and diagnosis to rehabilitation simulation.

2. The shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin as described in claim 1, characterized in that, The front-end visual intelligent physical examination module includes a 3D perception unit, an action modeling unit, and a temporal decoupling unit. The 3D perception unit completes multi-view data acquisition through a spatially gridded bionic compound eye array. The action modeling unit uses a Graph-LSTM-Attention model for action trajectory prediction. The temporal decoupling unit separates individual differences from action temporal features based on a spatiotemporal identity decoupling method, achieving data standardization and alignment. The output of the 3D perception unit is connected to the input of the action modeling unit, the output of the action modeling unit is connected to the input of the temporal decoupling unit, and the output of the temporal decoupling unit is connected to the input of the mid-end image-electromyography digital twin coupled diagnostic module. The state modeling and observation process of the front-end visual intelligent physical examination module follows the formula: ， in, The vector represents the human motion state (including spatial keypoints, motion velocity, and posture structure), and s is an individual identity feature variable. The data is the observation data from the i-th viewpoint (obtained by multi-view acquisition from the compound eye array). This represents the state evolution function driven by the ganglion prediction network. It represents a multi-view observation mapping function, which achieves independent modeling of individual differences and action patterns through spatiotemporal identity decoupling, and completes data standardization and alignment under a unified spatial grid benchmark.

3. The shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin as described in claim 1, characterized in that, The mid-range image-electromyography digital twin coupled diagnostic module adopts a Transformer-based cross-modal attention mechanism. Image data is processed using 3D CNN to extract spatial features, clinical data is encoded with BERT for textual features, and motion data is processed using LSTM to extract temporal features. Multimodal feature fusion is achieved through a cross-modal Transformer, and the fusion calculation follows the formula: Among them, F image F clinical F motion W represents the feature representations of imaging, clinical, and motion modalities, respectively. fusion To fuse the weight matrix, it is dynamically learned through an attention mechanism, and a federated averaging algorithm is used to achieve multi-center collaborative training. The parameter update follows the formula: in, Here are the global model parameters for round t, and k is the number of participating organizations. Let k be the amount of data for the k-th institution. The platform provides local model parameters for the k-th institution. It also employs differential privacy technology to add noise to the uploaded gradients, further protecting data privacy. Blockchain technology is used to ensure the traceability and integrity of the model parameters.

4. The shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin as described in claim 1, characterized in that, The backend personalized rehabilitation real-time simulation and optimization module comprises a three-layer cascaded structure: a load monitoring layer, a behavior capture layer, and a rework guarantee layer. The load monitoring layer integrates surface electromyography and video information to quantify muscle activation levels and exercise load. The behavior capture layer performs motion keypoint detection and trajectory reconstruction based on posture estimation. The rework guarantee layer establishes a mapping relationship between load response and functional recovery to achieve effect evaluation and feedback correction. This module models exercise prescription optimization as a Markov decision process and employs a proximal policy optimization algorithm for policy learning. The objective function follows the formula: in, This represents the probability ratio between the old and new strategies. For the estimation of the advantage function, To tailor the parameters, a real-time feedback mechanism is introduced, allowing the model to adjust the prescription parameters based on the patient's dynamic response during training, thereby achieving adaptive optimization of the exercise prescription.

5. The shoulder joint disorder diagnosis and rehabilitation simulation system based on image and digital twin as described in claim 1, characterized in that, The system adopts a multimodal deep learning framework to achieve multi-source heterogeneous data fusion, completes MRI image volume segmentation based on the 3D CNN (nnU-Net) model, uses the BERT-based NLP model to parse unstructured medical record text, and combines computer vision technology to analyze patients' movement patterns in real time and identify non-standard movements in rehabilitation training.

Images