Rheumatoid arthritis and its comorbidity diagnosis and treatment system based on dexterous hand and large model

Abstract

This invention belongs to the field of intelligent medical devices and artificial intelligence-assisted diagnosis and treatment technology, and particularly relates to a diagnosis and treatment system for rheumatoid arthritis and its comorbidities based on a dexterous hand and a large model. The system includes: a processor equipped with an integrated diagnosis, treatment, and rehabilitation large model; a depth vision sensor; a robotic arm; and a dexterous hand. The depth vision sensor is used to collect multi-dimensional image information of patients with rheumatoid arthritis and its comorbidities. The robotic arm is used to move the dexterous hand to the target operation position. The dexterous hand is used to move data acquisition components to collect the patient's pulse pressure data and joint feature scan data. The integrated diagnosis, treatment, and rehabilitation large model is used to generate a physical rehabilitation treatment plan based on the multi-dimensional image information, pulse pressure data, and joint feature scan data, and to control the dexterous hand to perform traditional Chinese medicine massage therapy. This invention achieves standardization, automation, and intelligence throughout the entire diagnosis and treatment process for rheumatoid arthritis and its comorbidities, realizing integrated closed-loop control from data acquisition to rehabilitation treatment.

Description

Technical Field

[0001] This invention relates to the fields of intelligent medical devices and artificial intelligence-assisted diagnosis and treatment, and in particular to a diagnosis and treatment system for rheumatoid arthritis and its comorbidities based on dexterous hands and large models. Background Technology

[0002] The traditional diagnosis and treatment of rheumatoid arthritis and its comorbidities often involves procedures such as pulse diagnosis, joint feature scanning, and physical rehabilitation massage. For example, doctors use their fingers to sense the patient's pulse to make a diagnosis, assess signs such as joint swelling and deformity through palpation and visual observation, and implement physical therapy such as massage according to the condition.

[0003] However, traditional clinical treatment models have many shortcomings, specifically in the following aspects: First, current data collection requires a high level of experience from doctors. Intern doctors often collect incomplete data and their procedures are not standardized, leading to data inaccuracies. Second, inflammatory physical therapy demands high levels of skill and experience from doctors, and the inconsistency in technique during prolonged rehabilitation significantly impacts the effectiveness of the treatment. Therefore, current treatment models heavily rely on doctors' manual operations and clinical experience, resulting in low standardization, poor repeatability, and difficulty in achieving full automation and intelligentization of the entire process. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a diagnostic and treatment system for rheumatoid arthritis and its comorbidities based on dexterous hands and large models, in order to solve the problems of low standardization, poor repeatability, and difficulty in achieving full-process automation and intelligence in existing technologies.

[0005] This invention provides a diagnostic and treatment system for rheumatoid arthritis and its comorbidities based on a dexterous hand and a large model, comprising: a processor equipped with an integrated diagnostic, therapeutic and rehabilitation large model, a depth vision sensor, a robotic arm and a dexterous hand; The depth vision sensor is used to collect multi-dimensional image information of patients with rheumatoid arthritis and its comorbidities according to the image acquisition instructions of the integrated diagnosis and rehabilitation model. The robotic arm is used to move the dexterous hand to the target operation position according to the movement control instructions of the integrated diagnosis and rehabilitation model; The dexterous hand, connected to the robotic arm, is used to move the data acquisition component to collect the patient's pulse pressure data and joint feature scan data according to the feature parameter acquisition instructions of the integrated diagnosis and rehabilitation model. The integrated diagnosis, treatment, and rehabilitation model is used to generate physical rehabilitation treatment plans based on the multi-dimensional image information, pulse pressure data, and joint feature scan data, so as to control the dexterous hand to perform traditional Chinese massage therapy.

[0006] Furthermore, the multi-dimensional image information includes at least: images of the patient's arthritis symptoms, tongue and facial features for auxiliary diagnosis, images of the patient's surgical site for determining the target operation location, and images of the operating status of the data acquisition component.

[0007] Furthermore, the data acquisition component includes at least: a pressure sensor integrated into the fingertip of the dexterous hand, and a detection device grasped by the dexterous hand; the feature parameter acquisition command includes a first feature parameter acquisition command and a second feature parameter acquisition command; The acquisition of the patient's pulse pressure data and joint feature scan data includes: Based on the first feature parameter acquisition command, the dexterous hand is switched to pulse diagnosis mode, and the pulse pressure data is acquired by simulating traditional Chinese medicine pulse diagnosis techniques using the pressure sensor; and Based on the second feature parameter acquisition command, the dexterous hand is switched to feature scanning mode, and the joint feature scanning data is acquired through the detection device.

[0008] Furthermore, the integrated diagnosis, treatment, and rehabilitation model includes at least: a traditional Chinese medicine pulse diagnosis sub-model for processing the pulse pressure data, a feature scanning sub-model for processing the joint feature scanning data, and a rheumatoid inflammation physical rehabilitation sub-model for generating the physical rehabilitation treatment plan.

[0009] Furthermore, the TCM pulse diagnosis model is used to: obtain multi-dimensional pulse analysis data based on the pulse pressure data, combined with the displacement information recorded by the robotic arm and the patient's wrist image acquired by the depth vision sensor; Based on the multi-dimensional pulse analysis data, the pulse type of the patient is identified according to the preset TCM palpation and tracing logic; Based on the pulse type, the arthritis symptom image, the tongue image, and the facial feature image, a pulse analysis report containing TCM diagnostic conclusions is generated.

[0010] Furthermore, the feature scanning sub-model is used to: obtain joint scan analysis data based on the joint feature scan data and combined with the arthritis symptom images; Based on the joint scan analysis data, the degree of inflammation and deformity of the patient's metacarpophalangeal joints and proximal interphalangeal joints are identified, and a joint feature analysis report is obtained.

[0011] Furthermore, the rheumatoid inflammation physical rehabilitation sub-model is used for: Based on the pulse analysis report and the joint feature analysis report, a personalized physical rehabilitation treatment plan is generated. The personalized physical rehabilitation treatment plan includes at least: acupressure points, pressure level, operation frequency, movement trajectory, and treatment duration.

[0012] Furthermore, the rheumatoid inflammation physical rehabilitation sub-model is also used for: According to the personalized physical rehabilitation treatment plan, the dexterous hand is controlled to switch to physical therapy mode to perform traditional Chinese massage therapy on the patient. During treatment, the massage parameters of the dexterous hand are dynamically adjusted or treatment is paused based on the pressure parameters fed back by the pressure sensor and the treatment process images monitored synchronously by the depth vision sensor.

[0013] Furthermore, the integrated diagnosis, treatment, and rehabilitation model is also used to generate control instructions based on the task type of the current diagnosis and treatment stage; The control commands include at least: an image acquisition command for triggering the depth vision sensor to perform an image acquisition task; a feature parameter acquisition command for controlling the dexterous hand to move the data acquisition component to perform a data acquisition task; a movement control command for controlling the movement of the robotic arm; and a treatment command for controlling the dexterous hand to perform traditional Chinese massage therapy.

[0014] Furthermore, the integrated diagnosis, treatment, and rehabilitation model adopts an architecture that fuses the ViT-Unet model with the medical LLM large language model.

[0015] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: On the one hand, unlike existing technologies which suffer from low standardization and poor repeatability, this invention utilizes a dexterous hand that can switch between pulse diagnosis and positioning modes, feature scanning modes, and physical therapy modes according to instructions from a large model. This allows for the execution of pulse acquisition, detection device scanning, and physical rehabilitation treatment processes, respectively, avoiding the subjective differences and instability inherent in manual operation. The robotic arm, in conjunction with a depth vision sensor, achieves precise positioning and movement, ensuring consistent position and controllable force in each operation, significantly improving the repeatability and safety of diagnostic and treatment procedures.

[0016] On the other hand, unlike existing technologies that struggle to achieve full-process automation and intelligence, this invention, through the collaborative work of a dexterous hand, robotic arm, depth vision sensor, and large-scale model, achieves standardization, automation, and intelligence throughout the entire diagnosis and treatment process of rheumatoid arthritis and its comorbidities, realizing integrated closed-loop control from data acquisition to rehabilitation treatment. Simultaneously, this invention possesses the ability to monitor the treatment process and iteratively optimize the treatment plan. Through real-time monitoring of the patient's facial expressions and joint status using a depth vision sensor, and feedback of massage intensity using a pressure sensor, the large-scale model can dynamically adjust treatment parameters, ensuring treatment safety. Furthermore, this system is entirely automated using intelligent devices, completely independent of human intervention, providing a new model for assisted medicine. It achieves a fully automated closed loop from data acquisition and analysis to intervention plan generation, with its output used to assist clinical treatment, and the final treatment action confirmed and executed by the physician. This enhances the system's information processing capabilities, thereby improving overall diagnostic efficiency and quality.

[0017] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0018] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a schematic diagram of the main modules of the rheumatoid arthritis and comorbidity diagnosis and treatment system based on dexterous hands and large models according to an embodiment of the present invention. Detailed Implementation

[0019] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0020] One specific embodiment of the present invention discloses a diagnostic and treatment system for rheumatoid arthritis and its comorbidities based on dexterous hands and large models. Rheumatoid arthritis is an autoimmune disease characterized primarily by chronic, symmetrical polyarticular synovitis, which can lead to progressive destruction of articular cartilage and bone tissue. During the disease progression, patients often experience one or more other diseases related to the pathogenesis, inflammatory state, or treatment of rheumatoid arthritis; these are called comorbidities, such as cardiovascular disease, osteoporosis, interstitial lung disease, Sjögren's syndrome, and metabolic syndrome. Figure 1As shown, this system specifically includes the following modules: a processor equipped with an integrated diagnostic and rehabilitation model, a depth vision sensor, a robotic arm, and a dexterous hand; The depth vision sensor is used to collect multi-dimensional image information of patients with rheumatoid arthritis and its comorbidities according to the image acquisition instructions of the integrated diagnosis and rehabilitation model. The robotic arm is used to move the dexterous hand to the target operation position according to the movement control instructions of the integrated diagnosis and rehabilitation model; The dexterous hand, connected to the robotic arm, is used to move the data acquisition component to collect the patient's pulse pressure data and joint feature scan data according to the feature parameter acquisition instructions of the integrated diagnosis and rehabilitation model. The integrated diagnosis, treatment, and rehabilitation model is used to generate physical rehabilitation treatment plans based on the multi-dimensional image information, pulse pressure data, and joint feature scan data, so as to control the dexterous hand to perform traditional Chinese massage therapy.

[0021] Specifically, embodiments of the present invention mainly include a processor equipped with an integrated diagnostic and rehabilitation model, and a depth vision sensor, a robotic arm, a dexterous hand, and data acquisition components controlled by the integrated diagnostic and rehabilitation model. The integrated diagnostic and rehabilitation model, as the core processing model of the present invention, enables intelligent operation throughout the entire process of data processing, analysis, diagnosis, and rehabilitation plan generation. The depth vision sensor is responsible for acquiring images and depth information (such as depth image information like the degree of joint swelling) throughout the entire process, achieving multi-dimensional capture of the patient's physical state. The robotic arm, as a mobile assistive mechanism, plays a crucial role in achieving precise positioning and flexible movement of the dexterous hand and data acquisition components, ensuring the orderly conduct of various operations. The dexterous hand, as the system's execution terminal, mainly undertakes functions such as grasping with the detection device, pulse diagnosis in traditional Chinese medicine, and massage.

[0022] It should be noted that the TCM massage therapy based on the physical rehabilitation treatment plan described in this system is not just mechanically pressing a single joint, but is implemented along the entire meridian path. It only represents a mechanical control method based on data processing results. The purpose is to have the robotic arm drive the dexterous hand to apply physical force to the patient's body surface according to the pressure parameters output by the model, simulating the massage techniques of a professional physician, so as to realize the closed-loop control process and information processing capabilities of the system.

[0023] It is understandable that traditional Chinese massage therapy can be effective in shortening the duration of morning stiffness and reducing joint swelling and pain in rheumatoid arthritis (and its comorbidities). Its techniques begin with acupressure, primarily focusing on unblocking the flow of Qi and blood throughout the meridians, supplemented by kneading, pressing, and pinching techniques. The massage pressure parameters and operation trajectories output by this system are automatically generated by a large model based on real-time data collection and do not constitute a diagnosis or treatment for any disease. In practical applications, relevant operation parameters should be reviewed and confirmed by a professional physician before execution; the system itself does not replace the physician's clinical judgment.

[0024] In addition, this system can adapt to the long course and slow changes of rheumatoid arthritis (and its comorbidities), and supports long-term, multiple physical intervention treatments and dynamic monitoring for patients with rheumatoid arthritis (and its comorbidities). Through data collection and feedback analysis before and after each treatment, it can achieve iterative optimization of physical operation parameters.

[0025] Preferably, the dexterous hand is a six-degree-of-freedom dexterous hand, with a depth vision sensor integrated at its wrist and a pressure sensor (measurement range 0-10N, accuracy 0.1N) integrated at its fingertips; the dexterous hand and the robotic arm are connected so that the depth vision sensor can be adjusted according to the patient's height and position (sitting / supine position) to ensure that the sensor can be accurately aligned with the patient's face, tongue and wrist pulse.

[0026] Preferably, the multi-dimensional image information acquired by the depth vision sensor includes at least: images of the patient's arthritis symptoms (such as redness, swelling, deformity, etc.) for auxiliary diagnosis, images of the tongue and facial features, images of the patient's surgical site for determining the target operation location, and images of the operational status of the data acquisition component. After acquisition, the depth vision sensor can perform preliminary preprocessing on the above data and transmit the multi-dimensional image information to the integrated diagnosis and rehabilitation model in real time, providing comprehensive data support for the model's analysis and ensuring that the model can acquire accurate raw data in a timely manner, thus guaranteeing subsequent analysis and diagnosis.

[0027] Furthermore, the data acquisition component includes at least: a pressure sensor integrated into the fingertip of the dexterous hand, and a detection device grasped by the dexterous hand; the feature parameter acquisition command includes a first feature parameter acquisition command and a second feature parameter acquisition command; the acquisition of the patient's pulse pressure data and joint feature scan data includes: Based on the first feature parameter acquisition command, the dexterous hand is switched to pulse diagnosis mode, and the pulse pressure data is acquired by simulating traditional Chinese medicine pulse diagnosis techniques using the pressure sensor. Based on the second feature parameter acquisition command, the dexterous hand is switched to feature scanning mode, and the joint feature scan data is acquired by the detection device. For example, the dexterous hand can grasp the probe of an ultrasound detector or a joint feature scanner, acquiring ultrasound grayscale images and color Doppler blood flow signals through the ultrasound detector; and acquiring two-dimensional or three-dimensional image data of the joint area through the joint feature scanner. Of course, it is also possible to first switch to feature scanning mode to perform joint feature scanning, and then switch to pulse diagnosis mode to acquire pulse pressure data; there is no limitation here.

[0028] During implementation, the working mode of the dexterous hand is triggered by the user to generate corresponding control commands from the integrated diagnosis and rehabilitation model, and the model controls the dexterous hand to switch modes according to the current diagnosis and treatment stage. The working modes of the dexterous hand may specifically include: pulse diagnosis and positioning mode, feature scanning mode and subsequent physical therapy mode. This invention does not restrict the execution order of various working modes of the dexterous hand, and the user can choose and determine them.

[0029] During the data acquisition phase of this invention, the dexterous hand primarily performs the following two core functions: (1) Detection device grasping: It can accurately grasp various functional detection devices (such as the probes of ultrasound detectors and joint feature scanners) and cooperate with the robotic arm to complete the detection operations of various parts of the patient's body, ensuring the stability and accuracy of the detection process. For example, in the joint feature scanning stage, the dexterous hand switches to feature scanning mode, first moves to the detection device storage location to grasp the corresponding detection device (such as an ultrasound probe), and then moves the detection device to the patient's joint to collect data. At this time, the pressure sensor integrated into the dexterous hand does not need to participate in the work.

[0030] (2) Traditional Chinese Medicine pulse diagnosis operation: The dexterous hand switches to the pulse diagnosis positioning mode. At this time, the dexterous hand does not grasp any detection device. The dexterous hand has a built-in high-precision pressure sensor in the fingertip, which can simulate the TCM pulse diagnosis method of "lifting, pressing and searching" to accurately perceive the core information such as the pressure, frequency and rhythm of the patient's pulse, providing basic data support for TCM diagnosis. At the same time, the pulse pressure data is transmitted to the large model in real time for analysis and processing.

[0031] Furthermore, during the physical rehabilitation treatment phase of this invention, the dexterous hand can switch to a physical therapy mode (massage mode). In this mode, it neither grasps the detection device nor collects pulse data via pressure sensors. Instead, it applies pressure to the patient's acupoints or joints according to the rehabilitation plan and provides real-time feedback on the massage intensity via pressure sensors. Therefore, through this mode-switching mechanism, the dexterous hand performs different tasks at different stages, avoiding operational conflicts and achieving fully automated collaboration throughout the entire process.

[0032] Preferably, the robotic arm, as a movement assist mechanism for a dexterous hand, mainly includes the following functions: (1) Target position movement: Based on the images of the patient's operating area collected by the depth vision sensor and the operating status images of the data acquisition components, the dexterous hand is accurately moved to the target areas such as the patient's pulse taking area, metacarpophalangeal joint (MCP), and proximal interphalangeal joint (PIP) to ensure the accuracy of the operation position.

[0033] (2) Assisted operation execution: It can work with the dexterous hand to complete a series of operations such as moving and scanning the detection device. According to the standardized process and the movement control instructions issued by the integrated diagnosis and rehabilitation model, the movement speed, angle and force can be adjusted to suit the physical characteristics of different patients and avoid causing discomfort to the patients during the operation.

[0034] In some preferred embodiments, the integrated diagnosis, treatment, and rehabilitation model adopts an architecture that fuses the ViT-Unet model (Visual Transformer-U-shaped Network) with a medical LLM large language model, and is trained using supervised learning. The ViT-Unet model is responsible for handling visual tasks, such as processing multi-dimensional image information; the LLM large language model is responsible for handling language tasks, such as logical reasoning in traditional Chinese medicine diagnosis and treatment plan generation.

[0035] Furthermore, the integrated diagnosis, treatment, and rehabilitation model includes at least: a traditional Chinese medicine pulse diagnosis sub-model for processing the pulse pressure data, a feature scanning sub-model for processing the joint feature scanning data, and a rheumatoid inflammation physical rehabilitation sub-model for generating the physical rehabilitation treatment plan.

[0036] Preferably, the TCM pulse diagnosis model is used to: obtain multi-dimensional pulse analysis data based on the pulse pressure data, combined with the displacement information recorded by the robotic arm and the patient's wrist image acquired by the depth vision sensor; Based on the multi-dimensional pulse analysis data, the pulse type of the patient is identified according to the preset TCM palpation and tracing logic; Based on the pulse type, the arthritis symptom image, the tongue image, and the facial feature image, a pulse analysis report containing TCM diagnostic conclusions is generated.

[0037] Specifically, the function of the TCM pulse diagnosis model in practical applications is to analyze pulse pressure data, simulate the TCM diagnostic logic, achieve accurate pulse identification and analysis, and generate a pulse analysis report. For example, by integrating pressure parameters from a dexterous fingertip pressure sensor, displacement information recorded by a robotic arm, and target images from a depth vision sensor, a multi-dimensional pulse data system is constructed to recreate the complete TCM pulse diagnosis scenario. Ultimately, a pulse analysis report is generated for the patient awaiting diagnosis. The pulse analysis can include: pulse type (such as floating pulse, deep pulse, wiry pulse), pulse characteristic parameters (pulse position, pulse strength, pulse rate, rhythm), and TCM diagnostic conclusions, such as Qi deficiency syndrome, blood stasis syndrome, Yang deficiency syndrome, and cold-dampness syndrome.

[0038] In some implementations, the TCM pulse-taking sub-model can be trained using supervised learning. Specifically, several experienced TCM physicians are invited to wear specialized sensor gloves and perform pulse-taking operations on patients with a history of rheumatoid arthritis (and its comorbidities) according to the standard TCM technique of "lifting, pressing, and searching." During the pulse-taking process, the sensor gloves simultaneously collect historical data such as the pressure parameters applied by the physicians' fingers, the spatial displacement trajectory of the fingers, the floating coordinate position, and the frequency of pressure changes. Simultaneously, the physicians record the patient's pulse type based on their clinical experience. The collected pressure parameters, displacement information, and wrist images are used as input features, and the pulse type labeled by the physicians is used as the output label to construct a training sample set. Then, deep learning algorithms (such as convolutional neural networks or LSTM long short-term memory networks) can be used to iteratively train the sub-model, enabling the model to learn to identify the mapping relationship between different pulse types from multiple types of pulse data. After training, the TCM pulse-taking sub-model can automatically output pulse type identification results based on real-time collected multi-dimensional pulse analysis data in practical applications.

[0039] Preferably, the feature scanning sub-model is used to: obtain joint scan analysis data based on the joint feature scan data and combined with the arthritis symptom images; Based on the joint scan analysis data, the degree of inflammation and deformity of the patient's metacarpophalangeal joints and proximal interphalangeal joints are identified, and a joint feature analysis report is obtained.

[0040] Specifically, the function of the feature scanning sub-model in the practical application stage is to: identify the detection device grasped by the dexterous hand, start the scanning process, ensure the standardization and accuracy of joint feature scanning, and generate a joint feature analysis report, which may include: the degree of joint inflammation (mild / moderate / severe), cartilage damage grading, and joint deformity, etc.

[0041] In some implementations, the training process of the feature scanning sub-model includes: (1) Training for identifying detection devices: Using image samples of various functional detection devices (such as ultrasonic detectors, joint scanners and their probes) as input, supervised learning enables the model to quickly identify the type, function and operating specifications of different detection devices. For example, image data of various functional detection devices under different postures, lighting and distance conditions are collected, and the type of device, grasping point and operation interface location are labeled to form a training database. After training, the sub-model can quickly identify the type, point and location of the currently available detection devices from the images collected by the depth vision sensor, and control the dexterous hand to perform grasping operations.

[0042] (2) Scanning process learning and training: Senior TCM physicians wear sensor gloves, grab different types of detection devices, and perform feature scans on the metacarpophalangeal joints (MCP), proximal interphalangeal joints (PIP) and other related parts of historical patients according to the preset operation process. The data such as the movement, trajectory and force of the entire operation process are synchronized to the sub-model, so that the model can master the standardized scanning operation process and guide the dexterous hand and robotic arm to complete autonomous scanning.

[0043] For example, based on standard scanning operation data from experienced physicians, and simultaneously recording the movement trajectory of the robotic arm, the grasping action of the dexterous hand, the scanning path, scanning angle, and scanning duration of the detection device, as well as corresponding historical joint feature scanning data and historical images of arthritis symptoms, the sub-model is trained using a combination of supervised learning and imitation learning, with "operational standardization" and "data acquisition completeness" as evaluation indicators. After training, the feature scanning sub-model can output standardized scanning operation control parameters, and can start the detection device grasped by the dexterous hand and perform joint feature scanning operations according to the current task instructions and a standardized scanning process. It can also adjust the motion parameters of the robotic arm and dexterous hand in real time based on feedback data, realizing the automation and precision of joint feature scanning.

[0044] (3) Joint feature data analysis training: The standard joint scan data of historical patients (including historical scan data of joint features, historical images of arthritis symptoms, etc.) are used as input, and the degree of joint inflammation and cartilage damage grade marked by clinicians are used as output labels. Supervised training is carried out so that the sub-model can automatically identify pathological features such as joint swelling and synovial hyperplasia from the scan data.

[0045] Therefore, after training, this feature scanning sub-model can not only control the dexterous hand and detection device to complete standardized scanning in practical applications, but also perform real-time analysis on the joint feature scanning data obtained by scanning, and output a joint feature analysis report including information such as inflammation grading and damage assessment.

[0046] Preferably, the rheumatoid inflammation physical rehabilitation sub-model is used for: Based on the pulse analysis report and the joint feature analysis report, a personalized physical rehabilitation treatment plan is generated. The personalized physical rehabilitation treatment plan includes at least: acupressure points, pressure level, operation frequency, movement trajectory, and treatment duration.

[0047] In some embodiments, the rheumatoid inflammation physical rehabilitation sub-model is also used for: According to the personalized physical rehabilitation treatment plan, the dexterous hand is controlled to switch to physical therapy mode to perform traditional Chinese massage therapy on the patient. During treatment, the massage parameters of the dexterous hand are dynamically adjusted or treatment is paused based on the pressure parameters fed back by the pressure sensor and the treatment process images monitored synchronously by the depth vision sensor.

[0048] Specifically, the function of the rheumatoid inflammation physical rehabilitation sub-model in the practical application stage is to generate personalized physical rehabilitation plans based on the pulse analysis report generated by the traditional Chinese medicine pulse diagnosis sub-model and the joint feature analysis report generated by the feature scanning sub-model, and to control the dexterous hand to perform massage operations.

[0049] In some implementations, the training process of the rheumatoid inflammation physical rehabilitation sub-model includes: (1) Human acupoint learning: Using human images with acupoints marked as input, the sub-model can master the location and function of human acupoints through supervised learning, especially the key acupoints related to the rehabilitation of rheumatoid inflammation.

[0050] (2) Basic massage technique training: Senior TCM physicians wearing sensor gloves perform rheumatoid inflammation rehabilitation massage. The gloves simultaneously record massage history data such as acupoint location, pressure magnitude, coordinate trajectory, and operation frequency during the massage process, and transmit it to the sub-model for learning, so that the sub-model can master basic rehabilitation massage techniques, and thus have the ability to output standardized massage parameters. The massage parameters include at least the range of force, trajectory pattern, and frequency range.

[0051] (3) Personalized massage training: Senior TCM physicians implement differentiated massage techniques for rheumatoid arthritis patients with different conditions and constitutions. The corresponding operation data is synchronized to the model. Through sample accumulation, the model can generate a suitable personalized rehabilitation plan based on the specific condition of the patient. For example, the diagnostic characteristics of historical patients and the corresponding massage operation data are synchronized as input samples. Through training with a large number of cases, the sub-model establishes a mapping relationship between symptom characteristics and massage parameters.

[0052] (4) Personalized rehabilitation plan generation training: Using historical patients' pulse analysis reports and joint feature analysis reports as input, and effective rehabilitation plans (including acupoints, intensity, frequency, duration, and trajectory) formulated by physicians as output, and combining the acupoint knowledge, basic manipulation parameters, and personalized manipulation mapping rules obtained from the previous training, the model is trained using supervised learning. After training, the model is able to automatically output personalized rehabilitation plans.

[0053] For example, severe cartilage damage corresponds to a smaller massage force (1-2N) to protect the joint, while mild damage can be appropriately increased (2-3N); patients in the active inflammatory phase should use a lower frequency (10-12 times / minute), while those in the remission phase can increase the frequency (13-15 times / minute), and so on. Of course, the correspondence between the above symptom characteristics and massage parameters can be achieved not only through the mapping function obtained by supervised learning training, but also by determining it according to a preset correspondence list; no restriction is placed here.

[0054] Furthermore, the integrated diagnosis, treatment, and rehabilitation model is also used to: generate control instructions based on the task type of the current diagnosis and treatment stage; The control commands include at least: an image acquisition command for triggering the depth vision sensor to perform an image acquisition task; a feature parameter acquisition command for controlling the dexterous hand to move the data acquisition component to perform a data acquisition task; a movement control command for controlling the movement of the robotic arm; and a treatment command for controlling the dexterous hand to perform traditional Chinese massage therapy.

[0055] In practice, the triggering of the above-mentioned instructions can be automatically generated by the integrated diagnosis and rehabilitation model based on the user's input operation instructions or preset diagnosis and treatment procedures, and sent to the corresponding execution components. For example, when the integrated diagnosis and rehabilitation model receives the user's input "start taking the pulse" operation instruction, it automatically generates an image acquisition instruction and sends it to the depth vision sensor to acquire an image of the wrist area. At the same time, it generates a motion control instruction and sends it to the robotic arm to move the dexterous hand to the pulse-taking position. After the positioning is completed, it generates a first feature parameter acquisition instruction and sends it to the dexterous fingertip pressure sensor or detection device to acquire pulse and joint feature data. Subsequently, when the system automatically enters the joint scanning stage according to the preset diagnosis and treatment process, the large model does not require user intervention. It automatically generates image acquisition instructions to obtain hand joint images, generates movement control instructions to move the dexterous hand to the storage position of the detection device and grasp the ultrasound probe, and then generates second feature parameter acquisition instructions to start the scan. After completing the data analysis, it automatically generates treatment instructions and sends them to the dexterous hand to perform rehabilitation treatment.

[0056] To further illustrate the present invention's system for diagnosing and treating rheumatoid arthritis and its comorbidities based on dexterous hands and large models, a specific embodiment is provided below, the execution process of which includes: First, use your dexterous hand to take the pulse of the patient and obtain pulse pressure data.

[0057] (1) Patient preparation: Guide the patient to a comfortable sitting position, expose the wrist area, remove the jewelry on the wrist, and ensure that the skin and the fingertips of the fingers are in full contact to avoid affecting the accuracy of pulse collection; at the same time, collect images of the patient's face and tongue through a depth vision sensor as auxiliary diagnostic data.

[0058] (2) Implement the operation, specifically including: Positioning and movement: The depth vision sensor is controlled to collect images of the patient's wrist and transmit them to the integrated diagnosis and rehabilitation model. After the model identifies the target location for pulse taking, it sends instructions to the robotic arm. The robotic arm then moves the dexterous hand to the patient's wrist, adjusting the angle and height to ensure that the dexterous fingertips are in close contact with the pulse taking area.

[0059] Pulse taking data: Based on the first feature parameter acquisition command sent by the TCM pulse taking sub-model, the dexterous hand senses the patient's pulse through the fingertip pressure sensor according to the preset "lifting, pressing and searching" technique, and records the pressure parameters simultaneously; the robotic arm records the displacement, floating trajectory and vertical coordinate position of the dexterous hand; the depth vision sensor simultaneously acquires images of the wrist and visual information related to the pulse, and all data is transmitted to the TCM pulse taking sub-model in real time.

[0060] (3) Data processing: The TCM pulse diagnosis model analyzes the multi-dimensional data such as pressure and displacement collected in real time, identifies the patient's pulse type (such as floating pulse, deep pulse, wiry pulse, etc.), and combines the tongue appearance and facial feature data to complete the preliminary diagnosis and generate a pulse analysis report, which provides a basis for the formulation of subsequent rehabilitation plans.

[0061] Second, a dexterous hand-grasping detection device is used to perform feature scans on the metacarpophalangeal joints (MCP) and proximal interphalangeal joints (PIP) of the patients to be treated, and to obtain joint feature scan data.

[0062] (1) Preliminary preparation: The depth vision sensor acquires images of the patient's hand, identifies the specific locations of the metacarpophalangeal joints (MCP) and proximal interphalangeal joints (PIP), generates instructions after analysis of the large model and transmits them to the robotic arm to complete the positioning and calibration of the dexterous hand and the detection device, ensuring that the scanning range covers the target joints.

[0063] (2) Implement the operation, specifically including: Device grasping: Based on the second feature parameter acquisition command sent by the feature scanning sub-model, the dexterous hand accurately grasps the designated detection device. The robotic arm adjusts its position and moves the detection device to the vicinity of the target joint in the patient's hand, maintaining an appropriate scanning distance.

[0064] Feature scanning: Following the pre-set scanning process of the large model (i.e., the standardized process based on the feature scanning sub-model training), the robotic arm drives the dexterous hand and the detection device to perform a full-range scan of the patient's metacarpophalangeal joints (MCP) and proximal interphalangeal joints (PIP). The depth vision sensor simultaneously collects images and joint depth information (such as the degree of joint swelling, the size of the joint space, etc.) during the scanning process. The joint feature scanning data (such as inflammatory marker-related data) collected by the detection device are transmitted to the feature scanning sub-model in real time.

[0065] (3) Data integration; The feature scanning sub-model integrates and analyzes the joint feature scanning data and arthritis symptom images collected in real time, identifies the inflammation degree and deformity of the patient's joints, and generates a joint feature analysis report; Preferably, the joint feature analysis report can also be combined with the previous pulse analysis report to form a complete physical status data file of the patient and transmit it to the rheumatoid inflammation physical rehabilitation sub-model.

[0066] Third, develop a physical rehabilitation treatment plan and conduct massage rehabilitation therapy.

[0067] (1) Rehabilitation program generation: The physical rehabilitation sub-model for rheumatoid inflammation generates a personalized physical rehabilitation treatment plan based on the pulse analysis report and joint feature analysis report of the patient to be treated, combined with the acupoint knowledge and rehabilitation massage techniques trained by the model. Based on the plan, the parameters such as acupoints, pressure, operation frequency, trajectory and treatment duration of rehabilitation massage are clearly defined to ensure that the plan is suitable for the patient's condition and physical condition.

[0068] (2) Implementation of rehabilitation treatment, specifically including: Plan Confirmation: After the rehabilitation plan is generated, this system supports manual review, which can be confirmed by medical staff. If adjustments are needed, parameters can be manually modified. Once confirmed to be correct, treatment instructions are sent to the dexterous hand and robotic arm for execution.

[0069] Massage operation: The robotic arm drives the dexterous hand to perform massage operations on the relevant acupoints and joints of the patient according to the parameters generated by the rehabilitation plan. The pressure sensor of the dexterous fingertip provides real-time feedback on the massage pressure. The robotic arm adjusts the movement trajectory and speed to ensure that the massage technique is consistent with the plan. The depth vision sensor monitors the treatment process in real time, collects image information such as the patient's facial expression and joint status, and transmits it to the rheumatoid inflammation physical rehabilitation sub-model in real time for plan optimization.

[0070] (3) Treatment monitoring and optimization Process monitoring: During treatment, the large model analyzes in real time the images of the treatment process collected by the depth vision sensor and the pressure parameters fed back by the dexterous hand to determine whether the patient is experiencing discomfort. If excessive pressure or abnormal patient expression occurs, the massage parameters are adjusted immediately or the treatment is paused to ensure the safety of the treatment.

[0071] Program optimization: After each treatment, all execution data and patient recovery status during the treatment process are integrated to iteratively optimize the rehabilitation program, gradually adjusting parameters such as massage techniques and duration to improve the rehabilitation effect; at the same time, the treatment data is stored in the patient's file.

[0072] It is understood that the above embodiments are only for ease of understanding and simplification of description, and should not be construed as limiting the present invention. The present invention does not specifically limit the type of data collection, the construction of the integrated diagnosis and rehabilitation model, the training method, etc.

[0073] Therefore, in this embodiment of the invention, on the one hand, the dexterous hand dynamically switches between pulse diagnosis and positioning mode, feature scanning mode, and physical therapy mode through relevant instructions from the integrated diagnosis, treatment, and rehabilitation model, respectively executing pulse acquisition, detection device grasping and scanning, and physical rehabilitation treatment processes, thus avoiding the subjective differences and instability of manual operation. Precise positioning and movement are achieved through the cooperation of a robotic arm and a depth vision sensor, ensuring consistent position and controllable force for each operation, significantly improving the repeatability and safety of diagnosis and treatment operations. On the other hand, through the collaborative work of the dexterous hand, robotic arm, depth vision sensor, and large model, the standardization, automation, and intelligence of the entire diagnosis and treatment process for rheumatoid arthritis and its comorbidities are realized, achieving integrated closed-loop control from data acquisition to rehabilitation treatment. Simultaneously, this invention possesses the ability to monitor the treatment process and iteratively optimize the treatment plan, thereby enabling the large model to dynamically adjust treatment parameters and ensure treatment safety.

[0074] It should be noted that the entire processing described in this system is completed collaboratively by hardware actuators and software control modules. The hardware component includes at least a dexterous hand, a robotic arm, depth vision sensors, and data acquisition components, while the software component includes an integrated diagnostic and rehabilitation model and its control instruction set. The core of this invention lies in analyzing, mathematically calculating, and mapping various types of multimodal data collected through the integrated diagnostic and rehabilitation model. The output physical rehabilitation treatment plan is only auxiliary information for treatment. This system does not directly perform any diagnostic or treatment operations on the living human body, nor does it output a final clinical diagnostic conclusion. Instead, it provides an information analysis method based on data processing. Therefore, the technical solution of this invention belongs to computer-implemented information processing methods.

[0075] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0076] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A diagnostic and treatment system for rheumatoid arthritis and its comorbidities based on dexterous hands and large models, characterized in that, include: It is equipped with a processor, depth vision sensor, robotic arm and dexterous hand for a large integrated diagnostic and rehabilitation model; The depth vision sensor is used to collect multi-dimensional image information of patients with rheumatoid arthritis and its comorbidities according to the image acquisition instructions of the integrated diagnosis and rehabilitation model. The robotic arm is used to move the dexterous hand to the target operation position according to the movement control instructions of the integrated diagnosis, treatment and rehabilitation model; The dexterous hand, connected to the robotic arm, is used to move the data acquisition component to collect the patient's pulse pressure data and joint feature scan data according to the feature parameter acquisition instructions of the integrated diagnosis and rehabilitation model. The integrated diagnosis, treatment, and rehabilitation model is used to generate physical rehabilitation treatment plans based on the multi-dimensional image information, pulse pressure data, and joint feature scan data, so as to control the dexterous hand to perform traditional Chinese massage therapy.

2. The system according to claim 1, wherein, The multi-dimensional image information includes at least: images of the patient's arthritis symptoms, tongue and facial features for auxiliary diagnosis, images of the patient's surgical site for determining the target operation location, and images of the operating status of the data acquisition component.

3. The system according to claim 2, wherein, The data acquisition component includes at least: a pressure sensor integrated into the fingertip of the dexterous hand, and a detection device grasped by the dexterous hand; the feature parameter acquisition command includes a first feature parameter acquisition command and a second feature parameter acquisition command; The acquisition of the patient's pulse pressure data and joint feature scan data includes: Based on the first feature parameter acquisition command, the dexterous hand is switched to pulse diagnosis mode, and the pulse pressure data is acquired by simulating traditional Chinese medicine pulse diagnosis techniques using the pressure sensor; and Based on the second feature parameter acquisition command, the dexterous hand is switched to feature scanning mode, and the joint feature scanning data is acquired through the detection device.

4. The dexterous hand and large model based rheumatoid arthritis and its comorbidity diagnosis and treatment system according to claim 3, characterized in that, The integrated diagnosis, treatment, and rehabilitation model includes at least: a traditional Chinese medicine pulse diagnosis sub-model for processing the pulse pressure data, a feature scanning sub-model for processing the joint feature scanning data, and a rheumatoid inflammation physical rehabilitation sub-model for generating the physical rehabilitation treatment plan.

5. The dexterous hand and large model based rheumatoid arthritis and its comorbidity diagnosis and treatment system according to claim 4, characterized in that, The TCM pulse diagnosis model is used to: obtain multi-dimensional pulse analysis data based on the pulse pressure data, combined with the displacement information recorded by the robotic arm and the patient's wrist image acquired by the depth vision sensor; Based on the multi-dimensional pulse analysis data, the pulse type of the patient is identified according to the preset TCM palpation and tracing logic; Based on the pulse type, the arthritis symptom image, the tongue image, and the facial feature image, a pulse analysis report containing TCM diagnostic conclusions is generated.

6. The dexterous hand and large model based rheumatoid arthritis and its comorbidity diagnosis and treatment system according to claim 5, characterized in that, The feature scanning sub-model is used to: obtain joint scan analysis data based on the joint feature scan data and combined with the arthritis symptom images; Based on the joint scan analysis data, the degree of inflammation and deformity of the patient's metacarpophalangeal joints and proximal interphalangeal joints are identified, and a joint feature analysis report is obtained.

7. The dexterous hand and large model based rheumatoid arthritis and its comorbidity diagnosis and treatment system according to claim 6, characterized in that, The rheumatoid inflammation physical rehabilitation sub-model is used for: Based on the pulse analysis report and the joint feature analysis report, a personalized physical rehabilitation treatment plan is generated. The personalized physical rehabilitation treatment plan includes at least: acupressure points, pressure level, operation frequency, movement trajectory, and treatment duration.

8. The dexterous hand and large model based rheumatoid arthritis and its comorbidity diagnosis and treatment system according to claim 7, characterized in that, The aforementioned physical rehabilitation sub-model for rheumatoid inflammation is also used for: According to the personalized physical rehabilitation treatment plan, the dexterous hand is controlled to switch to physical therapy mode to perform traditional Chinese massage therapy on the patient. During treatment, the massage parameters of the dexterous hand are dynamically adjusted or treatment is paused based on the pressure parameters fed back by the pressure sensor and the treatment process images monitored synchronously by the depth vision sensor.

9. The dexterous hand and large model based rheumatoid arthritis and its comorbidity diagnosis and treatment system according to claim 1, characterized in that, The integrated diagnosis, treatment, and rehabilitation model is also used to generate control instructions based on the task type of the current diagnosis and treatment stage. The control instructions include at least: an image acquisition instruction for triggering the depth vision sensor to perform an image acquisition task; a feature parameter acquisition instruction for controlling the dexterous hand to move the data acquisition component to perform a data acquisition task; a movement control instruction for controlling the movement of the robotic arm; and a treatment instruction for controlling the dexterous hand to perform traditional Chinese medicine massage therapy.

10. The rheumatoid arthritis and comorbidity diagnosis and treatment system based on dexterous hand and large model according to claim 1, characterized in that, The integrated diagnosis, treatment, and rehabilitation model adopts an architecture that fuses the ViT-Unet model with a medical LLM large language model.

Images