Orthopedic post-operation multi-modal data processing method, system and terminal
By collecting and processing multimodal data from orthopedic postoperative patients, and utilizing reinforcement learning models and medical knowledge graphs, personalized health education knowledge and monitoring programs are generated. This addresses the problem of insufficient data association in hospitals and improves rehabilitation outcomes and patient satisfaction.
Patent Information
- Application Number
- CN202511043397.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-28
- Publication Date
- 2025-11-11
AI Technical Summary
Existing technologies lack correlation between various data in hospitals, making it impossible to effectively process multimodal interactive data after orthopedic surgery and difficult to select the most suitable rehabilitation movements.
By collecting multimodal data from patients after orthopedic surgery, processing it using a reinforcement learning model, constructing a medical knowledge graph, matching target core entities, and generating target health education knowledge or monitoring plans, the effective processing of multimodal data is achieved.
It improved the accuracy of rehabilitation exercise selection, enhanced the personalization of health education knowledge and the efficiency of monitoring program implementation, and improved patient recovery speed and satisfaction.
Smart Images
Figure CN120932813A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing, and in particular to a method, system, terminal, and computer-readable storage medium for multimodal data processing after orthopedic surgery. Background Technology
[0002] Currently, during the postoperative rehabilitation period of orthopedic patients, monitoring and education are necessary. For example, patients need to be monitored according to a pre-set monitoring plan, including monitoring various physical indicators or complications. At the same time, the dissemination of postoperative health education knowledge is also particularly important to help patients understand health education knowledge, which can improve patients' awareness of their own disease and speed up recovery.
[0003] However, there is currently controversy regarding the priority of health education and monitoring programs for postoperative orthopedic patients. The priority of these two approaches varies depending on the patient's data. For example, health education should be prioritized for patients at high risk of falls or pressure injuries. Therefore, determining the priority requires consideration of multiple patient data points. However, the lack of correlation between various hospital data makes it difficult to effectively process multimodal interactive data and determine the most suitable rehabilitation exercises.
[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention
[0005] The main objective of this invention is to provide a method, system, and terminal for processing multimodal data after orthopedic surgery, aiming to solve the problem in the prior art that there is a lack of correlation between various data in hospitals, making it impossible to effectively process multimodal interactive data after orthopedic surgery, and thus making it difficult to select the most suitable rehabilitation movements.
[0006] To achieve the above objectives, the present invention provides a method for multimodal data processing after orthopedic surgery, the method comprising the following steps:
[0007] First and second multimodal data of patients after orthopedic surgery are collected from various interactive systems in the hospital. The first multimodal data is input into a pre-trained reinforcement learning model for processing and outputs a target action, which is to push health education knowledge or execute a monitoring plan.
[0008] A medical knowledge graph is constructed, and the second multimodal data is input into the medical knowledge graph for matching to obtain the target core entity. The target action content of the target core entity is obtained based on the target action, and the target action content is used as the target health education knowledge or the target monitoring plan.
[0009] If the target action content is a target monitoring plan, then the target monitoring plan is sent to the medical staff to notify them to monitor the patient regularly according to the target monitoring plan;
[0010] If the target action content is target health education knowledge, then the target health education knowledge is transformed and integrated to generate multimodal health education knowledge, and the multimodal health education is sent to the patient for learning.
[0011] Optionally, the method for processing multimodal data after orthopedic surgery, wherein collecting the patient's first and second multimodal data after orthopedic surgery from various interactive systems in the hospital specifically includes:
[0012] The hospital's medical information system, electronic medical record system, and nursing information system are connected via HL7 and FHIR protocols.
[0013] The first and second multimodal data of the patient after orthopedic surgery are collected from the information system, the electronic medical record system and the nursing information system through the multimodal interaction engine.
[0014] The first multimodal data includes pain scores, surgical types, historical behaviors, and real-time feedback, while the second multimodal data includes patient medical orders, test results, and nursing records.
[0015] Optionally, in the multimodal data processing method after orthopedic surgery, the reinforcement learning model includes: a Critic branch, an Actor branch, a dynamic reward module, and a knowledge distillation module.
[0016] Optionally, the method for processing multimodal data after orthopedic surgery, wherein inputting the first multimodal data into a pre-trained reinforcement learning model for processing and outputting the target action, specifically includes:
[0017] The pain score, the type of surgery, and the historical behavior are input into the OnlineQ network of the Critic branch and aligned with the Target Q network across modal modes using Transformer to obtain the aligned features. The aligned features are then fused across modal modes to obtain the first feature.
[0018] The first feature is regularized using Clipped Double Q-Learning. The regularized first feature is then input into the Online policy network and Target policy network of the Actor branch for processing, and the action and the corresponding score are output.
[0019] The real-time feedback is input into the dynamic reward module. Based on the real-time feedback, the dynamic reward module updates the reward weight in real time according to the ICD-10 and ICF standards, and adjusts the score corresponding to the action according to the updated reward weight to obtain the target score of each action.
[0020] The action with the highest target score is selected as the target action, the target action is output, and online distillation is performed through the knowledge distillation module to continuously transfer knowledge.
[0021] Optionally, the multimodal data processing method after orthopedic surgery, wherein the construction of the medical knowledge graph specifically includes:
[0022] Using core entities, action types, and action content as data structures, a medical knowledge graph is constructed based on these data structures.
[0023] The core entities include diseases, surgeries, and complications; the action types include pushing health education knowledge or implementing monitoring programs; and the action content represents the specific implementation of the action type.
[0024] Extract structured and unstructured data from historical patients, and map the structured and unstructured data into a knowledge graph;
[0025] The structured data includes test results and surgical records, while the unstructured data includes nursing records and inpatient medical records.
[0026] Optionally, the multimodal data processing method after orthopedic surgery, wherein the step of inputting the second multimodal data into the medical knowledge graph for matching to obtain the target core entity, and obtaining the target action content of the target core entity based on the target action, specifically includes:
[0027] Semantic analysis is performed on the second multimodal data to obtain the disease type, surgical type, and complication type contained in the second multimodal data;
[0028] The disease type, surgery type, and complication type are compared with the core entities of the medical knowledge graph to obtain the target core entity;
[0029] Based on the target action, the action type is determined, and the target action content of the target core entity under the action type is obtained.
[0030] Optionally, the multimodal data processing method after orthopedic surgery, wherein the step of converting and integrating the target health education knowledge to generate multimodal health education knowledge, and sending the multimodal health education to the patient for learning, specifically includes:
[0031] The target health education knowledge is input into the LLM model for transformation and integration to generate personalized health education knowledge text;
[0032] The personalized health education knowledge text is converted into voice health education knowledge and 3D animation health education knowledge, which are then sent to the patient for learning.
[0033] Furthermore, to achieve the above objectives, the present invention also provides a multimodal data processing system for post-orthopedic surgery, wherein the multimodal data processing system for post-orthopedic surgery includes:
[0034] The reinforcement learning module is used to collect first and second multimodal data of patients after orthopedic surgery from various interactive systems in the hospital. The first multimodal data is input into a pre-trained reinforcement learning model for processing, and the target action is output. The target action is to push health education knowledge or execute a monitoring plan.
[0035] The target action content acquisition module is used to construct a medical knowledge graph, input the second multimodal data into the medical knowledge graph for matching, obtain the target core entity, obtain the target action content of the target core entity based on the target action, and use the target action content as target health education knowledge or target monitoring scheme.
[0036] The monitoring plan push module is used to send the target monitoring plan to medical staff if the target action content is a target monitoring plan, so as to notify the medical staff to conduct regular monitoring of the patient according to the target monitoring plan;
[0037] The health education knowledge push module is used to convert and integrate the target health education knowledge if the target action content is target health education knowledge, generate multimodal health education knowledge, and send the multimodal health education to the patient for learning.
[0038] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a post-orthopedic multimodal data processing program stored in the memory and executable on the processor, wherein when the post-orthopedic multimodal data processing program is executed by the processor, it implements the steps of the post-orthopedic multimodal data processing method as described above.
[0039] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal data processing program for post-orthopedic surgery, which, when executed by a processor, implements the steps of the multimodal data processing method for post-orthopedic surgery as described above.
[0040] In this invention, first and second multimodal data are collected from patients after orthopedic surgery. The first multimodal data is input into a reinforcement learning model for processing, outputting a target action, which may be pushing health education knowledge or executing a monitoring plan. A medical knowledge graph is constructed, and the second multimodal data is input into the medical knowledge graph for matching to obtain target core entities. Based on the target action, the target action content of the target core entity is obtained, and the target action content is used as target health education knowledge or target monitoring plan. If the target action content is a target monitoring plan, the target monitoring plan is sent to medical staff for timed monitoring; if the target action content is target health education knowledge, the target health education knowledge is converted and integrated before being sent to the patient for learning. This invention can effectively process multimodal interactive data. Attached Figure Description
[0041] Figure 1 This is a flowchart of a preferred embodiment of the multimodal data processing method for orthopedic surgery of the present invention;
[0042] Figure 2 This is a structural diagram of a preferred embodiment of the multimodal data processing system for postoperative orthopedic surgery of the present invention;
[0043] Figure 3 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation
[0044] This application provides a method, system, and terminal for multimodal data processing after orthopedic surgery. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.
[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0046] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.
[0047] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to user analysis data, user stored data, user displayed data, etc.) and signals involved in this application are all information, data and signals authorized by the user or fully authorized by all parties; and the collection, use and processing of relevant information, data and signals comply with the laws, regulations and standards of relevant countries and regions.
[0048] The preferred embodiment of the multimodal data processing method after orthopedic surgery described in this invention, such as... Figure 1 As shown, the multimodal data processing method after orthopedic surgery includes the following steps:
[0049] Step S10: Collect the first and second multimodal data of the patient after orthopedic surgery from various interactive systems in the hospital. Input the first multimodal data into a pre-trained reinforcement learning model for processing and output the target action, which is to push health education knowledge or execute a monitoring plan.
[0050] The collection of postoperative first and second multimodal data from various interactive systems within the hospital specifically includes:
[0051] The hospital's medical information system, electronic medical record system, and nursing information system are connected via HL7 and FHIR protocols.
[0052] The first and second multimodal data of the patient after orthopedic surgery are collected from the information system, the electronic medical record system and the nursing information system through the multimodal interaction engine.
[0053] The first multimodal data includes pain scores, surgical types, historical behaviors, and real-time feedback, while the second multimodal data includes patient medical orders, test results, and nursing records.
[0054] Understandably, HL7 (Health Level Seven) and FHIR (Fast Healthcare Interoperability Resources) are important protocols in the healthcare field for achieving data exchange and interoperability. HL7 is an international standard designed to regulate the format and rules for electronic data exchange between healthcare information systems, facilitating data transmission and sharing between different healthcare institutions and systems. FHIR is a next-generation healthcare interoperability standard introduced by HL7, aiming to achieve efficient exchange and sharing of medical data through standardized resource formats.
[0055] In this embodiment, the hospital's medical information system (HIS), electronic medical record system (EMR), and nursing information system are connected via HL7 and the FHIR protocol. A multimodal interaction engine is used to collect first and second multimodal data of the patient after orthopedic surgery from the information system, the electronic medical record system, and the nursing information system. The first multimodal data includes pain score, surgical type, historical behavior, and real-time feedback, while the second multimodal data includes patient medical orders, test results, and nursing records.
[0056] The multimodal interaction engine integrates automatic speech recognition (ASR), an emotion-computing camera, and a touchscreen, supporting dialect voice interaction (such as Cantonese and Sichuanese) and gesture operation. After delivering health education content to the patient, the multimodal interaction engine collects real-time feedback information from the patient through various sensors and interaction modalities, and dynamically adjusts and interacts based on this information.
[0057] Furthermore, the reinforcement learning model includes: a Critic branch, an Actor branch, a dynamic reward module, and a knowledge distillation module. The knowledge distillation module employs KL divergence constraints for policy consistency online distillation to continuously transfer knowledge, and uses a teacher model and a student model for training on historical data and real-time interaction. The specific structure of the reinforcement learning model is shown in Table 1 below.
[0058] Table 1. Technical Features of Model Components
[0059]
[0060]
[0061]
[0062] The step of inputting the first multimodal data into a pre-trained reinforcement learning model for processing and outputting the target action specifically includes:
[0063] Step S201: Input the pain score, the surgical type, and the historical behavior into the Online Q network of the Critic branch and perform cross-modal Transformer alignment with the Target Q network to obtain the alignment features. Then, perform cross-modal feature fusion on the alignment features to obtain the first feature.
[0064] In this embodiment, the Critic branch includes an Online policy network and a Target policy network. The inputs to the Critic branch are the pain score, the surgery type, and the historical behavior input. The residual convolutional modules of the Online policy network and the Target policy network process the pain score, the surgery type, and the historical behavior input for unstructured data processing. The processed data is then aligned with multi-source features using a cross-modal Transformer. Finally, the aligned multi-source features are fused across modalities to obtain the first feature.
[0065] Step S202: Regularize the first feature using Clipped Double Q-Learning, input the regularized first feature into the Online policy network and Target policy network of the Actor branch for processing, and output the action and the corresponding score.
[0066] It is understood that this embodiment first uses Clipped Double Q-Learning to process the first feature to reduce overestimation. The Actor branch includes an Online policy network and a Target policy network. The Online policy network and the Target policy network output continuous actions (such as health education knowledge push strategy), and dynamically load pre-trained strategies according to the surgical type (such as TKA / THA), and perform real-time policy distillation to synchronize experience through a teacher-student model.
[0067] Step S203: Input the real-time feedback into the dynamic reward module. The dynamic reward module updates the reward weight in real time based on the ICD-10 and ICF standards according to the real-time feedback, and adjusts the score corresponding to the action according to the updated reward weight to obtain the target score of each action.
[0068] In this embodiment, the rule base embeds the ICD-10 and ICF standards, updates the reward weights in real time (such as increasing the reward for pushing health education knowledge when the risk of infection increases), and combines static rules with dynamic data-driven design to design hybrid rewards and calculate risk perception rewards: the reward value is adjusted based on XGBoost output.
[0069] Step S204: Select the action with the highest target score as the target action, output the target action, and perform online distillation through the knowledge distillation module to continuously transfer knowledge.
[0070] As an example, in the process of processing a reinforcement learning model, the input is: patient status: {pain score: 4 / 10, thrombosis risk: high, interaction history: [ankle pump exercise health education knowledge has been pushed 3 times]}, and the environmental feedback is: reward signal = -0.2 (because the health education knowledge task for today was not completed).
[0071] Data processing: First, the policy network (Actor) generates an action distribution based on the input: {Push health education knowledge: 0.7, Wait for monitoring: 0.3}. Then, the value network (Critic) evaluates the current state value V: V = 0.85. Finally, the output is: Selected action: Push health education knowledge.
[0072] As can be seen, the reinforcement learning model of this invention automatically generates data through environmental interaction without the need for labeling. Compared with traditional supervised learning, which requires labeled datasets, this invention reduces data dependence and lowers learning costs. Moreover, this invention dynamically adjusts by combining multimodal data and changing the push priority, exhibiting good user adaptability. In contrast, traditional supervised learning models are static models and cannot cope with changes in the patient's state. Finally, traditional supervised learning models are black-box models, making it difficult to trace the decision-making logic. In contrast, this invention uses a visual attention mechanism, which can trace the decision-making path and has strong interpretability. The advantages of the reinforcement learning model of this invention compared with traditional supervised learning models are shown in Table 2.
[0073] Table 2. Comparison between traditional supervised learning and the reinforcement learning model of this invention
[0074]
[0075] Step S20: Construct a medical knowledge graph, input the second multimodal data into the medical knowledge graph for matching, obtain the target core entity, obtain the target action content of the target core entity based on the target action, and use the target action content as target health education knowledge or target monitoring scheme.
[0076] The construction of the medical knowledge graph specifically includes: using core entities, action types, and action content as data structures, and constructing a medical knowledge graph based on the data structures.
[0077] The core entities include diseases, surgeries, and complications; the action types include pushing health education knowledge or implementing monitoring programs; and the action content represents the specific implementation of the action type.
[0078] Extract structured and unstructured data from historical patients, and map the structured and unstructured data into a knowledge graph;
[0079] The structured data includes test results and surgical records, while the unstructured data includes nursing records and inpatient medical records.
[0080] It is understandable that the relationship between medical knowledge graphs and patient medical orders, test results, and nursing records reflects dynamic data-driven knowledge reasoning and application.
[0081] Taking orthopedics as an example, the medical knowledge graph includes the following structure:
[0082] 1. Core Entity:
[0083] Diseases: fractures, osteoarthritis, scoliosis, and cervical and lumbar degenerative changes.
[0084] Surgery: Fracture reduction and internal fixation, total hip and knee replacement (THA), scoliosis correction, and percutaneous endoscopic discectomy.
[0085] Complications: Venous thromboembolism (VTE), incision infection, internal fixation failure.
[0086] 2. Action types: including pushing health education knowledge and implementing monitoring programs.
[0087] 3. Action Content: This includes the specific implementation of pushing health education knowledge or executing monitoring plans for the corresponding core entity. For example, the specific implementation of pushing health education knowledge for fractures is as follows:
[0088] What is a fracture? A fracture is a break in the integrity or continuity of a bone, often caused by direct violence (such as impact), indirect violence (such as sprain), or long-term strain (stress fracture). Common symptoms include pain, swelling, deformity, and limited mobility. Some patients may experience crepitus or abnormal movement.
[0089] The core principles of treatment are: Reduction: Restoring the fracture ends to their normal position through manual manipulation or surgery. Immobilization: Maintaining the reduced position using plaster casts, splints, plates, etc., to promote healing. Functional exercises: Early intervention to prevent joint stiffness and muscle atrophy.
[0090] Further, the second multimodal data is input into the medical knowledge graph for matching to obtain the target core entity. Based on the target action, the target action content of the target core entity is obtained, specifically including:
[0091] Semantic analysis is performed on the second multimodal data to obtain the disease type, surgical type, and complication type contained in the second multimodal data;
[0092] The disease type, surgery type, and complication type are compared with the core entities of the medical knowledge graph to obtain the target core entity;
[0093] Based on the target action, the action type is determined, and the target action content of the target core entity under the action type is obtained.
[0094] Step S30: If the target action content is a target monitoring plan, then the target monitoring plan is sent to the medical staff to notify them to monitor the patient regularly according to the target monitoring plan.
[0095] Specifically, if the target action is a target monitoring plan, the monitoring plan will be sent through multiple channels such as the HIS system and nursing communication APP. Then, the plan will be received, confirmed, and assigned to specific individuals: monitoring tasks will be automatically assigned to specific nurses according to the shift schedule (e.g., "Nurse Zhang is responsible for blood glucose monitoring of patient in bed 3 at 10:00"). Two-way confirmation: After receiving the plan, nurses must mark it as "received" in the system. If there are any questions about the plan (e.g., the monitoring frequency is too high), a "medical order questioning" process can be initiated through the system, and the doctor will respond within a fixed time.
[0096] Furthermore, the implementation and recording of scheduled monitoring should be standardized, and data collection and entry should be carried out, including:
[0097] Smart reminders: The mobile app pushes a vibration reminder 15 minutes in advance of the monitoring time (e.g., "Blood pressure needs to be measured for patient in bed 5 at 10:00"). If the reminder is not executed within the time limit, the reminder will be upgraded (a pop-up warning at the head nurse level).
[0098] Time zone synchronization: When patients are transferred across time zones (such as international medical centers), the monitoring time needs to be automatically switched to avoid missed detections due to time differences.
[0099] Data collection and entry: Data is automatically uploaded to the HIS via Bluetooth from devices such as blood pressure monitors and monitors, reducing human input errors. For example, pulse oximeter readings of 95% are automatically entered into the system. When nurses enter data, if the value reaches a warning threshold (such as a body temperature of 39.2℃), the system automatically highlights it in red and triggers the "Report Now" button, which allows for one-click access to a doctor.
[0100] Furthermore, the handover process and the early warning and handling of abnormal situations include:
[0101] Electronic handover log: The list of monitoring tasks to be performed is displayed in real time on the large screen at the nurse station (such as "Wound dressing for bed 8 needs to be changed from 14:00 to 16:00"). The handover is done by scanning the code during shift change, and unfinished tasks are automatically transferred to the next shift.
[0102] Early warning and handling of abnormal situations: A tiered early warning mechanism is used to report abnormal data from users and obtain the corresponding early warning level based on the abnormal data. Medical staff are then assigned to handle the situation according to the early warning level.
[0103] Step S40: If the target action content is target health education knowledge, then the target health education knowledge is transformed and integrated to generate multimodal health education knowledge, and the multimodal health education is sent to the patient for learning.
[0104] Specifically, if the target action content is target health education knowledge, then the target health education knowledge is input into the LLM model for transformation and integration to generate personalized health education knowledge text.
[0105] The LLM model is used for multimodal transformation. For example, the text generated by the LLM (such as "avoid massaging your calves") is processed by speech synthesis (dialect) + 3D animation generation, and the output is a multimodal output of speech + text and image.
[0106] Furthermore, the personalized health education knowledge text is converted to generate voice health education knowledge and 3D animation health education knowledge, which are then sent to the patient for learning. The LLM processing flow is shown in Table 3.
[0107] Table 3. LLM Processing Flowchart
[0108]
[0109] In this embodiment, the personalized health education knowledge text is converted into voice and 3D animation data (such as a demonstration of contraindicated movements after hip replacement surgery) to achieve multimodal output and improve the user's learning experience.
[0110] Furthermore, such as Figure 2 As shown, based on the above-mentioned multimodal data processing method for orthopedic surgery, the present invention also provides a multimodal data processing system for orthopedic surgery, wherein the multimodal data processing system for orthopedic surgery includes:
[0111] The reinforcement learning module 51 is used to collect first and second multimodal data of patients after orthopedic surgery from various interactive systems in the hospital, input the first multimodal data into a pre-trained reinforcement learning model for processing, and output a target action, which is to push health education knowledge or execute a monitoring plan.
[0112] The target action content acquisition module 52 is used to construct a medical knowledge graph, input the second multimodal data into the medical knowledge graph for matching, obtain the target core entity, obtain the target action content of the target core entity according to the target action, and use the target action content as target health education knowledge or target monitoring scheme.
[0113] The monitoring plan push module 53 is used to send the target monitoring plan to medical staff if the target action content is a target monitoring plan, so as to notify the medical staff to monitor the patient regularly according to the target monitoring plan;
[0114] The health education knowledge push module 54 is used to convert and integrate the target health education knowledge if the target action content is target health education knowledge, generate multimodal health education knowledge, and send the multimodal health education to the patient for learning.
[0115] Furthermore, such as Figure 3 As shown, based on the above-mentioned multimodal data processing method and system after orthopedic surgery, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 3 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.
[0116] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a multimodal data processing program 40 for post-orthopedic surgery, which can be executed by the processor 10 to implement the multimodal data processing method for post-orthopedic surgery described in this application.
[0117] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the multimodal data processing method after orthopedic surgery.
[0118] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visualized patient interface. The components of the terminal communicate with each other via a system bus.
[0119] In one embodiment, when the processor 10 executes the multimodal data processing program 40 for post-orthopedic surgery data in the memory 20, the following steps are performed:
[0120] First and second multimodal data of patients after orthopedic surgery are collected from various interactive systems in the hospital. The first multimodal data is input into a pre-trained reinforcement learning model for processing and outputs a target action, which is to push health education knowledge or execute a monitoring plan.
[0121] A medical knowledge graph is constructed, and the second multimodal data is input into the medical knowledge graph for matching to obtain the target core entity. The target action content of the target core entity is obtained based on the target action, and the target action content is used as the target health education knowledge or the target monitoring plan.
[0122] If the target action content is a target monitoring plan, then the target monitoring plan is sent to the medical staff to notify them to monitor the patient regularly according to the target monitoring plan;
[0123] If the target action content is target health education knowledge, then the target health education knowledge is transformed and integrated to generate multimodal health education knowledge, and the multimodal health education is sent to the patient for learning.
[0124] The collection of postoperative multimodal and second-mode data from various interactive systems within the hospital specifically includes:
[0125] The hospital's medical information system, electronic medical record system, and nursing information system are connected via HL7 and FHIR protocols.
[0126] The first and second multimodal data of the patient after orthopedic surgery are collected from the information system, the electronic medical record system and the nursing information system through the multimodal interaction engine.
[0127] The first multimodal data includes pain scores, surgical types, historical behaviors, and real-time feedback, while the second multimodal data includes patient medical orders, test results, and nursing records.
[0128] The reinforcement learning model includes: a Critic branch, an Actor branch, a dynamic reward module, and a knowledge distillation module.
[0129] Specifically, the step of inputting the first multimodal data into a pre-trained reinforcement learning model for processing and outputting the target action includes:
[0130] The pain score, the type of surgery, and the historical behavior are input into the OnlineQ network of the Critic branch and aligned with the Target Q network across modal modes using Transformer to obtain the aligned features. The aligned features are then fused across modal modes to obtain the first feature.
[0131] The first feature is regularized using Clipped Double Q-Learning. The regularized first feature is then input into the Online policy network and Target policy network of the Actor branch for processing, and the action and the corresponding score are output.
[0132] The real-time feedback is input into the dynamic reward module. Based on the real-time feedback, the dynamic reward module updates the reward weight in real time according to the ICD-10 and ICF standards, and adjusts the score corresponding to the action according to the updated reward weight to obtain the target score of each action.
[0133] The action with the highest target score is selected as the target action, the target action is output, and online distillation is performed through the knowledge distillation module to continuously transfer knowledge.
[0134] The construction of the medical knowledge graph specifically includes:
[0135] Using core entities, action types, and action content as data structures, a medical knowledge graph is constructed based on these data structures.
[0136] The core entities include diseases, surgeries, and complications; the action types include pushing health education knowledge or implementing monitoring programs; and the action content represents the specific implementation of the action type.
[0137] Extract structured and unstructured data from historical patients, and map the structured and unstructured data into a knowledge graph;
[0138] The structured data includes test results and surgical records, while the unstructured data includes nursing records and inpatient medical records.
[0139] Specifically, the step of inputting the second multimodal data into the medical knowledge graph for matching to obtain the target core entity, and obtaining the target action content of the target core entity based on the target action, includes:
[0140] Semantic analysis is performed on the second multimodal data to obtain the disease type, surgical type, and complication type contained in the second multimodal data;
[0141] The disease type, surgery type, and complication type are compared with the core entities of the medical knowledge graph to obtain the target core entity;
[0142] Based on the target action, the action type is determined, and the target action content of the target core entity under the action type is obtained.
[0143] Specifically, the process of transforming and integrating the target health education knowledge to generate multimodal health education knowledge, and then sending the multimodal health education to patients for learning, includes:
[0144] The target health education knowledge is input into the LLM model for transformation and integration to generate personalized health education knowledge text;
[0145] The personalized health education knowledge text is converted into voice health education knowledge and 3D animation health education knowledge, which are then sent to the patient for learning.
[0146] The present invention also provides a computer-readable storage medium storing a multimodal data processing program for post-orthopedic surgery, wherein the multimodal data processing program for post-orthopedic surgery, when executed by a processor, implements the steps of the multimodal data processing method for post-orthopedic surgery as described above.
[0147] In summary, this invention proposes a method, system, and terminal for processing multimodal data after orthopedic surgery. The method includes: collecting first and second multimodal data from various interactive systems in the hospital; inputting the first multimodal data into a pre-trained reinforcement learning model for processing, outputting a target action, which is either pushing health education knowledge or executing a monitoring plan; constructing a medical knowledge graph, inputting the second multimodal data into the medical knowledge graph for matching, obtaining target core entities, obtaining the target action content of the target core entities based on the target action, and using the target action content as target health education knowledge or target monitoring plan; if the target action content is a target monitoring plan, sending the target monitoring plan to medical staff to notify them to monitor the patient regularly according to the target monitoring plan; if the target action content is target health education knowledge, converting and integrating the target health education knowledge to generate multimodal health education knowledge, and sending the multimodal health education to the patient for learning. This invention can effectively process multimodal interactive data, improving the work efficiency of medical staff and patient satisfaction.
[0148] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.
[0149] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0150] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.
Claims
1. A method for processing multimodal data after orthopedic surgery, characterized in that, The aforementioned multimodal data processing method after orthopedic surgery includes: First and second multimodal data of patients after orthopedic surgery are collected from various interactive systems in the hospital. The first multimodal data is input into a pre-trained reinforcement learning model for processing and outputs a target action, which is to push health education knowledge or execute a monitoring plan. A medical knowledge graph is constructed, and the second multimodal data is input into the medical knowledge graph for matching to obtain the target core entity. The target action content of the target core entity is obtained based on the target action, and the target action content is used as the target health education knowledge or the target monitoring plan. If the target action content is a target monitoring plan, then the target monitoring plan is sent to the medical staff to notify them to monitor the patient regularly according to the target monitoring plan. If the target action content is target health education knowledge, then the target health education knowledge is transformed and integrated to generate multimodal health education knowledge, and the multimodal health education is sent to the patient for learning.
2. The method for processing multimodal data after orthopedic surgery according to claim 1, characterized in that, The collection of first and second multimodal data from various interactive systems within the hospital after orthopedic surgery specifically includes: The hospital's medical information system, electronic medical record system, and nursing information system are connected via HL7 and FHIR protocols. The first and second multimodal data of the patient after orthopedic surgery are collected from the information system, the electronic medical record system and the nursing information system through the multimodal interaction engine. The first multimodal data includes pain scores, surgical types, historical behaviors, and real-time feedback, while the second multimodal data includes patient medical orders, test results, and nursing records.
3. The method for processing multimodal data after orthopedic surgery according to claim 2, characterized in that, The reinforcement learning model includes: a Critic branch, an Actor branch, a dynamic reward module, and a knowledge distillation module.
4. The method for processing multimodal data after orthopedic surgery according to claim 3, characterized in that, The step of inputting the first multimodal data into a pre-trained reinforcement learning model for processing and outputting the target action specifically includes: The pain score, the type of surgery, and the historical behavior are input into the Online Q network of the Critic branch and aligned with the Target Q network across modal Transformer to obtain the aligned features. The aligned features are then fused across modal features to obtain the first feature. The first feature is regularized using Clipped Double Q-Learning. The regularized first feature is then input into the Online policy network and Target policy network of the Actor branch for processing, and the action and the corresponding score are output. The real-time feedback is input into the dynamic reward module. Based on the real-time feedback, the dynamic reward module updates the reward weight in real time according to the ICD-10 and ICF standards, and adjusts the score corresponding to the action according to the updated reward weight to obtain the target score of each action. The action with the highest target score is selected as the target action, the target action is output, and online distillation is performed through the knowledge distillation module to continuously transfer knowledge.
5. The method for processing multimodal data after orthopedic surgery according to claim 1, characterized in that, The construction of the medical knowledge graph specifically includes: Using core entities, action types, and action content as data structures, a medical knowledge graph is constructed based on these data structures. The core entities include diseases, surgeries, and complications; the action types include pushing health education knowledge or implementing monitoring programs; and the action content represents the specific implementation of the action type. Extract structured and unstructured data from historical patients, and map the structured and unstructured data into a knowledge graph; The structured data includes test results and surgical records, while the unstructured data includes nursing records and inpatient medical records.
6. The method for processing multimodal data after orthopedic surgery according to claim 5, characterized in that, The step of inputting the second multimodal data into the medical knowledge graph for matching to obtain the target core entity, and obtaining the target action content of the target core entity based on the target action, specifically includes: Semantic analysis is performed on the second multimodal data to obtain the disease type, surgical type, and complication type contained in the second multimodal data; The disease type, surgery type, and complication type are compared with the core entities of the medical knowledge graph to obtain the target core entity; Based on the target action, the action type is determined, and the target action content of the target core entity under the action type is obtained.
7. The method for processing multimodal data after orthopedic surgery according to claim 1, characterized in that, The process of transforming and integrating the target health education knowledge to generate multimodal health education knowledge, and then sending the multimodal health education to patients for learning, specifically includes: The target health education knowledge is input into the LLM model for transformation and integration to generate personalized health education knowledge text; The personalized health education knowledge text is converted into voice health education knowledge and 3D animation health education knowledge, which are then sent to the patient for learning.
8. A multimodal data processing system for post-orthopedic surgery, characterized in that, The multimodal data processing system for post-orthopedic surgery includes: The reinforcement learning module is used to collect first and second multimodal data of patients after orthopedic surgery from various interactive systems in the hospital. The first multimodal data is input into a pre-trained reinforcement learning model for processing, and the target action is output. The target action is to push health education knowledge or execute a monitoring plan. The target action content acquisition module is used to construct a medical knowledge graph, input the second multimodal data into the medical knowledge graph for matching, obtain the target core entity, obtain the target action content of the target core entity based on the target action, and use the target action content as target health education knowledge or target monitoring scheme. The monitoring plan push module is used to send the target monitoring plan to medical staff if the target action content is a target monitoring plan, so as to notify the medical staff to conduct regular monitoring of the patient according to the target monitoring plan; The health education knowledge push module is used to convert and integrate the target health education knowledge if the target action content is target health education knowledge, generate multimodal health education knowledge, and send the multimodal health education to the patient for learning.
9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a multimodal data processing program for post-orthopedic surgery stored in the memory and executable on the processor. When executed by the processor, the multimodal data processing program for post-orthopedic surgery implements the steps of the multimodal data processing method for post-orthopedic surgery as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multimodal data processing program for post-orthopedic surgery, which, when executed by a processor, implements the steps of the multimodal data processing method for post-orthopedic surgery as described in any one of claims 1-7.