Intelligent accompanying treatment nursing system based on AI

Abstract

The invention discloses an AI-based intelligent accompanying treatment and nursing system, and particularly relates to the technical field of medical assistance, and the system comprises the following modules: a virtual AR interaction module, an AI dialogue module, a relative and friend-medical care communication module, a treatment synchronization engine, a dynamic state analysis module, a virtual image generation module, and a multi-modal intervention module. According to the AI-based intelligent accompanying treatment nursing system, the treatment scene AR interface is generated through the virtual AR interaction module, information such as the focus position and the medicine use condition is displayed in a visual three-dimensional model mode, and compared with traditional complex examination lists and reports, relatives can more easily understand the current physical condition of a patient, and the treatment nursing efficiency is improved. And non-medical relatives can better participate in the treatment process of the patient, the attention and support strength of the relatives to the illness state of the patient are enhanced, a good treatment atmosphere can be created, the positive influence on the psychological state of the patient is generated, and the improvement of the treatment effect is indirectly promoted.

Description

Technical Field

[0001] The present invention relates to the field of medical assistance technologies, and particularly to an AI-based intelligent companion treatment and nursing system. Background Art

[0002] With the acceleration of the medical informatization process, remote consultation systems have become important auxiliary tools in long-term treatment scenarios such as tumor chemotherapy. However, existing technologies mainly focus on video communication between doctors and patients and sharing of electronic medical records, lacking effective solutions for the needs of continuous psychological intervention, relative collaborative nursing, and regional service adaptation during the patient's treatment cycle. Clinical studies have shown that the psychological stress index of chemotherapy patients due to treatment side effects is up to 3.2 times that of conventional patients, while traditional remote nursing systems can only achieve an average of 0.7 effective doctor-patient interactions per week, making it difficult to meet the urgent needs of patients for emotional support and dynamic nursing. Especially in Asian regions where the proportion of dialect speakers exceeds 38%, existing systems lack multi-modal interaction and intelligent decision-making capabilities, resulting in 15% - 22% of elderly patients having medication errors or treatment interruptions due to language barriers.

[0003] There are three core defects in existing remote consultation platforms:

[0004] Firstly, the frequency and quality of doctor-patient interaction are insufficient. Mainstream systems rely on video consultations at fixed times (average daily interaction duration ≤ 8 minutes) and cannot perceive changes in the patient's physiological state in real time. Research has confirmed that 68% of the psychological crisis events of chemotherapy patients occur during non-consultation periods, and traditional platforms lack continuous monitoring of physical sign data (such as peripheral circulation impedance, skin temperature fluctuations) and AI emotion analysis capabilities, resulting in a 53% proportion of cases where the crisis warning is delayed by more than 6 hours.

[0005] Secondly, the relative participation mechanism is weak. In current solutions, relatives can only receive information via text messages or one-way videos and cannot dynamically participate in treatment decisions. Data shows that the intervention of relatives can increase the medication compliance of patients by 41%, but traditional systems do not establish a relative feature modeling and personalized reminder mechanism, resulting in a reminder opening rate of less than 30% on the relative side.

[0006] Thirdly, the regional service ability is lacking. Existing systems mostly interact based on standard Mandarin, and the recognition accuracy of dialects such as Minnan and Cantonese is generally lower than 75%, and virtual image interaction with dialect lip synchronization cannot be achieved. Surveys have shown that the treatment misunderstanding rate of patients in dialect areas due to communication barriers is as high as 19%, significantly affecting the quality of nursing. Summary of the Invention

[0007] The main object of the present invention is to provide an AI-based intelligent companion treatment and nursing system, which can effectively solve the problems of low doctor-patient interaction frequency, weak relative participation, and dialect communication barriers existing in traditional remote consultation platforms.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] An AI-based intelligent companion therapy and nursing system includes the following modules:

[0010] A virtual AR interaction module that generates an AR interface for the treatment scenario and dynamically displays the chemotherapy plan, adverse reaction simulation, and precautions;

[0011] An AI dialogue module that supports multilingual interaction, analyzes the patient's emotional state, and generates a mental health risk score;

[0012] A relative-medical staff communication module that forms a treatment group through a mobile terminal and synchronously pushes the treatment progress and patient status;

[0013] A treatment synchronization engine that associates the hospital information system with the AR / APP side and triggers personalized education content;

[0014] A dynamic status analysis module that integrates the conversation content and vital sign data to generate a comprehensive nursing report;

[0015] A virtual image generation module that generates a digital portrait based on a relative's image and performs medication reminders and companion conversations;

[0016] A multimodal intervention module that pushes differentiated instructions to patients, medical staff, and relatives according to the risk score.

[0017] Preferably, the relative-medical staff communication module sets hierarchical permissions: relatives can only view the education content, and medical staff can edit the chemotherapy plan and mark high-risk periods.

[0018] Preferably, the AI dialogue module has a built-in dialect recognition sub-module, which optimizes dialect recognition using a transfer learning framework and updates the audio-lexicon correspondence data in real time according to the background data model.

[0019] Preferably, the dynamic status analysis module is connected to a vital sign bracelet to collect the body surface temperature and peripheral circulation impedance value in real time and monitor the patient's physical data.

[0020] Preferably, the virtual image generation module integrates a GAN network to convert a 2D photo into a 3D model with 40 expression control points, supporting dialect lip synchronization.

[0021] Preferably, the multimodal intervention module includes a medication compliance algorithm that activates the AR medicine box navigation and relative-side reminder when the patient fails to move on time.

[0022] Preferably, the virtual image generation module integrates an AI face-pinching function, which extracts facial feature points and voiceprint data by analyzing the image data uploaded by relatives, and generates a virtual image with emotional expression ability; when the patient's medication compliance is lower than a threshold, the virtual image automatically calls pre-recorded video, audio or uses a pre-stored voice library to automatically generate interactive reminders in the voice of relatives.

[0023] Preferably, the treatment synchronization engine is connected to the hospital PACS system, and a three-dimensional model of the imaging examination results of the day is superimposed on the AR interface.

[0024] Preferably, the nursing system also includes an offline instruction relay unit. When the network is disconnected, the AI ​​plays the pre-recorded video and audio messages of the relatives at the set time.

[0025] Preferably, the relatives and friends-medical care communication module also includes an intelligent monitoring robot within the group, which monitors the patient's physical data based on the data detected by the vital signs bracelet, and gives basic responses to questions raised by the patient's family through the background data model, and makes relevant reminders to medical staff and relatives within the group based on the patient's physical signs, expressions, demeanor and other data.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The present invention generates a treatment scene AR interface through a virtual AR interactive module, which displays information such as lesion location and medication status in the form of an intuitive three-dimensional model. Compared with traditional complex examination sheets and reports, relatives can more easily understand the patient's current physical condition, so that non-medical relatives can also better participate in the patient's treatment process, enhance family members' attention and support for the patient's condition, help create a good treatment atmosphere, have a positive impact on the patient's psychological state, and indirectly promote the improvement of treatment effects.

[0028] 2. The present invention allows relatives to view treatment progress visualization charts and educational videos. Medical staff can also edit chemotherapy plans and mark high-risk periods according to the patient's condition. At the same time, the intelligent monitoring robot in the group can provide basic responses to family members' questions and promptly remind relevant personnel of changes in the patient's physical data. This information sharing and interactive mechanism enables family members to no longer be bystanders in the treatment process, but to cooperate more actively with medical staff to provide support to patients and improve patient treatment compliance and overall treatment effectiveness.

[0029] 3. The AI ​​dialogue module in the present invention combines the dual-model architecture of BERT and BiLSTM and the dialect recognition submodule, which can accurately analyze the patient's emotional state and generate a mental health risk score. It also supports multi-language interaction, which helps medical staff to understand the patient's mental health status in a timely manner, provide patients with more targeted psychological care and intervention measures, and avoid emotional problems affecting the treatment effect. At the same time, the automatic speech recognition engine can process voice input in different language environments to ensure smooth information communication and reduce misunderstandings and delays caused by language barriers.

[0030] 4. The virtual image generation module provided in the present invention generates digital portraits based on relative images, performs medication reminders and companion conversations, and integrates AI face pinching function and dialect lip synchronization technology. When the patient's medication compliance is lower than the threshold, it can urge the patient to take medication through pre-recorded video, audio or generated customized voice reminders. This method is more friendly and targeted, and can effectively improve the patient's medication compliance, ensure that the patient takes the medicine on time and in the right amount, thereby ensuring the continuity and stability of the treatment effect.

[0031] 5. The multimodal intervention module provided in the present invention pushes differentiated instructions to patients, medical staff and relatives according to the risk score. The drug compliance algorithm integrates the mobile terminal GPS positioning and accelerometer data. When it is detected that the patient is not active on time or there is an abnormal situation, the AR medicine box navigation and relative reminder are activated. At the same time, the medical staff generates an early warning log. This multi-dimensional intervention measure can timely discover and correct problems in the patient's medication process, avoid treatment risks caused by missed doses, wrong doses, etc., and ensure the safety and effectiveness of the patient's medication.

[0032] 6. The present invention also provides an offline instruction relay unit for network disconnection situations. Even in network disconnection situations, the locally deployed lightweight model can be used to play relatives' pre-recorded messages and encrypt and store data, ensuring that patients can be effectively reminded of medication and provide nursing interventions in various network environments, further improving the safety of patients' medication and the reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0034] Figure 2 This is a schematic diagram of the core functional modules of the present invention;

[0035] Figure 3 It is a schematic diagram of a virtual image generation module of the present invention;

[0036] Figure 4 A schematic diagram of a family and friends-medical care communication module of the present invention;

[0037] Figure 5Schematic diagram of the multi-modal intervention module of the present invention. Detailed implementation manners

[0038] To make the technical means, creative features, achieved purposes and effects of the present invention easy to understand, the present invention will be further described below in conjunction with specific implementation manners.

[0039] As Figures 1 to 5 shown, an AI-based intelligent companion treatment and care system includes the following modules:

[0040] A virtual AR interaction module that generates an AR interface for the treatment scenario and dynamically displays the chemotherapy plan, adverse reaction simulation, and precautions;

[0041] An AR scene rendering framework is constructed through the Unity3D engine, a three-dimensional model is constructed, and the lesion location and medication situation are displayed on the three-dimensional model in real time. Relatives can understand the patient's current physical condition through the three-dimensional model. Compared with complex inspection sheets and reports, the three-dimensional model is more intuitive and easier to understand. For relatives who are not medical professionals, it is relatively simple to understand the condition and treatment situation.

[0042] An AI dialogue module that supports multi-language interaction, analyzes the patient's emotional state, and generates a mental health risk score;

[0043] Furthermore, the AI dialogue module is built-in with a dialect recognition sub-module, which optimizes dialect recognition using a transfer learning framework and updates the audio-lexicon correspondence data in real time according to the background data model.

[0044] A dual-model architecture combining BERT and BiLSTM is adopted for the deployment of the deep learning model to improve the accuracy and efficiency of natural language processing tasks

[0045] Among them, BERT is a pre-trained language representation model based on Transformer, which can capture bidirectional context information of text and is suitable for a variety of natural language processing tasks;

[0046] BiLSTM is a bidirectional recurrent neural network that can consider both previous and subsequent information and is suitable for processing sequence data.

[0047] The dual-model architecture combining BERT and BiLSTM can utilize BERT's powerful context understanding ability and further process sequence information through BiLSTM, thereby improving the accuracy and efficiency of natural language processing tasks.

[0048] On this basis, an automatic speech recognition (ASR) engine supporting multiple languages is also connected to the front end, enabling the system to process and understand speech inputs from different language environments;

[0049] The back-end emotion analysis model combines facial micro-expression recognition (using OpenFace to extract 17 facial action units) with speech prosody features (fundamental frequency, speaking rate, and pause intervals) to output a 0-10 level mental health risk index.

[0050] The dialect recognition submodule adopts a transfer learning framework to load the regional dialect acoustic model parameters based on the standard Mandarin model.

[0051] The family and friends-medical care communication module forms a treatment group through mobile terminals and simultaneously pushes treatment progress and patient status;

[0052] Furthermore, the relatives-medical care communication module sets up hierarchical permissions: relatives can only view the education content, and medical care can edit chemotherapy plans and mark high-risk periods.

[0053] Furthermore, the family and friends-medical care communication module also includes an intelligent monitoring robot within the group, which monitors the patient's physical data based on the data detected by the vital signs bracelet, and gives basic answers to questions raised by the patient's family through the background data model, and makes relevant reminders to medical staff and relatives within the group based on the patient's physical signs, expressions, demeanor and other data.

[0054] Relative accounts can only view treatment progress visualization charts and educational videos;

[0055] The medical account is equipped with a chemotherapy regimen editor and a risk period marking tool.

[0056] A conversational robot based on the Rasa framework is deployed in the treatment group to parse the vital signs data stream in real time (using the Kafka message queue). When out-of-range data is detected, such as a body temperature >38.5℃ for 30 minutes, the responsible nurse is automatically notified and an emergency treatment plan is pushed.

[0057] Synchronously, the conversational robot can route relatives’ questions to matching answers through the intent recognition engine.

[0058] Treatment synchronization engine, linking hospital information system and AR / APP end to trigger personalized education content;

[0059] The treatment synchronization engine is connected to the hospital's PACS system, and a three-dimensional model of the day's imaging results is superimposed on the AR interface.

[0060] The hospital HIS / PACS system is connected via the HL7 protocol. When the chemotherapy regimen is updated, the rule engine is triggered to match the patient feature library and extract personalized education content from the knowledge graph. The imaging data uses the MarchingCubes algorithm to reconstruct the 3D model, and multi-plane interactive browsing is achieved on the AR interface.

[0061] The dynamic status analysis module integrates the conversation content and vital sign data to generate a comprehensive nursing report;

[0062] Furthermore, the dynamic status analysis module is connected to a vital sign bracelet to collect the body surface temperature and peripheral circulation impedance value in real time, and monitor the patient's physical data.

[0063] Cooperate with the medical vital sign detection bracelet to collect the patient's vital sign data in real time, combine the results of the conversation emotion analysis, and generate a comprehensive nursing report through the random forest algorithm.

[0064] The vital sign detection bracelet is used to monitor in real time a number of data including but not limited to the patient's body temperature fluctuation, peripheral impedance fluctuation, activity compliance rate, myelosuppression risk index, neurotoxicity progression, probability of electrolyte imbalance, and digestive tract reaction, and summarize the detected data into a chart and store it on the local and background terminals.

[0065] The virtual image generation module generates a digital portrait based on the relative's image and performs medication reminders and companion conversations;

[0066] Furthermore, the virtual image generation module integrates a GAN network to convert a 2D photo into a 3D model with 40 expression control points, supporting dialect lip synchronization;

[0067] Even further, the virtual image generation module integrates an AI face pinching function, extracts facial feature points and voiceprint data by analyzing the video data uploaded by the relative, and generates a virtual image with emotional expression ability; when the patient's medication compliance is lower than the threshold, the virtual image automatically calls a pre-recorded video, audio or uses a pre-stored voice library to automatically generate an interactive reminder with the relative's voice line.

[0068] The 2D to 3D conversion is realized by using the StyleGAN2-ADA network, and a digital portrait containing 40 BlendShape expression control points is constructed.

[0069] The dialect lip synchronization uses the phoneme-visual position mapping technology and combines the LSTM network to predict the lip sequence.

[0070] When the analyzed medication compliance is lower than 70%, call a pre-recorded relative video or generate a customized voice reminder through Tacotron2 voice cloning, and after multiple reminders of not taking the medicine, the responsible doctor, nurse and relative are all notified in the group to facilitate the medical staff and relatives to urge the patient to take the medicine.

[0071] The multi-modal intervention module pushes differentiated instructions to the patient, medical staff and relatives according to the risk score.

[0072] The multi-modal intervention module includes a medication compliance algorithm. When the patient does not perform activities on time, it starts the AR medicine box navigation and relative end reminder.

[0073] The medication compliance algorithm integrates mobile GPS positioning (to determine the activity range and the distance from the medicine box) and accelerometer data (to calculate the amount of exercise). When an anomaly is detected:

[0074] ① An AR interface pops up the navigation path to the medicine box (using ARKit plane detection);

[0075] ② The relative's end receives an instant message notification;

[0076] ③ The medical staff's end generates an orange warning log.

[0077] The nursing system also includes an offline instruction relay unit. In the case of network disconnection, the AI plays pre-recorded video and audio messages of relatives at a set time.

[0078] A lightweight TensorFlowLite model is deployed locally. When the network is disconnected, an emergency protocol is started: cache content is played at a preset time, local interaction data is synchronously encrypted and stored, and after the network is restored, it is synchronized to the cloud through a differential update protocol.

[0079] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. An AI-based intelligent companion therapy and nursing system, characterized in that, Includes the following modules: Virtual AR interaction module generates an AR interface for treatment scenarios, dynamically displays chemotherapy plans, adverse reaction simulations, and precautions; AI dialogue module, which supports multi-language interaction and analyzes the patient's emotional state to generate a mental health risk score; The family and friends-medical care communication module forms a treatment group through mobile terminals and simultaneously pushes treatment progress and patient status; Treatment synchronization engine, linking hospital information system and AR / APP end to trigger personalized education content; Dynamic status analysis module, integrating conversation content and vital sign data to generate comprehensive nursing reports; Avatar generation module, which generates digital portraits based on images of relatives, performs medication reminders and companionship conversations; The multimodal intervention module pushes differentiated instructions to patients, medical staff and relatives based on risk scores.

2. The intelligent companion therapy and nursing system based on AI according to claim 1, characterized in that: The relatives and friends-medical care communication module sets hierarchical permissions: relatives can only view the education content, and medical care can edit chemotherapy plans and mark high-risk periods.

3. An AI-based intelligent companion therapy and care system according to claim 1, characterized in that: The AI ​​dialogue module has a built-in dialect recognition sub-module, uses a transfer learning framework to optimize dialect recognition, and updates audio-vocabulary correspondence data in real time based on the background data model.

4. An AI-based intelligent companion therapy and nursing system according to claim 1, wherein: The dynamic state analysis module is connected to the vital sign bracelet to collect body surface temperature and peripheral circulation impedance values ​​in real time to monitor the patient's physical data.

5. An AI-based intelligent companion therapy and nursing system according to claim 1, characterized in that: The avatar generation module integrates a GAN network to convert 2D photos into 3D models with 40 expression control points and supports dialect lip synchronization.

6. The intelligent companion therapy and nursing system based on AI according to claim 1, wherein: The multimodal intervention module includes a medication compliance algorithm, which activates AR medicine box navigation and relative-side reminders when the patient fails to move on time.

7. An AI-based intelligent companion therapy and nursing system according to claim 1, characterized in that: The virtual image generation module integrates an AI face-pinching function, which extracts facial feature points and voiceprint data by analyzing the image data uploaded by relatives, and generates a virtual image with emotional expression capabilities; when the patient's medication compliance is lower than a threshold, the virtual image automatically calls pre-recorded video, audio, or uses a pre-stored voice library to automatically generate interactive reminders in the voice of relatives.

8. An AI-based intelligent companion therapy and nursing system according to claim 1, characterized in that: The treatment synchronization engine is connected to the hospital PACS system and a three-dimensional model of the imaging results of the day is superimposed on the AR interface.

9. The intelligent companion therapy and nursing system based on AI according to claim 1, characterized in that: The nursing system also includes an offline command relay unit. In the event of a network outage, AI plays pre-recorded videos and audio messages from relatives at a set time.

10. An AI-based intelligent companion therapy and nursing system according to claim 1, characterized in that: The relatives and friends-medical care communication module also includes an intelligent monitoring robot within the group, which monitors the patient's physical data based on the data detected by the vital signs bracelet, and provides basic responses to questions raised by the patient's family through the background data model, and makes relevant reminders to medical staff and relatives within the group based on the patient's physical signs, expressions, demeanor and other data.

Images