Digital doctor multi-mode interaction system and training method

Abstract

The invention discloses a digital doctor multi-mode interaction system and a training method, and belongs to the technical field of medical learning, the digital doctor multi-mode interaction system comprises an interaction system main body and a terminal device which is set based on the interaction system, and the interaction system main body comprises a medical knowledge database module used for storing medical science popularization knowledge; the intelligent calculation module is used for providing system core calculation; the AR model simulation module is used for controlling establishment of a simulation training model; and the interactive perception module is used for realizing simulation operation. Based on a computer unit, an AR simulation device, an operation training device, an environment control device, an environment perception device, a medical knowledge database module, a smart computing module, an AR model simulation module and an interactive perception module, a digital doctor model and an illness state simulation training model are established to perform simulation learning on a user doctor. Simulation training of the illness state operation of the user is met, the effect of synchronous inquiry training and operation training is achieved, and experience accumulation of the doctor of the user can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of medical learning technology, and in particular to a digital doctor multimodal interaction system and training method. Background Art

[0002] With the popularization of digital doctors, it is more convenient for patients to seek medical treatment, and the channels for obtaining medical knowledge are more diversified. The use of digital doctors also reduces medical pressure;

[0003] In the medical system, doctors who are in internship or have no surgical experience often need to follow chief physicians and experienced physicians for a long time to learn before they can perform surgery and see patients themselves. Although learning can increase knowledge, it is impossible to perform actual operations and surgical training, which has little effect on the accumulation of experience. In addition, chief physicians also judge the training structure of interns based on long-term experience and cannot intuitively derive the learning structure of interns, which is not conducive to the training of physicians. In addition, some doctors are unable to perform simulated operations when encountering new conditions and surgeries, making it difficult to ensure the success rate of the surgery. Based on this, a digital doctor multimodal interaction system and training method are proposed. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the existing learning structure for interns cannot be intuitively derived, which is not conducive to the training of doctors, and some doctors are unable to perform simulated operations when encountering new diseases and surgeries, making it difficult to ensure the success rate of the operations. A digital doctor multimodal interaction system and training method are proposed to solve the problem that the existing learning structure for interns cannot be intuitively derived, which is not conducive to the training of doctors, and some doctors are unable to perform simulated operations when encountering new diseases and surgeries, making it difficult to ensure the success rate of the operations.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A digital doctor multimodal interactive system and training method, including an interactive system main body and a terminal device established based on the interactive system;

[0007] The interactive system main body includes:

[0008] A medical knowledge database module for storing medical popular science knowledge, wherein the medical knowledge database module includes real medical material data of patients based on the Internet of Things and corresponding symptom solutions;

[0009] An intelligent computing module for providing core computing of the system, the intelligent computing module comprising an artificial intelligence computing unit, a cloud computing unit, a computer vision computing unit and a neural rendering unit;

[0010] An AR model simulation module for controlling the establishment of a simulation training model, wherein the AR model simulation module includes a digital doctor model and a disease simulation training model;

[0011] An interactive perception module for realizing simulated surgery, the interactive perception module is based on an environment perception device terminal and an environment control device terminal;

[0012] The terminal device includes a computer unit, an AR simulation device, a surgical training device, an environmental control device and an environmental perception device.

[0013] Preferably, the patient medical real material data in the medical knowledge database module includes clinical consultation data and surgical condition data;

[0014] The corresponding symptom solution includes patient condition diagnosis data and surgical plan simulation data;

[0015] The clinical consultation data corresponds to the patient's condition diagnosis data, surgical condition data and surgical plan simulation data one by one to form a simulation training control data group;

[0016] Preferably, the intelligent computing module realizes information interaction processing between data and equipment, and is used to call interfaces of semantic understanding ability, emotion recognition ability, visual ability and cognitive ability to generate response data of each modality;

[0017] The computer vision computing unit in the intelligent computing module recognizes the movements and expressions of the simulated training personnel based on the human body recognition camera device in the environmental control device, and processes the expression and body movement information of the training personnel;

[0018] The neural rendering unit fuses the voice data, facial expression response data, and body movement data as the multimodal output data, and outputs it through the constructed digital doctor image.

[0019] Preferably, the digital doctor model in the AR model simulation module is based on the voice and image data of the teaching doctor, uses image learning and sound cloning to complete the replication of the image and voiceprint, uses computer graphics technology to create a virtual character that can interact with the real world, and projects the real AR simulation model in the AR simulation device based on AR technology.

[0020] Preferably, the disease simulation training model in the AR model simulation module is constructed based on the surgical disease data, a lesion simulation model is established based on 3D Slicer software technology, and a staged lesion model corresponding to the different surgical stages during the operation is established.

[0021] Preferably, the steps and conditions for triggering the staged lesion model are established based on the surgical training equipment, so that when the training steps and conditions meet the trigger conditions during the training process, the corresponding staged lesion model is triggered to continuously update the model, thereby achieving the effect of real continuous AR training. The human body recognition camera device in the environmental control equipment is used to identify the movements and expressions of the simulated training personnel, and the expression and limb movement information of the trainees is processed. In the disease consultation simulation stage, the artificial intelligence computing unit is used to judge the degree of fit between the training user's consultation data and the disease diagnosis data, and the user's consultation training is rated and scored. In the surgical model training simulation process, the moving state of the surgical training equipment and the accuracy and time of the training equipment usage steps are used for scoring, and the actual rating of the user doctor can be intuitively obtained through simulation.

[0022] A digital doctor multimodal interactive training method comprises the following steps:

[0023] S1. Disease mode selection: In the authorized local area network connected to the data doctor multimodal interactive system, internal users are identified through computers in the local area network. After the identification, the user enters the system to make an appointment for the disease training mode;

[0024] S2. Matching AR training model: According to the disease training mode selected by the user, the corresponding digital doctor model and disease simulation training model are matched in the system through the intelligent computing module, and the digital doctor model and the disease simulation training model generate corresponding identification codes and send them to the user. When the user uses the terminal device, he inputs the identification code to retrieve the corresponding digital doctor model and disease simulation training model;

[0025] S3. Clinical condition simulation: AR simulation equipment is used to simulate and generate an AR simulation physician, who describes the condition and simultaneously generates a patient simulation model through the AR simulation equipment;

[0026] S4. Digital doctor question-and-answer training: After the user has finished judging the condition, the AR simulated physician will ask the user questions based on the patient's clinical consultation data, obtain the consultation data of the simulated training user, compare the consultation data of the simulated training user with the condition diagnosis data, and make a judgment through the artificial intelligence computing unit;

[0027] S5. Simulation work preparation: Based on the surgical plan simulation data, a lesion simulation model is formed by using AR technology, and a surgical training device equipped with an environment sensing device terminal and an environment control device terminal is selected corresponding to the surgical plan simulation data;

[0028] S6. Model training simulation: During simulated surgical training, the movement status of the environment sensing device terminal and the environment control device terminal connected to the surgical training device and the steps of using the training device are used to trigger the staged lesion model for continuous model update and simulated surgical training;

[0029] S7. Data feedback scoring: During the medical consultation simulation stage, the artificial intelligence computing unit is used to judge the fit between the training user's medical consultation data and the medical diagnosis data, and the user's medical consultation training is rated and scored;

[0030] During the surgical model training simulation, scoring is performed based on the movement status of the surgical training equipment and the accuracy and time of the training equipment usage steps.

[0031] Preferably, the identity recognition includes but is not limited to face recognition, professional qualification recognition, identity information recognition or authorization.

[0032] Preferably, the identification code includes but is not limited to a biometric identification code, a digital feature identification code and a graphic feature identification code.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 1. The present invention is based on a computer unit, AR simulation equipment, surgical training equipment, environmental control equipment and environmental perception equipment, a medical knowledge database module, an intelligent computing module, an AR model simulation module, and an interactive perception module to establish a digital doctor model and a disease simulation training model to simulate learning for user doctors, meet the user's disease simulation training, achieve the effect of simultaneous consultation training and surgical training, and significantly improve the experience accumulation of user doctors.

[0035] 2. The present invention recognizes the movements and expressions of the personnel in the simulated training based on the human body recognition camera device in the environmental control equipment, and processes the expressions and body movement information of the trainees. In the stage of disease consultation simulation, the artificial intelligence computing unit is used to judge the degree of fit between the training user's consultation data and the disease diagnosis data, and the user's consultation training is rated and scored. In the process of surgical model training simulation, the moving state of the surgical training equipment and the accuracy and time of the training equipment usage steps are used for scoring, and the actual rating of the user doctor can be intuitively obtained through simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a module of a digital doctor multimodal interaction system proposed by the present invention;

[0037] Figure 2 A flowchart of a multimodal interactive training method for digital doctors proposed in the present invention. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] Example, see Figure 1-2 , a digital doctor multimodal interactive system and training method, including an interactive system main body and a terminal device established based on the interactive system;

[0040] The main body of the interactive system includes a medical knowledge database module for storing medical popular science knowledge, and the medical knowledge database module includes real medical material data of patients based on the Internet of Things and corresponding symptom solutions;

[0041] The patient medical real material data in the medical knowledge database module includes clinical consultation data and surgical condition data;

[0042] The corresponding symptom solution includes patient diagnosis data and surgical plan simulation data; the clinical consultation data corresponds to the patient diagnosis data, surgical condition data and surgical plan simulation data one by one to form a simulation training control data group;

[0043] The intelligent computing module is used to provide the core computing of the system. The intelligent computing module includes an artificial intelligence computing unit, a cloud computing unit, a computer vision computing unit and a neural rendering unit. The intelligent computing module is set based on the terminal device. The system is a prior art and will not be described in detail here.

[0044] The intelligent computing module realizes information interaction processing between data and devices, and is used to call interfaces of semantic understanding ability, emotion recognition ability, visual ability and cognitive ability to generate response data of each modality;

[0045] The computer vision computing unit in the intelligent computing module recognizes the movements and expressions of the simulated training personnel based on the human body recognition camera device in the environmental control device, and processes the expression and body movement information of the training personnel;

[0046] The neural rendering unit fuses voice data, facial expression response data, and body movement data as multimodal output data, and outputs it through the constructed digital doctor image.

[0047] An AR model simulation module for controlling the establishment of a simulation training model, wherein the AR model simulation module includes a digital doctor model and a disease simulation training model;

[0048] The digital doctor model in the AR model simulation module is based on the voice and image data of the teaching doctor. It uses image learning and sound cloning to complete the replication of the image and voiceprint. It uses computer graphics technology to create a virtual character that can interact with the real world. Based on AR technology, it projects in the AR simulation device to establish a real AR simulation model. The virtual digital person can simulate human behavior, enhance visual effects and enhance user experience.

[0049] The disease simulation training model in the AR model simulation module is constructed based on the surgical condition data, and the lesion simulation model is established based on 3D Slicer software technology. In addition, the staged lesion models corresponding to the different surgical stages during the operation are established.

[0050] An interactive perception module for realizing simulated surgery, the interactive perception module is based on an environment perception device terminal and an environment control device terminal;

[0051] The terminal device includes a computer unit, an AR simulation device, a surgical training device, an environmental control device and an environmental sensing device.

[0052] Based on the surgical training equipment, the steps and conditions for triggering the staged lesion model are established, so that when the training steps and conditions meet the triggering conditions during the training process, the corresponding staged lesion model is triggered for continuous model updating, thereby achieving the effect of real continuous AR training.

[0053] A training method proposed by a digital doctor multimodal interaction system includes: in an authorized local area network connected to the digital doctor multimodal interaction system, an internal user performs identity recognition through a computer in the local area network, and the user who passes the identity recognition enters the system to perform an appointment for a disease training mode, and the identity recognition includes but is not limited to face recognition, professional qualification recognition, identity information recognition or authorization;

[0054] According to the disease training mode selected by the user, the corresponding digital doctor model and disease simulation training model are matched in the system through the intelligent computing module, and the digital doctor model and the disease simulation training model generate corresponding identification codes and send them to the user. When the user uses the terminal device, he inputs the identification code to retrieve the corresponding digital doctor model and disease simulation training model. The identification code includes but is not limited to a biometric identification code, a digital feature identification code and a graphic feature identification code;

[0055] The AR simulation device is used to simulate and generate an AR simulation physician, who describes the condition and simultaneously generates a patient simulation model through the AR simulation device;

[0056] After the user has finished judging the condition, the AR simulated physician will inquire the user based on the patient's clinical consultation data, obtain the consultation data of the simulated training user, compare the consultation data of the simulated training user with the condition diagnosis data, and make a judgment through the artificial intelligence computing unit;

[0057] According to the surgical plan simulation data, a lesion simulation model is formed by using AR technology, and a surgical training device equipped with an environment sensing device terminal and an environment control device terminal is selected corresponding to the surgical plan simulation data, wherein the environment sensing device terminal includes but is not limited to various sensors, cameras, infrared devices, etc.;

[0058] During the simulated surgery training, the movement status of the environment sensing device terminal and the environment control device terminal connected to the surgical training device and the steps of using the training device are used to trigger the staged lesion model for continuous model update and simulated surgery training;

[0059] In the disease consultation simulation stage, the artificial intelligence computing unit is used to judge the fit between the training user's consultation data and the disease diagnosis data, and the user's consultation training is rated and scored;

[0060] During the surgical model training simulation, scoring is performed based on the movement status of the surgical training equipment and the accuracy and time of the training equipment usage steps.

[0061] The present invention is based on a computer unit, an AR simulation device, a surgical training device, an environmental control device, an environmental perception device, a medical knowledge database module, an intelligent computing module, an AR model simulation module, and an interactive perception module to establish a digital doctor model and a disease simulation training model to simulate learning for the user doctor, meet the user's disease surgery simulation training, and achieve the effect of simultaneous consultation training and surgical training, which can significantly improve the user doctor's experience accumulation; based on the human body recognition camera device in the environmental control device, the movements and expressions of the simulated training personnel are recognized, and the expression and body movement information of the trainee are processed. In the disease consultation simulation stage, the artificial intelligence computing unit is used to judge the degree of fit between the training user's consultation data and the disease diagnosis data, and the user's consultation training is rated and scored. In the surgical model training simulation process, the moving state of the surgical training equipment and the accuracy and time of the training equipment usage steps are used to score, and the actual rating of the user doctor can be intuitively obtained through simulation.

[0062] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A digital doctor multimodal interaction system, characterized in that: It includes an interactive system main body and a terminal device established based on the interactive system; The interactive system main body includes: A medical knowledge database module for storing medical popular science knowledge, wherein the medical knowledge database module includes real medical material data of patients based on the Internet of Things and corresponding symptom solutions; An intelligent computing module for providing core computing of the system, the intelligent computing module comprising an artificial intelligence computing unit, a cloud computing unit, a computer vision computing unit and a neural rendering unit; An AR model simulation module for controlling the establishment of a simulation training model, wherein the AR model simulation module includes a digital doctor model and a disease simulation training model; An interactive perception module for realizing simulated surgery, the interactive perception module is based on an environment perception device terminal and an environment control device terminal; The terminal device includes a computer unit, an AR simulation device, a surgical training device, an environmental control device and an environmental perception device.

2. A digital doctor multimodal interaction system according to claim 1, characterized in that: The patient medical real material data in the medical knowledge database module includes clinical consultation data and surgical condition data; The corresponding symptom solution includes patient condition diagnosis data and surgical plan simulation data; The clinical consultation data corresponds one-to-one with the patient's condition diagnosis data, surgical condition data and surgical plan simulation data to form a simulation training control data group.

3. A digital doctor multimodal interaction system according to claim 1, characterized in that: The intelligent computing module realizes information interaction processing between data and devices, and is used to call interfaces of semantic understanding ability, emotion recognition ability, visual ability and cognitive ability to generate response data of each modality; The computer vision computing unit in the intelligent computing module recognizes the movements and expressions of the personnel in the simulation training based on the human body recognition camera device in the environmental control device, and processes the expression and body movement information of the training personnel; The neural rendering unit fuses the voice data, facial expression response data, and body movement data as the multimodal output data, and outputs it through the constructed digital doctor image.

4. A digital doctor multimodal interaction system according to claim 1, characterized in that: The digital doctor model in the AR model simulation module is based on the voice and image data of the teaching doctor, uses image learning and sound cloning to complete the replication of the image and voiceprint, uses computer graphics technology to create a virtual character that can interact with the real world, and projects in the AR simulation device based on AR technology to establish a real AR simulation model.

5. A digital doctor multimodal interaction system according to claim 2, characterized in that: The disease simulation training model in the AR model simulation module is constructed based on the surgical disease data, a lesion simulation model is established based on 3D Slicer software technology, and a staged lesion model corresponding to the different surgical stages during the operation is established.

6. A digital doctor multimodal interaction system according to claim 5, characterized in that: Based on the surgical training equipment, the steps and conditions for triggering the staged lesion model are established, so that when the training steps and conditions meet the triggering conditions during the training process, the corresponding staged lesion model is triggered for continuous model updating, thereby achieving the effect of real continuous AR training.

7. The training method proposed by any one of claims 1 to 6 for the digital doctor multimodal interaction system is characterized in that: The following steps are involved: S1. Disease mode selection: In the authorized local area network connected to the data doctor multimodal interactive system, internal users are identified through computers in the local area network. After the identification, the user enters the system to make an appointment for the disease training mode; S2. Matching AR training model: According to the disease training mode selected by the user, the corresponding digital doctor model and disease simulation training model are matched in the system through the intelligent computing module, and the digital doctor model and the disease simulation training model generate corresponding identification codes and send them to the user. When the user uses the terminal device, he inputs the identification code to retrieve the corresponding digital doctor model and disease simulation training model; S3. Clinical condition simulation: AR simulation equipment is used to simulate and generate an AR simulation physician, who describes the condition and simultaneously generates a patient simulation model through the AR simulation equipment; S4. Digital doctor question-and-answer training: After the user has finished judging the condition, the AR simulated physician will ask the user questions based on the patient's clinical consultation data, obtain the consultation data of the simulated training user, compare the consultation data of the simulated training user with the condition diagnosis data, and make a judgment through the artificial intelligence computing unit; S5. Simulation work preparation: Based on the surgical plan simulation data, a lesion simulation model is formed by using AR technology, and a surgical training device equipped with an environment sensing device terminal and an environment control device terminal is selected corresponding to the surgical plan simulation data; S6. Model training simulation: During simulated surgical training, the movement status of the environment sensing device terminal and the environment control device terminal connected to the surgical training device and the steps of using the training device are used to trigger the staged lesion model for continuous model update and simulated surgical training; S7. Data feedback scoring: During the medical consultation simulation stage, the artificial intelligence computing unit is used to judge the fit between the training user's medical consultation data and the medical diagnosis data, and the user's medical consultation training is rated and scored; During the surgical model training simulation, scoring is performed based on the movement status of the surgical training equipment and the accuracy and time of the training equipment usage steps.

8. A digital doctor multimodal interactive training method according to claim 1, characterized in that: The identity recognition includes but is not limited to face recognition, professional qualification recognition, identity information recognition or authorization.

9. A digital doctor multimodal interactive training method according to claim 1, characterized in that: The identification code includes but is not limited to a biometric identification code, a digital feature identification code, and a graphic feature identification code.

Images