Intelligent online auxiliary diagnosis and treatment system and method based on virtual digital human
By building medical knowledge graphs and deep learning technologies, the virtual digital human system achieves comprehensive consultation and precise department positioning, solving the problems of doctors' difficulties in obtaining medical history and patients' choice of departments, providing efficient and accurate diagnosis and treatment services, breaking medical barriers, Provide full-time companionship for elderly patients.
Patent Information
- Application Number
- CN202411271341.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-06
AI Technical Summary
In the prior art, it is difficult for doctors to obtain the patient's medical history in a comprehensive manner, resulting in a high misdiagnosis rate and it is difficult for patients to accurately select departments, which increases the difficulty of visiting the doctor.
An intelligent online auxiliary diagnosis and treatment system based on virtual digital people can realize comprehensive consultation, department precise positioning and visual medical record output by building a medical knowledge graph, and combine deep learning and large language models to provide a question-and-answer diagnosis and treatment service. And provide diagnostic, treatment and rehabilitation consultation after the diagnosis and treatment.
Comprehensive auxiliary diagnosis and treatment services have been realized to ensure that patients can register accurately, query medical records and reports at any time, understand disease-related knowledge, break down medical barriers, and provide customized services for aging patients, improving diagnosis and treatment efficiency and accuracy.
Smart Images

Figure CN119092102A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of online auxiliary diagnosis and treatment systems, and specifically to an intelligent online auxiliary diagnosis and treatment system and method based on virtual digital humans. Background Art
[0002] With the progress of society and the reform of the medical industry, the construction of "Internet + Medical Health" has been continuously promoted. Smart hospitals have gradually spread from urban areas to towns and villages, covering more areas. The expectations of ordinary people for medical services have also become higher and higher, and the contradiction between supply and demand between doctors and patients has become increasingly prominent. With the continuous advancement of artificial intelligence technology and the optimization of algorithms, the interactive ability and diagnostic accuracy of virtual digital people will be significantly improved. Through deep learning and data mining, virtual digital people can continuously learn the latest medical knowledge and cases and improve their service level in the medical industry. In addition, with the development of technologies such as 5G and the Internet of Things, the virtual digital human consultation project will achieve more efficient data transmission and telemedicine services, break geographical restrictions, and enable high-quality medical resources to be more widely distributed. With the intensification of social aging and the increase in patients with chronic diseases, people's demand for convenient and efficient medical services is growing.
[0003] At present, people still have problems such as incomplete consultation and high medical barriers during the medical treatment process, which will lead to doctors being unable to fully obtain patients' medical history information, increase the misdiagnosis rate, and even cause medical disputes. At the same time, the proposal and practice of refined department classification emphasizes that diseases need to be treated by specialists, but ignores the problem that patients have difficulty choosing the right department due to their lack of medical knowledge.
[0004] Therefore, we proposed an intelligent online assisted diagnosis and treatment system and method based on virtual digital humans to solve the above problems. Summary of the invention
[0005] The purpose of the present invention is to provide an intelligent online auxiliary diagnosis and treatment system and method based on a virtual digital human to solve the problems in the current market raised by the above background technology.
[0006] To achieve the above-mentioned purpose, the present invention provides the following technical solution: an intelligent online auxiliary diagnosis and treatment system based on virtual digital human, comprising:
[0007] A database, wherein the information sources of the database include cutting-edge literature, expert opinions, clinical treatment guidelines and diagnostics, and a medical knowledge graph constructed based on the information sources;
[0008] An online consultation module is used to identify medical questions raised by users, match, filter and analyze them in the medical knowledge graph, form appropriate answer statements, and conduct human-computer interaction through virtual humans in the form of audio and video;
[0009] Department location module, which is used to collect and organize all relevant medical information provided by the user in the online consultation module, and conduct in-depth analysis to identify potential pathogenic factors related to the user's health status and the user's symptom information, match this information with the data in the medical knowledge graph, select a department according to the user's condition and the probability of disease occurrence, and output it to the user;
[0010] A medical record output module, which is used to simplify and visualize the final data of the online consultation module and the department location module through a visualization tool, output a complete medical record in text and image versions, and output the visualized medical record to the user;
[0011] The prognosis consultation module is used to store all information including diagnosis, treatment plan, auxiliary examination report details, and provide medical consultation related to diagnosis, treatment, and rehabilitation.
[0012] Preferably, it also includes a virtual image construction module for constructing a virtual human image that simulates the facial features and voice intonation of a designated real person, and the virtual image construction module includes a portrait input unit and an audio input unit.
[0013] Preferably, the online consultation module includes an input unit, an intention recognition unit, a language synthesis unit, and a virtual human output unit.
[0014] Preferably, the input unit is used for the user to input medical consultation information, and the medical consultation information includes the user's basic information, chief complaint, current medical history, past medical history, personal history, and family history.
[0015] Preferably, the intention recognition unit classifies the medical inquiry information input by the user based on the large language model LLM and the retrieval enhanced generation RAG method to determine the user's medical inquiry intention.
[0016] Preferably, the language synthesis unit performs analysis based on the user's medical inquiry intention to form a suitable answer sentence.
[0017] Preferably, the virtual human output unit converts text into speech and outputs it in the form of a virtual digital human to achieve human-computer interaction with the user.
[0018] Preferably, the prognosis consultation module includes a storage unit and a question-and-answer unit, the storage unit is used to store all information including diagnosis, treatment plan, and auxiliary examination report details, and the question-and-answer unit is used to provide medical consultation related to diagnosis, treatment, and rehabilitation through a virtual digital human.
[0019] The diagnosis and treatment method of the intelligent online auxiliary diagnosis and treatment system based on virtual digital human as described above includes the following steps:
[0020] S1: First, the user communicates with the virtual digital human through the online consultation module and provides relevant medical information. The virtual digital human conducts a one-on-one question-and-answer inquiry with the user based on the database to complete the preliminary consultation;
[0021] S2: The department location module conducts in-depth analysis of all relevant medical information provided in the online consultation module to identify potential pathogenic factors related to the user's health status and the user's symptom information, matches this information with the data in the medical knowledge graph, selects a department according to the user's condition and the probability of disease occurrence, and outputs it to the user;
[0022] S3: The medical record output module simplifies and visualizes the final data of the online consultation module and the department location module through visualization tools, outputs the complete medical record in text and image versions, and outputs the visualized medical record to the user;
[0023] S4: After the diagnosis and treatment, the prognosis consultation module stores all the information including the diagnosis, treatment plan, and auxiliary examination report details, and continues to provide medical consultation related to diagnosis, treatment, and rehabilitation through virtual digital people.
[0024] Compared with the prior art, the present invention has the following beneficial effects:
[0025] The present invention realizes comprehensive medical consultation, precise department positioning, and visual output of medical records by constructing a medical knowledge graph and training virtual digital humans, ensuring that patients can register accurately, check medical records at any time, and understand disease-related knowledge; after the consultation, the present invention collects patient diagnosis, treatment, and auxiliary examination information and provides prognosis consulting services, breaking down medical barriers and providing patients with comprehensive auxiliary diagnosis and treatment services; at the same time, the present invention pays attention to the issue of aging, and virtual relative services can be customized for elderly patients to provide full companionship during the diagnosis and treatment process.
[0026] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a schematic diagram of the process of the intelligent online auxiliary diagnosis and treatment system based on virtual digital human of the present invention;
[0028] Figure 2 This is a flow chart of the intelligent online auxiliary diagnosis and treatment method based on virtual digital human of the present invention. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] See also Figure 1 , an intelligent online auxiliary diagnosis and treatment system based on virtual digital human, including a database, the information sources of which include cutting-edge literature, expert opinions, clinical treatment guidelines and diagnostics, and a medical knowledge map built based on the information sources;
[0031] Knowledge graph is a structured semantic knowledge base used to quickly describe concepts and their relationships in the physical world. Neo4j and Graph XR can be used as important tools for building knowledge graphs. They can edit and process the content in the knowledge graph and convert abstract content into visual two-dimensional and three-dimensional content. The construction of medical knowledge graph mainly plays the role of summarizing and organizing data and establishing relationships between data, laying the foundation for subsequent training of virtual digital humans and giving them logic.
[0032] It also includes an online consultation module, which is used to identify medical-related questions raised by users, and match, screen and analyze them in the medical knowledge graph to form appropriate answer statements, and conduct human-computer interaction through virtual humans in the form of audio and video.
[0033] Specifically, the online consultation module includes an input unit, an intention recognition unit, a language synthesis unit, and a virtual human output unit.
[0034] The input unit is used for the user to input medical consultation information, which includes the user's basic information, chief complaint, current medical history, past history, personal history, and family history.
[0035] User basic information: including user's full name, age, ethnicity, address, contact information and other basic information.
[0036] Current medical history: mainly includes the duration of the patient's illness, the site of the disease, and the patient's treatment status, etc. On the one hand, it can give a general understanding of the patient's condition, and on the other hand, it can also understand what the patient's main expectations are for coming to the doctor.
[0037] Past history: mainly includes whether the patient has a history of surgery, vaccination history, infectious disease history, major disease conditions, allergies, etc. Its main purpose is to help doctors better judge whether the disease is a recurrence or a new one and the precautions for prescribing drugs.
[0038] Personal history: mainly includes the patient's habits and hobbies, marital history, and reproductive history. Knowing whether the patient has bad habits, marital history, and reproductive history that affect health will help to better issue personalized medical advice based on the patient's actual situation. For example, if a long-term smoker comes to see a doctor for lung infection, the doctor can advise the patient to quit smoking on the basis of routine medical advice. More personalized customization will make the patient feel at home.
[0039] Family history: Knowing about the major diseases suffered by the patient's blood relatives can help determine the possibility that the patient's disease is inherited and the possibility that the patient's children will be sick, so as to take preventive measures in advance.
[0040] The intention recognition unit classifies the user's input medical consultation information based on the large language model LLM and retrieval enhanced generation RAG method to determine the user's medical consultation intention.
[0041] LLM can read and understand the user's intention through intent recognition. LLM can establish a link between the knowledge graph and the virtual digital human, allowing the content of the knowledge graph to be smoothly extracted and expressed, giving the virtual digital human the ability to recognize intentions.
[0042] The LLM intent classifier is a new intent classifier that uses a large language model (LLM) to classify intents. The LLM intent classifier relies on the retrieval-augmented generation (RAG) method, combining the advantages of retrieval-based and generation-based methods.
[0043] The steps of LLM intent classifier are described as follows:
[0044] (1) During the training process, the classifier embeds all intent examples and stores their embeddings in a vector repository.
[0045] (2) During the prediction process, the classifier embeds the current message and uses the embedding to find similar intent examples in the vector repository.
[0046] (3) The retrieved examples are ranked according to their similarity to the current message, and the most similar examples will be included in the LLM prompt. This prompt guides the LLM to predict the intent of the message.
[0047] (4) LLM predicts an intent label.
[0048] (5) The generated label is mapped to an intent in the domain. LLM can also predict labels that are not in the training data. In this case, the intent in the domain that is most similar to its embedding will be predicted.
[0049] The virtual digital human relies on the large language model Q (LLM) and retrieval-augmented generation (RAG) method to classify the patient's natural language description, determine the patient's consultation intention, and then the digital human selects relevant questions for consultation.
[0050] The language synthesis unit analyzes the user's inquiry intention and forms an appropriate answer sentence.
[0051] The virtual human output unit converts text into speech and outputs it in the form of a virtual digital human to achieve human-computer interaction with the user. A virtual digital human is a digital character image close to a human image created using digital technology. It is mainly composed of three technical sections, including: Lip Sync, TTS and VITS.
[0052] Lip Sync is an audio and video synchronization technology based on deep learning. It can achieve high-precision lip synchronization effects by analyzing audio signals and video frames. The algorithm first converts the input audio into a spectrogram, then matches the best mouth shape position in the video frame, and performs lip shape transformation at that position according to the audio signal. TTS converts text into the voice of a virtual digital person, so that the virtual digital person can speak with intonation and express relevant emotions. VITS uses input language to train the neural network to enable the virtual digital person to output a similar language technology. VITS is a highly expressive speech synthesis model that combines variational reasoning, normalized flow, and adversarial training. VITS connects the acoustic model and vocoder in speech synthesis through hidden variables rather than spectrum, performs random modeling on the hidden variables, and uses random duration predictors to improve the diversity of synthesized speech. The same text can be input to synthesize speech with different tones and rhythms.
[0053] The smooth operation of virtual digital humans depends on multiple system models such as the encoder-decoder speech recognition system, dialect speech recognition system, dialogue system construction, diffusion model construction, etc. The most critical one is the encoder-decoder speech recognition system: based on encoder and decoder, virtual digital humans can recognize any text, voice, image, and video data, enable virtual digital humans to communicate with patients, and convert patients' language descriptions into natural language descriptions.
[0054] It also includes a department positioning module, which is used to collect and organize all relevant medical information provided by users in the online consultation module, and conduct in-depth analysis to identify potential pathogenic factors related to the user's health status and the user's symptom information, match this information with the data in the medical knowledge graph, select the department according to the user's condition and the probability of disease occurrence, and output it to the user to achieve accurate department positioning, ensuring that the user can get the treatment and services that best suit their condition.
[0055] It also includes a medical record output module, which is used to simplify and visualize the final data of the online consultation module and the department positioning module through the visualization tool numpy pandas, output text and image versions of the complete medical record, and output the visualized medical record to the user.
[0056] The present invention realizes comprehensive medical consultation, precise department positioning, and visual medical record output by constructing a medical knowledge graph and training virtual digital humans, ensuring that patients can register accurately, check medical record reports at any time, and understand disease-related knowledge.
[0057] It also includes a prognosis consultation module, which is used to store all information including diagnosis, treatment plan, and auxiliary examination report details, and provides medical consultation related to diagnosis, treatment, and rehabilitation. The prognosis consultation module includes a storage unit and a question-and-answer unit. The storage unit is used to store all information including diagnosis, treatment plan, and auxiliary examination report details, and the question-and-answer unit is used to provide medical consultation related to diagnosis, treatment, and rehabilitation through a virtual digital person. Users only need to enter questions or communicate directly with the virtual digital person to understand knowledge related to the disease.
[0058] The present invention is provided with a prognosis consultation module, which collects patient diagnosis, treatment, and auxiliary examination information and provides prognosis consultation services after the consultation, thereby breaking down medical barriers and providing patients with comprehensive auxiliary diagnosis and treatment services.
[0059] It also includes a virtual image construction module, which is used to construct a virtual human image that simulates the facial features and voice intonation of a designated real person. The virtual image construction module includes a portrait input unit and an audio input unit.
[0060] The virtual person in the initial state of the present invention is an image automatically generated by AI. The specified virtual image can be customized through the virtual image construction module. Virtual relative services can be customized for elderly patients to achieve full companionship during the diagnosis and treatment process.
[0061] See also Figure 2 , as described above, the diagnosis and treatment method based on the intelligent online auxiliary diagnosis and treatment system of virtual digital human comprises the following steps:
[0062] S1: First, the user communicates with the virtual digital human through the online consultation module and provides relevant medical information. The virtual digital human conducts a one-on-one question-and-answer inquiry with the user based on the database to complete the preliminary consultation;
[0063] S2: The department location module conducts in-depth analysis of all relevant medical information provided in the online consultation module to identify potential pathogenic factors related to the user's health status and the user's symptom information, matches this information with the data in the medical knowledge graph, selects a department according to the user's condition and the probability of disease occurrence, and outputs it to the user;
[0064] S3: The medical record output module simplifies and visualizes the final data of the online consultation module and the department location module through visualization tools, outputs the complete medical record in text and image versions, and outputs the visualized medical record to the user;
[0065] S4: After the diagnosis and treatment, the prognosis consultation module stores all the information including the diagnosis, treatment plan, and auxiliary examination report details, and continues to provide medical consultation related to diagnosis, treatment, and rehabilitation through virtual digital people.
[0066] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.
[0067] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0068] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0069] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0070] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by hardware associated with program instructions, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps in the method embodiment.
[0071] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0072] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. Intelligent online auxiliary diagnosis and treatment system based on virtual digital human, It is characterized in that include: A database, wherein the information sources of the database include cutting-edge literature, expert opinions, clinical treatment guidelines and diagnostics, and a medical knowledge graph constructed based on the information sources; An online consultation module is used to identify medical questions raised by users, match, filter and analyze them in the medical knowledge graph, form appropriate answer statements, and conduct human-computer interaction through virtual humans in the form of audio and video; Department location module, which is used to collect and organize all relevant medical information provided by the user in the online consultation module, and conduct in-depth analysis to identify potential pathogenic factors related to the user's health status and the user's symptom information, match this information with the data in the medical knowledge graph, select a department according to the user's condition and the probability of disease occurrence, and output it to the user; A medical record output module, which is used to simplify and visualize the final data of the online consultation module and the department location module through a visualization tool, output a complete medical record in text and image versions, and output the visualized medical record to the user; The prognosis consultation module is used to store all information including diagnosis, treatment plan, auxiliary examination report details, and provide medical consultation related to diagnosis, treatment, and rehabilitation.
2. The intelligent online auxiliary diagnosis and treatment system based on virtual digital human according to claim 1, Features: It also includes a virtual image construction module for constructing a virtual human image that simulates the facial features and voice intonation of a designated real person. The virtual image construction module includes a portrait input unit and an audio input unit.
3. The intelligent online auxiliary diagnosis and treatment system based on virtual digital human according to claim 1, Features: The online consultation module includes an input unit, an intention recognition unit, a language synthesis unit, and a virtual human output unit.
4. The intelligent online auxiliary diagnosis and treatment system based on virtual digital human according to claim 3, Features: The input unit is used for the user to input medical consultation information, and the medical consultation information includes the user's basic information, chief complaint, current medical history, past medical history, personal history, and family history.
5. The intelligent online auxiliary diagnosis and treatment system based on virtual digital human according to claim 4, Features: The intention recognition unit classifies the medical inquiry information input by the user based on the large language model LLM and the retrieval enhanced generation RAG method to determine the user's medical inquiry intention.
6. The intelligent online auxiliary diagnosis and treatment system based on virtual digital human according to claim 5, Features: The language synthesis unit analyzes the user's inquiry intention and forms a suitable answer sentence.
7. The intelligent online auxiliary diagnosis and treatment system based on virtual digital human according to claim 6, Features: The virtual human output unit converts text into voice and outputs it in the form of a virtual digital human to achieve human-computer interaction with the user.
8. The intelligent online auxiliary diagnosis and treatment system based on virtual digital human according to claim 1, Features: The prognosis consultation module includes a storage unit and a question-and-answer unit. The storage unit is used to store all information including diagnosis, treatment plan, and auxiliary examination report details. The question-and-answer unit is used to provide medical consultation related to diagnosis, treatment, and rehabilitation through a virtual digital person.
9. The diagnosis and treatment method based on the intelligent online auxiliary diagnosis and treatment system of virtual digital human according to any one of claims 1 to 8, It is characterized in that The steps include: S1: First, the user communicates with the virtual digital human through the online consultation module and provides relevant medical information. The virtual digital human conducts a one-on-one question-and-answer inquiry with the user based on the database to complete the preliminary consultation; S2: The department location module conducts in-depth analysis of all relevant medical information provided in the online consultation module to identify potential pathogenic factors related to the user's health status and the user's symptom information, matches this information with the data in the medical knowledge graph, selects a department according to the user's condition and the probability of disease occurrence, and outputs it to the user; S3: The medical record output module simplifies and visualizes the final data of the online consultation module and the department location module through visualization tools, outputs the complete medical record in text and image versions, and outputs the visualized medical record to the user; S4: After the diagnosis and treatment, the prognosis consultation module stores all the information including the diagnosis, treatment plan, and auxiliary examination report details, and continues to provide medical consultation related to diagnosis, treatment, and rehabilitation through virtual digital people.
Citation Information
Cited By
Traditional Chinese medicine robot diagnosis and treatment system based on multi-modal large model
CN121885150A
