Intelligent inquiry method and device, electronic equipment and storage medium

Abstract

The invention provides an intelligent inquiry method and device, electronic equipment and a storage medium, and the intelligent inquiry method comprises the steps: obtaining model mapping parameters according to user body feature data, mapping the model mapping parameters to a basic three-dimensional body model, and obtaining a user three-dimensional model; binding the user disease model with the first user three-dimensional model to obtain a second three-dimensional model; extracting inquiry keywords of the user inquiry data to generate a doctor model; performing a first inquiry dialogue between the doctor model and the second three-dimensional model through the interactive interface, extracting a symptom description of the user according to the first inquiry dialogue through the language recognition model, and updating the second three-dimensional model to obtain a third three-dimensional model; and performing a second inquiry dialogue between the doctor model and the third three-dimensional model through the interactive interface, and performing inquiry processing according to the second inquiry dialogue through the language recognition model to obtain an inquiry result. The method has the beneficial effects that privacy leakage during inquiry of the user is avoided, and the inquiry efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer artificial intelligence technology, and in particular to an intelligent diagnosis method, device, electronic equipment and storage medium. Background Art

[0002] The rise of big data and artificial intelligence has enabled big models and digital humans to play an important role in the field of medical consultation. The application of these technologies has not only improved the efficiency of medical services, but also improved the accuracy of diagnosis.

[0003] The prior art has the following problems: (1) How to visualize complex medical data so that doctors and patients can understand the disease more intuitively; (2) How to protect patient privacy and prevent the abuse of medical data; (3) How to improve the human-computer interaction experience between users and doctors to make it more in line with people’s communication habits. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to provide an intelligent medical consultation method, device, electronic device and storage medium to avoid privacy leakage during user medical consultation and improve medical consultation efficiency.

[0005] One aspect of the present invention provides an intelligent medical consultation method, comprising: According to the consultation request, obtain the user's consultation data, user's physical characteristic data and user's case data; According to the user's body feature data, a model mapping parameter is obtained, and the model mapping parameter is mapped to a basic three-dimensional body model to obtain a first user three-dimensional model, wherein the basic three-dimensional body model is a parameter-adjustable model; Extracting user disease characteristics from the user case data, obtaining a user disease model based on the user disease characteristics and a preset disease model, and binding the user disease model with the first user three-dimensional model to obtain a second three-dimensional model; Extracting the medical inquiry keywords of the user's medical inquiry data, and determining the doctor model according to the medical inquiry keywords; Conducting a first consultation dialogue between the doctor model and the second three-dimensional model through an interactive interface, extracting a user's symptom description according to the first consultation dialogue through a language recognition model, and updating the second three-dimensional model according to the symptom description to obtain a third three-dimensional model; A second consultation dialogue is conducted between the doctor model and the third three-dimensional model through the interactive interface, and consultation processing is performed according to the second consultation dialogue through the language recognition model to obtain a consultation result.

[0006] According to the intelligent medical consultation method, the model mapping parameters are obtained according to the user's body feature data, and the model mapping parameters are mapped to the basic three-dimensional body model to obtain a first user three-dimensional model, including: Obtaining the user's authorized physical characteristic data, wherein the user's physical characteristic data includes height, weight, body shape, muscle mass and fat ratio; Mapping the user's body feature data according to preset values ​​to obtain model mapping parameters; The model mapping parameters are mapped to the basic three-dimensional body model through a visual interface, and the parameters of the obtained first user three-dimensional model are tuned to obtain the first user three-dimensional model.

[0007] According to the intelligent medical consultation method, the method for determining the basic three-dimensional body model includes: Obtaining body anatomical data, wherein the body anatomical data includes medical literature, anatomical atlases, CT scan images, and MRI images; Determine the positional relationship of body structures through medical literature and anatomical atlases, and determine the three-dimensional body structure view through CT scan images and MRI images, wherein the three-dimensional body structure view is used to maintain and optimize the basic three-dimensional body model; Preprocessing the CT scan images and MRI images, and using Mimics to segment the preprocessed CT scan images and MRI images to obtain segmented anatomical structure images; The segmented anatomical structure image is converted into a point cloud, which is processed by surface reconstruction to obtain a surface model, and the surface model is combined according to the positional relationship of the body structure to obtain a basic three-dimensional body model.

[0008] According to the intelligent medical consultation method, the user disease characteristics are extracted from the user case data, a user disease model is obtained according to the user disease characteristics and a preset disease model, and the user disease model is bound to the first user three-dimensional model to obtain a second three-dimensional model, including: Acquire disease image data collected from different angles, wherein the disease image data includes disease anatomical structure, disease tissue changes, lesion location, lesion range and lesion degree; Preprocessing the disease image data to obtain a lesion area image through an image segmentation algorithm; The images of the lesion area at different angles are used to construct point cloud data, and the point cloud data is converted using a surface reconstruction algorithm, and then smoothed to obtain a three-dimensional surface model of the disease; The disease three-dimensional surface model is annotated according to the user's disease characteristics, and the annotated disease three-dimensional surface model is subjected to fluid dynamics simulation and stress simulation according to the lesion location and lesion range, so that the disease three-dimensional surface model is bound to the first user three-dimensional model.

[0009] According to the intelligent consultation method, determining the doctor model according to the consultation keywords includes: Obtaining the doctor's physical data and the doctor's voice data, wherein the doctor's physical data includes facial features, body features, habit features, and clothing features; Generating a doctor model by using three-dimensional digital modeling on the doctor's body data, wherein the three-dimensional digital modeling includes facial modeling, action modeling, body modeling and clothing modeling; The doctor's voice data is subjected to speech preprocessing and speech feature extraction to obtain the doctor's voice features, and the doctor's voice feature data is trained using one of deep learning and adversarial learning to obtain a doctor's voice model.

[0010] According to the intelligent consultation method, a first consultation dialogue is conducted between a doctor model and a second three-dimensional model through an interactive interface, a user's symptom description is extracted according to the first consultation dialogue through a language recognition model, and the second three-dimensional model is updated according to the symptom description to obtain a third three-dimensional model, including: Generating a three-dimensional animation of a doctor on a user client according to the doctor voice model and the doctor model; Acquire a first consultation dialogue when the user and the doctor's three-dimensional animation conduct interactive question and answer, and use a language recognition model to recognize the symptom description of the first consultation dialogue, wherein the language recognition model recognition includes word segmentation, part-of-speech tagging and named entity recognition processing; The symptom description of the first consultation dialogue updates the user disease model in the second three-dimensional model to obtain a third three-dimensional model.

[0011] According to the intelligent consultation method, a second consultation dialogue is conducted between the doctor model and the third three-dimensional model through an interactive interface, and a consultation process is conducted according to the second consultation dialogue through a language recognition model to obtain a consultation result, including: Obtain the inspection information and image data uploaded by the user, format the inspection information and image data, and convert them into an input matrix to obtain multimodal data; Processing the multimodal data through the language model to obtain analysis results of the inspection information and image data; Searching for matching symptoms from a medical knowledge base based on the analysis results of the test information and image data, and searching for the relationship between symptoms, diseases, and drugs to obtain search results; A diagnosis suggestion is generated based on the language model and the medical knowledge base, and structured data with risk level, body part, and diagnosis suggestion annotations processed by the language model is stored in a database.

[0012] Another aspect of the embodiment of the present invention provides an intelligent medical consultation device, wherein the intelligent medical consultation system includes a doctor end, a patient end, and a server end, wherein the server end is connected to the doctor end and the patient end respectively, and includes: The first module is used to obtain user consultation data, user physical characteristic data and user case data according to the consultation request; A second module is used to obtain a model mapping parameter according to the user's body feature data, and map the model mapping parameter to a basic three-dimensional body model to obtain a first user three-dimensional model, wherein the basic three-dimensional body model is a parameter-adjustable model; A third module is used to extract user disease characteristics from the user case data, obtain a user disease model according to the user disease characteristics and a preset disease model, and bind the user disease model to the first user three-dimensional model to obtain a second three-dimensional model; The fourth module is used to extract the medical inquiry keywords of the user's medical inquiry data and determine the doctor model according to the medical inquiry keywords; A fifth module is used to conduct a first consultation dialogue between the doctor model and the second three-dimensional model through an interactive interface, extract a user's symptom description according to the first consultation dialogue through a language recognition model, and update the second three-dimensional model according to the symptom description to obtain a third three-dimensional model; The sixth module is used to conduct a second consultation dialogue between the doctor model and the third three-dimensional model through the interactive interface, and to perform consultation processing according to the second consultation dialogue through the language recognition model to obtain a consultation result.

[0013] Another aspect of an embodiment of the present invention provides an electronic device, including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method described above.

[0014] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method described above.

[0015] The beneficial effects of the present invention are as follows: by acquiring the user's image data and symptom description, a personalized three-dimensional model of the user is first constructed, and then the user's disease characteristics are adaptively mapped to the user's three-dimensional model by using a model mapping method; similarly, a corresponding three-dimensional model is constructed according to the doctor's personal characteristics, habits, clothing, etc., and visual communication with the user is carried out through a voice model and a motion model, and the user's three-dimensional model is displayed on the doctor's client, thereby improving the efficiency of consultation; a language model is used to perform disease matching and generate diagnostic suggestions based on symptom descriptions and image data, thereby avoiding privacy leakage during user consultation and improving the efficiency of consultation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which: Figure 1 Schematic diagram of an intelligent medical consultation system according to an embodiment of the present invention.

[0017] Figure 2 It is a schematic diagram of the intelligent medical consultation process of an embodiment of the present invention.

[0018] Figure 3 It is a schematic diagram of the user three-dimensional model generation process according to an embodiment of the present invention.

[0019] Figure 4 It is a schematic diagram of the basic three-dimensional model generation process of an embodiment of the present invention.

[0020] Figure 5 It is a flowchart of binding disease characteristics with user models according to an embodiment of the present invention.

[0021] Figure 6 It is a schematic diagram of the doctor model generation process according to an embodiment of the present invention.

[0022] Figure 7 It is a schematic diagram of the user three-dimensional model update process according to an embodiment of the present invention.

[0023] Figure 8 It is a schematic diagram of the user consultation process of an embodiment of the present invention.

[0024] Fig. 9 It is a schematic diagram of the patient-side interaction process of an embodiment of the present invention.

[0025] Fig.10 It is a schematic diagram of the service process of the embodiment of the present invention.

[0026] Fig.11 It is a schematic diagram of the doctor-side business process of an embodiment of the present invention.

[0027] Fig.12 Schematic diagram of an intelligent medical consultation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. In the subsequent description, the use of suffixes such as "module", "component" or "unit" used to represent elements is only for the purpose of facilitating the description of the present invention, and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used in a mixed manner. "First", "second" and the like are only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the order of the indicated technical features. In this subsequent description, the continuous numbering of the method steps is for the convenience of review and understanding. In combination with the overall technical solution of the present invention and the logical relationship between the various steps, adjusting the implementation order between the steps will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0029] refer to Figure 1 The schematic diagram of the intelligent consultation system shown in the figure includes a patient side, a server side and a doctor side, wherein: On the patient side, this module can display a virtual doctor image in front of the patient by calling and rendering the doctor data in the database, and then use the language model to conduct a conversation with the patient through the digital intelligent person (i.e. the virtual doctor image). After fully understanding the patient's symptoms and needs, the consultation result data is transmitted to the server for analysis and storage.

[0030] The server side is responsible for storing and rendering virtual physician images, running language model algorithms for interaction, and storing medical data. The server side digitally models the existing physicians in the hospital, including images, voices, and some habitual movements, and stores the digital models in the server database. The language model will parse the patient's description, and store the patient's description in a structured form in the database.

[0031] On the doctor's side, universal human body modeling is used to divide the three-dimensional model of the human body into multiple layers such as skin, muscles, blood vessels, nerves, and bones, and then display it on the doctor's side. The patient description structured by the language model is called to quickly mark different symptoms on the three-dimensional model, and different colors are used to mark the severity determined by the system. Finally, a recommended diagnosis and treatment plan is output for the doctor's confirmation. At the same time, the large language model will continuously fine-tune the parameters based on the doctor's feedback.

[0032] In some embodiments, the language model adopts a large language model, wherein the large language model is trained by deep learning using massive texts.

[0033] Figure 2 Schematic diagram of the intelligent medical consultation process of an embodiment of the present invention. It includes but is not limited to steps S100 to S600: S100, obtaining user consultation data, user physical characteristic data and user case data according to the consultation request.

[0034] In some embodiments, the user consultation data is the consultation text input by the user, such as "My knee hurts", etc.; the user body characteristic data is the user's age, gender, fat percentage, etc.; the user case data is the imaging data, medical record data, etc. uploaded by the user. It can be understood that these data are stored confidentially.

[0035] S200, obtaining model mapping parameters according to user body feature data, and mapping the model mapping parameters to a basic three-dimensional body model to obtain a first user three-dimensional model, wherein the basic three-dimensional body model is a parameter-adjustable model.

[0036] In some embodiments, reference Figure 3 The user 3D model generation process diagram shown in FIG. includes but is not limited to steps S210 to S230: S210, obtaining user's physical characteristic data authorized by the user, wherein the user's physical characteristic data includes height, weight, body shape, muscle mass and fat ratio; S220, mapping the user's body feature data according to preset values ​​to obtain model mapping parameters; S230, mapping the model mapping parameters to the basic three-dimensional model through a visual interface, and performing parameter tuning on the obtained first user three-dimensional model to obtain the first user three-dimensional model.

[0037] In some embodiments, the patient's description is parsed through a large language model. This process may involve some natural language processing technologies, such as word segmentation, part-of-speech tagging, named entity recognition, etc. According to these technologies, the patient's description is matched with the medical knowledge in the database to find matching symptoms and the relationship between these symptoms and diseases and drugs. The data processing of the large model can refer to the following steps: Obtain patient information, collect patient descriptions through conversations, and upload test information and imaging data, all of which are formatted into matrix information that can be input into the large model. Obtain disease image data from the imaging data, including disease anatomical structure, disease tissue changes, lesion location, lesion range, and lesion severity; Process multimodal data (patient information) through large models, including test data, imaging data, and patient descriptions. For example, by parsing the data in the test report, you can understand the image information in the imaging data and the text information in the patient description.

[0038] Match medical knowledge, combine the patient's description, the big model's understanding of the patient's description with the medical knowledge in the database, find matching symptoms, and the relationship between these symptoms and diseases and drugs.

[0039] In some embodiments, after obtaining the above-mentioned user disease-related data, a model mapping process is performed.

[0040] In some embodiments, reference Figure 4 The basic three-dimensional model generation process diagram shown in FIG. includes but is not limited to steps S240 to S270: S240, acquiring body anatomical data, wherein the body anatomical data includes medical literature, anatomical atlas, CT scan images, and MRI images; S250, determining the positional relationship of body structures through medical literature and anatomical atlases, and determining a three-dimensional body structure view through CT scan images and MRI images, wherein the three-dimensional body structure view is used to maintain and optimize the basic three-dimensional body model; In some embodiments, by checking the three-dimensional body structure view, the doctor or expert can make optimization suggestions for its existing defects, and regularly maintain and optimize the three-dimensional body structure view according to the optimization suggestions, so that the obtained basic three-dimensional body model is more accurate.

[0041] S260, preprocessing the CT scan image and the MRI image, and performing image segmentation on the preprocessed CT scan image and the MRI image using Mimics to obtain a segmented anatomical structure image; S270, converting the segmented anatomical structure image into a point cloud, processing the point cloud by using a surface reconstruction method to obtain a surface model, and combining the surface model according to the positional relationship of the body structure to obtain a basic three-dimensional body model.

[0042] Among them, Mimics is a software tool for medical image processing and 3D modeling.

[0043] In some embodiments, the basic 3D model further includes the following process: Data collection and analysis, obtaining human anatomical data, based on medical literature, anatomical atlases, CT scans, MRI images, etc., including anatomical structure data including the morphology and position information of bones, muscles, blood vessels, nerves, etc.; physiological function data including the functions of organs and tissues, such as the pumping function of the heart and the breathing function of the lungs, etc.; tissue characteristic data, such as the physical properties of different tissues (such as skin, fat, muscle), such as elasticity, density, etc.; 3D modeling, creating the basic framework of the human body based on the skeletal structure; adding muscles and soft tissues: adding muscles and other soft tissues to the skeletal framework; simulating blood vessels and nerves, including simulating the vascular network and nervous system, including the main blood vessels and nerve pathways; detail optimization, refining the model, including skin texture, facial features, etc., to improve the realism of the model.

[0044] Model verification and optimization: after the preliminary basic three-dimensional model is constructed, verification and optimization are required; including expert review, in which experts review the model and make modification suggestions; functional testing, in which the functionality of the model is tested, such as simulating blood circulation and nerve conduction; user feedback, in which feedback from users (such as doctors and medical students) is collected; and the model is iteratively improved based on feedback and test results.

[0045] Model integration application integrates verified and optimized 3D basic models into medical information systems, such as integrating 3D models with electronic health records, diagnostic tools and other systems for doctors to use; interactive design: through the user interface, ensure that doctors can interact with 3D models, such as rotating, zooming, and viewing different levels of tissue.

[0046] In some embodiments, surface reconstruction refers to a process of generating a continuous geometric surface model from a discrete three-dimensional point set (point cloud).

[0047] S300, extracting user disease characteristics from user case data, obtaining a user disease model based on the user disease characteristics and a preset disease model, and binding the user disease model to the first user three-dimensional model to obtain a second three-dimensional model.

[0048] In some embodiments, reference Figure 5 The flowchart of binding disease characteristics with user models shown in FIG. 1 includes but is not limited to steps S310 to S340: S310, acquiring disease image data collected at different angles, wherein the disease image data includes disease anatomical structure, disease tissue changes, lesion location, lesion range and lesion degree; S320, preprocessing the disease image data, and obtaining a lesion area image through an image segmentation algorithm; S330, constructing point cloud data from the images of the lesion area at different angles, converting the point cloud data using a surface reconstruction algorithm, and then performing smoothing to obtain a three-dimensional surface model of the disease; S340, annotating the disease three-dimensional surface model according to the user's disease characteristics, and performing fluid dynamics simulation and stress simulation on the annotated disease three-dimensional surface model according to the lesion location and lesion range, so as to complete the binding of the disease three-dimensional surface model with the first user three-dimensional model.

[0049] S400, extracting the medical inquiry keywords from the user's medical inquiry data, and determining the doctor model according to the medical inquiry keywords.

[0050] In some embodiments, reference Figure 6 The doctor model generation process diagram shown in the figure includes but is not limited to steps S410 to S430: S410, obtaining doctor's physical data and doctor's voice data, wherein the doctor's physical data includes facial features, body features, habit features and clothing features; S420, generating a doctor model by using three-dimensional digital modeling on the doctor's body data, wherein the three-dimensional digital modeling includes facial modeling, action modeling, body modeling and clothing modeling; S430, perform speech preprocessing and speech feature extraction on the doctor's voice data to obtain the doctor's voice features, and train the doctor's voice feature data using one of deep learning and adversarial learning to obtain a doctor's voice model.

[0051] In some embodiments, digital modeling of the physician model comprises the following steps: Build a physician database, which includes a large amount of physician information, various attributes of physician information (such as name, specialty, experience, etc.), and relationships representing the association between physicians and medical institutions, diseases, drugs, etc.; Render and display the virtual doctor image on the front-end display interface through a three-dimensional engine; Modeling the image of the physician, converting the physician data in the physician database into a three-dimensional model, and performing mapping, modeling, rendering, etc. through a three-dimensional engine to display the physician's image in a three-dimensional form; Call the voice model to convert the doctor's conversation information into the doctor's corresponding voice intonation through the voice model; Rendering, through the three-dimensional engine, integrates the image, behavior and voice of the modeled physician, and renders a virtual physician image that is closest to a real physician through texture mapping, lighting calculation, animation rendering and other methods.

[0052] In some embodiments, the physician's process includes: Through the interactive elements of the patient-side interface, such as dialog boxes and buttons, patients can interact with virtual doctors and fully understand their symptoms and needs. Using a large language model, the digital intelligent person (i.e., the virtual doctor image) interviews the patient in a conversational manner. Through natural language processing, the patient's description is matched with the medical knowledge in the database to find matching symptoms and the relationship between these symptoms and diseases and drugs. The data on patients' symptoms and needs is transmitted to the server for analysis and storage.

[0053] In some embodiments, the process of physician's speech modeling includes: speech collection, collecting the physician's speech through microphone recording and speech encoding; speech preprocessing, performing noise reduction, normalization, frame division and other preprocessing on the collected speech; speech feature extraction, the preprocessed speech is converted into a feature vector using methods such as MFCC, LPCC, PLP, etc. The feature vector includes pitch, timbre, sound intensity, etc.; speech model training, generating a speech model and storing it through the obtained feature vector using deep learning and adversarial generative learning methods.

[0054] S500, a first consultation dialogue is conducted between the doctor model and the second three-dimensional model through an interactive interface, a user's symptom description is extracted according to the first consultation dialogue through a language recognition model, and the second three-dimensional model is updated according to the symptom description to obtain a third three-dimensional model.

[0055] In some embodiments, reference Figure 7 The user 3D model update process diagram shown in FIG. includes but is not limited to steps S510 to S530: S510, generating a doctor's three-dimensional animation on the user client according to the doctor's voice model and the doctor model; S520, obtaining a first consultation dialogue when the user and the doctor perform interactive question and answer in a three-dimensional animation, and using a language recognition model to recognize a symptom description of the first consultation dialogue, wherein the language recognition model recognition includes word segmentation, part-of-speech tagging, and named entity recognition processing; S530, the symptom description of the first consultation dialogue updates the user disease model in the second three-dimensional model to obtain a third three-dimensional model.

[0056] S600, a second consultation dialogue is conducted between the doctor model and the third three-dimensional model through the interactive interface, and consultation processing is conducted according to the second consultation dialogue through the language recognition model to obtain a consultation result.

[0057] In some embodiments, different symptoms are quickly annotated on the three-dimensional model. The patient's symptoms and test information have been processed by the large model, processed according to the body part, severity and recommended diagnostic information, and stored in the database. The doctor can directly call these data and annotate them with different colors according to the severity determined by the system, and annotate the symptom information on the corresponding human model level and the corresponding part; the doctor modifies and confirms the annotated diagnostic suggestions, and stores the confirmed diagnostic results in the database. The multimodal large model will regularly fine-tune the model parameters according to the doctor's modifications to the diagnostic results to update the effect of the model reasoning; refer to Figure 8 , Figure 8 This is a schematic diagram of the user consultation process of an embodiment of the present invention. It includes but is not limited to steps S610 to S640: S610, obtaining the inspection information and image data uploaded by the user, formatting the inspection information and image data and converting them into an input matrix to obtain multimodal data; S620, processing the multimodal data through a language model to obtain analysis results of the inspection information and the image data; S630, searching for matching symptoms from a medical knowledge base according to the analysis results of the test information and the image data, and searching for relationships among symptoms, diseases, and drugs, to obtain search results; S640, generating a diagnosis suggestion based on the language model and the medical knowledge base, and storing the structured data processed by the language model with annotations of risk level, body part, and diagnosis suggestion in a database.

[0058] In some embodiments, the doctor's three-dimensional model, the user's three-dimensional model, the basic model, etc. may also be updated based on the doctor's feedback on the user's three-dimensional model.

[0059] In some embodiments, reference Fig. 9 Go, where Fig. 9 The patient-side module is a flow chart of the patient-side interaction. The patient-side module calls and renders the physician data in the database, displays a virtual physician image to the patient, and fully understands the patient's symptoms and demands through the interaction process with the patient.

[0060] In some embodiments, Fig.10 This is the server-side business process of the embodiment of the present invention. The doctor side needs to digitally model the existing doctors in the hospital, including images, voices, and some habitual movements, etc., wherein the digital model is stored in the server database.

[0061] In some embodiments, Fig.11This is a schematic diagram of the doctor-side interaction process of an embodiment of the present invention. In the medical service system, the doctor-side adopts universal human body modeling, divides the human body three-dimensional model into multiple layers such as skin, muscle, blood vessel, nerve, and bone, and displays it on the doctor-side, and calls the patient description structured by the large language model to quickly annotate different symptoms on the three-dimensional model.

[0062] Fig.12 12 is a diagram of an intelligent medical consultation and analysis device according to an embodiment of the present invention. The device includes a first module 1210 , a second module 1220 , a third module 1230 , a fourth module 1240 , a fifth module 1250 and a sixth module 1260 .

[0063] Among them, the first module 1210 is used to obtain user consultation data, user body feature data and user case data according to the consultation request; the second module 1220 is used to obtain model mapping parameters according to the user body feature data, map the model mapping parameters to the basic three-dimensional body model, and obtain a first user three-dimensional model, wherein the basic three-dimensional body model is a parameter-adjustable model; the third module 1230 is used to extract user disease characteristics from the user case data, obtain a user disease model according to the user disease characteristics and the preset disease model, and bind the user disease model to the first user three-dimensional model to obtain a second three-dimensional model; the fourth module 1240 is used to extract the consultation keywords of the user consultation data, and determine the doctor model according to the consultation keywords; the fifth module 1250 is used to conduct a first consultation dialogue between the doctor model and the second three-dimensional model through an interactive interface, extract the user's symptom description according to the first consultation dialogue through a language recognition model, and update the second three-dimensional model according to the symptom description to obtain a third three-dimensional model; the sixth module 1260 is used to conduct a second consultation dialogue between the doctor model and the third three-dimensional model through an interactive interface, and conduct consultation processing according to the second consultation dialogue through a language recognition model to obtain a consultation result.

[0064] Exemplarily, under the cooperation of the first module 1210, the second module 1220, the third module 1230, the fourth module 1240, the fifth module 1250 and the sixth module 1260 in the device, the embodiment device can implement any of the aforementioned intelligent consultation methods, that is, according to the consultation request, obtain the user consultation data, the user body feature data and the user case data; according to the user body feature data, obtain the model mapping parameters, map the model mapping parameters to the basic three-dimensional body model, and obtain the first user three-dimensional model, wherein the basic three-dimensional body model is a parameter adjustable model; extract the user disease characteristics from the user case data, and according to the user disease The invention provides a method for obtaining a user disease model by using a user's image data and a preset disease model, and binding the user disease model with the first user three-dimensional model to obtain a second three-dimensional model; extracting the inquiry keywords of the user's inquiry data, and determining the doctor model according to the inquiry keywords; conducting a first inquiry dialogue between the doctor model and the second three-dimensional model through an interactive interface, extracting the user's symptom description according to the first inquiry dialogue through a language recognition model, and updating the second three-dimensional model according to the symptom description to obtain a third three-dimensional model; conducting a second inquiry dialogue between the doctor model and the third three-dimensional model through an interactive interface, and performing inquiry processing according to the second inquiry dialogue through a language recognition model to obtain an inquiry result. The beneficial effects of the invention are as follows: by acquiring the user's image data and symptom description, first constructing a user personalized three-dimensional model, and then using a model mapping method to adaptively map the user's disease features to the user three-dimensional model; similarly, constructing a corresponding three-dimensional model according to the doctor's personal characteristics, habits, and clothing, visually communicating with the user through a voice model and an action model, and displaying the user three-dimensional model on the doctor's client, thereby improving the inquiry efficiency; using a language model to generate disease matching and diagnostic suggestions based on symptom description and image data, thereby avoiding privacy leakage during user inquiry and improving the inquiry efficiency.

[0065] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory; The memory stores a program; The processor executes the program to perform the aforementioned intelligent consultation method; the electronic device has the function of carrying and running the intelligent consultation software system provided by the embodiment of the present invention, for example, a personal computer, a minicomputer, a main frame, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicates with a charged particle tool or other imaging device, etc.

[0066] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the intelligent diagnosis method as described above.

[0067] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0068] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the aforementioned intelligent consultation method.

[0069] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0070] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0072] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0073] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0074] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0075] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0076] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An intelligent medical consultation method, characterized in that: include: According to the consultation request, obtain the user's consultation data, user's physical characteristic data and user's case data; According to the user's body feature data, a model mapping parameter is obtained, and the model mapping parameter is mapped to a basic three-dimensional body model to obtain a first user three-dimensional model, wherein the basic three-dimensional body model is a parameter-adjustable model; Extracting user disease characteristics from the user case data, obtaining a user disease model based on the user disease characteristics and a preset disease model, and binding the user disease model with the first user three-dimensional model to obtain a second three-dimensional model; Extracting the medical inquiry keywords of the user's medical inquiry data, and determining the doctor model according to the medical inquiry keywords; Conducting a first consultation dialogue between the doctor model and the second three-dimensional model through an interactive interface, extracting a user's symptom description according to the first consultation dialogue through a language recognition model, and updating the second three-dimensional model according to the symptom description to obtain a third three-dimensional model; A second consultation dialogue is conducted between the doctor model and the third three-dimensional model through the interactive interface, and consultation processing is performed according to the second consultation dialogue through the language recognition model to obtain a consultation result.

2. The intelligent medical consultation method according to claim 1, characterized in that: The method of obtaining a model mapping parameter according to the user body feature data, and mapping the model mapping parameter to the basic three-dimensional body model to obtain a first user three-dimensional model includes: Obtaining the user's authorized physical characteristic data, wherein the user's physical characteristic data includes height, weight, body shape, muscle mass and fat ratio; Mapping the user's body feature data according to preset values ​​to obtain model mapping parameters; The model mapping parameters are mapped to the basic three-dimensional body model through a visual interface, and the parameters of the obtained first user three-dimensional model are tuned to obtain the first user three-dimensional model.

3. The intelligent medical consultation method according to claim 1, characterized in that: The method for determining the basic three-dimensional body model comprises: Obtaining body anatomical data, wherein the body anatomical data includes medical literature, anatomical atlases, CT scan images, and MRI images; Determine the positional relationship of body structures through medical literature and anatomical atlases, and determine the three-dimensional body structure view through CT scan images and MRI images, wherein the three-dimensional body structure view is used to maintain and optimize the basic three-dimensional body model; Preprocessing the CT scan images and MRI images, and using Mimics to segment the preprocessed CT scan images and MRI images to obtain segmented anatomical structure images; The segmented anatomical structure image is converted into a point cloud, which is processed by surface reconstruction to obtain a surface model, and the surface model is combined according to the positional relationship of the body structure to obtain a basic three-dimensional body model.

4. The intelligent medical consultation method according to claim 3, characterized in that: The extracting of the user disease characteristics from the user case data, obtaining the user disease model according to the user disease characteristics and the preset disease model, and binding the user disease model with the first user three-dimensional model to obtain the second three-dimensional model include: Acquire disease image data collected from different angles, wherein the disease image data includes disease anatomical structure, disease tissue changes, lesion location, lesion range and lesion degree; Preprocessing the disease image data to obtain a lesion area image through an image segmentation algorithm; The images of the lesion area at different angles are used to construct point cloud data, and the point cloud data are converted using a surface reconstruction algorithm, and then smoothed to obtain a three-dimensional surface model of the disease; The disease three-dimensional surface model is annotated according to the user's disease characteristics, and the annotated disease three-dimensional surface model is subjected to fluid dynamics simulation and stress simulation according to the lesion location and lesion range, so that the disease three-dimensional surface model is bound to the first user three-dimensional model.

5. The intelligent medical consultation method according to claim 1, characterized in that: Determining the doctor model according to the consultation keywords includes: Obtaining the doctor's physical data and the doctor's voice data, wherein the doctor's physical data includes facial features, body features, habit features, and clothing features; The doctor's body data is subjected to three-dimensional digital modeling to generate a doctor model, wherein the three-dimensional digital modeling includes facial modeling, action modeling, body modeling and clothing modeling; The doctor's voice data is subjected to speech preprocessing and speech feature extraction to obtain the doctor's voice features, and the doctor's voice feature data is trained using one of deep learning and adversarial learning to obtain a doctor's voice model.

6. The intelligent medical consultation method according to claim 5, characterized in that: The first consultation dialogue between the doctor model and the second three-dimensional model is conducted through the interactive interface, the user's symptom description is extracted according to the first consultation dialogue through the language recognition model, and the second three-dimensional model is updated according to the symptom description to obtain the third three-dimensional model, including: Generating a three-dimensional animation of a doctor on a user client according to the doctor voice model and the doctor model; Acquire a first consultation dialogue when the user and the doctor's three-dimensional animation conduct interactive question and answer, and use a language recognition model to recognize the symptom description of the first consultation dialogue, wherein the language recognition model recognition includes word segmentation, part-of-speech tagging and named entity recognition processing; The symptom description of the first consultation dialogue updates the user disease model in the second three-dimensional model to obtain a third three-dimensional model.

7. The intelligent medical consultation method according to claim 6, characterized in that: The second consultation dialogue between the doctor model and the third three-dimensional model is conducted through the interactive interface, and consultation processing is performed according to the second consultation dialogue through the language recognition model to obtain the consultation result, including: Obtain the inspection information and image data uploaded by the user, initialize the inspection information and image data and convert them into input matrices to obtain multimodal data; Processing the multimodal data through the language model to obtain analysis results of the inspection information and image data; Searching for matching symptoms from a medical knowledge base based on the analysis results of the test information and image data, and searching for the relationship between symptoms, diseases, and drugs to obtain search results; A diagnosis suggestion is generated based on the language model and the medical knowledge base, and structured data with risk level, body part, and diagnosis suggestion annotations processed by the language model is stored in a database.

8. An intelligent medical consultation device, characterized in that: include: The first module is used to obtain user consultation data, user physical characteristic data and user case data according to the consultation request; A second module is used to obtain a model mapping parameter according to the user's body feature data, and map the model mapping parameter to a basic three-dimensional body model to obtain a first user three-dimensional model, wherein the basic three-dimensional body model is a parameter-adjustable model; A third module is used to extract user disease characteristics from the user case data, obtain a user disease model according to the user disease characteristics and a preset disease model, and bind the user disease model to the first user three-dimensional model to obtain a second three-dimensional model; The fourth module is used to extract the medical inquiry keywords of the user's medical inquiry data and determine the doctor model according to the medical inquiry keywords; A fifth module is used to conduct a first consultation dialogue between the doctor model and the second three-dimensional model through an interactive interface, extract a user's symptom description according to the first consultation dialogue through a language recognition model, and update the second three-dimensional model according to the symptom description to obtain a third three-dimensional model; The sixth module is used to conduct a second consultation dialogue between the doctor model and the third three-dimensional model through the interactive interface, and to perform consultation processing according to the second consultation dialogue through the language recognition model to obtain a consultation result.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the intelligent medical consultation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the intelligent medical consultation method according to any one of claims 1 to 7.

Images