AI-based intelligent medical diagnostic engine

Abstract

This invention relates to the field of artificial intelligence technology, specifically to an intelligent medical diagnostic engine based on a large AI model. The engine includes an auxiliary diagnostic module that collects patient electronic medical records, medical images, and other data, and uses ensemble learning methods to predict auxiliary outcomes; a drug recommendation module that analyzes drug recommendation data using the large AI model to predict the potential efficacy and side effects of new drugs; and a surgical simulation module that combines virtual reality technology to construct a surgical environment and human body model. This invention enables rapid processing and analysis of large amounts of medical data through auxiliary diagnosis, shortening diagnostic time and reducing unnecessary invasive examinations and misdiagnosis. Drug recommendation integrates patients' genetic, physiological, and pathological multi-omics data to provide patients with personalized drug treatment plans and doctors with comprehensive and accurate diagnostic evidence and treatment recommendations. Furthermore, by accurately simulating real surgical scenarios, including mechanical and optical properties, it provides doctors and medical students with a more realistic training experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an intelligent medical diagnosis engine based on an AI large model. BACKGROUND

[0002] With the rapid development of information technology, artificial intelligence (AI) technology is increasingly widely used in the medical field. AI is increasingly widely used in the medical and health field, showing great potential in aspects such as auxiliary diagnosis and personalized treatment, and greatly promoting the innovation and upgrading of medical service models. Intelligent medicine, as a new medical service model, has become a development trend in the medical industry by using AI technology to improve the efficiency and quality of medical services.

[0003] However, despite the remarkable achievements, current AI-based intelligent medical diagnosis models still face some challenges and shortcomings, which are manifested in the following aspects: 1. Although some AI-based auxiliary diagnosis systems have appeared in recent years, these systems can only handle simple cases and are difficult to analyze cases comprehensively in combination with multiple data, which can easily lead to misdiagnosis; 2. Drug therapy is one of the main means of modern medicine, but different patients may have great differences in response to the same drug, although there are currently some AI-based drug recommendation systems, these systems often ignore the individual differences of patients, and cannot provide doctors with comprehensive and accurate diagnosis basis and treatment recommendations; 3. Although virtual reality (VR) and augmented reality (AR) technologies have been used for surgical training and planning, existing surgical simulation systems still have a lot of room for improvement in terms of realism, force feedback, and seamless integration with actual surgical environments.

[0004] Therefore, it is necessary to propose an intelligent medical diagnosis engine based on an AI large model to solve the above problems. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides an intelligent medical diagnosis engine based on an AI large model, which solves the problems of existing intelligent medical diagnosis systems that are prone to misdiagnosis, cannot provide doctors with accurate diagnosis basis and treatment recommendations, and surgical simulation systems have poor realism.

[0006] To achieve the above purpose, the present application is realized by the following technical scheme:

[0007] An intelligent medical diagnosis engine based on an AI large model, the medical diagnosis engine comprising:

[0008] The auxiliary diagnosis module collects patient electronic medical record data (EMR), medical image data, real-time monitoring data and genomic data, and pre-processes and extracts features from the data. The learning model is trained on the extracted data set, and the integrated learning method is used to predict the results.

[0009] The medical literature assistant: doctors query relevant medical literature and research results through voice or text input, recommend treatment plans or suggest diagnostic tests.

[0010] The drug recommendation module uses AI large models to analyze drug recommendation data, including compound structure, biological activity, and clinical trial results. Through deep learning algorithms, it predicts the potential efficacy and side effects of new drugs, providing valuable reference information for drug recommendations. At the same time, combined with the patient's genetic data, it recommends drug treatment plans for patients.

[0011] The surgery simulation module combines medical image data and surgical operation knowledge, and uses virtual reality technology to build a surgical environment and human model. Doctors can practice surgery in a virtual environment to improve their surgical skills and ability to handle complex surgical situations. At the same time, the system can provide real-time feedback and suggestions based on the doctor's operation performance to help doctors continuously improve their surgical skills.

[0012] Optionally, the electronic medical record data (EMR) in the auxiliary diagnosis module includes patient history, diagnosis records, treatment process and laboratory test results. Medical image data includes X-ray, CT scan, and magnetic resonance imaging (MRI). Real-time monitoring data includes heart rate, blood pressure, and blood glucose physiological indicators. Genomic data includes patient genetic information.

[0013] The pre-processing includes removing noise data, filling missing values and standardizing data format. Feature extraction includes extracting partial features from original data.

[0014] Optionally, the specific unit architecture of the medical literature assistant includes:

[0015] Database: includes medical literature, disease database and patient medical record information data. Data is collected and processed through the data interface layer to provide a basis for subsequent analysis and matching.

[0016] Auxiliary model unit: use natural language processing technology to analyze text, extract text keywords and sentences, and match them with existing medical knowledge base. At the same time, use machine learning algorithms to analyze large-scale case databases to find similarities and rules between patients, and display similar medical literature and research results.

[0017] Intelligent recommendation unit: the system will recommend treatment plans or suggest further diagnosis and examination for patients based on medical guidelines and practical experience.

[0018] Optionally, the specific unit architecture of the drug recommendation module includes:

[0019] Data acquisition unit: responsible for collecting and organizing multi-source heterogeneous data and gene expression data related to drug recommendation, including compound, drug target, biological activity, clinical trial data, compound database contains the chemical structure, biological activity and pharmacological characteristics information of known compounds, gene expression data includes patient's gene expression profile, which helps personalized medicine, clinical trial data includes the results of historical clinical trials and patient response data;

[0020] Data processing unit: clean, deduplicate and standardize the data, convert it into the format input by the machine learning model, use cheminformatics methods to convert compound molecules into computer recognizable feature vectors, including compound molecular fingerprint, gene mutation site, SMILES string, fingerprint descriptor;

[0021] Drug analysis unit: apply deep neural network (DNN) for unsupervised pre-training, learn the potential efficacy and side effects of compounds, combine network pharmacology model to predict the interaction between drugs and targets.

[0022] Optionally, the specific operation steps of the surgery simulation module are as follows:

[0023] S1: Data acquisition: including medical image data, surgical tools, human anatomy data and surgical operation data, data is collected and processed through data acquisition technology, providing basis for subsequent modeling and simulation;

[0024] S2: Model construction: use deep learning algorithm to analyze and process medical images to extract feature information, at the same time, combine virtual reality technology to build realistic surgical environment and human model, provide immersive surgical experience for learners, use computer graphics model and physical simulation technology to build three-dimensional model of operating room, surgical instruments, patient model and other elements, and simulate various mechanical responses and interactive behaviors in the surgical process;

[0025] S3: Application operation: apply the constructed virtual surgery scene to actual surgery training, doctors operate through virtual reality headsets and controller devices;

[0026] S4: Evaluation feedback: the system evaluates the operation ability of the operator in real time and provides feedback.

[0027] Optionally, the auxiliary diagnosis module and the drug recommendation module introduce a generative pre-training model (such as GPT) to capture long-distance dependencies in the input sequence, generate a weighted sum representation by calculating the similarity between query (Query), key (Key) and value (Value), and capture context information, each word has its corresponding query, key and value vectors, which are generated by linear transformation, and the relevance between each word and other words is obtained by calculating the dot product of query and key, and scaling and normalizing the result, to achieve complex natural language processing and deep text analysis capabilities, which will improve the accuracy and efficiency of the system in medical literature retrieval and personalized recommendation generation.

[0028] Optionally, the auxiliary diagnosis module and the drug recommendation module introduce a generative pre-training model to capture long-distance dependencies in the input sequence, generate a weighted sum representation by calculating the similarity between query (Query), key (Key) and value (Value), and capture context information, each word has its corresponding query, key and value vectors, which are generated by linear transformation, and the relevance between each word and other words is obtained by calculating the dot product of query and key, and scaling and normalizing the result, to achieve complex natural language processing and deep text analysis capabilities.

[0029] Optionally, self-supervised learning and transfer learning methods are applied to the processing of medical images and multi-modal data in the intelligent medical diagnosis engine, and the specific architecture of self-supervised learning is as follows:

[0030] Data preprocessing: cleaning, normalization and other operations are performed on the original data to make it suitable for model processing;

[0031] Feature learning: select a self-supervised learning strategy and train a self-supervised learning model;

[0032] The specific architecture of transfer learning is as follows:

[0033] Parameter transfer: directly apply the trained model parameters to the new task for fine-tuning;

[0034] Knowledge transfer: apply the trained model knowledge to the new task;

[0035] Structural transfer: apply the trained model structure to the new task;

[0036] Model transfer: apply the learned features to the model of the new task and fine-tune.

[0037] Optionally, the auxiliary diagnosis module and the drug recommendation module introduce a knowledge graph (KG), and the specific architecture and mathematical formula of the knowledge graph are as follows:

[0038] Data collection and preprocessing: Collect data from various sources, clean, de-duplicate, standardize, and perform other preprocessing operations on the data;

[0039] Entity recognition and relationship extraction: Use natural language processing (NLP) techniques to identify entities in the text and extract relationships between entities, forming triples;

[0040] Knowledge graph construction: Store the extracted entities and relationships in a graph database and construct a knowledge graph, which is usually represented as a directed graph where nodes represent entities and edges represent relationships between entities;

[0041] Inference and application: Use graph neural network (GNN) algorithms to perform inference on the knowledge graph, which can be used to discover new entity relationships, predict drug interactions, and assist in diagnosis.

[0042] Optionally, in the surgery simulation module and the auxiliary diagnosis module, multi-modal data fusion and cross-modal learning are introduced, and the specific architecture is as follows:

[0043] Data preprocessing layer:

[0044] Image data processing: Standardize medical images; Text data processing: Clean, segment, and remove stop words from electronic medical records and diagnostic report text data; Gene data processing: Perform quality control, mutation detection, and functional annotation analysis on gene sequence data to extract genetic information related to diseases;

[0045] Feature extraction and representation layer: Use deep learning models to extract features from different modalities of data, including visual features for images, semantic features for text, and biological features for gene data. Further process the features through feature transformation or dimensionality reduction techniques to reduce dimensions and retain key information;

[0046] Cross-modal fusion layer: Use early fusion, late fusion, or hybrid fusion strategies to fuse features from different modalities;

[0047] Decision and inference layer: Based on the fused feature vectors, use machine learning or deep learning models for decision-making and inference, outputting diagnosis results, surgery planning suggestions, or other related information.

[0048] Optionally, the medical diagnosis engine incorporates an explainable model (SHAP or LIME tool) to explain the prediction results of the model, and the specific architecture is as follows:

[0049] Data preparation layer: Collect and organize the data set that needs to be explained, including input features and target variables, and perform preprocessing on the data;

[0050] Model training layer: use the selected machine learning algorithm to train the model on the training set, evaluate the performance of the model, and ensure that it has good generalization ability on the test set;

[0051] Interpreter building layer: according to the selected interpretation tool (SHAP or LIME), build the corresponding interpreter object, for LIME, specify a subset of the training data as the local neighborhood, and train a simplified model within the neighborhood, for SHAP, select the appropriate interpreter according to the model type;

[0052] Explanation generation layer: use the interpreter to explain specific instances or the entire dataset, generate the contribution or importance score of each feature, and visualize the explanation results;

[0053] Result analysis layer: analyze the explanation results, understand the decision-making process of the model and the importance of the features.

[0054] The present application provides an intelligent medical diagnosis engine based on AI large model, which has the following beneficial effects:

[0055] 1、The present application can quickly process and analyze a large amount of medical data through intelligent auxiliary diagnosis, shorten the diagnosis time, and the doctor can rely on the real-time reasoning and analysis results of the system during the diagnosis process, improve the work efficiency, reduce the waiting time of the patient, optimize the allocation of medical resources, reduce unnecessary invasive examination and misdiagnosis, and the accurate diagnosis suggestion provided by the system can reduce unnecessary medical intervention, thereby avoiding the waste of medical resources and improving the overall medical service quality.

[0056] 2、The drug recommendation of the present application can integrate the multi-omics data of the patient's gene, physiology and pathology, and provide personalized drug treatment plan for the patient, such as analyzing the genetic characteristics and clinical data of the patient, the drug recommendation can recommend the most suitable drug and dosage for the patient, improve the treatment effect, and the drug recommendation technology can provide comprehensive and accurate diagnosis basis and treatment suggestion for the doctor.

[0057] 3、The present application can provide doctors and medical students with a more realistic training experience by accurately simulating real surgery scenes, including mechanical and optical properties, simulation technology can significantly improve the mastery of surgical skills, and the highly realistic virtual surgery environment can help doctors familiarize with the use of various surgical tools and master the surgical process, doctors can better prepare for complex surgical situations, thereby reducing the risk in actual surgery. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 It is the running process schematic diagram of the intelligent medical diagnosis engine of the present application;

[0059] Figure 2 It is the running process schematic diagram of the surgery simulation module of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application. Embodiment 1

[0061] Please refer to FIG. 1, the intelligent medical diagnosis engine based on AI large model, the medical diagnosis engine includes:

[0062] The auxiliary diagnosis module collects patient electronic medical record data (EMR), medical image data, real-time monitoring data and genomic data, and pre-processes and extracts features from the data, trains the extracted data set using a learning model, and predicts the result using an ensemble learning method. The electronic medical record data (EMR) includes the patient's medical history, diagnosis record, treatment process and laboratory examination result, the medical image data includes X-ray film, CT scan, magnetic resonance imaging (MRI), the real-time monitoring data includes heart rate, blood pressure, blood glucose physiological index, and the genomic data includes the patient's genetic information. The pre-processing includes removing noise data, filling missing values and standardizing data format, and the feature extraction includes extracting part of the features from the original data, such as extracting the features of the lesion area from the medical image;

[0063] The learning model is a logistic regression model, and the logistic regression is a classification algorithm for binary classification problems, and its mathematical formula is:

[0064]

[0065] Where Y is the target variable (0 or 1), X is the feature variable, and β is the model parameter;

[0066] The ensemble learning method is a random forest, which makes the final prediction by constructing multiple decision trees and taking their average or majority vote, and its mathematical formula is:

[0067]

[0068] Where, is the final prediction result, N is the number of decision trees, and T i is the prediction result of the i-th decision tree;

[0069] The medical literature assistant: the doctor queries the relevant medical literature and research results through voice or text input, recommends the treatment scheme or suggests the diagnosis and examination;

[0070] Drug recommendation module: using AI large model to analyze drug recommendation data, including compound structure, biological activity, clinical trial results, through deep learning algorithm, to predict the potential efficacy and side effects of new drugs, and provide valuable reference information for drug recommendation, at the same time, combined with the gene data of patients, to recommend drug treatment plan for patients;

[0071] Surgery simulation module: combined with medical image data and surgical operation knowledge, combined with virtual reality technology to build surgical environment and human body model, doctors can practice surgery in virtual environment to improve surgical skills and ability to deal with complex surgical situations, at the same time, the system can provide real-time feedback and suggestions according to the operation performance of doctors to help doctors continuously improve surgical skills.

[0072] Assuming that there is an intelligent auxiliary diagnosis system based on convolutional neural network (CNN) for identifying tumors in medical images, first of all, we need to collect a large number of labeled medical image data as training set, then use these data to train a CNN model, in actual application, when doctors upload new medical images, the system will use the trained CNN model to infer, identify the tumor position and size in the image, and display the results in the form of heat map to the doctors, doctors can make more accurate diagnosis decision according to these information.

[0073] In this embodiment, through intelligent auxiliary diagnosis, using deep learning and big data analysis technology, it can identify the disease patterns and characteristics that are difficult to detect, through learning a large number of medical data, the system can provide more accurate diagnosis suggestions, significantly improve the accuracy of disease diagnosis;

[0074] It can quickly process and analyze a large amount of medical data, shorten the diagnosis time, doctors can rely on the real-time inference and analysis results of the system in the diagnosis process, improve work efficiency, reduce patient waiting time, and optimize medical resource allocation;

[0075] Through reducing unnecessary invasive examination and misdiagnosis, intelligent auxiliary diagnosis can help reduce medical cost, accurate diagnosis suggestions provided by the system can reduce unnecessary medical intervention, so as to avoid waste of medical resources and improve overall medical service quality. Embodiment 2

[0076] This embodiment is further optimized on the basis of embodiment 1, specifically, the specific unit framework of medical literature assistant includes:

[0077] Database: including medical literature, disease database and patient medical record information data, data is collected and processed through data interface layer, providing basis for subsequent analysis and matching;

[0078] The auxiliary model unit: using natural language processing technology, the voice description is analyzed, and the text keywords and sentences are extracted, and matched with the existing medical knowledge base, at the same time, combined with machine learning algorithm, the similarity and rule between patients are found out by analyzing large-scale case database, and similar medical literature and research results are displayed;

[0079] The intelligent recommendation unit: the system will recommend treatment plan or suggest further diagnosis and examination for patients according to medical guidelines and practical experience;

[0080] The natural language processing technology is word embedding technology, which is used to convert words into vector representation, specifically word2vec, and the optimization objective in the training process is mathematically represented as:

[0081]

[0082] Wherein, T is the total number of words in the corpus, c is the size of the context window, w t is the target word, w t+j is the context word, P （ w t+j∣ w t） is the probability of the context word appearing under the given target word;

[0083] The machine learning algorithm is support vector machine (SVM) classification algorithm, for linearly separable data, the goal of SVM is to find an optimal hyperplane, so that the distance from the hyperplane to the sample of different categories is maximized, and the optimization problem is mathematically represented as:

[0084] ,

[0085] ,

[0086] Wherein, w is the weight vector, b is the bias term, y i is the label of the ith sample, x i is the feature vector of the ith sample (i=1,2,...,n).

[0087] In this embodiment, by quickly querying relevant medical literature and information, doctors can quickly obtain the latest research results and clinical guidelines, so as to improve the accuracy and efficiency of diagnosis, the system can recommend the best treatment plan or suggest doctors to make further diagnosis and examination according to medical guidelines and practical experience, which is helpful to make more personalized and precise treatment plan, by analyzing large-scale case database, the system can find the similarity and rule between patients, assist doctors in diagnosis and treatment decision, so as to reduce the medical risk;

[0088] The intelligent medical literature assistant can be personalized according to the specific conditions of the patient, enabling doctors to develop surgical plans for specific patients in a simulated environment, and helping to achieve more precise and personalized medical services. Embodiment 3

[0089] This embodiment is further optimized on the basis of Embodiment 1, specifically, the specific unit framework of the drug recommendation module includes:

[0090] Data acquisition unit: responsible for collecting and organizing multi-source heterogeneous data and gene expression data related to drug recommendation, including compound, drug target, biological activity, clinical trial data, compound database contains chemical structure, biological activity and pharmacological characteristics information of known compounds, gene expression data includes patient's gene expression profile, which helps personalized medicine, clinical trial data includes historical clinical trial results and patient response data;

[0091] Data processing unit: clean, deduplicate and standardize the data, convert it into the format input by the machine learning model, use cheminformatics methods to convert compound molecules into computer recognizable feature vectors, including compound molecular fingerprint, gene mutation site, SMILES string, fingerprint descriptor;

[0092] Drug analysis unit: apply deep neural network (DNN) for unsupervised pre-training, learn the potential efficacy and side effects of compounds, combine network pharmacology model to predict the interaction between drugs and targets;

[0093] The network pharmacology model is the PageRank algorithm, which combines graph theory and network analysis technology to predict the interaction between drugs and targets. Its basic principle is to construct a drug-target interaction network and analyze the importance of nodes and edges in the network. Its mathematical formula is:

[0094]

[0095] Where PR(p i ) is the PageRank value of node p i , M(p i ) is the set of nodes pointing to p i , L(p j ) is the out-degree of node p j , d is the damping coefficient, and N is the total number of nodes.

[0096] Assuming there is a DNN-based intelligent drug recommendation assistant for predicting the potential efficacy and side effects of new drugs, first, a large amount of compound and clinical trial data needs to be collected as a training set, then a DNN model is trained using these data, in actual application, when researchers input new compound structures, the system uses the trained DNN model to infer, predict the potential efficacy and possible side effects of the compound, and display the results in the form of a report to the researchers, who can make more scientific and reasonable research and development decisions based on this information.

[0097] In this embodiment, drug recommendation can integrate patients' genetic, physiological and pathological multi-omics data to provide personalized drug treatment plans for patients, such as by analyzing patients' genetic characteristics and clinical data, drug recommendation can recommend the most suitable drugs and dosages for patients to improve treatment effectiveness;

[0098] Drug recommendation technology can provide doctors with comprehensive and accurate diagnostic basis and treatment recommendations to assist doctors in making more scientific decisions, such as by analyzing patients' medical records and medical imaging data, drug recommendation can assist doctors in developing personalized treatment plans, and by accurately predicting the efficacy and side effects of drugs, drug recommendation can help doctors avoid prescribing unnecessary drugs and reduce patients' burden;

[0099] Drug recommendation technology can help researchers discover new drug targets and provide new directions for drug research and development, drug recommendation can simulate the interaction between drugs and biological entities to discover potential drug combinations and provide a basis for combination therapy, for rare disease drug research and development, due to the small sample size of patients, traditional methods are difficult to conduct large-scale research, drug recommendation technology can accelerate the research and development process of rare disease drugs through simulation and prediction. Embodiment 4

[0100] Please refer to Figure 2, this embodiment is further optimized on the basis of embodiment 1, specifically, the specific operation steps of the surgery simulation module are as follows:

[0101] S1: Data acquisition: including medical image data, surgical tools, human anatomy data and surgical operation data, data is collected and processed through data acquisition technology to provide a basis for subsequent modeling and simulation;

[0102] S2: Model construction: using deep learning algorithms to analyze and process medical images to extract feature information, at the same time, combining virtual reality technology to build realistic surgical environment and human model to provide learners with immersive surgical experience, using computer graphics models and physical simulation technology to model the operating room, surgical instruments, patient models and other elements in three dimensions, and simulate various mechanical responses and interactive behaviors in the surgical process;

[0103] S3: Application operation: apply the built virtual surgery scene to actual surgery training, and the doctor operates through the virtual reality headset and controller device;

[0104] S4: Evaluation feedback: the system evaluates the operator's operation ability in real time and provides feedback;

[0105] In virtual surgery simulation, the deep learning algorithm is the core part. Taking a three-dimensional convolutional neural network as an example, its forward propagation formula can be expressed as:

[0106]

[0107] where a （l） is the activation value of the lth layer neuron, a （l−1） is the activation value of the l−1 layer neuron, W （l） is the weight matrix of the lth layer, b （l） is the bias vector of the lth layer, and g is the activation function.

[0108] The computer graphics model is the lighting model, specifically the Phong Illumination Model, whose formula is:

[0109]

[0110] where I is the final lighting intensity, Ia is the ambient light intensity, Ka is the ambient light reflection coefficient, Id is the diffuse reflection light intensity, Kd is the diffuse reflection coefficient, N is the normal vector, L is the light source direction vector, Is is the specular reflection light intensity, Ks is the specular reflection coefficient, R is the reflection vector, V is the view direction vector, and n is the highlight coefficient.

[0111] The physical simulation technology is the physical engine for physical simulation. The mechanical equation in physical simulation is represented as F=ma, where F is the force, m is the mass, and a is the acceleration. The motion state of the object under the force is calculated to simulate various physical phenomena in the surgery process.

[0112] In this embodiment, by accurately simulating the real surgery scene, including mechanical and optical properties, a more realistic training experience is provided for doctors and medical students. Simulation technology can significantly improve the mastery of surgical skills. The highly realistic virtual surgery environment allows doctors to practice a large number of surgical operations in the simulation, thereby improving their surgical skills. It helps doctors to familiarize themselves with the use of various surgical tools, master the surgical process, and accumulate valuable experience before operating on actual patients. Virtual surgery simulation allows doctors to try new surgical techniques and methods without risk. Through repeated practice in the simulation environment, doctors can better prepare for complex surgical situations, thereby reducing the risk in actual surgery.

[0113] Through virtual surgery simulation, medical institutions can provide more surgery training opportunities for doctors without increasing the actual surgery burden, which helps to optimize the allocation of medical resources and improve the efficiency of medical services. Embodiment 5

[0114] This embodiment is further optimized on the basis of Embodiment 1, specifically, a generative pre-training model (such as GPT) is introduced into the auxiliary diagnosis module and drug recommendation module to capture long-distance dependencies in the input sequence, and the framework of GPT is as follows:

[0115] Transformer architecture: GPT is based on the Transformer architecture, which is a deep learning model structure with powerful natural language processing capabilities. The core of Transformer is the self-attention mechanism (Self-Attention), which can capture long-distance dependencies in the input sequence.

[0116] Self-attention mechanism: The self-attention mechanism calculates the similarity between the query (Query), key (Key), and value (Value) to generate a weighted sum representation to capture context information. Specifically, each word has its corresponding query, key, and value vectors, which are generated through linear transformation. By calculating the dot product of the query and the key, and scaling and normalizing the result, the relevance (i.e., attention weight) between each word and other words is obtained.

[0117] Pre-training and fine-tuning: The training of the GPT model is divided into two stages: pre-training and fine-tuning. In the pre-training stage, the language model is trained on large-scale text data, with the goal of maximizing the conditional probability P(wt∣w1, w2,..., wt−1). In the fine-tuning stage, the pre-trained model is supervised learning using task-specific data to improve the model's performance on that task.

[0118] In this embodiment, by introducing a generative pre-training model, more complex natural language processing and deep text analysis capabilities are achieved, which will improve the accuracy and efficiency of the system in medical literature retrieval and personalized suggestion generation. The generative pre-training model can quickly generate high-quality diagnosis reports or suggestions, greatly reducing the time for doctors to write medical records or diagnosis reports, which is of great significance to improve the overall work efficiency of the hospital and patient satisfaction. Through learning a large amount of medical literature, medical record data, the generative pre-training model can capture complex medical knowledge and patterns, thereby providing more accurate diagnosis suggestions for doctors, which helps to reduce the risk of misdiagnosis and missed diagnosis, and improve the quality of medical services. Embodiment 6

[0119] This embodiment is further optimized on the basis of Embodiment 1, specifically, self-supervised learning and transfer learning methods are applied to the processing of medical images and multi-modal data in the intelligent medical diagnosis engine, reducing the dependence on labeled data, while enhancing the adaptability of the model in new fields and improving data utilization efficiency;

[0120] Self-supervised learning refers to the process in which the model learns features and patterns from unlabeled data during training, thereby training. It utilizes the structure and relationship of the data itself, and converts unlabeled data into labeled data through a predefined function to realize the training of the model.

[0121] The specific architecture of self-supervised learning is as follows:

[0122] Data preprocessing: cleaning, normalizing, and other operations are performed on the original data to make it suitable for model processing.

[0123] Feature learning: select a self-supervised learning strategy (such as generative adversarial networks GANs, variational autoencoders VAEs, self-supervised pre-training), and train a self-supervised learning model.

[0124] Transfer learning refers to the application of knowledge from a trained model to different but related fields or problems to improve the performance of new tasks.

[0125] The specific architecture of transfer learning is as follows:

[0126] Parameter transfer: directly apply the trained model parameters to the new task for fine-tuning.

[0127] Knowledge transfer: apply the trained model knowledge (such as features, rules) to the new task.

[0128] Structural transfer: apply the trained model structure (such as the number of layers, connection method) to the new task.

[0129] Model transfer: apply the learned features to the model of the new task and fine-tune.

[0130] In this embodiment, the annotation cost of medical images and multi-modal data is high and time-consuming, as professional medical knowledge is required to ensure the accuracy of the annotation. Self-supervised learning pre-trains the model by constructing a series of self-supervised tasks, allowing the model to learn features and patterns from unlabeled data, thereby reducing the dependence on labeled data. Self-supervised learning enables the model to be pre-trained on a large amount of unlabeled data, extracting more useful features that can be transferred and applied in different medical tasks, improving the generalization ability of the model in new tasks.

[0131] Self-supervised learning and transfer learning methods can help models capture the latest medical research findings and clinical practice guidelines, providing timely and effective knowledge support for doctors, which helps to promote the inheritance and update of medical knowledge and improve the overall level of the medical industry. Through automatic processing of large amounts of medical images and multi-modal data, self-supervised learning and transfer learning methods can reduce the workload of doctors and reduce labor costs. Embodiment 7

[0132] This embodiment is a further optimization based on Embodiment 1, specifically, in the medical auxiliary diagnosis module and drug recommendation module, a knowledge graph (KG) is introduced, which can significantly improve the intelligent level of the system by enhancing the reasoning ability. The knowledge graph can help the system to associate and reason when analyzing patient data and related medical knowledge, providing more in-depth decision support. The specific architecture and mathematical formula of the knowledge graph are as follows:

[0133] Data collection and preprocessing: Collect data from various data sources (such as medical literature, medical records, drug databases), and perform cleaning, deduplication, standardization, and other preprocessing operations on the data;

[0134] Entity recognition and relationship extraction: Use natural language processing (NLP) techniques to identify entities (such as diseases, symptoms, drugs, genes) in text and extract relationships between entities to form triples (entity-relation-entity);

[0135] Knowledge graph construction: Store the extracted entities and relationships in a graph database and construct a knowledge graph, which is usually represented as a directed graph, where nodes represent entities and edges represent relationships between entities;

[0136] Inference and application: Use graph neural network (GNN) algorithms to perform reasoning on the knowledge graph, which can be used to discover new entity relationships, predict drug interactions, and assist in diagnosis;

[0137] In the knowledge graph, the reasoning process often involves graph embedding learning, which converts nodes and edges in the graph into low-dimensional vectors for processing by machine learning algorithms. Graph embedding learning methods and their mathematical formulas:

[0138] TransE model: TransE model is one of the classic methods of knowledge graph embedding, which assumes that the difference between the head entity vector and the tail entity vector is equal to the relationship vector, h + r ≈ t, where h, r, t represent the vector representation of the head entity, relationship, and tail entity, respectively;

[0139] TransH model: TransH model is another classic method of knowledge graph embedding, which assumes that the relationship vector is a linear combination of the head entity vector and the tail entity vector, h + r ≈ t, where h, r, t represent the vector representation of the head entity, relationship, and tail entity, respectively.ComplEx model: The ComplEx model is a complex version of the TransE model that considers more complex relationship patterns, h1+r-h2≈t, where h1 and h2 represent two different head entity vectors.

[0140] In this embodiment, the knowledge graph integrates a large amount of medical knowledge and data to provide doctors with fast and accurate information retrieval and decision support, thereby shortening the diagnosis time. In image diagnosis, the knowledge graph can assist doctors in identifying abnormal patterns in images, improving the accuracy and efficiency of diagnosis. The knowledge graph contains rich medical entities and relationships, which can help doctors capture complex medical knowledge and patterns, reducing the risk of misdiagnosis and missed diagnosis.

[0141] Through reasoning and analysis of the information in the knowledge graph, doctors can obtain a more comprehensive assessment of the patient's condition, thereby making more accurate diagnosis decisions. The knowledge graph can generate personalized treatment plans or recommendations based on the patient's medical history, test results, and other information, helping doctors better understand the patient's condition and make more reasonable treatment decisions. In drug recommendation, the knowledge graph can combine the patient's specific condition and drug characteristics to recommend the most suitable drug treatment plan, improving treatment effectiveness. Embodiment 8

[0142] This embodiment is further optimized based on Embodiment 1, specifically, in the surgery simulation module and auxiliary diagnosis module, multi-modal data (such as medical images, text, and genetic data) fusion and cross-modal learning are introduced, the specific architecture and mathematical formula are as follows:

[0143] Data preprocessing layer:

[0144] Image data processing: Standardize medical images, including normalizing pixel values, adjusting contrast and brightness, and involving image segmentation and feature extraction steps; Text data processing: Clean, segment, and remove stop words from electronic medical records and diagnostic report text data, and extract key features; Genetic data processing: Perform quality control, mutation detection, and functional annotation analysis on genetic sequence data to extract disease-related genetic information.

[0145] Feature extraction and representation layer:

[0146] Use deep learning models (such as convolutional neural networks CNN for images, recurrent neural networks RNN or Transformer for text) to extract features from different modalities of data, these features are visual features of images, semantic features of text, and biological features of genetic data, further process the features through feature transformation or dimensionality reduction techniques (such as principal component analysis PCA, autoencoder AE) to reduce dimensions and retain key information.

[0147] Cross-modal fusion layer:

[0148] Early fusion, late fusion or hybrid fusion strategy is adopted to fuse features of different modalities, early fusion: fusion is performed immediately after feature extraction to form a unified feature vector, for example, fusion is performed at the feature level, features are extracted from each modality first, then fusion is performed, late fusion: after feature extraction and modeling of each modality, the results are fused, for example, different modalities are processed separately, then the results are fused, hybrid fusion: combining the advantages of early fusion and late fusion, fusion is performed at different levels, for example, feature extraction based on single modality is performed first, then fusion is performed at a higher level, using specific fusion algorithms (such as attention mechanism, bidirectional multi-modal attention mechanism) to discover and encode the mutual relationship between different modalities;

[0149] Decision and reasoning layer:

[0150] Based on the fused feature vector, machine learning or deep learning models (such as support vector machine SVM, random forest RF, neural network NN) are used for decision and reasoning, outputting diagnosis results, surgery planning suggestions or other related information;

[0151] The following are some common mathematical concepts and formulae in the application of cross-modal learning:

[0152] Feature extraction: for image data, convolutional neural network (CNN) is used for feature extraction, its mathematical expression usually involves convolution operation, pooling operation; for text data, word embedding (such as Word2Vec, BERT) can be used to convert text into vector representation;

[0153] Feature fusion: in early fusion, different modalities of feature vectors are directly spliced or weighted summed, in late fusion, integrated learning methods (such as weighted average, voting) are used to combine the outputs of different models.

[0154] In this embodiment, multi-modal data fusion can comprehensively consider various information of patients, such as medical images, electronic medical records, genetic data, etc., so as to provide more comprehensive and accurate diagnosis results, for example, by integrating CT images and electronic medical record data, pulmonary embolism can be more accurately identified and common causes of acute respiratory failure can be distinguished; in surgery simulation and auxiliary diagnosis, doctors need to make decisions based on a large amount of complex information, multi-modal data fusion can provide more rich context information to help doctors make more reasonable and personalized treatment decisions. Embodiment 9

[0155] This embodiment is further optimized based on Embodiment 1, specifically, a reinforcement learning algorithm is applied in the drug recommendation module and the surgery simulation module, the framework and mathematical formula of the reinforcement learning algorithm are as follows:

[0156] Environment definition: In the drug recommendation module, the environment includes the patient's electronic health record (EHR), current symptoms, drug inventory, in the surgery simulation module, the environment involves the patient's specific physiological state, the state of surgical equipment, the environmental conditions of the operating room;

[0157] Agent: Agent is the core of reinforcement learning algorithm, it makes decisions according to the observed environment state, and executes the corresponding action, in the drug recommendation module, the agent is a drug recommendation system, it recommends appropriate drugs and dosage according to the patient's condition and historical medication data, in the surgery simulation module, the agent is a virtual surgeon or surgery planning system, it formulates surgery plan according to the surgery environment and patient state;

[0158] Action space: Action space is the set of all possible actions that the agent can execute, in the drug recommendation module, the action includes selecting a specific drug, adjusting the drug dosage, adding or deleting a drug, in the surgery simulation module, the action includes executing a specific surgical step, adjusting the parameters of surgical equipment, changing the surgical strategy;

[0159] Reward function: Reward function is used to evaluate the good or bad of the action taken by the agent, and gives the agent corresponding rewards or punishments according to the evaluation results; in the drug recommendation module, the reward function is designed based on the efficacy, side effects, cost and other factors of the drug, in the surgery simulation module, the reward function is designed based on the success rate of surgery, postoperative recovery, surgery time and other factors;

[0160] Learning process: Agent learns how to make better decisions by interacting with the environment, it will update its internal strategy according to the observed environment state and reward function, so as to make better decisions when encountering similar situations in the future.

[0161] In this embodiment, the reinforcement learning algorithm can develop personalized treatment strategies according to the individual characteristics and condition of the patient, through learning historical cases and clinical trial data, the algorithm can predict the effect of different treatment schemes, and guide the doctor to choose the best treatment scheme;

[0162] Reinforcement learning algorithms can optimize surgical procedures, improve surgical success rates, and learn and improve surgical strategies by simulating surgical processes, reducing surgical time and complications; through precise drug recommendations, unnecessary drug use can be reduced, and drug treatment costs can be reduced, while optimizing treatment plans can reduce hospital stays and medical costs, optimizing surgical processes and reducing surgical time can reduce surgical costs and improve operating room utilization;

[0163] Personalized drug recommendations can improve treatment outcomes and reduce side effects. By learning from a large number of medical records, the algorithm can provide more accurate drug combinations and dosage recommendations. Through simulation and optimization of surgical processes, reinforcement learning algorithms can help doctors choose the best surgical plan, improve surgical success rates, and improve postoperative recovery outcomes.

[0164] Example 10

[0165] This embodiment is further optimized based on Example 1, specifically, an explainable model (SHAP or LIME tool) is added to the medical diagnosis engine to explain the prediction results of the model, the specific architecture and mathematical formula are as follows:

[0166] Data preparation layer: collect and organize the data set that needs to be explained, including input features and target variables, and preprocess the data such as cleaning, standardization, and normalization;

[0167] Model training layer: use selected machine learning algorithms (such as random forest, neural network) to train the model on the training set, evaluate the performance of the model, and ensure its good generalization ability on the test set;

[0168] Interpreter construction layer: according to the selected explainable tool (SHAP or LIME), construct the corresponding interpreter object, for LIME, specify a subset of training data as the local neighborhood and train a simplified model within the neighborhood, for SHAP, select the appropriate interpreter according to the model type, such as TreeExplainer for tree models;

[0169] Explanation generation layer: use the interpreter to explain specific instances or the entire data set, generate the contribution or importance score of each feature, and visualize the explanation results, such as feature importance charts, SHAP value charts;

[0170] Result analysis layer: analyze the explanation results, understand the decision-making process of the model and the importance of the features.

[0171] In this embodiment, by explaining the prediction results of the model, doctors and patients can better understand how the model works, thereby increasing the trust in the model, and the transparency helps to ensure the fairness and rationality of medical decisions, especially when sensitive or critical decisions are involved;Tools such as LIME and SHAP can reveal the contribution of each feature to the model's prediction, helping doctors understand which factors have a significant impact on the diagnosis result.

[0172] By explaining the prediction results of the model, doctors can better understand the complexity and diversity of diseases, thereby promoting the inheritance and update of medical knowledge, and doctors can formulate more personalized treatment plans based on the explanation results of the model combined with their professional knowledge and experience.

[0173] The application further discloses an online service device based on an AI large model, which comprises a processor, an input device, an output device and a memory, and the processor, the input device, the output device and the memory are connected with each other, wherein the memory is used for storing a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions and execute the method of any one of the above-mentioned embodiments.

[0174] The application further discloses a computer readable storage medium, and the computer storage medium stores a computer program, the computer program comprises program instructions, and the program instructions make the processor execute the method of any one of the above-mentioned embodiments when the processor executes the program instructions.

[0175] In summary: the application can quickly process and analyze a large amount of medical data through intelligent auxiliary diagnosis, shorten the diagnosis time, doctors can rely on the real-time reasoning and analysis results of the system during the diagnosis process, improve the work efficiency, reduce the waiting time of patients, optimize the allocation of medical resources, reduce unnecessary invasive examination and misdiagnosis through the application, the precise diagnosis suggestion provided by the system can reduce unnecessary medical intervention, thereby avoiding the waste of medical resources, and improving the overall medical service quality;

[0176] Drug recommendation can integrate multi-omics data such as patient's gene, physiology and pathology, and provide personalized drug treatment plan for patients, such as by analyzing the genetic characteristics and clinical data of patients, the drug recommendation can recommend the most suitable drug and dosage for patients, improve the treatment effect, and the drug recommendation technology can provide comprehensive and accurate diagnosis basis and treatment suggestion for doctors, and assist doctors to make more scientific decisions;

[0177] By accurately simulating real surgery scenes, including mechanical and optical properties, the simulation technology can significantly improve the mastery of surgical skills, and the highly realistic virtual surgery environment can enable doctors to practice a large number of surgical operations in the simulation, thereby improving surgical skills, helping doctors to be familiar with the use of various surgical tools and master the surgical process, and doctors can better prepare for complex surgical situations, thereby reducing the risk in actual surgery.

[0178] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will fall within the protection scope of the present application.

Claims

1. An AI large model-based intelligent medical diagnosis engine, characterized in that, The medical diagnosis engine comprises: an auxiliary diagnosis module: collecting patient electronic medical record data, medical image data, real-time monitoring data and genomic data, and pre-processing and feature extraction of the data, training the extracted data set by using a learning model, and predicting the result by using an integrated learning method; a medical literature assistant: doctors query relevant medical literature and research results through voice or text input, recommend treatment plans or suggest diagnostic tests; the drug recommendation module uses an AI large model to analyze drug recommendation data, including compound structure, biological activity, and clinical trial results, predicts the potential efficacy and side effects of new drugs through a deep learning algorithm, and recommends drug treatment plans for patients in combination with patient genetic data; the surgical simulation module: combines medical image data and surgical operation knowledge, and constructs a surgical environment and a human body model by combining virtual reality technology, doctors perform surgical practice in the virtual environment, and the system can provide real-time feedback and suggestions according to the operation performance of the doctors; The specific unit architecture of the drug recommendation module further comprises: a data acquisition unit: responsible for collecting and organizing multi-source heterogeneous data and gene expression data related to drug recommendation; a drug analysis unit: applying a deep neural network for unsupervised pre-training to learn the potential efficacy and side effects of compounds, and combining a network pharmacology model to predict the interaction between drugs and targets, the network pharmacology model is a PageRank algorithm, and the mathematical formula is: ； Wherein, PR(pi) is the PageRank value of node pi, M(pi) is the node set pointing to pi, L(pj) is the out-degree of node pj, d is the damping coefficient, and N is the total number of nodes; A generative pre-training model is introduced in the auxiliary diagnosis module and the drug recommendation module to capture long-distance dependencies in the input sequence, generate a weighted sum representation by calculating the similarity between queries (Query), keys (Key) and values (Value) to capture context information, each word has its corresponding Query, Key and Value vector, which is generated by linear transformation, the relevance between each word and other words is obtained by calculating the dot product of Query and Key and scaling and normalizing the result, to realize complex natural language processing and deep text analysis capabilities; In the auxiliary diagnosis module and the drug recommendation module, a knowledge graph (KG) is introduced; Graph neural network (GNN) algorithm is used for reasoning on the knowledge graph, which can be used to discover new entity relationships, predict drug interactions and assist in diagnosis; In the surgical simulation module, data acquisition technology is used to collect and process medical image data, surgical tools, human anatomy data and surgical operation data; deep learning algorithms are used to analyze and process medical images to extract feature information, and virtual reality technology is used to construct realistic surgical environments and human body models; computer graphics models and physical simulation technology are used to model elements including operating rooms, surgical instruments and patient models in three dimensions, and simulate various mechanical responses and interactive behaviors during the surgical process; the system evaluates the operation ability of the operator in real time and provides feedback; In the surgical simulation module and the auxiliary diagnosis module, the genetic sequence data is subjected to quality control, variation detection and functional annotation analysis to extract genetic information related to diseases; deep learning models are used to extract features from data in different modalities, which are visual features of images, semantic features of texts and biological features of genetic data; the features are further processed by dimension reduction techniques to reduce dimensions and retain key information; a hybrid fusion strategy is used to fuse features in different modalities; based on the fused feature vectors, deep learning models are used for decision-making and reasoning to output diagnosis results and surgical planning suggestions. 2.The AI large model-based intelligent medical diagnosis engine according to claim 1, characterized in that: The electronic medical record data in the auxiliary diagnosis module includes the patient's medical history, diagnosis record, treatment process and laboratory examination results; the medical image data includes X-ray film, CT scan, nuclear magnetic resonance imaging, real-time monitoring data includes heart rate, blood pressure, blood glucose physiological indicators; the genomic data includes the patient's genetic information; the preprocessing includes removing noise data, filling missing values and standardizing data format, and the feature extraction includes extracting part of the features from the original data. 3.The AI large model-based intelligent medical diagnosis engine according to claim 1, characterized in that: The specific unit architecture of the medical literature assistant includes: a database including medical literature, disease database and patient medical record information data, the data is collected and processed through a data interface layer to provide a basis for subsequent analysis and matching; an auxiliary model unit: using natural language processing technology to analyze the text, extract the text keywords and sentences, and match them with the existing medical knowledge base, at the same time, combining machine learning algorithm to analyze large-scale case database, find the similarity and regularity between patients, and display similar medical literature and research results; an intelligent recommendation unit: the system will recommend treatment plan or suggest further diagnosis and examination for patients according to medical guidelines and practical experience. 4.The AI large model-based intelligent medical diagnosis engine according to claim 1, characterized in that: The specific unit architecture of the drug recommendation module also includes: a data acquisition unit: responsible for collecting and organizing multi-source heterogeneous data and gene expression data related to drug recommendation, including compounds, drug targets, biological activity, clinical trial data, compound database contains chemical structure, biological activity and pharmacological property information of known compounds, gene expression data includes patient's gene expression profile, clinical trial data includes historical clinical trial results and patient response data; a data processing unit: cleaning, deduplication, standardization processing of data, converting to the format input by machine learning model, using cheminformatics method to convert compound molecules into computer recognizable feature vectors, including compound molecular fingerprint, gene mutation site, SMILES string, fingerprint descriptor; a drug analysis unit: applying deep neural network for unsupervised pre-training to learn the potential efficacy and side effects of compounds, combining network pharmacology model to predict the interaction between drugs and targets. 5.The AI large model-based intelligent medical diagnosis engine according to claim 1, characterized in that: Self-supervised learning and transfer learning methods are applied to the processing of medical images and multi-modal data in the intelligent medical diagnosis engine. The specific architecture of self-supervised learning: data preprocessing: cleaning and normalizing the original data to make it suitable for model processing; Feature learning: Choose a self-supervised learning strategy and train a self-supervised learning model; Specific architecture of transfer learning: Parameter transfer: directly apply the trained model parameters to the new task and fine-tune; Knowledge Transfer: apply the knowledge of the trained model to the new task; Structural transfer: apply the structure of the trained model to the new task; Model transfer: apply the learned features to the model of the new task and fine-tune. 6.The AI large model-based intelligent medical diagnosis engine according to claim 1, characterized in that: In the auxiliary diagnosis module and drug recommendation module, introduce knowledge graph (KG), the specific architecture of knowledge graph is as follows: data collection and preprocessing: collect data from multiple data sources, clean, remove duplicates, and standardize the data; entity recognition and relationship extraction: use natural language processing (NLP) technology to identify entities in the text and extract relationships between entities to form triples; knowledge graph construction: store the extracted entities and relationships in a graph database to construct a knowledge graph, which is usually represented as a directed graph, where nodes represent entities and edges represent relationships between entities; reasoning and application: use graph neural network (GNN) algorithms to reason on the knowledge graph, reasoning can be used to discover new entity relationships, predict drug interactions, and assist in diagnosis. 7.The AI large model-based intelligent medical diagnosis engine according to claim 1, characterized in that: Add an explainable model to the medical diagnosis engine to explain the prediction results of the model through SHAP or LIME tools, the specific architecture is as follows: data preparation layer: collect and organize the data set that needs to be explained, including input features and target variables, and preprocess the data; model training layer: use the selected machine learning algorithm to train the model on the training set, evaluate the performance of the model, and ensure its good generalization ability on the test set; interpreter construction layer: according to the selected explanation tool, construct the corresponding interpreter object, for LIME, specify a subset of the training data as the local neighborhood and train a simplified model within the neighborhood, for SHAP, select the appropriate interpreter according to the model type; explanation generation layer: use the interpreter to explain specific instances or the entire data set, generate the contribution or importance score of each feature, and visualize the explanation results; result analysis layer: analyze the explanation results, understand the decision-making process of the model and the importance of the features.

Images