Hospital information intelligent analysis and decision-making system based on multi-modal large model

Abstract

The invention discloses a hospital information intelligent analysis and decision-making system based on a multi-modal large model, and relates to the technical field of hospital information analysis and decision-making. The system comprises a data acquisition module, a preprocessing and fusion module, a large model construction and training module, an intelligent analysis module, a decision support module, a knowledge graph construction and application module, a data security and privacy protection module, a system evaluation and optimization module, a multi-hospital cooperation and data sharing module, a mobile application module and the like, and all the modules cooperate to realize hospital information intelligent processing and decision making. The system integrates multi-modal data, assists in precise diagnosis, recommends a treatment scheme, predicts resource demands, monitors medical quality, assists medical research, manages patient health and the like, comprehensively improves the intelligent level of hospitals, optimizes medical services and benefits doctors and patients.

Description

Technical Field

[0001] The present invention relates to the technical field of information analysis and decision-making systems, and in particular to a hospital information intelligent analysis and decision-making system based on a multimodal large model. Background Art

[0002] In the medical field, with the rapid development of information technology, hospitals have accumulated vast amounts of information, covering patient medical records, medical images, test reports, and more. However, traditional hospital information systems are often independently developed by various departments, resulting in fragmented data and a lack of effective integration. For example, when making a diagnosis, doctors must switch between different systems to search for various patient information, which consumes a significant amount of time and effort, making comprehensive analysis difficult. This can easily lead to delayed or inaccurate diagnoses, compromising patient treatment outcomes.

[0003] Medical data is multimodal, containing rich information across diverse data types, including text, images, and numerical values. However, existing analytical methods are often limited to processing data in a single modality. For example, diagnosing a disease solely based on medical records ignores key pathological information that may be present in images. Alternatively, analyzing images alone without integrating the patient's medical history and test results can lead to one-sided analysis results, failing to fully tap the data's value and meeting the demands of precision medicine and scientific decision-making in complex medical scenarios.

[0004] Hospitals also face numerous challenges in their daily operations and management. In terms of resource allocation, there is a lack of accurate forecasting methods, which often leads to irrational allocation of resources such as beds, medicines, and medical equipment. For example, during the peak influenza season, hospitals are unable to accurately estimate the number of patients in advance, resulting in bed shortages and affecting patients' medical experience. In terms of medical quality monitoring, traditional methods rely on manual regular statistical analysis, which is inefficient and prone to omissions, making it difficult to detect potential medical quality issues in a timely manner and intervene. In addition, in terms of medical research and patient health management, existing systems are unable to provide comprehensive and intelligent support, which restricts the innovative development of medical technology and the effectiveness of long-term patient health management. Therefore, there is an urgent need for a hospital information system that can integrate multimodal data and realize intelligent analysis and decision-making to improve the level of medical services and optimize hospital operations and management. Summary of the Invention

[0005] The present invention proposes a hospital information intelligent analysis and decision-making system based on a multimodal large model to solve the problems mentioned in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a hospital information intelligent analysis and decision-making system based on a multimodal large model, comprising:

[0007] Data acquisition module: This module collects data from various hospital information systems, including text data in the electronic medical record system, medical image data in the image archiving and communication system, and test numerical data in the laboratory information system. For text data, relevant records in the electronic medical record system are captured in real time through the interface. For medical image data, the module uses the DICOM standard protocol to obtain data from imaging devices, and performs preliminary format conversion and compression processing on the images to save storage space. The compression algorithm uses JPEG-XR. Test numerical data is connected to the database, and the results in the laboratory information system are extracted regularly. The data collection frequency is set according to the data type. Text data is updated every hour, image data is collected after the examination is completed, and test numerical data is obtained in real time after the report is generated.

[0008] Data preprocessing and fusion module: Preprocess the collected data of different modalities. For text data, first clean the text to remove characters and stop words, and use the stemming and part-of-speech tagging methods in the NLTK natural language toolkit to convert the text into a form that is easy to analyze. For medical image data, use the image enhancement algorithm to improve the contrast and clarity of the image. The formula is: Where s is the pixel value after transformation, L is the total number of gray levels of the image, n is the total number of pixels in the image, and n k is the number of occurrences of gray level k, j is the gray level of the current pixel; the test numerical data is normalized using the Z-score normalization formula: Where x is the original data, μ is the data mean, and σ is the standard deviation; then, a feature fusion model is constructed to fuse the preprocessed different modal data, and a fusion method based on the attention mechanism is adopted to assign different weights to different modal data to highlight information.

[0009] Furthermore, it also includes:

[0010] Model building and training module: A Transformer-based model is constructed, including a text encoder, an image encoder, and a numerical encoder, which perform feature extraction on text, images, and test numerical data, respectively. The text encoder uses the BERT structure, the image encoder uses the VisionTransformer structure, and the numerical encoder is designed as a fully connected neural network structure. During the training process, the hospital's historical medical records, images, and test results are used, combined with contrastive learning and self-supervised learning methods, to enable the model to learn the associations and feature representations between different modal data. Contrastive learning optimizes the model by maximizing the similarity between positive samples and minimizing the similarity between negative samples. The formula is: Where sim is the similarity function, is a positive sample pair, is a negative sample pair, τ is a temperature hyperparameter, and N is the number of negative samples. Self-supervised learning enables the model to learn the intrinsic characteristics of the data without manual annotation by designing mask language modeling and image filling self-supervision tasks.

[0011] Analysis module: Analyzes the fused data through the trained model. In terms of disease diagnosis assistance, the patient's data is input, and the model outputs the predicted probability distribution of the disease. The model output is converted into a probability value through the Softmax function. The formula is: Where z is the model output vector, j is the category index, and K is the total number of categories. Regarding treatment plan recommendations, the model retrieves and matches recommended treatment plans based on the patient's disease diagnosis results and historical treatment history. It also provides efficacy predictions and risk assessments for these plans. Regarding medical resource demand forecasting, the model combines the hospital's historical visit data and seasonal factors, using a combination of time series analysis and machine learning to predict future bed demand, drug consumption, and medical equipment usage frequency across different departments, providing a basis for resource allocation in the hospital.

[0012] Decision support module: The results of the analysis module are presented to hospital managers and medical staff through a visual interface. For disease diagnosis auxiliary results, the probability of various diseases is displayed in the form of a probability bar chart. Treatment plan recommendations are presented in the form of a list, and each plan is accompanied by a description of the efficacy and risks. The results of the medical resource demand forecast are displayed in a combination of line charts and bar charts to show the resource demand situation at different time points. At the same time, the system provides decision-making recommendations for doctors to examine and take treatment measures. Based on the medical resource demand forecast, managers are advised to adjust bed allocation, purchase medicines, and maintain medical equipment in advance.

[0013] Knowledge graph construction and application module: Construct a medical knowledge graph based on hospital data; the nodes in the graph include diseases, symptoms, treatment methods, drugs, and medical devices, and the edges represent the relationships between them. During the construction process, natural language processing technology is used to extract entities and relationships from text data, identify the connection between disease characteristics and related entities from imaging data, and mine knowledge related to disease diagnosis and treatment from test numerical data; the knowledge graph is used for auxiliary diagnosis. When a patient's symptoms are input, the system retrieves diseases and diagnostic methods through the knowledge graph; in the formulation of treatment plans, reasonable treatment pathways and drug combinations are recommended based on the knowledge graph to improve the standardization of treatment;

[0014] Data security and privacy protection module: This module uses technology to ensure data security and patient privacy. For data storage, the AES encryption algorithm is used for patient personal information in electronic medical records. SSL / TLS protocols are used to encrypt data transmission during data transmission. Different permissions are assigned to different users through access control mechanisms, enabling authorized users to access required data.

[0015] System evaluation and optimization module: Establish a system evaluation indicator system, including diagnostic accuracy, satisfaction with treatment plan recommendations, and medical resource demand prediction error rate; diagnostic accuracy is calculated by comparing with the gold standard diagnostic results, treatment plan recommendation satisfaction is statistically analyzed through questionnaires of medical staff and patients, and the medical resource demand prediction error rate is calculated using the root mean square error (RMSE) formula: where y i is the actual value, is the predicted value, n is the number of samples; according to the evaluation results, the model parameters are adjusted, training data is added, and the model structure is improved;

[0016] Hospital collaboration and data sharing module: A hospital collaboration and data sharing platform is established within the region. Each hospital uploads desensitized data to the platform for encrypted storage. When collaborative diagnosis or research is needed, the hospital requests access to related data through a contract. The contract specifies the rights and scope of data use.

[0017] Mobile application module: Develop mobile applications to facilitate medical staff's access to the system. Mobile applications connect to the hospital information system through a secure interface to obtain patient data and the system's analytical and decision-making results. Medical staff can view patients' medical records, images, and test reports on their mobile devices, and receive auxiliary information for disease diagnosis and treatment recommendations. The mobile application also supports online consultations, allowing patients to communicate with doctors via video calls, and doctors use the system to make preliminary diagnoses and recommendations based on the patient's description and uploaded data.

[0018] Medical expense prediction module: The model combines the patient's disease diagnosis, treatment plan, medical insurance policy and hospital charging standard information to predict the patient's medical expenses. By learning from historical patient medical expense data, a cost prediction model is constructed using regression analysis. The formula is: y = β0 + β1x1 + β2x2 + ... + β n x n +∈, where y is medical expenses, x i is the factor affecting the cost, β i is the regression coefficient, ∈ is the error term; the prediction results are presented in the form of a cost list, including the estimated amount and proportion of each cost, to help patients and hospitals plan medical expenses.

[0019] Furthermore, it also includes:

[0020] Medical quality monitoring module: extracts indicators related to medical quality from the data, and compares and analyzes the indicators with industry standards and the hospital's own historical data through real-time monitoring; when an indicator is abnormal, the system automatically issues an early warning, prompting hospital managers and departments to make quality improvements.

[0021] Medical research support module: Analyzes and integrates medical research literature, clinical data, and experimental results through models to provide medical researchers with research ideas and data support. Through knowledge graphs and semantic analysis technologies, it mines research literature and builds a medical research knowledge network, enabling researchers to understand the current status and trends of medical record research and promote innovation and development in medical research.

[0022] Patient health management module: establishes a health profile for the patient based on the patient's case data, including medical history, physical examination results, and lifestyle information; provides personalized health advice to the patient through model analysis of the patient's health risk factors; at the same time, the patient interacts with the doctor through the mobile application module and provides feedback on his health status. The doctor adjusts the health management plan based on the patient's feedback.

[0023] Drug management module: Analyze drug usage, inventory information, and efficacy data through models, and evaluate the clinical value of drugs by analyzing the frequency of drug use and therapeutic effects; combine the hospital's procurement plan and inventory management strategy to predict drug demand and optimize drug procurement and inventory management.

[0024] Furthermore, it also includes:

[0025] Hospital Operation Efficiency Analysis Module: Extracts operational efficiency indicators from hospital data, including bed turnover rate, average length of stay, and waiting time for outpatient registration; analyzes these indicators through models to identify factors affecting hospital operational efficiency and provide hospital managers with recommendations for optimizing operational efficiency.

[0026] Medical equipment management module: This module uses a model to analyze the operating data, maintenance records, and fault information of medical equipment. By analyzing the equipment's operating status data, it predicts the probability of equipment failure. This module uses a method that combines fault tree analysis and machine learning. The formula is: Where P(F) is the probability of equipment failure, P(X i ) is the probability of occurrence of basic event i that causes equipment failure; arrange equipment maintenance and servicing plans in advance, and provide support for equipment procurement decisions. According to the hospital's business needs, equipment performance and cost factors, medical equipment procurement plans are recommended through optimization algorithms.

[0027] Medical Education Module: Provides resources and support for medical education through models, integrates medical images, medical records, and surgical video data, creates a virtual case library and simulated diagnosis and treatment environment for medical students to learn and practice; and provides students with learning advice and guidance based on their learning situation and answer results through the tutoring system.

[0028] Emergency Management Module: Monitors and warns of public health events and natural disasters through models. By analyzing epidemic data, meteorological information, and geographic location data, it predicts the impact of emergencies on hospitals and formulates emergency plans including bed allocation, medical supplies reserves, and personnel scheduling plans.

[0029] Compared with the existing technology, the beneficial effects of the present invention are:

[0030] In medical diagnosis, it can fully integrate multimodal patient data to assist doctors in accurately diagnosing the condition, significantly improving diagnostic accuracy, reducing misdiagnoses and missed diagnoses, and maximizing treatment opportunities for patients. For example, in diagnosing complex diseases, combining medical records, images, and test results provides doctors with a more comprehensive and accurate diagnostic reference, improving diagnostic reliability.

[0031] In terms of treatment recommendations, the system recommends personalized, scientifically sound treatment plans based on individual patient conditions and multimodal data, and assesses efficacy and risks, helping doctors develop better treatment plans and improve treatment outcomes. Furthermore, the system accurately predicts medical resource needs and rationally allocates beds, medications, and equipment, avoiding resource waste and shortages, improving hospital operational efficiency, and enhancing the patient experience.

[0032] By building a knowledge graph, we provide rich knowledge resources for medical research, helping researchers discover topics, screen subjects, and analyze results, thereby promoting innovative development in medical research. In terms of patient health management, we create personalized profiles for patients, analyze health risks, provide customized health advice, and encourage patients to actively participate in health management and improve their long-term health.

[0033] In medical quality monitoring, key indicators are monitored in real time, providing timely warnings of anomalies, helping hospitals implement targeted improvements and enhance medical quality. The drug management module optimizes procurement inventory and ensures medication safety. The multi-hospital collaboration module leverages blockchain to enable secure data sharing and improve regional medical collaboration. Mobile applications facilitate 24 / 7 access to information for medical staff, improving work efficiency. The emergency management module effectively responds to emergencies, ensuring stable hospital operations and patient care, comprehensively enhancing the hospital's information and intelligent capabilities, benefiting both doctors and patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic block diagram of the hospital information intelligent analysis and decision-making system based on a multimodal large model proposed by the present invention;

[0035] Figure 2 This is a schematic diagram comparing the diagnostic accuracy of different diseases in the hospital information intelligent analysis and decision-making system based on a multimodal large model proposed by the present invention;

[0036] Figure 3This is a schematic diagram of the comparison of medical resource demand prediction errors of the hospital information intelligent analysis and decision-making system based on the multimodal large model proposed in the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0039] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below with reference to the accompanying drawings.

[0040] Reference Figure 1-Figure 3 : A specific implementation method of hospital information intelligent analysis and decision-making system based on multimodal large model.

[0041] 1. Multimodal Data Acquisition Module

[0042] The hospital information system architecture is deeply integrated with the electronic medical record (EMR), picture archiving and communication system (PACS), and laboratory information system (LIS). From the EMR system, text data such as patient symptom descriptions, diagnosis conclusions, and treatment processes are regularly obtained hourly using the HL7 interface. In the PACS system, once the imaging device completes the scan, the image data is instantly transmitted using the JPEG-XR compression algorithm (compression ratio 10:1-20:1) in accordance with the DICOM standard protocol. For example, a CT device generates 50GB of raw images per day, which are compressed to only 2.5GB-5GB. The LIS system is directly connected to the database, and upon confirmation of the test report, numerical data for blood routine, biochemistry, and other tests are immediately extracted.

[0043] 2. Data Preprocessing and Fusion Module

[0044] Text data preprocessing: Use Python's NLTK library to clean the text. Write regular expressions to remove special characters, load a stop word list to filter out meaningless words, use NLTK's stemming tool to restore words, such as changing "running" to "run," and annotate the part of speech.

[0045] Image data preprocessing: Image enhancement based on Python OpenCV library. Histogram equalization is used for low-contrast X-ray images. For example, in an 8-bit X-ray image with grayscale level L = 256 and 512×512 pixels n = 512×512 = 262144, a pixel with grayscale level j = 100 has a number of occurrences n. 100 , according to the formula Calculate the transformed pixel values ​​to improve image clarity.

[0046] Preprocessing of test numerical data: For the test numerical data, use Python's pandas library to calculate the mean μ and standard deviation σ, according to the formula Perform Z-score normalization. For example, for a set of blood glucose values ​​[5.5, 6.2, 4.8, 7.0, 5.8], μ = 5.86 and σ = 0.84, which after normalization become [-0.43, 0.40, -1.26, 1.36, -0.07].

[0047] Data fusion: We built a fusion model based on the attention mechanism, implemented using the Python TensorFlow framework. We assigned different weights to text, image, and numerical data. For example, in heart disease diagnosis, image data is crucial for identifying structural abnormalities, so we assigned a higher weight to highlight key information and improve fusion accuracy.

[0048]

[0049] As the table shows, preprocessing significantly improved the quality of data from each modality. Unnecessary information was removed from the text, image clarity was enhanced, and the test values ​​were standardized. Data fusion increased diagnostic accuracy by 15%, demonstrating the effectiveness of preprocessing and fusion, laying a solid foundation for subsequent analysis.

[0050] III. Multimodal Large Model Construction and Training Module Model Architecture: Based on the Transformer architecture, a multimodal large model was constructed using Python's TensorFlow and Keras libraries. The text encoder uses the BERT structure, loading the pre-trained BERT-base-uncased model weights to perform word embedding and positional encoding on the input text, and then extracts features through multiple layers of Transformer blocks. The image encoder uses the VisionTransformer (ViT) structure to segment the image into fixed-size image blocks, map them into vectors, and input them into the Transformer blocks. The numerical encoder is designed as a fully connected neural network, inputting standardized test numerical data, and extracting features through multiple layers of neurons. Training Data Preparation: The hospital collected historical medical records, imaging, and test result data from the past five years, totaling 100,000 patient records. The data was preprocessed and annotated, such as disease diagnosis and treatment effect. The data was divided into training, validation, and test sets in an 8:1:1 ratio. Training Process: Training combines contrastive learning and self-supervised learning. In contrastive learning, construct positive sample pairs (same patient data from different modalities) and negative sample pairs (different patients data from different modalities), and use the formula Calculate loss and optimize model parameters. Self-supervised learning is used to design tasks such as masked language modeling and image completion. For example, in masked language modeling, 15% of words in a text are randomly masked, and the model predicts the masked words. Through multiple rounds of training, the model's ability to understand multimodal data is improved.

[0051]

[0052] The table shows that as training progresses, the loss value decreases and the accuracy improves. The model goes from initial non-convergence to good convergence in the later stages, indicating that the training is effective, the model performance is continuously optimized, and it can better extract multimodal data features.

[0053] 4. Intelligent Analysis Module Disease Diagnosis Assistance: When a patient seeks medical treatment, the system integrates their multimodal data into a trained multimodal large model. The model outputs the disease prediction probability distribution, which is then processed by the Softmax function. Converted into probability values. For example, if a patient with chest pain is input, the model outputs a probability of 0.6 for coronary heart disease, 0.2 for cardiomyopathy, and 0.2 for other diseases, assisting doctors in diagnosis. Treatment recommendation: Based on disease diagnosis results, multimodal data on the patient's physical condition, and historical treatment cases, the model searches a case library for matches. For example, for lung cancer patients, it recommends surgery, chemotherapy, and targeted therapy, and provides efficacy predictions for each option (e.g., 5-year survival rate for surgery is 70%, for chemotherapy 40%, for targeted therapy 50%) and risk assessments (probability of surgical complications is 10%, for chemotherapy side effects is 80%, for targeted therapy resistance is 30%). Medical resource demand forecasting: Combining historical hospital visit data, seasonal factors, epidemic situation data, and other textual modalities, forecasts are made using a combination of the Python Prophet library (time series analysis) and LightGBM (machine learning). For example, during flu season, a model trained on flu season data from the past five years predicts that peak bed demand for that department this year will increase by 20% compared to last year, providing a basis for hospitals to allocate resources in advance.

[0054]

[0055] As can be seen from the table, the intelligent analysis module significantly improves the accuracy of disease diagnosis and treatment plan recommendations, and the forecast of medical resource demand optimizes resource allocation, reducing waste and shortages, reflecting the effectiveness of the module in medical decision-making and resource management.

[0056] 5. Decision Support Module

[0057] Visualization Interface Design: We developed a visualization interface using the Python Dash library. Disease diagnosis results are presented as probability bar charts, allowing for intuitive comparison of the probabilities of different diseases. Recommended treatment options are presented in a list, with each option accompanied by a description of its efficacy and risks. Medical resource demand forecasts use line charts to display time trends, while bar charts illustrate the required quantities of different resources.

[0058] Decision-making recommendations: Recommendations are generated based on intelligent analysis results. For diseases with a high probability of diagnosis, doctors are advised to conduct further examinations to confirm the diagnosis. Treatment options are recommended based on efficacy and risk, taking into account patient preferences. Medical resource demand forecasts advise managers to allocate resources in advance, such as increasing fever clinic beds before flu season.

[0059] Implementation of interactive functions: The interface has a text input box where users can enter questions, such as "What are the countermeasures for the side effects of this treatment plan?" The system will call upon the multimodal large model knowledge reserve to answer the questions and provide relevant medical knowledge and case references.

[0060]

[0061] The table shows that doctors and managers have a high satisfaction rate with the visual interface, a high adoption rate of decision suggestions, and convenient interactive functions, indicating that the decision support module can meet user needs and assist in scientific decision-making.

[0062] 6. Knowledge Graph Construction and Application Module

[0063] Knowledge Graph Construction: A medical knowledge graph was constructed using the Python Neo4j library. Natural language processing techniques were used to extract entities (such as diseases, symptoms, and medications) and relationships (such as the association between diseases and symptoms and medications treating diseases) from text data. For example, imaging data was used to establish a relationship between lung cancer imaging features and lung cancer entities. Numerical data mining was tested to correlate disease diagnosis and treatment knowledge. For example, a relationship between abnormal blood sugar levels and a diabetes diagnosis was found and stored in the knowledge graph.

[0064] Knowledge Graph Application: When assisting with diagnosis, the system searches the knowledge graph for related diseases and diagnostic methods by entering patient symptoms. For example, if you enter "cough, hemoptysis," diseases such as lung cancer and tuberculosis, along with their corresponding diagnostic tests, will be retrieved. When formulating treatment plans, the knowledge graph recommends appropriate pathways and drug combinations. For example, for lung cancer treatment, genetic testing may be recommended first, with targeted drugs or chemotherapy selected based on the results, improving treatment standardization.

[0065]

[0066] The table shows that the application of knowledge graphs improves the accuracy of auxiliary diagnosis and the rationality score of treatment plans, indicating that knowledge graphs have important value in the medical process and provide knowledge support for medical decision-making.

[0067] 7. Data Security and Privacy Protection Module

[0068] Data storage encryption: For sensitive patient information, such as ID numbers and contact information in electronic medical records, we use the Python cryptography library to implement the AES-256 encryption algorithm. A 256-bit random key is generated during encryption, and data is encrypted and stored block by block. For example, if a patient medical record is stored, the encrypted data cannot be directly read, ensuring data storage security.

[0069] Data transmission encryption: SSL / TLS protocols are used at the data transmission layer. SSL / TLS encrypted channels are established for communication between internal hospital systems and external interfaces, such as for data reporting to higher-level medical regulatory authorities. Certificates are configured using the OpenSSL library to ensure data transmission is protected from theft and tampering.

[0070] Access control: Assign permissions to users based on their roles. Doctors can view and modify their own patient records; nurses can only view basic patient information and nursing records; administrators can view hospital-wide statistics but cannot modify patient diagnoses. User authentication and permission management are implemented using Python's Flask-Login library.

[0071] Security vulnerability scanning: We perform comprehensive system scans monthly using the Nessus vulnerability scanning tool. This scan covers servers, network devices, and applications. If a SQL injection vulnerability is discovered during a scan, it will be promptly fixed to ensure system security.

[0072]

[0073] The table shows that all security measures are effective, no security incidents have occurred, the risk of data leakage has been greatly reduced, user permissions are compliant, and the safe and stable operation of the hospital information system is guaranteed.

[0074] 8. System Evaluation and Optimization Module

[0075] Evaluation indicator calculation: Calculate the diagnostic accuracy and compare it with the gold standard diagnosis results. For example, for 100 patients, 80 were diagnosed correctly by the gold standard, while 75 were correctly diagnosed by the system, with an accuracy rate of 75%. Satisfaction with treatment plan recommendations is assessed through a questionnaire survey of medical staff and patients. For example, if 80 out of 100 questionnaires were satisfied, the satisfaction rate is 80%. The error rate of medical resource demand prediction is calculated using the root mean square error (RMSE) formula: For example, when predicting bed demand for 10 days, the RMSE between the actual and predicted values ​​is 5.

[0076] Model Optimization: Optimize large multimodal models based on evaluation results. If accuracy is low, increase training data and adjust model parameters, such as increasing the number of BERT layers and adjusting the ViT image block size. For example, increasing training data by 20% can improve diagnostic accuracy by 5%.

[0077] Functional module optimization: Optimized data preprocessing processes, such as regular expression optimization for text cleaning, resulting in a 20% speed increase. Improved visual interface interactivity, such as adjusting button positions, increased the user convenience score from 8 to 9.

[0078]

[0079] The table shows that through evaluation and optimization, various system indicators have improved, indicating that the optimization measures have effectively improved system performance and user experience.

[0080] 9. Multi-hospital collaboration and data sharing module

[0081] Blockchain Platform Construction: A blockchain platform was built based on the Hyperledger Fabric framework. Each hospital within the alliance served as a node, and each node deployed blockchain network components. For example, Hospital A deployed Peer and Orderer nodes, responsible for storing and verifying data and sorting transactions.

[0082] Data upload and storage: Hospitals upload encrypted, desensitized multimodal data. For example, if Hospital B uploads 1,000 patient records, sensitive information such as patient names and ID numbers is first desensitized, encrypted using AES, and written to the blockchain as a transaction. Data is stored in a distributed ledger, synchronized across all nodes, ensuring data security and tamper-proofing.

[0083] Smart contract development: Develop smart contracts for data usage. For example, in rare disease research, if Hospital C requests data from Hospital D, the smart contract stipulates that Hospital C can only view specific fields for research on a specific disease, with a one-month usage period, ensuring the legal use of the data.

[0084]

[0085] The table shows that the multi-hospital collaboration and data sharing module improved data sharing efficiency, prevented any safety incidents, and promoted the output of research results, indicating that the module plays an important role in regional medical collaboration.

[0086] 10. Mobile Application Module

[0087] Application Development: Develop a cross-platform mobile app using the React Native framework. The interface is simple and user-friendly. For example, the homepage features a patient list, allowing users to view detailed medical records, images, and test reports.

[0088] Functionality: Connects to the hospital information system via a secure interface to access data in real time. Supports online consultations, video calls between doctors and patients, and patients can upload photos of their symptoms and simple test results. Doctors then analyze the results to make preliminary diagnoses. Reminders are also available, such as alerting doctors to check patients' latest test results and reminding patients to take medications, all delivered through mobile phone notifications.

[0089]

[0090] The table shows that the mobile application functions have received high satisfaction from doctors and patients, and the usage rate and efficiency of online consultation and reminder functions are high, indicating that the application meets user needs and improves the convenience of medical services.

[0091] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A hospital information intelligent analysis and decision-making system based on a multimodal large model, characterized by: include: Data acquisition module: This module collects data from various hospital information systems, including text data in the electronic medical record system, medical image data in the image archiving and communication system, and test numerical data in the laboratory information system. For text data, relevant records in the electronic medical record system are captured in real time through the interface. For medical image data, the module uses the DICOM standard protocol to obtain data from imaging devices, and performs preliminary format conversion and compression processing on the images to save storage space. The compression algorithm uses JPEG-XR. Test numerical data is connected to the database, and the results in the laboratory information system are extracted regularly. The data collection frequency is set according to the data type. Text data is updated every hour, image data is collected after the examination is completed, and test numerical data is obtained in real time after the report is generated. Data preprocessing and fusion module: preprocesses the collected data of different modalities; For text data, we first clean the text to remove characters and stop words, and use the stemming and part-of-speech tagging methods in the NLTK natural language toolkit to convert the text into a form that is easy to analyze. For medical image data, we use image enhancement algorithms to improve the contrast and clarity of the image. The formula is: Where s is the pixel value after transformation, L is the total number of gray levels of the image, n is the total number of pixels in the image, and n k is the number of occurrences of gray level k, j is the gray level of the current pixel; the test numerical data is normalized using the Z-score normalization formula: Where x is the original data, μ is the data mean, and σ is the standard deviation; then, a feature fusion model is constructed to fuse the preprocessed different modal data, and a fusion method based on the attention mechanism is adopted to assign different weights to different modal data to highlight information.

2. The hospital information intelligent analysis and decision-making system based on a multimodal large model according to claim 1 is characterized in that: Also includes: Model building and training module: A Transformer-based model is constructed, including a text encoder, an image encoder, and a numeric encoder, which perform feature extraction on text, images, and test numeric data, respectively. The text encoder uses the BERT structure, the image encoder uses the VisionTransformer structure, and the numeric encoder is designed as a fully connected neural network structure. During the training process, the hospital's historical medical records, images, and test results are used, combined with comparative learning and self-supervised learning methods, to enable the model to learn the associations and feature representations between different modal data. Contrastive learning optimizes the model by maximizing the similarity between positive samples and minimizing the similarity between negative samples. The formula is: Where sim is the similarity function, is a positive sample pair, is a negative sample pair, τ is a temperature hyperparameter, and N is the number of negative samples. Self-supervised learning enables the model to learn the intrinsic characteristics of the data without manual annotation by designing mask language modeling and image filling self-supervision tasks. Analysis module: Analyzes the fused data through the trained model. In terms of disease diagnosis assistance, the patient's data is input, and the model outputs the predicted probability distribution of the disease. The model output is converted into a probability value through the Softmax function. The formula is: Where z is the model output vector, j is the category index, and K is the total number of categories. Regarding treatment plan recommendations, the model retrieves and matches recommended treatment plans based on the patient's disease diagnosis results and historical treatment history. It also provides efficacy predictions and risk assessments for these plans. Regarding medical resource demand forecasting, the model combines the hospital's historical visit data and seasonal factors, using a combination of time series analysis and machine learning to predict future bed demand, drug consumption, and medical equipment usage frequency across different departments, providing a basis for resource allocation in the hospital. Decision support module: The results of the analysis module are presented to hospital managers and medical staff through a visual interface. For disease diagnosis auxiliary results, the probability of various diseases is displayed in the form of a probability bar chart. Treatment plan recommendations are presented in the form of a list, and each plan is accompanied by a description of the efficacy and risks. The results of the medical resource demand forecast are displayed in a combination of line charts and bar charts to show the resource demand situation at different time points. At the same time, the system provides decision-making recommendations for doctors to examine and take treatment measures. Based on the medical resource demand forecast, managers are advised to adjust bed allocation, purchase medicines, and maintain medical equipment in advance. Knowledge graph construction and application module: Construct a medical knowledge graph based on hospital data; the nodes in the graph include diseases, symptoms, treatment methods, drugs, and medical devices, and the edges represent the relationships between them. During the construction process, natural language processing technology is used to extract entities and relationships from text data, identify the connection between disease characteristics and related entities from imaging data, and mine knowledge related to disease diagnosis and treatment from test numerical data; the knowledge graph is used for auxiliary diagnosis. When a patient's symptoms are input, the system retrieves diseases and diagnostic methods through the knowledge graph; in the formulation of treatment plans, reasonable treatment pathways and drug combinations are recommended based on the knowledge graph to improve the standardization of treatment; Data security and privacy protection module: This module uses technology to ensure data security and patient privacy. For data storage, the AES encryption algorithm is used for patient personal information in electronic medical records. SSL / TLS protocols are used to encrypt data transmission during data transmission. Different permissions are assigned to different users through access control mechanisms, enabling authorized users to access required data. System evaluation and optimization module: Establish a system evaluation indicator system, including diagnostic accuracy, satisfaction with treatment plan recommendations, and medical resource demand prediction error rate; diagnostic accuracy is calculated by comparing with the gold standard diagnostic results, treatment plan recommendation satisfaction is statistically analyzed through questionnaires of medical staff and patients, and the medical resource demand prediction error rate is calculated using the root mean square error (RMSE) formula: where y i is the actual value, is the predicted value, n is the number of samples; according to the evaluation results, the model parameters are adjusted, training data is added, and the model structure is improved; Hospital collaboration and data sharing module: A hospital collaboration and data sharing platform is established within the region, and each hospital uploads desensitized data to the platform for encrypted storage; When collaborative diagnosis or research is needed, hospitals request access to related data through contracts; the contracts specify the rights and scope of use of the data; Mobile application module: Develop mobile applications to facilitate medical staff's access to the system. The mobile application connects to the hospital information system through a secure interface to obtain patient data and the system's analysis and decision-making results; Medical staff can view patients' medical records, images, and test reports on their mobile devices, and receive auxiliary information for disease diagnosis and treatment recommendations. The mobile application also supports online consultation, where patients can communicate with doctors via video calls. Doctors use the system to make preliminary diagnoses and recommendations based on the patient's description and uploaded data. Medical expense prediction module: The model combines the patient's disease diagnosis, treatment plan, medical insurance policy and hospital charging standard information to predict the patient's medical expenses. By learning from historical patient medical expense data, a cost prediction model is constructed using regression analysis. The formula is: y = β0 + β1x1 + β2x2 + ... + β n x n +∈, where y is medical expenses, x i is the factor affecting the cost, β i is the regression coefficient, ∈ is the error term; the prediction results are presented in the form of a cost list, including the estimated amount and proportion of each cost, to help patients and hospitals plan medical expenses.

3. The hospital information intelligent analysis and decision-making system based on a multimodal large model according to claim 1 is characterized in that: Also includes: Medical quality monitoring module: extracts indicators related to medical quality from the data, and compares and analyzes the indicators with industry standards and the hospital's own historical data through real-time monitoring; when an indicator is abnormal, the system automatically issues an early warning, prompting hospital managers and departments to make quality improvements.

4. The hospital information intelligent analysis and decision-making system based on a multimodal large model according to claim 1 is characterized in that: Also includes: Medical research support module: Analyzes and integrates medical research literature, clinical data, and experimental results through models to provide medical researchers with research ideas and data support. Through knowledge graphs and semantic analysis technologies, it mines research literature and builds a medical research knowledge network, enabling researchers to understand the current status and trends of medical record research and promote innovation and development in medical research.

5. The hospital information intelligent analysis and decision-making system based on multimodal large model according to claim 1 is characterized in that: Also includes: Patient health management module: establish health records for patients based on their case data, including medical history, physical examination results, and lifestyle information; The model analyzes the patient's health risk factors and provides personalized health advice to the patient. At the same time, the patient interacts with the doctor through the mobile application module and provides feedback on his or her health status. The doctor adjusts the health management plan based on the patient's feedback.

6. The hospital information intelligent analysis and decision-making system based on multimodal large model according to claim 1 is characterized in that: Also includes: Drug management module: Analyze drug usage, inventory information, and efficacy data through models, and evaluate the clinical value of drugs by analyzing the frequency of drug use and therapeutic effects; combine the hospital's procurement plan and inventory management strategy to predict drug demand and optimize drug procurement and inventory management.

7. The hospital information intelligent analysis and decision-making system based on multimodal large model according to claim 1 is characterized in that: Also includes: Hospital Operation Efficiency Analysis Module: Extracts operational efficiency indicators from hospital data, including bed turnover rate, average length of stay, and waiting time for outpatient registration; analyzes these indicators through models to identify factors affecting hospital operational efficiency and provide hospital managers with recommendations for optimizing operational efficiency.

8. The hospital information intelligent analysis and decision-making system based on multimodal large model according to claim 1 is characterized in that: Also includes: Medical equipment management module: This module uses a model to analyze the operating data, maintenance records, and fault information of medical equipment. By analyzing the equipment's operating status data, it predicts the probability of equipment failure. This module uses a method that combines fault tree analysis and machine learning. The formula is: Where P(F) is the probability of equipment failure, P(X i ) is the probability of occurrence of basic event i that causes equipment failure; arrange equipment maintenance and servicing plans in advance, and provide support for equipment procurement decisions. According to the hospital's business needs, equipment performance and cost factors, medical equipment procurement plans are recommended through optimization algorithms.

9. The hospital information intelligent analysis and decision-making system based on multimodal large model according to claim 1 is characterized in that: Also includes: Medical Education Module: Provides resources and support for medical education through models, integrates medical images, medical records, and surgical video data, creates a virtual case library and simulated diagnosis and treatment environment for medical students to learn and practice; and provides students with learning advice and guidance based on their learning situation and answer results through the tutoring system.

10. The hospital information intelligent analysis and decision-making system based on multimodal large model according to claim 1 is characterized in that: Also includes: Emergency Management Module: Monitors and warns of public health events and natural disasters through models. By analyzing epidemic data, meteorological information, and geographic location data, it predicts the impact of emergencies on hospitals and formulates emergency plans including bed allocation, medical supplies reserves, and personnel scheduling plans.

Images