Artificial intelligence assisted diagnosis and treatment system for urinary system diseases based on internet hospital

Abstract

The application relates to the field of software and discloses an artificial intelligence auxiliary diagnosis and treatment system for urinary system diseases based on an Internet hospital, which comprises a data acquisition module, a feature extraction module, a knowledge graph module, a cross-modal semantic fusion module and an auxiliary diagnosis and treatment generation module.The feature extraction module converts multi-modal data into feature vectors with consistent dimensions; the knowledge graph module stores knowledge in the field of urinary system diseases, and comprises entities such as diseases, symptoms, test indexes, image manifestations and evidence levels of evidence-based medicine and associated relationships; the cross-modal semantic fusion module maps the feature vectors to a knowledge graph entity space, generates multi-modal fusion feature vectors through semantic alignment and weighted fusion; and the auxiliary diagnosis and treatment generation module generates diagnosis suggestions and recommended examination item lists based on the fusion feature vectors.Through deep fusion of multi-modal features and field knowledge, the application improves the accuracy and interpretability of auxiliary diagnosis.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence software, and in particular to an AI-assisted diagnosis and treatment system for urinary system diseases based on an internet hospital. Background Technology

[0002] With the rapid development of internet healthcare, online screening, chronic disease management, and postoperative follow-up for urinary system diseases are gradually migrating online. Urinary system diseases encompass various types, including infections, stones, tumors, and prostate diseases. Accurate diagnosis heavily relies on the comprehensive analysis of multimodal information, including patient symptom descriptions, laboratory test results (such as urinalysis, renal function indicators, and PSA), and medical imaging (such as ultrasound, CT, and MRI). In internet hospital scenarios, patients submit online consultation texts, upload test reports, and imaging data through the platform. Doctors need to remotely integrate this heterogeneous data to make diagnostic and treatment decisions. However, specialist physician resources are limited, and facing a large amount of concurrent multimodal data, there is an urgent need for intelligent auxiliary diagnostic and treatment methods to improve triage and diagnostic efficiency.

[0003] To address these needs, several AI-based assisted diagnostic systems have emerged in the current technology landscape. Early research focused primarily on unimodal analysis, such as lesion identification models based on medical images or expert systems based on laboratory indicators. These unimodal systems suffer from incomplete information utilization, significantly limiting their diagnostic accuracy. To overcome this limitation, multimodal fusion technology has been gradually introduced. This technology achieves information complementarity by concatenating, weighting, or fusing features from text, numerical values, and images using general neural networks. However, these general fusion methods have significant shortcomings: they lack explicit utilization of structured knowledge in the field of urological diseases. In clinical practice, the contribution of different modalities to the diagnosis of different diseases naturally varies, and this difference is clearly defined and quantified within the medical knowledge system. Existing data-driven fusion architectures fail to embed this domain knowledge into the feature fusion process, resulting in the fusion weights of each modality being purely determined by the statistical distribution of the training data, lacking medical basis. When data quality is uneven or samples are sparse, the accuracy and robustness of the fusion results are affected.

[0004] Furthermore, knowledge graph technology has already been applied in the medical field. By constructing entities such as diseases, symptoms, laboratory indicators, and imaging manifestations, along with their relationships, it can provide knowledge support for clinical reasoning. Some existing systems use knowledge graphs as independent modules for interpreting diagnostic results or retrieving similar cases, but they do not deeply integrate knowledge graphs with the feature extraction and fusion process of multimodal data. This loose coupling means that the semantic relationships between entities contained in the knowledge graph cannot directly participate in the weighted fusion calculation of multimodal features. The semantic gap between multimodal data and domain knowledge has not been effectively bridged, which to some extent restricts the performance of auxiliary diagnostic systems in complex clinical scenarios.

[0005] In summary, existing internet-based auxiliary diagnostic technologies for urological diseases lack a technical solution capable of semantically aligning multimodal data with a urological disease knowledge graph at the feature level and dynamically guiding multimodal feature fusion based on knowledge associations. Therefore, how to achieve knowledge graph-driven multimodal deep semantic fusion to improve the accuracy and interpretability of auxiliary diagnosis is a pressing technical problem to be solved in this field. Summary of the Invention

[0006] This invention addresses the technical problems of insufficient integration of multimodal data and domain knowledge, and inadequate diagnostic accuracy and interpretability in existing internet-based auxiliary diagnosis and treatment technologies for urinary system diseases. It proposes an artificial intelligence-assisted diagnosis and treatment system and method for urinary system diseases based on internet hospitals.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: An AI-assisted diagnosis and treatment system for urological diseases based on an internet hospital includes a data acquisition module, a feature extraction module, a knowledge graph module, a cross-modal semantic fusion module, and an auxiliary diagnosis and treatment generation module. The data acquisition module is configured to acquire multimodal diagnostic and treatment data from the internet hospital platform, specifically including text description data submitted by the patient through the online consultation interface, laboratory test numerical data submitted through the test report upload function, and medical imaging data submitted through the image data upload function. The text description data includes textual information such as the patient's chief complaint, present illness, and past medical history. The laboratory test data covers quantitative test results such as urinalysis indicators, renal function indicators, and prostate-specific antigen (PSA). The medical imaging data includes ultrasound images, CT scan images, and magnetic resonance imaging (MRI) images. The feature extraction module is configured to perform feature vectorization processing on the collected multimodal data, specifically including: using a pre-trained language coding network to semantically encode the patient's online consultation text and extract text feature vectors that can characterize the symptom description features; using a numerical processing network to standardize and feature map laboratory test indicators and extract laboratory feature vectors that can characterize the degree of abnormality of test results; using a medical image analysis network to extract features from ultrasound, CT, and MRI images and extract image feature vectors that can characterize the image performance features; the dimensions of each feature vector are kept consistent to facilitate subsequent fusion calculations; The knowledge graph module is configured to store structured knowledge of the urinary system disease domain. The knowledge graph contains various types of entity nodes and semantic relationships between nodes. Specifically, the entity nodes include disease entities, clinical symptom entities, laboratory test indicator entities, and medical imaging manifestation entities. The relationships include the co-occurrence relationship between disease and symptoms, the diagnostic relationship between disease and abnormal laboratory indicators, and the feature correspondence between disease and imaging manifestations. Each relationship carries an evidence-based medicine-based quantitative value to characterize the reliability of the relationship in clinical diagnosis. The cross-modal semantic fusion module is the core innovative unit of this invention. It is configured to achieve semantic alignment and deep fusion between multimodal feature vectors and knowledge graph entity space; specifically, it includes a semantic alignment unit and a weighted fusion unit. The semantic alignment unit is configured to map the extracted text feature vectors, laboratory feature vectors, and image feature vectors into the entity embedding space of the knowledge graph, respectively. By calculating the similarity measure between each feature vector and the corresponding type entity vector in the knowledge graph, the semantic association matching between patient data features and medical knowledge entities is realized. Specifically, the similarity is calculated between the text feature vector and symptom entities, the similarity is calculated between the laboratory feature vector and test indicator entities, and the similarity is calculated between the image feature vector and image performance entities. The weighted fusion unit is configured to calculate the fusion weights of each modality feature vector based on the semantic alignment results and perform a weighted fusion operation. Specifically, it uses an attention mechanism to learn the association strength between each feature vector and its corresponding knowledge entity to obtain a preliminary association weight matrix. Further, it performs a weighted correction on the association weights based on the evidence level quantification value of each association in the knowledge graph, so that the fusion weights reflect both data-driven semantic relevance and the domain knowledge constraints of evidence-based medicine. Through the above fusion mechanism, a multimodal fusion feature vector that can comprehensively reflect the patient's multidimensional diagnosis and treatment information is generated. This fusion feature vector contains both the patient's original data features and the semantic guidance of domain expert knowledge. The auxiliary diagnosis and treatment generation module is configured to generate auxiliary diagnosis and treatment information based on the multimodal fusion feature vector; specifically, it includes a diagnostic reasoning unit and an information output unit. The diagnostic reasoning unit takes a multimodal fusion feature vector as input and outputs candidate diagnostic hypotheses and their confidence scores through a neural network reasoning model. The information output unit generates auxiliary diagnosis and treatment suggestion texts for doctors and patients based on the diagnostic reasoning results, as well as a list of recommended examination items required for the current diagnosis. The generated auxiliary diagnosis and treatment information is sent to the Internet hospital platform through an interface for medical staff to refer to during the diagnosis and treatment process.

[0008] Furthermore, the cross-modal semantic fusion module of the present invention also includes a heterogeneous graph construction unit and a graph neural network processing unit; The heterogeneous graph construction unit is configured to insert the patient's multimodal feature vector as a node into the entity relationship network of the knowledge graph, establish connection edges between patient data nodes and knowledge graph entity nodes, and form a heterogeneous graph structure containing patient nodes, knowledge entity nodes and the edges between them; wherein, the patient data node establishes connection edges with the corresponding symptom entity node, test indicator entity node and image manifestation entity node respectively, and the weight of the edge is determined according to the similarity score calculated by the semantic alignment unit; The graph neural network processing unit is configured to execute a graph attention message passing mechanism on the constructed heterogeneous graph. By iteratively updating the feature representation of each node, the patient data node can fully aggregate domain semantic information from the knowledge graph. Specifically, a multi-head attention mechanism is used to calculate the information transmission weight between nodes, and then the aggregated information from different neighboring nodes is weighted and summed to update the final feature representation of the patient data node. Finally, by performing graph-level aggregation on all modal features of the patient data node, a fused multimodal fusion feature vector is obtained.

[0009] Furthermore, the cross-modal semantic fusion module of the present invention also includes a modality importance evaluation network; The modality importance assessment network is configured to evaluate the contribution of different modalities of data to diagnosis based on the patient's specific symptoms. Specifically, sub-vectors related to the chief complaint are extracted from the text feature vectors and input into the trained importance assessment network. The outputs three importance weights corresponding to the text modality, laboratory modality, and imaging modality, respectively. These weights reflect the differences in the diagnostic reference value of each modality of data in the context of the current patient's symptoms. When calculating the association weights between each feature vector and the knowledge entity, the attention mechanism uses the modality importance weight to scale the initially calculated association weights, so that modalities with high diagnostic contributions receive larger weight coefficients in the final fusion, thereby improving the sensitivity of the fusion results to key diagnostic information.

[0010] Furthermore, the relationships stored in the knowledge graph module of the present invention include evidence level quantification values ​​based on evidence-based medicine guidelines; Specifically, the association between diseases and symptoms is classified into three levels according to the level of evidence-based medicine: strong association, moderate association, and weak association, and each level is assigned a different quantitative weight coefficient. For example, the association between urinary tract infection and symptoms of urinary frequency, urgency, and pain is strong, and the quantitative value of the evidence level is set to a high value; while the association between some atypical symptoms and diseases is weak, and the quantitative value of the evidence level is set to a low value. After calculating the association weights between each feature vector and the knowledge entity, the cross-modal semantic fusion module further uses the evidence level quantification value to weight and correct the association weights, making the fusion process more in line with the medical logic of clinical diagnosis and improving the medical rationality of the fusion results.

[0011] Furthermore, the auxiliary diagnosis and treatment generation module of the present invention includes a sequence generation unit and a logic constraint unit; The sequence generation unit is configured to generate diagnostic suggestion text in natural language form based on multimodal fusion feature vectors; specifically, it adopts an autoregressive generative neural network, using multimodal fusion feature vectors as the initial hidden state, and generates complete diagnostic suggestion statements by progressively predicting the next word; The logical constraint unit is configured to incorporate medical knowledge constraints during sequence generation to avoid generating diagnostic suggestions that violate medical logic. Specifically, at each step of the generation network output, diagnostic logic rules related to the currently generated sequence are retrieved from the knowledge graph, these rules are encoded into constraint vectors, and then the constraint vectors are multiplied element-wise with the word probability distribution output by the generation network to suppress the probability of unreasonable words, thereby ensuring that the generated diagnostic suggestions comply with the diagnostic and treatment guidelines for urinary system diseases.

[0012] Furthermore, the present invention also includes a federated learning module configured to support collaborative training of multi-center internet hospitals while protecting patient privacy data; The federated learning module controls each Internet hospital node to calculate the gradient information of the model parameters locally using its own diagnosis and treatment data, and uploads the gradient information to the central aggregation server after encrypting it. The central aggregation server decrypts and performs weighted aggregation operations on the received encrypted gradients to generate updated global model parameters, and distributes the global model parameters back to each node for local model updates. Through this federated learning mechanism, the diagnosis and treatment data of each Internet hospital are always stored locally, and only encrypted gradient information is transmitted. This effectively solves the technical problem of protecting patient privacy in multi-center collaborative learning, while also improving the generalization ability of the model by utilizing larger-scale aggregated data.

[0013] Furthermore, the data acquisition module of the present invention is also configured to acquire the patient's demographic characteristics data and past medical history data; The feature extraction module is further configured to perform feature encoding on the demographic data and past medical history data to obtain a background feature vector containing patient background information; The cross-modal semantic fusion module also includes a background attention unit, configured to achieve personalized diagnostic and treatment information fusion using patient background features. Specifically, the background feature vector is used as a query vector, and similarity calculation is performed with the embedding vectors of each symptom entity, test indicator entity, and image manifestation entity in the knowledge graph to obtain a personalized attention distribution for the current patient. This attention distribution reflects the relative importance of different diagnostic dimensions to the patient based on factors such as age, gender, and past medical history. The personalized attention distribution is used to perform weighted fusion of text feature vectors, laboratory feature vectors, and image feature vectors to obtain a more personalized multimodal fusion feature vector.

[0014] Furthermore, the present invention also includes an incremental update module configured to enable knowledge retention of the model during continuous learning; The incremental update module monitors the cumulative number of newly labeled diagnostic data. When the new data reaches a preset threshold, the incremental training process is triggered. The incremental training process includes: firstly, calculating the importance weight of each model parameter to the historical diagnostic task. This importance weight is determined by calculating the gradient magnitude of the parameter with respect to the loss function of the historical task. A regularization term is added to the loss function of the incremental training. This regularization term applies an update penalty to parameters whose importance weight is higher than a set threshold, so that these important parameters remain stable during the incremental training process and avoid catastrophic forgetting of the model. Through this incremental update mechanism, the system can continuously learn new diagnostic and treatment models for urinary system diseases while maintaining its ability to retain the knowledge it has already learned.

[0015] Furthermore, the feature extraction module described in this invention obtains optimal initial parameters through self-supervised pre-training before formal deployment; Specifically, for the text feature extraction part, a masked language model is pre-trained using a large-scale unlabeled medical literature corpus, enabling the network to learn the semantic representation of medical terms and contextual relationships; for the image feature extraction part, a contrastive learning task is used to pre-train the medical image coding network, and by maximizing the similarity between the feature representations obtained after different data augmentation processes of the same medical image, the network learns image feature representations with medical semantic discriminativeness. Through the aforementioned pre-training strategy, the feature extraction module can obtain better initial feature representation capabilities when faced with specific diagnostic and treatment data of urinary system diseases, thereby improving the model performance of downstream diagnostic tasks.

[0016] Furthermore, the present invention also includes a multimodal conflict resolution module, configured to handle situations where there are inconsistencies in diagnostic opinions among multimodal data; The multimodal conflict resolution module first calculates the support score of each modality feature vector for the candidate diagnostic hypothesis; then it compares the support score differences between different modalities. When the support difference between any two modalities exceeds the preset conflict judgment threshold, it determines that there is a multimodal conflict in the current diagnosis and treatment data. Upon detecting a conflict, a conflict resolution operation is triggered: the system queries the prior reliability weights of each modality in the knowledge graph under the corresponding diagnostic hypothesis. Modalities with higher reliability receive greater participation weights during fusion, while modalities with lower reliability have their weight proportion in the fusion features reduced. At the same time, for modal data with significantly abnormal support, the system generates targeted re-examination prompts, suggesting that patients supplement or re-perform the corresponding test and examination items in order to obtain more consistent diagnostic and treatment evidence.

[0017] Compared with the prior art, the present invention has the following beneficial effects: First, by semantically aligning multimodal feature vectors with knowledge graph entities, a deep fusion of patient data features and medical knowledge at the feature level is achieved. This results in a fusion that includes both data-driven statistical features and evidence-based medical knowledge constraints, significantly improving the accuracy and interpretability of assisted diagnosis. Second, by introducing a weight correction mechanism based on the quantitative value of evidence-based medicine, the fusion weights of different modal data conform to the medical logic of clinical diagnosis, avoiding the problem of decreased fusion effect when the pure data-driven method has sparse samples or poor data quality. Third, by incorporating diagnostic logic constraints from knowledge graphs into the auxiliary diagnosis and treatment generation process, the generated diagnostic suggestions are ensured to conform to medical standards, avoiding erroneous suggestions that violate clinical common sense, thus improving the clinical usability of the system. Fourth, the design of the federated learning module achieves an effective balance between multi-center collaborative learning and patient privacy protection, providing technical support for the deployment and application of the system in a wider range of Internet hospitals; Fifth, through the multimodal conflict resolution mechanism, it can automatically detect and handle diagnostic opinion conflicts between different modal data, providing doctors with prompts for supplementary examinations, thereby improving the robustness and clinical applicability of the auxiliary diagnosis and treatment system. Attached Figure Description

[0018] Figure 1 This is a system structure diagram provided for an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the reference numerals used in the following description are only for illustrating specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

[0020] System Overall Architecture As attached Figure 1 As shown, the AI-assisted diagnosis and treatment system for urological diseases based on an internet hospital provided by this invention mainly includes a data acquisition module, a feature extraction module, a knowledge graph module, a cross-modal semantic fusion module, an auxiliary diagnosis and treatment generation module, a federated learning module, an incremental update module, and a multimodal conflict resolution module. These modules are connected through standardized data interfaces to form a complete data processing chain.

[0021] The output of the data acquisition module is connected to the input of the feature extraction module via a high-speed data bus. The data acquisition module is responsible for collecting raw patient diagnosis and treatment data from various business systems within the internet hospital. The output of the feature extraction module is simultaneously connected to the inputs of the knowledge graph module and the cross-modal semantic fusion module, sending the extracted feature vectors to these modules respectively. The output of the knowledge graph module is connected to the input of the cross-modal semantic fusion module, sending the constructed knowledge graph entities and relation vectors to it. The output of the cross-modal semantic fusion module is connected to the input of the auxiliary diagnosis and treatment generation module, sending the fused multimodal feature vectors to it. The output of the auxiliary diagnosis and treatment generation module serves as the output of the entire system, returning auxiliary diagnosis and treatment suggestions to the internet hospital's information system. The federated learning module is bidirectionally connected to both the feature extraction module and the auxiliary diagnosis and treatment generation module, enabling distributed training and updating of model parameters. The incremental update module connects its input to the output of the assisted diagnosis generation module, and its output connects to the input of the knowledge graph module, enabling continuous knowledge updates. The multimodal conflict resolution module connects to the outputs of each sub-module of the feature extraction module to handle conflicts and contradictions between different modalities of data.

[0022] Data acquisition module The data acquisition module includes text data acquisition, image data acquisition, physiological parameter acquisition, and laboratory data acquisition units. The text data acquisition unit establishes a data connection with the electronic medical record system of the internet hospital, extracting the patient's medical history, chief complaint information, present illness, past medical history, and personal history from the electronic medical record system. The text data acquisition unit has a built-in data cleaning engine that standardizes the extracted text, including removing special characters, standardizing the encoding format, performing word segmentation, and part-of-speech tagging. The image data acquisition unit connects to the medical image archiving and communication system via the DICOM protocol, acquiring the patient's urinary system CT scan images, MRI images, ultrasound images, and X-ray images. The image data acquisition unit performs format conversion and quality assessment on the acquired raw images, filtering out medical image data that meets quality requirements. The physiological parameter acquisition unit connects to the monitoring equipment data interface of the internet hospital, acquiring the patient's heart rate, blood pressure, body temperature, and respiratory rate data. The test data acquisition unit connects to the test information system of the Internet hospital to collect patients' routine urine test results, routine blood test results, biochemical test results, and bacterial culture results.

[0023] The data acquisition module also includes a data synchronization controller, which coordinates the acquisition sequence of each acquisition unit to ensure that text data, image data, physiological parameter data, and test data of the same patient are aligned and correlated according to timestamps. The data synchronization controller has a built-in timestamp calibration algorithm to uniformly calibrate the acquisition time of each data source, eliminating data misalignment problems caused by system time differences. The output end of the data acquisition module is equipped with a data buffer queue, which adopts a first-in-first-out data storage structure to buffer data to be processed and achieve smooth data transmission.

[0024] Feature extraction module As attached Figure 1 As shown, the feature extraction module includes a text feature extraction submodule, an image feature extraction submodule, a physiological feature extraction submodule, and a test feature extraction submodule. The input of the text feature extraction submodule is connected to the output of the text data acquisition unit of the data acquisition module. The text feature extraction submodule integrates a pre-trained language model and uses a bidirectional long short-term memory network structure to extract features from the patient's textual diagnostic information, obtaining a text feature vector. The word embedding layer of the text feature extraction submodule has a dimension of 512, the hidden layer has a dimension of 1024, and the output layer outputs a 512-dimensional text feature vector.

[0025] The input of the image feature extraction submodule is connected to the output of the image data acquisition unit of the data acquisition module. The submodule employs a deep convolutional neural network architecture for image feature extraction. The convolutional layers of the submodule utilize a residual network structure with a depth of 50 layers. The submodule extracts image feature vectors from medical images. The global average pooling layer of the submodule outputs a 2048-dimensional feature vector, which is then reduced to a 512-dimensional feature vector by a fully connected layer.

[0026] The input of the physiological feature extraction submodule is connected to the output of the physiological parameter acquisition unit of the data acquisition module. This submodule extracts temporal features from the acquired physiological parameters such as heart rate, blood pressure, body temperature, and respiratory rate. The physiological feature extraction submodule uses a one-dimensional convolutional neural network for temporal feature extraction. The kernel size of the one-dimensional convolutional neural network is set to 3, the number of kernels is set to 64, and the output is a 512-dimensional physiological feature vector.

[0027] The input of the test feature extraction submodule is connected to the output of the test data acquisition unit of the data acquisition module. This submodule extracts features from test data such as urinalysis, blood routine tests, biochemistry, and bacterial culture. The submodule employs a fully connected neural network structure. The number of nodes in the input layer is determined by the number of test indicators. The hidden layers have three layers with 256, 512, and 512 nodes respectively. The output layer outputs a 512-dimensional test feature vector.

[0028] The feature extraction module also includes a feature standardization unit. The input of the feature standardization unit is connected to the outputs of the text feature extraction submodule, image feature extraction submodule, physiological feature extraction submodule, and test feature extraction submodule, respectively. The feature standardization unit performs L2 standardization on the feature vectors output by each submodule to ensure consistent scale across modal feature vectors. The output of the feature standardization unit is connected to the cross-modal semantic fusion module and the knowledge graph module, respectively.

[0029] Knowledge Graph Module The knowledge graph module includes four types of nodes: disease entity nodes, clinical symptom entity nodes, laboratory test indicator entity nodes, and medical imaging manifestation entity nodes, as well as three types of relationship edges: accompanying relationship edges, diagnostic relationship edges, and feature correspondence relationship edges. The knowledge graph module stores nodes using a graph database structure. Each entity node stores the entity ID, entity name, entity attributes, and entity vector. Accompanying relationship edges store the start node ID, end node ID, relationship type, and relationship weight. Diagnostic relationship edges store the start node ID, end node ID, relationship type, and evidence level quantification value. Feature correspondence relationship edges store the start node ID, end node ID, relationship type, and feature matching degree.

[0030] The knowledge graph module also includes a graph construction unit. The input of the graph construction unit is connected to the output of the feature extraction module. The graph construction unit performs similarity matching between the extracted patient feature vectors and entities in the knowledge graph. The graph construction unit first uses a vector index structure to find the K entity nodes in the entity vector space that are closest to the patient feature vectors. Then, it determines the correspondence between these entity nodes and the patient features based on the feature correspondence edges.

[0031] The knowledge graph module also includes an evidence level quantification unit. This unit labels the diagnostic relationship edges with evidence levels, and the quantification values ​​are set according to the evidence level standards of evidence-based medicine. The evidence level quantification unit divides the diagnostic relationship edges into five levels: Level 1 evidence corresponds to a quantification value of 1.0, Level 2 evidence corresponds to a quantification value of 0.8, Level 3 evidence corresponds to a quantification value of 0.6, Level 4 evidence corresponds to a quantification value of 0.4, and Level 5 evidence corresponds to a quantification value of 0.2.

[0032] The output of the knowledge graph module is connected to the input of the cross-modal semantic fusion module. The knowledge graph module serializes the constructed subgraph structure and then sends it to the cross-modal semantic fusion module. The knowledge graph module is also connected to the output of the incremental update module, receiving incremental knowledge data from the incremental update module to achieve dynamic updates of the knowledge graph.

[0033] Cross-modal semantic fusion module The cross-modal semantic fusion module includes a semantic alignment unit, a weighted fusion unit, a heterogeneous graph construction unit, and a graph neural network processing unit. The input of the semantic alignment unit is connected to the output of both the feature extraction module and the knowledge graph module, and the semantic alignment unit maps feature vectors from different modalities into a unified semantic space.

[0034] The semantic alignment unit internally incorporates multiple modality mapping networks. The text feature mapping network maps 512-dimensional text feature vectors to a 256-dimensional semantic space, with a weight matrix of 512×256. Similarly, the image feature mapping network maps 512-dimensional image feature vectors to a 256-dimensional semantic space, with a weight matrix of 512×256. The physiological feature mapping network maps 512-dimensional physiological feature vectors to a 256-dimensional semantic space, with a weight matrix of 512×256. The test feature mapping network maps 512-dimensional test feature vectors to a 256-dimensional semantic space, with a weight matrix of 512×256. Finally, the knowledge graph mapping network maps knowledge graph entity vectors to a 256-dimensional semantic space.

[0035] The semantic alignment unit also includes a semantic similarity calculation module, which calculates the cosine similarity between the patient's multimodal feature vector and the knowledge graph entity vector. The output of the semantic similarity calculation module forms a semantic alignment matrix, the dimension of which is the product of the number of patient features and the number of knowledge graph entities.

[0036] The input of the weighted fusion unit is connected to the output of the semantic alignment unit. The weighted fusion unit calculates the fusion weights of each modality feature based on the semantic alignment matrix. The weighted fusion unit uses an attention mechanism to calculate the fusion weights, with a single attention dimension of 64 and 4 attention heads. The output of the weighted fusion unit is the fused multimodal feature vector, which has a dimension of 256.

[0037] The input of the heterogeneous graph construction unit is connected to the output of the weighted fusion unit. The heterogeneous graph construction unit fuses the fused multimodal feature vectors with the graph structure of the knowledge graph to construct a heterogeneous graph structure. The heterogeneous graph construction unit first adds the fused feature vectors as new graph nodes to the knowledge graph, and then establishes edge connections between the new nodes and existing graph nodes based on feature similarity. The heterogeneous graph generated by the heterogeneous graph construction unit includes five types of nodes: patient nodes, symptom nodes, disease nodes, laboratory indicator nodes, and imaging manifestation nodes.

[0038] The input of the graph neural network processing unit is connected to the output of the heterogeneous graph construction unit. The graph neural network processing unit performs graph convolution processing on the constructed heterogeneous graph. The message passing mechanism used by the graph neural network processing unit includes three steps: aggregation, update, and concatenation. In the aggregation step, each node collects the feature information of its neighboring nodes and calculates the feature aggregation vector of the neighboring nodes using the mean aggregation strategy. In the update step, each node concatenates its own features with the aggregated neighboring features, and then updates the features through a fully connected network. In the concatenation step, the output features of multiple layers of graph convolution are concatenated to form the final feature representation.

[0039] The output of the graph neural network processing unit is connected to the input of the auxiliary diagnosis generation module. The graph neural network processing unit performs a readout operation on the processed graph node features to form a fixed-length output feature vector with a dimension of 512.

[0040] Assisted diagnosis and treatment generation module The auxiliary diagnosis generation module includes a diagnostic reasoning unit, a sequence generation unit, and a logical constraint module. The input of the diagnostic reasoning unit is connected to the output of the cross-modal semantic fusion module, and the diagnostic reasoning unit performs disease diagnosis reasoning based on the fused multimodal feature vectors.

[0041] The diagnostic reasoning unit contains a diagnostic reasoning network, which employs a multilayer perceptron structure. The input layer has 512 nodes, and the hidden layers consist of three layers with 1024, 512, and 256 nodes respectively. The number of nodes in the output layer is determined based on the type of urinary system disease. The diagnostic reasoning unit first calculates the diagnostic probability for each disease category, then sorts the diseases according to their probabilities, and outputs the Top-K candidate diagnoses. The output of the diagnostic reasoning unit includes a list of diagnoses and their corresponding diagnostic confidence values.

[0042] The input of the sequence generation unit is connected to the output of both the diagnostic reasoning unit and the cross-modal semantic fusion module. The sequence generation unit employs a sequence-to-sequence generation model to convert the diagnostic reasoning results into natural language-based auxiliary treatment suggestions. The encoder of the sequence generation unit uses a bidirectional GRU network with a hidden layer dimension of 512. The encoder encodes the diagnostic reasoning results and the fused feature vector into a context vector. The decoder of the sequence generation unit uses a GRU network with an attention mechanism, with both the hidden layer and context vector dimensions set to 512. The output of the sequence generation unit is the generated treatment suggestion text sequence, with a maximum length of 256 tokens.

[0043] The input of the logical constraint module is connected to the output of the sequence generation unit. The logical constraint module performs logical consistency checks on the generated treatment suggestion text. Internally, the logical constraint module contains a medical knowledge rule base that stores logical rules related to the diagnosis and treatment of urinary system diseases. The logical constraint module performs rule matching checks on the generated text. When content violating logical rules is detected, the logical constraint module triggers a text regeneration mechanism, requiring the sequence generation unit to regenerate treatment suggestions that conform to the logical rules. The output of the logical constraint module serves as the final output of the auxiliary treatment generation module, outputting the auxiliary treatment suggestion text to the external information system.

[0044] Federated Learning Module The federated learning module consists of multiple internet hospital nodes and a central aggregation server. Each internet hospital node is connected to the feature extraction module and the auxiliary diagnosis generation module to update and train local model parameters.

[0045] Each internet hospital node contains a local model training unit and a gradient encryption unit. The local model training unit uses stochastic gradient descent to train model parameters on local data, with a learning rate of 0.001, a batch size of 32, and 10 training epochs. The gradient encryption unit encrypts the trained gradient parameters using a homomorphic encryption algorithm. The encrypted gradient parameters are then transmitted to the central aggregation server via a secure communication protocol.

[0046] The input end of the central aggregation server is connected to the output end of each Internet hospital node. The central aggregation server receives encrypted gradient parameters from each Internet hospital node. The central aggregation server has a gradient aggregation unit, which uses a federated averaging algorithm to aggregate the gradient parameters uploaded by each node. The federated averaging algorithm first decrypts the gradient parameters of each node, and then performs a weighted average according to the data volume weight of each node to obtain the global model update gradient.

[0047] The output of the central aggregation server is connected to the input of each Internet hospital node. The central aggregation server distributes the aggregated global model parameters to each Internet hospital node through a secure communication protocol. After receiving the global model parameters, each Internet hospital node updates its local model parameters to the global model parameters, completing one round of federated learning iteration.

[0048] Incremental update module The input of the incremental update module is connected to the output of the auxiliary diagnosis and treatment generation module to collect feedback information on the auxiliary diagnosis and treatment suggestions. The incremental update module includes a feedback acquisition unit, a knowledge extraction unit, and a graph update unit.

[0049] The feedback collection unit gathers information on doctors' adoption and modification suggestions for auxiliary diagnosis and treatment. This unit interfaces with the internet hospital's medical information system to obtain doctors' feedback data in real time. The feedback collection unit indexes and stores the feedback data according to patient ID and suggestion ID, forming a feedback dataset.

[0050] The input of the knowledge extraction unit is connected to the output of the feedback acquisition unit, and the knowledge extraction unit extracts new medical knowledge from the feedback data. The knowledge extraction unit uses a knowledge extraction algorithm to extract disease symptom correlations, diagnostic rules, and treatment plans from the doctor's modification opinions. The output of the knowledge extraction unit is structured incremental knowledge data.

[0051] The input of the graph update unit is connected to the output of the knowledge extraction unit. The graph update unit updates the knowledge graph module with incremental knowledge data. The graph update unit first verifies the validity of the incremental knowledge data. After successful verification, the incremental knowledge data is inserted into the corresponding nodes and edges of the knowledge graph, realizing the dynamic incremental update of the knowledge graph.

[0052] Multimodal conflict resolution module The multimodal conflict resolution module is connected to the outputs of each sub-module of the feature extraction module to handle conflicts and contradictions between features of different modalities. The multimodal conflict resolution module includes a conflict detection unit and a conflict resolution unit.

[0053] The input of the conflict detection unit is connected to the outputs of the text feature extraction submodule, image feature extraction submodule, physiological feature extraction submodule, and test feature extraction submodule, respectively. The conflict detection unit calculates the similarity between feature vectors of each modality. When the similarity between feature vectors of different modalities is lower than a preset threshold, the conflict detection unit determines that an intermodal conflict exists. The conflict detection unit outputs a conflict detection report, which includes the modality type involved in the conflict, the specific content of the conflict, and the severity of the conflict.

[0054] The input of the conflict resolution unit is connected to the output of the conflict detection unit, and the conflict resolution unit uses a weighted voting strategy for conflict resolution. The conflict resolution unit first assigns initial weights based on the reliability of each modal feature, and then dynamically adjusts the weights according to the conflict content. The output of the conflict resolution unit is a unified feature vector after resolution, which is then fed to the cross-modal semantic fusion module for further processing.

[0055] Detailed description of system workflow The workflow of the entire system is as follows: The data acquisition module collects patient diagnosis and treatment data from various business systems of the Internet Hospital, including text data, image data, physiological parameter data, and test data. After preprocessing, the collected data is sent to the feature extraction module, which extracts the corresponding feature vectors from each type of data to form a multimodal feature representation.

[0056] The feature vectors output by the feature extraction module are simultaneously fed into the knowledge graph module and the cross-modal semantic fusion module. The knowledge graph module retrieves relevant medical knowledge nodes and relationships from the knowledge graph based on the patient's feature vectors, forming a knowledge subgraph related to the patient's diagnosis and treatment. The cross-modal semantic fusion module first maps feature vectors from different modalities to a unified semantic space using a semantic alignment unit, then calculates the fusion weights for each modality's features using a weighted fusion unit, next fuses the fused features with the knowledge graph using a heterogeneous graph construction unit to construct a heterogeneous graph, and finally performs deep processing on the heterogeneous graph using a graph neural network processing unit, outputting the fused multimodal feature vectors.

[0057] The auxiliary diagnosis and treatment generation module receives the fused feature vector output by the cross-modal semantic fusion module, performs disease diagnosis reasoning through the diagnosis reasoning unit, generates auxiliary diagnosis and treatment suggestion text through the sequence generation unit, performs logical consistency verification on the generated text through the logical constraint module, and finally outputs auxiliary diagnosis and treatment suggestions that conform to medical logic.

[0058] The federated learning module enables collaborative model training across internet hospitals. Each internet hospital node trains its model on local data and uploads encrypted gradients. The central aggregation server aggregates the gradients from each node and distributes the global model parameters to each node, achieving collaborative learning with privacy protection.

[0059] The incremental update module collects feedback from doctors on auxiliary diagnosis and treatment suggestions, extracts new medical knowledge from the feedback, and updates it to the knowledge graph module, enabling the system to continuously learn and accumulate knowledge.

[0060] The multimodal conflict resolution module detects conflicts between features of different modalities during the feature extraction stage and resolves conflicts through a weighted voting strategy, ensuring the quality and consistency of the feature vectors input to the cross-modal semantic fusion module.

[0061] The above description is only a specific embodiment of the present invention. Equivalent transformations and improvements made by those skilled in the art based on the technical solutions of the present invention should all fall within the protection scope of the present invention.

Claims

1. An AI-assisted diagnosis and treatment system for urological diseases based on an internet hospital, characterized in that: include: The data acquisition module is used to acquire patient online consultation text data, laboratory test data, and medical imaging data from the Internet hospital platform; The feature extraction module is used to extract text features from the online consultation text data to obtain a text feature vector, extract numerical features from the laboratory test data to obtain a laboratory feature vector, and extract image features from the medical image data to obtain an image feature vector. The knowledge graph module is used to store a knowledge graph of urinary system diseases, which includes disease entities, symptom entities, laboratory indicator entities, imaging manifestation entities, and the relationships between these entities. The cross-modal semantic fusion module is used to semantically align the text feature vector, laboratory feature vector, and image feature vector with the corresponding symptom entities, laboratory indicator entities, and image manifestation entities in the urinary system disease knowledge graph, respectively, and to calculate the association weight between each feature vector and its corresponding entity using an attention mechanism. Based on the association weight, the text feature vector, laboratory feature vector, and image feature vector are weighted and fused to obtain a multimodal fused feature vector. The auxiliary diagnosis and treatment generation module is used to generate auxiliary diagnosis and treatment information based on the multimodal fusion feature vector. The auxiliary diagnosis and treatment information includes diagnostic suggestions and recommended examination items, and is sent to the Internet hospital platform.

2. The system according to claim 1, characterized in that, The cross-modal semantic fusion module includes a heterogeneous graph construction unit and a heterogeneous graph attention network unit; the heterogeneous graph construction unit maps the text feature vector, laboratory feature vector and image feature vector to the entity space of the knowledge graph, and establishes connections with the corresponding symptom entities, laboratory indicator entities and image performance entities to form a multimodal heterogeneous graph; The heterogeneous graph attention network unit applies a graph attention mechanism to the multimodal heterogeneous graph, updates the feature representation of each node through attention message passing between nodes, and aggregates the updated patient data node features to obtain the multimodal fusion feature vector.

3. The system according to claim 1, characterized in that, The cross-modal semantic fusion module further includes a modality importance evaluation network, which takes the chief symptom sub-vector in the text feature vector as input and outputs modality importance weights corresponding to the text modality, laboratory modality, and image modality; the attention mechanism uses the modality importance weights to scale the association weights when calculating the association weights between each feature vector and its corresponding entity.

4. The system according to claim 1, characterized in that, In the knowledge graph of urinary system diseases, the association between disease entities and symptom entities, laboratory indicator entities and imaging manifestation entities includes evidence level quantification values ​​based on evidence-based medicine; after calculating the association weights between each feature vector and its corresponding entity, the cross-modal semantic fusion module uses the evidence level quantification values ​​to weight and correct the association weights.

5. The system according to claim 1, characterized in that, The auxiliary diagnosis generation module includes a sequence generation unit and a logical constraint unit. The sequence generation unit takes the multimodal fusion feature vector as input and generates diagnostic suggestion text word by word through an autoregressive method. In each step of the sequence generation unit, the logical constraint unit obtains the diagnostic logic rules related to the currently generated sequence in the knowledge graph, generates a constraint vector according to the diagnostic logic rules, and performs element-wise operations on the constraint vector and the word prediction probability distribution output by the sequence generation unit to correct the word prediction probability distribution.

6. The system according to claim 1, characterized in that, It also includes a federated learning module, which controls the system deployed on different Internet hospital nodes to calculate the gradient of model parameters locally using their respective privacy data, encrypt the gradient of model parameters and upload it to the aggregation server; and controls the aggregation server to decrypt and aggregate the received gradient of model parameters, update the global model parameters using the aggregated gradient, and distribute the updated global model parameters to each node.

7. The system according to claim 1, characterized in that, The data acquisition module is also used to acquire the patient's demographic data and past medical history data; the feature extraction module is also used to perform feature encoding on the demographic data and past medical history data to obtain a background feature vector; The cross-modal semantic fusion module further includes a background attention unit. The background attention unit uses the background feature vector as a query vector, calculates the similarity between the query vector and the entity vectors of each symptom entity, laboratory indicator entity, and image performance entity in the knowledge graph, obtains a personalized attention distribution, and uses the personalized attention distribution to perform weighted fusion of the text feature vector, laboratory feature vector, and image feature vector.

8. The system according to claim 1, characterized in that, It also includes an incremental update module, which is used to trigger incremental training of the feature extraction module, cross-modal semantic fusion module and auxiliary diagnosis generation module when the number of newly labeled data reaches a preset amount; The incremental training process includes: calculating the importance weights of each model parameter for the historical task; A regularization term is added to the loss function, which includes an update penalty for parameters whose importance weights are above a threshold.

9. The system according to claim 1, characterized in that, Before being put into use, the feature extraction module obtains initial parameters through the following self-supervised pre-training: for text feature extraction, it is pre-trained on a large-scale unlabeled medical text corpus using a masked language model task; for image feature extraction, it is pre-trained by maximizing the similarity between different enhanced views of the same medical image using a contrastive learning task.

10. The system according to claim 1, characterized in that, It also includes a multimodal conflict resolution module, which is used to: calculate the support of the text feature vector, laboratory feature vector and image feature vector for the candidate diagnostic hypothesis; determine that a multimodal conflict has occurred when the support difference between any two modalities is greater than a preset conflict threshold; and trigger a conflict resolution operation, which includes reducing the participation weight of the feature vector of the support-abnormal modality in the cross-modal semantic fusion according to the reliability prior weight of each modality in the knowledge graph under the corresponding diagnostic hypothesis, or generating a re-examination prompt message for the support-abnormal modality.

Images