Ophthalmology remote intelligent consultation system and method based on multimodal data fusion

Abstract

The present invention belongs to the field of intelligent ophthalmic medical technology. The present invention provides an ophthalmic remote intelligent consultation system and method based on multimodal data fusion, including: collecting and integrating ophthalmic multimodal data, generating a multimodal feature set and inputting it into an improved cross-modal attention model, generating final fusion features through dynamic weight fusion and a two-way interaction mechanism, outputting the probability distribution of the patient's ophthalmic disease in combination with the local ophthalmic knowledge graph, quantifying the detailed information of ophthalmic experts into expert feature vectors, calculating the matching degree between the probability distribution of the patient's ophthalmic disease and the expert feature vectors, screening ophthalmic experts with high matching degree for consultation, calculating the recommended value of the treatment plan provided by the ophthalmic experts and generating an electronic consultation form, supplementing local case data to train the local knowledge graph, and the central server generating a global knowledge graph by weighted aggregation of local knowledge graphs of multiple institutions and providing feedback and updates to the local knowledge graph.

Description

Technical Field

[0001] The present invention belongs to the field of ophthalmic medical intelligent technology, and specifically to an ophthalmic remote intelligent consultation system and method based on multimodal data fusion. Background Art

[0002] In the traditional ophthalmic medical consultation process, patients usually need to go to a specific medical institution for a preliminary examination and diagnosis by a local ophthalmologist. If they encounter difficult cases, local doctors will transfer the patient's medical information to superiors or more professional ophthalmologists for consultation through more traditional methods such as written reports and film transmission. This model has many significant disadvantages. Written reports may not be able to fully and accurately describe the patient's symptoms, medical history and other key information. Films are also prone to wear and loss during the transmission process, resulting in incomplete or inaccurate information received by experts, which in turn affects the accuracy of the diagnosis. Patients need to travel long distances to the consultation location, which consumes a lot of time and energy. Especially for patients with serious conditions and mobility difficulties, this undoubtedly increases the difficulty and pain of seeking medical treatment. At the same time, expert resources are unevenly distributed, and patients in some areas find it difficult to obtain consultation opinions from authoritative experts in a short period of time.

[0003] With the rapid development of information technology, some intelligent ophthalmic diagnostic systems have gradually emerged. However, these systems still have obvious technical defects when dealing with complex ophthalmic disease consultations. In existing technologies, ophthalmic disease diagnosis often requires comprehensive information from multiple modalities such as image data, text data, and detection data. However, most existing systems can only process data from a single or a few modalities and are unable to effectively integrate and deeply analyze multimodal data, resulting in incomplete and inaccurate judgments on diseases.

[0004] On the other hand, the intelligent diagnostic systems in existing technologies mainly rely on preset rules or simple machine learning models to diagnose diseases. They lack in-depth understanding and reasoning ability of ophthalmic medical knowledge, and are therefore unable to accurately judge the type and development trend of the disease, let alone provide patients with personalized treatment plans. When expert consultation is required, existing technologies often simply assign patients to a certain expert without fully considering the matching degree between the patient's condition and the expert's professional field, experience, etc. This may result in the patient being unable to consult with the most suitable expert, affecting the consultation and treatment effects.

[0005] In response to the above problems, the present invention proposes an ophthalmology remote intelligent consultation system and method based on multimodal data fusion. Summary of the Invention

[0006] In order to make up for the deficiencies of the prior art, at least one technical problem raised in the background technology is solved.

[0007] The technical solution adopted by the present invention to solve the technical problem is: an ophthalmology remote intelligent consultation system based on multimodal data fusion, comprising:

[0008] Data acquisition module: collects and integrates ophthalmic multimodal data to generate a multimodal feature set;

[0009] Fusion Diagnosis Module: This module inputs the multimodal feature set into an improved cross-modal attention model, generates the final fused features through dynamic weight fusion and a two-way interaction mechanism, and then performs graph neural network reasoning based on the local ophthalmic knowledge graph to output the probability distribution of the patient's ophthalmic disease.

[0010] Consultation matching module: Quantify detailed information about ophthalmologists into expert feature vectors, calculate the matching degree between the probability distribution of the patient's ophthalmic disease and the expert feature vectors, select ophthalmologists with high matching degrees for consultation, calculate the recommended value of the treatment plan provided by the ophthalmologist, and generate an electronic consultation form containing the diagnosis conclusion and treatment suggestions based on the recommended value;

[0011] Iterative update module: desensitizes the electronic consultation form and supplements local case data to train the local knowledge graph. The central server generates a global knowledge graph by weighted aggregation of local knowledge graphs of multiple institutions, and provides feedback and updates to the local knowledge graph based on the generated global knowledge graph.

[0012] Further preferably, the multimodal feature set is generated in the following manner:

[0013] Collecting and preprocessing multimodal ophthalmic data of patients to be treated, including image data, text data, and test data; performing lesion segmentation and geometric feature extraction on the preprocessed image data to obtain image feature vectors; performing semantic feature extraction on the text data to obtain text feature vectors; and performing structured feature mapping on the test data to obtain test feature vectors;

[0014] The three modalities of image feature vector, text feature vector and detection feature vector are integrated to generate a multimodal feature set in a unified format.

[0015] Further preferably, the probability distribution of the patient's ophthalmic disease is obtained in the following manner:

[0016] Based on clinical guidelines and local case data, a directed local ophthalmology knowledge graph containing three types of core entities was constructed. The three types of core entities include disease entities, symptom entities and detection entities. The final fusion features were obtained as the initial feature vector of the disease entity node feature vector. A multimodal attention aggregation function was designed to iteratively update the node feature vector with the initial feature vector as the starting point. The fully connected layer and Softmax function were applied to the iteratively updated node feature vector to obtain the probability distribution of patients' ophthalmic diseases.

[0017] More preferably, the final fusion feature is obtained in the following manner:

[0018] Obtain a weight matrix and design a bidirectional fusion mechanism based on it. Each modal feature in the multimodal feature set serves as a query to extract features from other modalities and also as a key to provide its own features. Combined with the weight matrix, intra-modal fusion and inter-modal fusion are calculated. Intra-modal fusion retains the core features of each modality, while inter-modal fusion captures cross-modal associations.

[0019] The final fusion features are obtained by combining intra-modal fusion and inter-modal fusion through layer normalization and stable training.

[0020] More preferably, the weight matrix is ​​obtained as follows:

[0021] The image feature vector, text feature vector and detection feature vector in the multimodal feature set are mapped to the shared latent space through the fully connected layer, and the multimodal feature set is converted into a shared latent space representation of the same dimension. The trimodal interaction matrix is ​​defined to represent the similarity between the modalities, and the attention weight between any two modal features is calculated and integrated to obtain the weight matrix.

[0022] More preferably, the electronic consultation form is obtained in the following manner:

[0023] In a remote consultation, the ophthalmologists who responded to the request provided treatment plans. The Jaccard similarity was used to calculate the similarity between any two treatment plans. Two treatment plans with a similarity greater than or equal to the similarity threshold were classified into the same similarity plan set. The number of treatment plans in each similarity plan set was normalized and marked as the consensus feature value of the treatment plans in the similarity plan set.

[0024] Based on any treatment plan, the matching degree of the ophthalmologist providing the treatment plan is obtained as the weight of the treatment plan. Combined with the consensus characteristic value of the treatment plan, the recommended value of the treatment plan is obtained by weighted calculation. The treatment plans are sorted in descending order according to the recommended value, and the sorted treatment plans and corresponding diagnosis results are filled in the electronic consultation form to generate an electronic consultation form.

[0025] Further preferably, the matching method of the consulting ophthalmologist who responds to the request is:

[0026] Obtain the matching degree of all ophthalmologists in the platform, sort the ophthalmologists in descending order according to the matching degree, and mark the ophthalmologists with selection numbers according to the order to obtain a sorted list of ophthalmologists, and select ophthalmologists with selection numbers less than or equal to the selection threshold in the ophthalmologist list to form a consulting ophthalmologist set;

[0027] A consultation request is sent to ophthalmologists in the consulting ophthalmologist set, and a consulting ophthalmologist who responds to the request is matched.

[0028] More preferably, the matching degree is obtained in the following manner:

[0029] Detailed information of all ophthalmologists in the platform is collected and features are extracted and quantified to obtain expert feature vectors. The probability distribution of ophthalmic diseases of patients to be treated is obtained, and the probability distribution of ophthalmic diseases of patients is constructed as disease feature vectors of patients to be treated. Data processing is performed on the disease feature vectors of patients to be treated and the expert feature vectors of each ophthalmologist to obtain the matching degree between the probability distribution of ophthalmic diseases of patients and the expert feature vectors.

[0030] Further preferably, the method of providing feedback and updating the local knowledge graph is:

[0031] Obtain multimodal ophthalmic data of patients to be treated and the generated electronic consultation form and perform desensitization processing to generate desensitized ophthalmic treatment data. Continuously record the desensitized ophthalmic treatment data to generate a desensitized dataset and add it to the local case data. Each medical institution conducts incremental training on the local knowledge graph based on the local case data to optimize the symptom-disease association strength. After the training is completed, the graph structure parameters and node feature parameters are extracted to update the local knowledge graph.

[0032] The central server uses the FedAvg algorithm to aggregate the local knowledge graph parameters of multiple institutions, calculates the aggregation weight and performs parameter fusion based on the amount of local case data of each medical institution, obtains the global edge weight matrix and global node embedding vector of the global knowledge graph, and distributes them to each medical institution to replace the local knowledge graph parameters, and performs iterative feedback updates on the local knowledge graph.

[0033] The ophthalmology remote intelligent consultation method based on multimodal data fusion includes the following steps:

[0034] Collect and integrate ophthalmic multimodal data to generate a multimodal feature set;

[0035] The multimodal feature set is input into the improved cross-modal attention model, and the final fused features are generated through dynamic weight fusion and bidirectional interaction mechanism. The graph neural network reasoning is combined with the local ophthalmic knowledge graph to output the probability distribution of the patient's ophthalmic disease.

[0036] Quantify the detailed information of ophthalmologists into expert feature vectors, calculate the matching degree between the probability distribution of patients' ophthalmic diseases and the expert feature vectors, select ophthalmologists with high matching degree for consultation, calculate the recommended value of the treatment plan provided by the ophthalmologists, and generate an electronic consultation form containing the diagnosis conclusion and treatment suggestions based on the recommended value;

[0037] The electronic consultation form is desensitized and supplemented with local case data to train the local knowledge graph. The central server generates a global knowledge graph by weighted aggregation of local knowledge graphs of multiple institutions, and provides feedback and updates to the local knowledge graph based on the generated global knowledge graph.

[0038] The beneficial effects of the present invention are as follows:

[0039] 1. By collecting and integrating multimodal ophthalmic data and conducting in-depth fusion analysis, the present invention can comprehensively mine patient condition information, combine local ophthalmic knowledge graphs to infer the association between symptoms and diseases, and accurately generate disease probability distributions, greatly improving the accuracy of ophthalmic disease diagnosis. At the same time, it quantifies expert information and calculates the matching degree with the patient's condition, accurately screening highly matched experts for consultation, ensuring that patients receive the most suitable diagnosis and treatment recommendations, effectively avoiding the problem of inaccurate expert matching in traditional consultations, and providing patients with higher-quality, personalized medical services.

[0040] 2. The present invention desensitizes electronic consultation forms and supplements local case data, and updates the local knowledge graph based on incremental data training, so that the knowledge graph can timely reflect the latest clinical experience and medical achievements, and continuously improve diagnostic and treatment capabilities. The central server uses the FedAvg algorithm to aggregate the local knowledge graphs of multiple institutions to generate a global knowledge graph, breaking down data barriers between institutions, realizing knowledge sharing and collaboration, promoting the common improvement of the medical level of each institution, and promoting the overall field of ophthalmic medical care to move towards intelligence and precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be further described below with reference to the accompanying drawings.

[0042] Figure 1 1 is a system module architecture diagram of an ophthalmology remote intelligent consultation system based on multimodal data fusion according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart of the steps of the ophthalmic remote intelligent consultation method based on multimodal data fusion described in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0045] Example 1

[0046] See also Figure 1 As shown, the ophthalmology remote intelligent consultation system based on multimodal data fusion according to the embodiment of the present invention includes the following modules:

[0047] Data acquisition module: collects and integrates ophthalmic multimodal data to generate a multimodal feature set;

[0048] Collect multimodal ophthalmic data from patients undergoing treatment. Multimodal ophthalmic data includes three modalities: image data, text data, and test data. Image data includes fundus OCT images and fundus color photographs, text data includes medical records, and test data includes visual field test data and intraocular pressure test data.

[0049] preprocessing the collected ophthalmic multimodal data to obtain preprocessed ophthalmic multimodal data;

[0050] Specifically, the imaging data were spatially aligned. A mutual information-based rigid registration algorithm was used for the fundus OCT image and the fundus color photograph. The corresponding feature points in the fundus OCT image and the fundus color photograph were first extracted using the SIFT algorithm and matched into matching point pairs. The RANSAC algorithm was used to remove mismatched point pairs. The rotation matrix and translation vector were calculated using the least squares method to achieve geometric alignment of the fundus OCT image and the fundus color photograph in the retinal global coordinate system. For images with inconsistent resolutions, the bicubic interpolation method was used to uniformly resize them to 512×512 pixels to preserve edge details. When the resolution was insufficient, zero padding and Gaussian smoothing were used to avoid artifacts.

[0051] After spatial alignment of the image data, the ophthalmic multimodal data of the patients to be treated are all temporally aligned. Based on the consultation time of the patients to be treated, the ophthalmic multimodal data with a time difference between the detection time and the consultation time that is less than or equal to the time difference threshold are screened to ensure that the ophthalmic multimodal data reflect the same stage of the disease. The ophthalmic multimodal data, including the image data, text data, and test data, are format-standardized, noise-filtered, and outlier-processed to obtain preprocessed ophthalmic multimodal data.

[0052] Extract features from pre-processed ophthalmic multimodal data and construct a multimodal feature set;

[0053] Specifically, the imaging data was segmented and geometric features extracted. A U-Net++ network was trained using a public ophthalmology dataset combined with internally labeled data from the hospital. The preprocessed imaging data was then fed into the U-Net++ network for pixel-level segmentation. The network structure consisted of a four-layer encoder, a bottleneck layer, and a four-layer decoder. It output a probabilistic segmentation mask of the same size as the input. Core lesion indices were then calculated based on the probabilistic segmentation mask. These core lesion indices included basic features, morphological features, grayscale features, inter-layer features of fundus OCT images, and color image features.

[0054] The basic features include lesion area, lesion perimeter, and lesion volume; morphological features include lesion circularity and lesion eccentricity; grayscale features include lesion mean grayscale value, lesion grayscale standard deviation, and the contrast between the lesion and surrounding normal tissue; fundus OCT image interlayer features include nerve fiber layer thickness and outer nuclear layer integrity score; color image features include lesion vascular density and lesion exudate quantity. The core lesion indicators are integrated to generate the image feature vector V;

[0055] Semantic features are extracted from text data. Medical record text is input into a clinical BERT model pre-trained on the MIMIC-III medical corpus. A multi-layer Transformer encoder is used to generate word-level embeddings, and the text feature vector T is obtained through average pooling.

[0056] Perform structured feature mapping on the test data, normalize the visual field test data and convert it into a feature vector, take the average of the intraocular pressure values ​​of both eyes in the intraocular pressure test data and perform interval mapping, map the normal range to [0, 0.7], and linearly expand the abnormal values ​​to [0, 1], and integrate them to obtain the test feature vector D;

[0057] Integrate the three modalities of image feature vector V, text feature vector T and detection feature vector D to generate a multimodal feature set {V, T, D} in a unified format;

[0058] It should be noted that this module uses the SIFT-RANSAC algorithm to achieve geometric alignment between OCT and color images, resolving image spatial misalignment and ensuring cross-modal consistency of lesion location. It also designs core lesion indicators for ophthalmic data and combines the U-Net++ and BERT models to achieve cross-modal feature extraction from pixel-level segmentation to semantic-level understanding, preserving key clinical information.

[0059] Fusion Diagnosis Module: This module inputs the multimodal feature set into an improved cross-modal attention model, generates the final fused features through dynamic weight fusion and a two-way interaction mechanism, and then performs graph neural network reasoning based on the local ophthalmic knowledge graph to output the probability distribution of the patient's ophthalmic disease.

[0060] Specifically, the image feature vector V, text feature vector T and detection feature vector D in the multimodal feature set are mapped to the shared latent space through the fully connected layer, and the multimodal feature set is converted into a shared latent space representation of the same dimension. ;

[0061] in, 、 、 They represent the image feature vector V, text feature vector T, and detection feature vector D respectively after being mapped to the shared latent space through the fully connected layer. The converted shared latent space representations of the same dimension represent the feature vectors of the image feature vector V, text feature vector T, and detection feature vector D in the shared latent space.

[0062] Define the trimodal interaction matrix S and calculate the attention weight between any two modal features to reflect the feature dependency.

[0063]

[0064] in, is the element in the i-th row and j-th column of the trimodal interaction matrix S, which represents the similarity between the i-th modal feature and the j-th modal feature, i, j∈{1,2,3}, where the first modal feature corresponds to the image feature vector, the second modal feature corresponds to the text feature vector, and the third modal feature corresponds to the detection feature vector. 、 、 Respectively represent the similarity of the image feature vector to the three modal features, 、 、 Respectively represent the similarity of the text feature vector to the three modal features, 、 、 , respectively represent the similarity of the detection feature vector to the three modal features;

[0065] Normalize the similarity and calculate the weight matrix ,in It represents the attention weight of the i-th modal feature to the j-th modal feature, and the formula is:

[0066]

[0067] Among them, a, b∈{1,2,3}, Represents the element in the ath row and bth column of the trimodal interaction matrix S, which represents the similarity between the ath modal feature and the bth modal feature. d represents the dimension of the shared latent space. Represents the weight matrix The matrix element at row i and column j in ;

[0068] Weight Matrix satisfy:

[0069]

[0070] For example, is the attention weight of the image feature to the text feature, indicating the degree of attention paid by the image feature to the text feature. is the attention weight of the detection feature on the image feature, indicating the degree of attention paid by the detection feature to the image feature;

[0071] Based on the weight matrix, a bidirectional fusion mechanism is designed. Each modal feature is used as a query to extract other modal features and as a key to provide its own features:

[0072] Computational Intra-Modal Fusion , the formula is:

[0073]

[0074] in, Represented as the matrix element in the first row and first column, Represented as the matrix element in the second row and second column, is the matrix element in the third row and third column;

[0075] Computational inter-modality fusion , the formula is:

[0076]

[0077] Among them, intra-modal fusion retains the core features of each modality, and inter-modal fusion captures cross-modal feature correlations;

[0078] Combine intra-modal fusion and inter-modal fusion to calculate the final fusion features , the formula is:

[0079]

[0080] Among them, LayerNorm represents layer normalization, which stabilizes the training through layer normalization and outputs the final fusion feature containing the intra-modal characteristics and cross-modal interaction information ;

[0081] Construct a local ophthalmology knowledge graph based on clinical guidelines and local case data. The graph contains three core entities: disease entities, symptom entities, and detection entities. Symptom entities contain image features, text features, and detection features. Define the associations between core entities, define the associations between symptom entities and disease entities, and set weights. Indicates the support of the symptom entity to the disease entity, defines the association between the detection entity and the disease entity, and sets the weight Indicates the correlation between the detection entity and the disease entity, defines the association between the disease entity and the disease entity, and sets the weight represents the probability of comorbidity;

[0082] The local ophthalmology knowledge graph is stored using the triple {head entity, association, tail entity, weight}. The graph has a total of approximately 500 nodes and approximately 3,000 edges. It is stored in the Neo4j graph database to support efficient graph query and reasoning.

[0083] The final fusion feature As the initial features of the disease entity, the features of the symptom entity and the detection entity are predefined by the clinical knowledge base. , the initial eigenvector Contains disease entities, symptom entities, and detection entities, where the disease entity corresponds to the final fusion feature , symptom entities correspond to one-hot encoding, and detection entities correspond to normalized values ​​of detection indicators, where k represents the node number;

[0084] Design multimodal attention aggregation function:

[0085]

[0086] Iteratively update the node feature vector to capture the association between symptoms and diseases, where Represents the feature vector of node m in the L+1th layer, which represents the information representation of node m after aggregation update. represents the feature vector of the L-th layer neighbor node n, Represents the activation function, which is used to introduce nonlinearity so that the model can learn complex patterns. represents all neighbor nodes n of node m, represents the weight reflecting the strength of the association between node m and its neighbor node n, Represents the modal weight coefficient, i∈{1,2,3}, corresponding to image features, text features and detection features respectively, and W represents the neighbor node features The weight matrix for linear transformation;

[0087] Apply the fully connected layer and Softmax function to the iteratively updated node feature vector to output the probability distribution P of the eye disease of the patient to be treated;

[0088]

[0089] in, represents the probability that the patient to be treated suffers from the ophthalmic disease corresponding to the disease entity numbered c, c = {1, 2, ..., q}, q represents the total number of disease entities in the local ophthalmology knowledge graph;

[0090] It should be noted that the role of this module is to map multimodal features to a shared latent space through a fully connected layer, calculate cross-modal dependency weights using a trimodal attention mechanism, and combine a bidirectional fusion mechanism to generate a final fusion feature that includes intra-modal features and cross-modal interactions. This module then constructs a local ophthalmic knowledge graph, iteratively updates node features through a graph neural network, and outputs the probability distribution P of ophthalmic diseases, enabling symptom-to-disease association reasoning.

[0091] Consultation matching module: Quantify detailed information about ophthalmologists into expert feature vectors, calculate the matching degree between the probability distribution of the patient's ophthalmic disease and the expert feature vectors, select ophthalmologists with high matching degrees for consultation, calculate the recommended value of the treatment plan provided by the ophthalmologist, and generate an electronic consultation form containing the diagnosis conclusion and treatment suggestions based on the recommended value;

[0092] Obtain the probability distribution P of the patient's eye disease to be treated, and accurately match the patient with an ophthalmologist for remote consultation;

[0093] Specifically, we collected detailed information about all ophthalmologists on the platform, including their professional fields, years of practice, types of ophthalmic diseases they specialize in, and the number of cases they have consulted. We then extracted and quantified the details of these ophthalmologists and converted them into computable expert feature vectors. For each ophthalmologist, the corresponding expert feature vector consists of the following: professional field features, years of practice features, and historical consultation case features:

[0094] Among them, the professional field feature uses one-hot encoding to represent the types of diseases that ophthalmologists are good at. For the ophthalmological diseases corresponding to the q disease entities covered in the local ophthalmology knowledge graph, if the ophthalmologist is good at the ophthalmological disease corresponding to the cth disease entity, the cth element of the feature vector is 1, and the rest are 0. The years of practice feature is obtained by normalizing the years of practice of the ophthalmologist. The historical consultation case feature is obtained by calculating the proportion of the number of consultation cases for each ophthalmological disease by the ophthalmologist to the total number of cases and the consultation success rate;

[0095] The professional field features, years of experience features and historical consultation case features in the expert feature vector are weighted and fused to obtain the expert feature vector of the ophthalmologist. ;

[0096]

[0097] Where f represents the serial number of the ophthalmologist in the system, represents the fusion feature value of ophthalmologists on the ophthalmic disease corresponding to the disease entity with serial number c, c = {1, 2, ..., q}, q represents the total number of disease entities in the local ophthalmology knowledge graph;

[0098] Obtain the probability distribution P of ophthalmic diseases of patients to be treated, and construct the probability distribution P of ophthalmic diseases into the disease feature vector of patients to be treated , the disease feature vector directly adopts the disease probability distribution P, ;

[0099] The cosine similarity is used to calculate the matching degree between the disease feature vector of the patient to be treated and the expert feature vector of each ophthalmologist, and the matching degree between the patient to be treated and the ophthalmologist with serial number f is calculated. The calculation formula is:

[0100]

[0101] in, represents the dot product of the disease feature vector and the expert feature vector, and Represent the modulus of disease feature vector and expert feature vector respectively;

[0102] Calculate the matching degree of all ophthalmologists, sort the ophthalmologists in descending order according to the matching degree, and mark the ophthalmologists with selection numbers according to the order to obtain a sorted ophthalmologist list, and select ophthalmologists with selection numbers less than or equal to the selection threshold in the ophthalmologist list to form a consulting ophthalmologist set;

[0103] The system automatically sends consultation requests to ophthalmologists in the consulting ophthalmologist set, including detailed information such as the patient's basic information, disease probability distribution, and multimodal characteristics, and arranges remote consultations for the consulting ophthalmologists who respond to the request;

[0104] In a remote consultation, the ophthalmologists who responded to the request provided treatment plans. Based on any two treatment plans, the similarity between the two treatment plans was calculated using the Jaccard similarity test. Specifically, the number of identical measures in the two treatment plans was calculated by dividing the total number of measures in the two treatment plans by the ratio, and the similarity between the two treatment plans was obtained.

[0105] Two treatment plans with similarity greater than or equal to the similarity threshold are classified into the same similar plan set. All treatment plans are summarized and integrated to obtain several similar plan sets. The number of treatment plans in each similar plan set is normalized and marked as the consensus feature value of the treatment plan in the similar plan set.

[0106] Based on any treatment plan, the matching degree of the ophthalmologist who provided the treatment plan is used as the weight of the treatment plan. Combined with the consensus feature value of the plan, the weighted calculation is used to obtain the recommended value of the treatment plan.

[0107] Sort the treatment plans in descending order according to the recommended values, and fill in the sorted treatment plans and corresponding diagnosis results into the electronic consultation form to generate an electronic consultation form for the patient to be treated;

[0108] It should be noted that the function of this module is to quantify expert information into feature vectors that include professional fields, years of experience, and case experience. The matching degree between patient disease characteristics and expert characteristics is calculated through cosine similarity, and highly matched experts are selected to form a consultation set. Treatment plans are clustered based on Jaccard similarity, and the recommended value is calculated by combining expert weights and plan consensus. An electronic consultation form containing diagnosis conclusions and treatment recommendations is generated.

[0109] Iterative update module: Desensitizes electronic consultation forms and supplements local case data to train the local knowledge graph. The central server generates a global knowledge graph by weighted aggregation of local knowledge graphs from multiple institutions, and then provides feedback and updates to the local knowledge graph based on the generated global knowledge graph.

[0110] Based on the generated electronic consultation form, treatment is provided to the patient in question, along with the patient's multimodal ophthalmic data and the generated electronic consultation form. Patient information is removed, and homomorphic encryption is used to blur the coordinates of the lesion area to ensure pixel-level privacy protection. Natural language processing (NLP) is used to identify and replace sensitive information such as names and medical record numbers, while retaining symptom descriptions and diagnostic terms. This generates desensitized ophthalmic treatment data, preserving clinical features while meeting privacy protection requirements and providing secure input for federated learning.

[0111] Continuously record desensitized ophthalmic treatment data, generate a desensitized dataset, and add it to the local case data. Each medical institution conducts incremental training on the local knowledge graph based on the local case data to optimize the symptom-disease association strength. After training is completed, the graph structure parameters and node feature parameters are extracted to update the local knowledge graph.

[0112] The central server uses the FedAvg algorithm to aggregate the local knowledge graph parameters of multiple institutions. The aggregation weight is calculated based on the amount of local case data of each medical institution to ensure that institutions with rich data contribute more to the global model. The global edge weight matrix and global node embedding vector of the global knowledge graph are obtained by combining the aggregation weight. The global edge weight matrix and global node embedding vector are then distributed to the local knowledge graph of each medical institution, replacing the local knowledge graph parameters and iteratively updating the local knowledge graph.

[0113] It should be noted that this step desensitizes the electronic consultation form data and generates a secure dataset that can be used for federated learning. Each institution incrementally trains the knowledge graph based on local case data, extracting the graph structure and node feature parameters. The central server aggregates the multi-institutional parameters using the FedAvg algorithm, generates a global knowledge graph, and distributes updates to achieve cross-institutional collaborative optimization.

[0114] The technical solution of the embodiment of the present invention is: collecting and integrating ophthalmic multimodal data, generating a multimodal feature set including image feature vectors, text feature vectors, and detection feature vectors through standardization processing and feature extraction, deeply fusing the multimodal feature set, inputting the constructed local ophthalmic knowledge graph, performing association reasoning between symptoms and diseases on the multimodal feature set, generating a probability distribution of ophthalmic diseases of patients to be treated, quantifying detailed information of ophthalmic experts into expert feature vectors, calculating the matching degree between the probability distribution of ophthalmic diseases of patients and the expert feature vectors, screening ophthalmic experts with high matching degree for consultation, calculating the recommended value of treatment plan provided by ophthalmic experts, generating an electronic consultation form including diagnostic conclusions and treatment suggestions based on the recommended value, desensitizing the electronic consultation form to supplement local case data, incrementally training the local knowledge graph based on local case data, and the central server weightedly aggregates the local knowledge graphs of multiple institutions through the FedAvg algorithm to generate a global knowledge graph and update the local knowledge graph.

[0115] Example 2

[0116] like Figure 2 As shown, the ophthalmology remote intelligent consultation method based on multimodal data fusion according to the embodiment of the present invention includes the following steps:

[0117] Step 1: Collect and integrate ophthalmic multimodal data to generate a multimodal feature set;

[0118] Step 2: The multimodal feature set is input into the improved cross-modal attention model, and the final fused features are generated through dynamic weight fusion and bidirectional interaction mechanism. The final fused features are then combined with the local ophthalmic knowledge graph for graph neural network reasoning to output the probability distribution of the patient's ophthalmic disease.

[0119] Step 3: Quantify the ophthalmologist's detailed information into an expert feature vector, calculate the match between the patient's ophthalmic disease probability distribution and the expert feature vector, select ophthalmologists with high matching scores for consultation, calculate the ophthalmologist's recommended treatment plan, and generate an electronic consultation form containing the diagnosis conclusion and treatment suggestions based on the recommended value;

[0120] Step 4: Desensitize the electronic consultation form and supplement local case data to train the local knowledge graph. The central server generates a global knowledge graph by weighted aggregation of local knowledge graphs of multiple institutions, and updates the local knowledge graph based on the generated global knowledge graph.

[0121] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent remote ophthalmology consultation system based on multimodal data fusion, characterized by: include: Data acquisition module: collects and integrates ophthalmic multimodal data to generate a multimodal feature set; Fusion Diagnosis Module: This module inputs the multimodal feature set into an improved cross-modal attention model, generates the final fused features through dynamic weight fusion and a two-way interaction mechanism, and then performs graph neural network reasoning based on the local ophthalmic knowledge graph to output the probability distribution of the patient's ophthalmic disease. Consultation matching module: Quantify detailed information about ophthalmologists into expert feature vectors, calculate the matching degree between the probability distribution of the patient's ophthalmic disease and the expert feature vectors, select ophthalmologists with high matching degrees for consultation, calculate the recommended value of the treatment plan provided by the ophthalmologist, and generate an electronic consultation form containing the diagnosis conclusion and treatment suggestions based on the recommended value; Iterative update module: Desensitizes electronic consultation forms and supplements local case data to train the local knowledge graph. The central server generates a global knowledge graph by weighted aggregation of local knowledge graphs from multiple institutions, and then provides feedback and updates to the local knowledge graph based on the generated global knowledge graph. The probability distribution of the patient's ophthalmic disease is obtained as follows: Based on clinical guidelines and local case data, a directed local ophthalmology knowledge graph was constructed containing three core entities: disease entities, symptom entities, and detection entities. The final fused features were used as the initial feature vectors of the disease entity node feature vectors. A multimodal attention aggregation function was designed to iteratively update the node feature vectors starting from the initial feature vectors. A fully connected layer and a softmax function were applied to the iteratively updated node feature vectors to obtain the probability distribution of the patient's ophthalmic disease. The matching degree is obtained as follows: Detailed information of all ophthalmologists on the platform is collected and features are extracted and quantified to obtain expert feature vectors. The probability distribution of ophthalmic diseases in patients to be treated is obtained, and the probability distribution of ophthalmic diseases in patients is constructed into disease feature vectors for patients to be treated. The disease feature vectors of patients to be treated are then processed with the expert feature vectors of each ophthalmologist to obtain the matching degree between the probability distribution of ophthalmic diseases in patients and the expert feature vectors. The method of providing feedback and updating the local knowledge graph is as follows: Obtain multimodal ophthalmic data of patients to be treated and the generated electronic consultation form and perform desensitization processing to generate desensitized ophthalmic treatment data. Continuously record the desensitized ophthalmic treatment data to generate a desensitized dataset and add it to the local case data. Each medical institution conducts incremental training on the local knowledge graph based on the local case data to optimize the symptom-disease association strength. After the training is completed, the graph structure parameters and node feature parameters are extracted to update the local knowledge graph. The central server uses the FedAvg algorithm to aggregate the local knowledge graph parameters of multiple institutions, calculates the aggregation weight and performs parameter fusion based on the amount of local case data of each medical institution, obtains the global edge weight matrix and global node embedding vector of the global knowledge graph, and distributes them to each medical institution to replace the local knowledge graph parameters, and performs iterative feedback updates on the local knowledge graph.

2. The ophthalmology remote intelligent consultation system based on multimodal data fusion according to claim 1 is characterized by: The multimodal feature set is generated as follows: Collecting and preprocessing multimodal ophthalmic data of patients to be treated, including image data, text data, and test data; performing lesion segmentation and geometric feature extraction on the preprocessed image data to obtain image feature vectors; performing semantic feature extraction on the text data to obtain text feature vectors; and performing structured feature mapping on the test data to obtain test feature vectors; The three modalities of image feature vector, text feature vector and detection feature vector are integrated to generate a multimodal feature set in a unified format.

3. The ophthalmology remote intelligent consultation system based on multimodal data fusion according to claim 2, characterized in that: The final fusion feature is obtained as follows: Obtain a weight matrix and design a bidirectional fusion mechanism based on it. Each modal feature in the multimodal feature set serves as a query to extract features from other modalities and also as a key to provide its own features. Combined with the weight matrix, intra-modal fusion and inter-modal fusion are calculated. Intra-modal fusion retains the core features of each modality, while inter-modal fusion captures cross-modal associations. The final fusion features are obtained by combining intra-modal fusion and inter-modal fusion through layer normalization and stable training.

4. The ophthalmology remote intelligent consultation system based on multimodal data fusion according to claim 3 is characterized by: The weight matrix is ​​obtained as follows: The image feature vector, text feature vector and detection feature vector in the multimodal feature set are mapped to the shared latent space through the fully connected layer, and the multimodal feature set is converted into a shared latent space representation of the same dimension. The trimodal interaction matrix is ​​defined to represent the similarity between the modalities, and the attention weight between any two modal features is calculated and integrated to obtain the weight matrix.

5. The ophthalmology remote intelligent consultation system based on multimodal data fusion according to claim 1, characterized in that: The method for obtaining the electronic consultation form is: In a remote consultation, the ophthalmologists who responded to the request provided treatment plans. The Jaccard similarity was used to calculate the similarity between any two treatment plans. Two treatment plans with a similarity greater than or equal to the similarity threshold were classified into the same similarity plan set. The number of treatment plans in each similarity plan set was normalized and marked as the consensus feature value of the treatment plans in the similarity plan set. Based on any treatment plan, the matching degree of the ophthalmologist providing the treatment plan is obtained as the weight of the treatment plan. Combined with the consensus characteristic value of the treatment plan, the recommended value of the treatment plan is obtained by weighted calculation. The treatment plans are sorted in descending order according to the recommended value, and the sorted treatment plans and corresponding diagnosis results are filled in the electronic consultation form to generate an electronic consultation form.

6. The ophthalmology remote intelligent consultation system based on multimodal data fusion according to claim 5, characterized in that: The matching method of the consulting ophthalmologist who responds to the request is: Obtain the matching degree of all ophthalmologists in the platform, sort the ophthalmologists in descending order according to the matching degree, and mark the ophthalmologists with selection numbers according to the order to obtain a sorted list of ophthalmologists, and select ophthalmologists with selection numbers less than or equal to the selection threshold in the ophthalmologist list to form a consulting ophthalmologist set; A consultation request is sent to ophthalmologists in the consulting ophthalmologist set, and a consulting ophthalmologist who responds to the request is matched.

7. An ophthalmological remote intelligent consultation method based on multimodal data fusion, applied to the ophthalmological remote intelligent consultation system based on multimodal data fusion according to any one of claims 1 to 6, characterized in that: The following steps are involved: Collect and integrate ophthalmic multimodal data to generate a multimodal feature set; The multimodal feature set is input into the improved cross-modal attention model, and the final fused features are generated through dynamic weight fusion and bidirectional interaction mechanism. The graph neural network reasoning is combined with the local ophthalmic knowledge graph to output the probability distribution of the patient's ophthalmic disease. Quantify the detailed information of ophthalmologists into expert feature vectors, calculate the matching degree between the probability distribution of patients' ophthalmic diseases and the expert feature vectors, select ophthalmologists with high matching degree for consultation, calculate the recommended value of the treatment plan provided by the ophthalmologists, and generate an electronic consultation form containing the diagnosis conclusion and treatment suggestions based on the recommended value; The electronic consultation form is desensitized and supplemented with local case data to train the local knowledge graph. The central server generates a global knowledge graph by weighted aggregation of local knowledge graphs of multiple institutions, and provides feedback and updates to the local knowledge graph based on the generated global knowledge graph.

Images