Pathological section intelligent auxiliary differential diagnosis system based on multi-modal fusion

Abstract

The invention relates to a pathological section intelligent auxiliary differential diagnosis system based on multi-modal fusion, in particular to the field of clinical pathology, semantic unification of multi-modal data is achieved through meta-task construction and a cross-modal alignment technology, and transferable diagnostic knowledge is extracted by utilizing a meta-learning framework; the method combines a generative model and knowledge constraints to generate high-quality synthetic data, and finally fuses real and synthetic samples through a self-adaptive diagnosis mechanism, thereby remarkably improving the differential diagnosis capability of rare lesions, effectively solving the problem of model generalization in a training data scarcity scene, and improving the accuracy of model identification. And efficient and reliable intelligent auxiliary decision support is provided for clinical pathological diagnosis.

Description

Technical Field

[0001] This invention relates to the field of clinical pathology, and more specifically, to an intelligent auxiliary differential diagnosis system for pathological slides based on multimodal fusion. Background Technology

[0002] In clinical pathology diagnosis practice, the widespread adoption of digital pathology slide scanning technology and its integration with AI-assisted diagnostic systems have brought enormous potential for improving diagnostic efficiency and consistency. However, existing systems face significant challenges when applied to primary healthcare institutions or specialized centers focusing on rare disease diagnosis and treatment. These institutions often handle cases with high heterogeneity and sparsity, such as certain histological subtypes of sarcomas, rare tumors of the hematopoietic and lymphatic systems, or lesions related to hereditary tumor syndromes. Due to the extremely low incidence of these diseases, collecting multimodal datasets, including digitized pathology slide images that meet diagnostic requirements, as well as corresponding clinical history, laboratory test results, immunohistochemical marker expression profiles, and even molecular genetic testing reports, is exceptionally difficult and costly. Furthermore, differences in data acquisition equipment, staining procedures, diagnostic standards, and reporting formats among different medical institutions further exacerbate the complexity of high-quality, standardized multimodal data aggregation. In this reality of scarce and extremely unevenly distributed data, the deployment effectiveness of intelligent assisted diagnostic systems faces severe challenges.

[0003] Currently, technical solutions aimed at improving the generalization ability of models mainly rely on transfer learning techniques based on large-scale data pre-training, or data augmentation methods such as geometric transformation and color perturbation for single samples of image modalities. These methods have shown some effectiveness in diagnostic tasks for common and frequently occurring diseases, but their core limitation lies in their failure to fully explore and utilize the deep correlations and complementarities between the multimodal information inherent in the pathological diagnosis process. For example, when dealing with an extremely rare alveolar soft tissue sarcoma, the transfer learning model may fail to identify key morphological features because its feature extractor is mainly trained on common cancer types; simply relying on image augmentation cannot synthesize clinical background information specifically related to the disease, such as patient age. The deeper technical problem with rare disease-specific gene fusion events lies in the fact that existing few-shot learning algorithms, such as metric-based prototype networks or optimization-based meta-learning algorithms, are usually designed around single-modal data. They struggle to effectively coordinate high-dimensional pathological image features with discrete clinical text or continuous genomic data. This lack of a multimodal collaborative mechanism prevents the model from efficiently learning a cross-modal representation that can distinguish subtle pathological changes and has strong discriminative power from extremely limited multimodal samples of rare diseases. Consequently, in real clinical applications, the sensitivity and diagnostic specificity for identifying rare pathological patterns are not ideal, severely limiting the practical value and feasibility of such systems in comprehensively assisting pathologists in differential diagnosis. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing an intelligent auxiliary differential diagnosis system for pathological slides based on multimodal fusion. The system solves the problems mentioned in the background by using a meta-task construction module, a meta-knowledge learning module, a data generation and refinement module, and an adaptive diagnosis module.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a meta-task construction module, a meta-knowledge learning module, a data generation and refinement module, and an adaptive diagnosis module, wherein;

[0006] Meta-task construction module: This module receives multimodal data of common diseases from a database. The multimodal data includes digital pathological slide images and corresponding clinical text reports and their pathological classification labels. Based on pathological classification standards, it constructs a set of meta-tasks simulating small-sample diagnostic tasks. Each meta-task includes a support set and a query set. The meta-task construction module also uses a cross-modal alignment loss function to perform semantic space alignment processing on the image features extracted from the digital pathological slide images and the text features extracted from the clinical text reports, and outputs a set of meta-tasks composed of multiple feature-aligned meta-tasks.

[0007] Meta-knowledge learning module: It is used to receive the meta-task set output by the meta-task construction module and perform cross-task training using an improved model-independent meta-learning framework. The inner loop gradient update process of this framework integrates the classification loss based on the support set in the meta-task and the alignment loss calculated by the cross-modal alignment loss function. It learns and obtains a set of meta-learner parameters through the cross-task training.

[0008] The data generation and refinement module receives conditional information about the target rare disease from external input and meta-learner parameters obtained by the meta-knowledge learning module. It then uses a conditional generative adversarial network (GAN) to generate synthetic multimodal data. The conditional input of the generator in the GAN includes prototype features obtained by processing the target rare disease information with the meta-learner parameters. The generation process is constrained by a differentiable pathological knowledge constraint function. The data generation and refinement module also performs performance evaluation and screening on the generated synthetic multimodal data using the meta-learner parameters and outputs a refined set of synthetic multimodal samples.

[0009] Adaptive Diagnosis Module: This module receives a small number of real samples of the target rare disease from external input, forming a real sample set of the target rare disease. It then combines the refined synthetic multimodal sample set output by the data generation and refinement module with the real sample set of the target rare disease to form an enhanced support set. Subsequently, it uses the meta-learner parameters obtained by the meta-knowledge learning module to adapt a diagnostic model using fast gradient descent. The diagnostic model is a neural network model initialized by the meta-learner parameters. Finally, the adapted diagnostic model outputs the differential diagnosis result of the query case.

[0010] In a preferred embodiment, the specific process of constructing a set of meta-tasks simulating small-sample diagnostic tasks based on pathological classification criteria in the meta-task construction module includes:

[0011] First, the meta-task construction module classifies the common disease multimodal data according to the World Health Organization's tumor classification standards, and groups samples with the same fine-grained pathological classification labels into the same pathological category.

[0012] Next, the meta-task construction module performs meta-task sampling, randomly selects one or more categories from the classified pathological categories as the target category of the current meta-task, and then randomly extracts a first preset number of samples from each target category to form the support set of the meta-task, and randomly extracts a second preset number of samples from the target category or other categories that have differential diagnostic significance to form the query set of the meta-task, wherein the second preset number is greater than the first preset number.

[0013] Finally, by repeating the above meta-task sampling process, a meta-task set consisting of multiple independent meta-tasks is generated, where the support set of each meta-task simulates the limited reference cases that can be obtained when diagnosing rare diseases, and the query set simulates new cases that need to be diagnosed.

[0014] In a preferred embodiment, the specific process by which the meta-task construction module performs semantic space alignment processing on image features extracted from pathological slide image data and text features extracted from clinical text reports using a cross-modal alignment loss function includes:

[0015] First, the meta-task construction module uses a pre-trained deep convolutional neural network model to extract high-dimensional image feature vectors from pathological slide image data, and uses a pre-trained text encoder model to extract high-dimensional text feature vectors from clinical text reports.

[0016] Subsequently, the meta-task construction module optimizes two learnable mapping functions by minimizing the cross-modal alignment loss function, which is defined based on optimal transport theory. The image feature vectors constitute an empirical distribution of image features, and the text feature vectors constitute an empirical distribution of text features. The cross-modal alignment loss function achieves semantic alignment by minimizing a total objective value, which consists of two parts:

[0017] The first part is the total expected cost between the image feature empirical distribution and the text feature empirical distribution under the optimal joint transport coupling. This total expected cost is defined by the distance metric function between each pair of image feature vectors and text feature vectors after they are mapped to a unified semantic space by the mapping function.

[0018] The second part is a regularization term, which is characterized by calculating the entropy of the optimal joint transmission coupling and multiplying it by a regularization coefficient; the total objective value is defined as the sum of the total expected cost and the regularization term;

[0019] By minimizing the cross-modal alignment loss function, the two learnable mapping functions are optimized such that the vector obtained by transforming the image feature vector from the same sample through one mapping function and the vector obtained by transforming the text feature vector from the same sample through the other mapping function are close to each other in the unified semantic space, thereby achieving semantic space alignment of features.

[0020] After semantic space alignment processing, the meta-task construction module outputs a set of feature-aligned meta-tasks, where the samples in the support set and query set of each meta-task are represented as aligned feature representations in the unified semantic space.

[0021] In a preferred embodiment, the specific process of cross-task training using an improved model-independent meta-learning framework in the meta-knowledge learning module includes:

[0022] The meta-knowledge learning module receives a set of feature-aligned meta-tasks output by the meta-task construction module. For each meta-task in the set, the meta-knowledge learning module first calculates the multi-task loss function on the support set of the meta-task.

[0023] The multi-task loss function is formed by adding a first part and a second part. The first part is the classification loss based on the support set, which is used to measure the classification accuracy of the current meta-learner parameters on the support set samples. The second part is the alignment loss calculated by the cross-modal alignment loss function, which is used to measure the degree of alignment between the image feature vectors extracted from digital pathology slide images and the text feature vectors extracted from clinical text reports in the unified semantic space. The multi-task loss function is the sum of the classification loss and the alignment loss multiplied by a preset alignment loss weight coefficient.

[0024] In a preferred embodiment, the processing steps of the meta-knowledge learning module after calculating the multi-task loss function include:

[0025] The meta-knowledge learning module performs adaptive update of the inner loop parameters, which means subtracting the product of the gradient of the multi-task loss function with respect to the meta-learner parameters and an inner loop learning rate hyperparameter from the current meta-learner parameters, thereby obtaining the updated meta-learner parameters for the current meta-task.

[0026] Next, the meta-knowledge learning module performs outer loop meta-knowledge optimization. The outer loop meta-knowledge optimization includes defining the outer loop loss function as the mathematical expectation of the query set classification loss on all meta-tasks. That is, the outer loop loss function is the average value of the task adaptation parameters of each meta-task on the query set of that meta-task, and updating the original meta-learner parameters by minimizing the outer loop loss function.

[0027] The update rule for the meta-learner parameters is as follows: subtract the outer loop learning rate hyperparameter from the current meta-learner parameters, and multiply the gradient of the outer loop loss function with respect to the meta-learner parameters.

[0028] In a preferred embodiment, the process by which the meta-knowledge learning module learns and obtains meta-learner parameters through cross-task training includes:

[0029] The meta-knowledge learning module repeatedly executes the process of multi-task loss function calculation, inner loop parameter adaptive update and outer loop meta-knowledge optimization for each meta-task, traversing a large number of meta-tasks in the feature-aligned meta-task set until the meta-learner parameters converge to a stable state.

[0030] The final meta-learner parameters are a set of meta-learner parameters that are highly sensitive to multimodal few-shot learning problems.

[0031] In a preferred embodiment, the specific process of generating synthetic multimodal data using a conditional generative adversarial network in the data generation and refinement module includes:

[0032] First, the data generation and refinement module receives conditional information about the target rare disease from an external source and meta-learner parameters obtained by the meta-knowledge learning module. The meta-learner parameters are used to define and parameterize a feature extraction function. The data generation and refinement module uses the feature extraction function to process the conditional information and maps the conditional information into prototype features representing the core features of the rare disease through forward propagation calculation.

[0033] Then, the data generation and refinement module uses the prototype features and random noise vectors as conditional inputs to the generator in the conditional generative adversarial network (GAN). Through the forward propagation of the generator, it calculates and synthesizes multimodal data samples, which include image feature vectors and text feature vectors. During this generation process, the data generation and refinement module introduces a differentiable pathological knowledge constraint function to restrict the generation process. The calculation of the pathological knowledge constraint function includes:

[0034] First, a pre-trained pathological image feature extractor is used to map the image feature vectors in the generated data to the morphological feature space, and the square of the Euclidean distance between the mapped features and the ideal morphological feature vector corresponding to the rare disease retrieved from the pathological knowledge base is calculated.

[0035] Secondly, calculate the KL divergence between the semantic distribution of the text feature vectors in the generated data and the distribution of the standard clinical text description of the rare disease;

[0036] Finally, the square of the Euclidean distance is multiplied by the KL divergence by a preset weighting coefficient and then added together to obtain the pathological knowledge constraint function value.

[0037] This generates synthetic multimodal data.

[0038] In a preferred embodiment, the specific process of performance evaluation and screening of the generated synthetic multimodal data using meta-learner parameters includes:

[0039] For each generated synthetic multimodal data, the data generation and refining module calculates its performance improvement score, and the calculation method for the performance improvement score is as follows:

[0040] First, select a preset number of NE meta-tasks. For each meta-task, add the synthetic multimodal data to the query set of that meta-task to form an enhanced query set.

[0041] Then, the meta-learner parameters are quickly adapted on the support set of the meta-task to obtain the adapted parameters, and the difference between the classification loss of the adapted parameters on the augmented query set and the classification loss on the original query set is calculated.

[0042] Finally, the average of the differences calculated on all NE meta-tasks is used as the performance improvement score for the synthetic multimodal data sample.

[0043] Subsequently, the data generation and refinement module compares the performance improvement score with a preset threshold and performs a filtering operation to retain only synthetic multimodal data samples whose performance improvement score is greater than the preset threshold; the synthetic multimodal data samples retained after this filtering operation constitute the refined synthetic multimodal sample set.

[0044] In a preferred embodiment, the specific process of constructing an enhanced support set by combining the refined synthetic multimodal sample set with the target rare disease real sample set in the adaptive diagnosis module includes:

[0045] First, the adaptive diagnosis module receives a target rare disease real sample set from an external source. This target rare disease real sample set contains a small number of real samples to simulate the extreme small sample scenario of rare disease diagnosis. At the same time, it receives a refined synthetic multimodal sample set output by the data generation and refinement module. This synthetic multimodal sample set contains multiple synthetic samples.

[0046] Then, the adaptive diagnostic module assigns weight values ​​to each real sample and each synthetic sample respectively. For each real sample, its weight value is calculated based on the sample quality confidence score, which is obtained by evaluating the image clarity, staining quality and clinical text report integrity of the real sample. The confidence score value is between 0 and 1. The weight value of the real sample is 1 plus a preset real sample weight coefficient and the sample quality confidence score of the real sample.

[0047] For each synthetic sample, its weight value is calculated based on the sample quality score, which is obtained by the data generation and refining module through the evaluation of meta-learner parameters during the generation process. The evaluation method is to calculate the average of the performance improvement scores brought by the synthetic sample after adding it to the query set of multiple meta-tasks. The quality score value is between 0 and 1. The weight value of the synthetic sample is the product of a preset synthetic sample weight coefficient and the sample quality score of the synthetic sample.

[0048] The weight coefficients of the real samples and the weight coefficients of the synthetic samples are hyperparameters, which are set through cross-validation.

[0049] Finally, the adaptive diagnostic module fuses all real samples with all weight values ​​assigned with all synthetic samples with all weight values ​​assigned to form an enhanced support set. Each element in the enhanced support set contains four parts of information: image feature vector, text feature vector, class label, and weight value of a sample.

[0050] In a preferred embodiment, the specific process of the adaptive diagnosis module adapting the diagnostic model using meta-learner parameters via fast gradient descent includes:

[0051] First, the adaptive diagnosis module retrieves the meta-learner parameters obtained by the meta-knowledge learning module and uses the meta-learner parameters to initialize the network parameters of the diagnosis model, that is, the initial parameters of the diagnosis model are set to the meta-learner parameters.

[0052] The diagnostic model is a multimodal neural network, whose architecture includes three parts: an image branch, a text branch, and a fusion module. The image branch is composed of multiple residual convolutional blocks connected in series, which are used to extract diagnostic-related features from the input image feature vector.

[0053] The text branch consists of a Transformer encoding layer, used to extract semantic features from the input text feature vector;

[0054] The fusion module uses a cross-modal attention mechanism to fuse the output features of the image branch and the text branch, and outputs the class probability through a softmax layer;

[0055] Then, the adaptive diagnostic module performs fast gradient descent adaptation on the enhanced support set. The adaptation process adopts a meta-learning inner loop optimization strategy. The optimization objective is to minimize the multi-task loss function on the enhanced support set. This multi-task loss function is composed of the sum of the classification loss and the alignment loss. The classification loss adopts the cross-entropy loss function, and the alignment loss adopts the same function as the cross-modal alignment loss function defined in the meta-task construction module.

[0056] During the adaptation process, the update rule for the diagnostic model parameters is as follows: calculate the gradient of the multi-task loss function with respect to the current parameters of the diagnostic model, use a preset fast adaptation learning rate and the product of the gradient as the parameter update amount, and subtract the update amount from the current parameters of the diagnostic model to obtain the new parameters; the value range of the fast adaptation learning rate is 0.01 to 0.05, and the value range of the weight coefficient of the alignment loss function in the multi-task loss function is 0.1 to 0.5;

[0057] By repeating the above parameter update process until the performance of the diagnostic model on the enhanced support set reaches the convergence condition, the adapted diagnostic model parameters are obtained.

[0058] Finally, when diagnosing the queried cases, the adaptive diagnosis module operates according to the following steps:

[0059] The Monte Carlo Dropout method is used to sample the parameters of the adapted diagnostic model 10 to 20 times to generate multiple perturbation versions of the parameters.

[0060] For each parameter perturbation version, calculate the probability distribution of the query case belonging to each pathological category; take the arithmetic mean of the probability distributions generated by all parameter perturbation versions to obtain the calibrated probability distribution; select the category with the highest probability value from the calibrated probability distribution as the primary diagnosis result, multiply the highest probability value by one minus the entropy of the probability distribution as the confidence score, and list all categories and their probability values ​​in descending order of probability value as the differential diagnosis list.

[0061] Finally, the output includes the differential diagnosis result, which contains the primary diagnosis, confidence score, and a list of differential diagnoses.

[0062] The beneficial effects of this invention are: semantic unification of multimodal data is achieved through meta-task construction and cross-modal alignment technology; transferable diagnostic knowledge is extracted using a meta-learning framework; high-quality synthetic data is generated by combining generative models and knowledge constraints; and finally, real and synthetic samples are fused through an adaptive diagnostic mechanism, which significantly improves the ability to differentiate and diagnose rare diseases, effectively solves the model generalization problem in scenarios with scarce training data, and provides efficient and reliable intelligent auxiliary decision support for clinical pathology diagnosis. Attached Figure Description

[0063] Figure 1 This is a flowchart of the method of the present invention;

[0064] Figure 2 This is a block diagram of the system structure of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0067] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0068] Example 1

[0069] This embodiment provides, for example Figure 1-2 The system illustrates an intelligent auxiliary differential diagnosis system for pathological slides based on multimodal fusion, specifically comprising: a meta-task construction module, a meta-knowledge learning module, a data generation and refinement module, and an adaptive diagnosis module, wherein;

[0070] The meta-task construction module receives multimodal data of common diseases from the database. The multimodal data of common diseases includes digital pathological slide images and corresponding clinical text reports and their pathological classification labels. Based on the pathological classification criteria, it constructs a set of meta-tasks to simulate small-sample diagnostic tasks. Each meta-task includes a support set and a query set. The meta-task construction module also uses a cross-modal alignment loss function to perform semantic space alignment processing on the image features extracted from digital pathological slide images and the text features extracted from clinical text reports. The output is a set of meta-tasks composed of multiple feature-aligned meta-tasks.

[0071] Meta-knowledge learning module: It is used to receive the meta-task set output by the meta-task building module and to perform cross-task training using an improved model-independent meta-learning framework. The inner loop gradient update process of this framework integrates the classification loss based on the support set in the meta-task and the alignment loss calculated by the cross-modal alignment loss function. It learns and obtains a set of meta-learner parameters through cross-task training.

[0072] The data generation and refinement module receives conditional information about the target rare disease from external input and meta-learner parameters obtained from the meta-knowledge learning module. It then uses a conditional generative adversarial network (GAN) to generate synthetic multimodal data. The conditional input of the generator in the GAN includes prototype features obtained by processing the target rare disease information with the meta-learner parameters. The generation process is constrained by a differentiable pathological knowledge constraint function. The data generation and refinement module also evaluates and filters the generated synthetic multimodal data through the meta-learner parameters and outputs a refined set of synthetic multimodal samples.

[0073] The adaptive diagnosis module receives a small number of real samples of the target rare disease from external input, forming a real sample set of the target rare disease. It then combines the refined synthetic multimodal sample set output by the data generation and refinement module with the real sample set of the target rare disease to form an enhanced support set. Subsequently, it uses the meta-learner parameters obtained by the meta-knowledge learning module to adapt a diagnostic model using fast gradient descent. The diagnostic model is a neural network model initialized by the meta-learner parameters. Finally, the adapted diagnostic model outputs the differential diagnosis result of the query case.

[0074] In this embodiment, it is specifically necessary to explain the process of constructing a set of meta-tasks for simulating small-sample diagnostic tasks based on pathological classification standards in the meta-task construction module, which includes:

[0075] First, the meta-task construction module classifies common disease multimodal data into fine-grained pathological categories (such as "gastric adenocarcinoma, intestinal type" and "lung squamous cell carcinoma, keratotic type") according to the World Health Organization's tumor classification standards. Samples with the same fine-grained pathological classification labels are grouped into the same pathological category. The division of fine-grained pathological categories strictly follows the latest tumor classification standards published by the World Health Organization, which is the globally recognized gold standard for pathological diagnosis, ensuring the pathological rationality of the constructed meta-task. For example, "gastric adenocarcinoma, intestinal type" has specific morphological characteristics and clinical prognosis, and there is a need for differential diagnosis with other subtypes. This classification method enables the constructed meta-task to simulate real clinical differential diagnosis scenarios.

[0076] Next, the meta-task construction module performs meta-task sampling, randomly selecting one or more categories from the classified pathology categories as the target category for the current meta-task. Then, it randomly extracts a first preset number of samples from each target category (the value is set between 1 and 5, aiming to accurately simulate the core challenge of extremely few reference cases in the diagnosis of rare diseases in clinical practice) to jointly form the support set for the meta-task. A second preset number of samples are randomly extracted from the target category or other categories with differential diagnostic significance to form the query set for the meta-task, where the second preset number is greater than the first preset number (depending on actual needs). (Adaptive settings required); Other categories with differential diagnostic significance refer to disease types that are easily confused with the target category in terms of morphology, immunophenotype, or clinical manifestations based on pathological knowledge; for example, if the target category is "classical Hodgkin lymphoma", its differential diagnostic categories may include "anaplastic large cell lymphoma" or "large B-cell lymphoma rich in T cells / histocytes"; This sampling strategy forces the meta-learner to learn to extract key differential features from a very small number of reference cases and effectively distinguish similar diseases, thereby significantly improving the meta-learner's generalization ability and robustness in the diagnosis of real rare diseases;

[0077] Finally, by repeating the above meta-task sampling process, a meta-task set consisting of multiple independent meta-tasks is generated. The support set of each meta-task simulates the limited reference cases available when diagnosing rare diseases, and the query set simulates new cases that need to be diagnosed. The meta-task set serves as an intermediate result and is input into the subsequent semantic space alignment process. After alignment, the feature-aligned meta-task set is output. By repeating this sampling process extensively, the final generated meta-task set covers a variety of disease types and diagnostic scenarios of different difficulty levels, providing rich and high-quality training data for the subsequent meta-knowledge learning module. This design ensures that the entire system can effectively perform few-shot learning, and the quality of the constructed meta-tasks directly determines the upper limit of the meta-learner's performance in rare disease diagnosis.

[0078] The specific process of semantic space alignment processing of image features extracted from pathological slide image data and text features extracted from clinical text reports using the cross-modal alignment loss function in the meta-task construction module includes:

[0079] First, the meta-task construction module uses a pre-trained deep convolutional neural network model to extract high-dimensional image feature vectors from pathological slide image data, and a pre-trained text encoder model to extract high-dimensional text feature vectors from clinical text reports. The pre-trained deep convolutional neural network model is preferably a deep convolutional neural network model (such as ResNet or EfficientNet) pre-trained on large-scale natural image datasets (such as ImageNet) and / or large-scale pathological image datasets to obtain powerful general visual feature extraction capabilities. The pre-trained text encoder model is preferably a domain-specific Transformer model (such as Clinical BERT or BioBERT) trained on medical literature and clinical reports to accurately understand the semantic information of medical terms and clinical descriptions.

[0080] Subsequently, the meta-task construction module optimizes two learnable mapping functions by minimizing a cross-modal alignment loss function. The cross-modal alignment loss function is defined based on optimal transport theory, where image feature vectors constitute an empirical distribution of image features, and text feature vectors constitute an empirical distribution of text features. The cross-modal alignment loss function achieves semantic alignment by minimizing an overall objective value, which consists of two parts:

[0081] The first part is the total expected cost between the empirical distribution of image features and the empirical distribution of text features under the optimal joint transport coupling. This total expected cost is defined by the distance metric function between each pair of image feature vectors and text feature vectors after they are mapped to a unified semantic space by a mapping function.

[0082] The second part is a regularization term, which is characterized by calculating the entropy of the optimal joint transport coupling and multiplying it by a regularization coefficient. Its role is to constrain the complexity of the optimal joint transport coupling to ensure the smoothness of the solution and computational feasibility. The total objective value is defined as the sum of the total expected cost and the regularization term. Optimal transport theory transforms the alignment problem of different modal features into a quality transport optimization problem between two probability distributions (image feature distribution and text feature distribution). The distance metric function is usually cosine distance or Euclidean distance. The introduction of the regularization term (entropy-based regularization) is key. By encouraging more "dispersed" transport coupling, it avoids the alignment algorithm from getting trapped in trivial local optima (such as assigning all image features to the same text feature), thus ensuring that the alignment algorithm can efficiently solve for the optimal alignment method that reflects the true semantic association. The value of the regularization coefficient is determined by cross-validation to balance alignment accuracy and computational efficiency.

[0083] By minimizing the cross-modal alignment loss function, two learnable mapping functions are optimized so that the vector obtained by transforming the image feature vector from the same sample through one mapping function and the vector obtained by transforming the text feature vector from the same sample through the other mapping function are spatially close to each other in a unified semantic space, thus achieving semantic space alignment of features. The technical effect of this alignment process is significant. After alignment, for example, a pathological slide image showing the feature of "lung adenocarcinoma" and a text report describing the typical clinical manifestations of "lung adenocarcinoma" will have very similar feature representations in a unified semantic space, even if they come from different cases. Conversely, features of different diseases will be far apart. This lays a high-quality, semantically consistent feature foundation for subsequent meta-learning and greatly enhances the ability of the meta-learner to learn discriminative features from small samples.

[0084] After semantic space alignment, the meta-task construction module outputs a feature-aligned meta-task set, where the samples in the support set and query set of each meta-task are represented as aligned feature representations in a unified semantic space. The final output feature-aligned meta-task set has each sample as a compact feature vector that integrates image and text semantic information. This set is the direct input to the subsequent meta-knowledge learning module. Compared to the original heterogeneous multimodal data, this aligned feature set eliminates modal differences and highlights semantic commonalities, enabling the meta-learner to focus more on learning general discrimination rules across tasks rather than being interfered with by modal differences. This is a key preprocessing step to improve the system's performance in rare disease diagnosis.

[0085] In this embodiment, it is specifically necessary to explain the process by which the meta-knowledge learning module uses an improved model-independent meta-learning framework for cross-task training, including:

[0086] The meta-knowledge learning module receives a set of feature-aligned meta-tasks output by the meta-task construction module. For each meta-task in the set, the meta-knowledge learning module first calculates the multi-task loss function on the support set of the meta-task.

[0087] The multi-task loss function is composed of the sum of the first and second parts. The first part is the classification loss based on the support set, which measures the classification accuracy of the current meta-learner parameters on the support set samples. The second part is the alignment loss calculated by the cross-modal alignment loss function, which measures the degree of alignment between the image feature vectors extracted from digital pathology slide images and the text feature vectors extracted from clinical text reports in the unified semantic space. The multi-task loss function is the sum of the classification loss and the alignment loss multiplied by a preset alignment loss weight coefficient. The alignment loss weight coefficient is set to a value between 0.1 and 1.0. The specific values ​​are determined on the validation set through grid search or Bayesian optimization methods to balance the importance of classification accuracy and modality alignment, and to avoid a single loss term dominating the training process. By constructing this multi-task loss function, the meta-learner needs to improve classification accuracy and maintain semantic consistency between multimodal features when adapting to each meta-task. This dual-loss joint optimization design ensures that the meta-learner will not destroy the multimodal semantic alignment relationship established in the meta-task construction stage due to excessive focus on classification accuracy when quickly adapting to new tasks, thereby significantly improving the robustness and generalization ability of the meta-learner in real multimodal diagnostic scenarios.

[0088] The meta-knowledge learning module's processing after calculating the multi-task loss function includes:

[0089] The meta-knowledge learning module performs adaptive updates of the inner loop parameters. This involves subtracting the gradient of the multi-task loss function with respect to the meta-learner parameters from the current meta-learner parameters and multiplying it by an inner loop learning rate hyperparameter (which controls the step size for parameter adaptation). This yields the updated meta-learner parameters for the current meta-task. The inner loop learning rate hyperparameter is typically set to a small value, such as below 0.01, to ensure the stability of the inner loop update process and avoid training oscillations caused by excessively large single-step updates.

[0090] Next, the meta-knowledge learning module performs outer loop meta-knowledge optimization. This optimization involves defining the outer loop loss function as the mathematical expectation of the query set classification loss for all meta-tasks. In other words, the outer loop loss function is the average of the task adaptation parameters of each meta-task and their classification loss on the query set of that meta-task. The original meta-learner parameters are updated by minimizing this outer loop loss function. Furthermore, the loop optimization process is essentially the core of meta-knowledge extraction. Its goal is to find a set of initial model parameters that allow the meta-learner to achieve excellent performance with only a few inner loop updates when facing new tasks.

[0091] The update rule for the meta-learner parameters is as follows: subtract the outer loop learning rate hyperparameter from the current meta-learner parameters and multiply it by the gradient of the outer loop loss function with respect to the meta-learner parameters. This gradient calculation involves the second derivative (Hessian matrix) and is implemented through automatic differentiation. The outer loop learning rate hyperparameter is set on the order of 0.001 to ensure the stable convergence of the meta-learner parameters. Although the calculation of the second derivative is complex, it can be efficiently implemented with the help of the automatic differentiation mechanism of modern deep learning frameworks (such as PyTorch and TensorFlow), ensuring the feasibility of the entire training process.

[0092] The process by which the meta-knowledge learning module learns and obtains meta-learner parameters through cross-task training includes:

[0093] The meta-knowledge learning module repeatedly executes the process of multi-task loss function calculation, inner loop parameter adaptive update, and outer loop meta-knowledge optimization for each meta-task, traversing a large number of meta-tasks in the feature-aligned meta-task set until the meta-learner parameters converge to a stable state. The convergence criterion is set as follows: the outer loop loss function no longer decreases significantly (the decrease is less than a preset threshold, such as 0.0001) or reaches the maximum number of training epochs (such as 10,000 iterations) in multiple consecutive epochs (e.g., 10 epochs) on the validation meta-task set.

[0094] The final meta-learner parameters are a set of parameters that are highly sensitive to multimodal few-shot learning problems. These parameters can quickly adapt to new rare disease diagnosis tasks with a very small number of gradient update steps. Furthermore, these meta-learner parameters inherently maintain multimodal semantic consistency during rapid adaptation, providing crucial prior knowledge for subsequent data generation and refinement modules. The technical effect of the final meta-learner parameters is that they encapsulate the ability to "learn quickly" learned from a large number of common disease meta-tasks. When faced with a completely new rare disease diagnosis task, only 1-5 supporting samples are needed. The diagnostic model based on these meta-learner parameters can achieve high diagnostic accuracy through 1-3 internal loop updates, while ensuring the consistency of image features and clinical text features in the semantic space. This greatly enhances the practical value of the system in clinical practice.

[0095] In this embodiment, it is specifically necessary to explain the process of generating synthetic multimodal data using conditional generative adversarial networks in the data generation and refinement module, which includes:

[0096] First, the data generation and refinement module receives conditional information about the target rare disease from an external source, as well as meta-learner parameters obtained from the meta-knowledge learning module. These meta-learner parameters define and parameterize a feature extraction function. The data generation and refinement module then uses this feature extraction function to process the conditional information, mapping it to prototype features representing the core characteristics of the rare disease through forward propagation. Specifically, the feature extraction function is the forward propagation network within the meta-learner, whose parameters are trained by the meta-knowledge learning module. This network can transform abstract disease conditional information into discriminative feature representations. The prototype features are set to a dimension of 256-1024 to ensure they fully represent the essential characteristics of the disease.

[0097] Then, the data generation and refinement module uses the prototype features and random noise vectors as conditional inputs to the generator in the conditional generative adversarial network (GAN). Through the generator's forward propagation, it calculates and synthesizes multimodal data samples, which include image feature vectors and text feature vectors. During this generation process, the data generation and refinement module introduces a differentiable pathological knowledge constraint function to restrict the generation process. The calculation of the pathological knowledge constraint function includes:

[0098] First, a pre-trained pathological image feature extractor is used to map the image feature vectors in the generated data to the morphological feature space. The square of the Euclidean distance between the mapped features and the ideal morphological feature vector corresponding to the rare disease retrieved from the pathological knowledge base is calculated. The pre-trained pathological image feature extractor is a convolutional neural network, such as ResNet50 or EfficientNet-B4, pre-trained on a large-scale pathological image dataset, with an output feature dimension of 512-2048. The ideal morphological feature vector is a digital representation of the typical morphological features of the disease extracted from authoritative pathological atlases.

[0099] Secondly, the KL divergence between the semantic distribution of the text feature vectors in the generated data and the standard clinical text description distribution of the rare disease is calculated. The semantic distribution of the text feature vectors is represented by the output probability distribution of the language model, and the standard clinical text description distribution is the probability distribution of the typical clinical manifestations of the disease obtained from standard medical literature.

[0100] Finally, the square of the Euclidean distance and the KL divergence are multiplied by a preset weighting coefficient and then added together to obtain the pathological knowledge constraint function value. The weighting coefficient is set to a range of 0.5-2.0 to balance the relative importance of image modality and text modality constraints. The specific value is determined through cross-validation.

[0101] Furthermore, the generation process requires that the value of the pathological knowledge constraint function does not exceed a preset constraint threshold (the constraint threshold is set at 0.1-0.5, which ensures the consistency between the generated data and medical knowledge while allowing a certain degree of generation diversity). This ensures that the consistency between the generated data and medical knowledge is within the allowable range, thereby generating synthetic multimodal data that has both data augmentation value and medical rationality. This knowledge-constrained generation method ensures that the synthetic data is not only realistic in appearance, but more importantly, accurate in medical semantics. It avoids generating invalid data that has no diagnostic value or contradicts medical common sense, significantly improving the practicality and reliability of the generated data.

[0102] The specific process of performance evaluation and screening of the generated synthetic multimodal data using meta-learner parameters includes:

[0103] For each generated synthetic multimodal dataset, the data generation and refining module calculates its performance improvement score. The calculation method for the performance improvement score is as follows:

[0104] First, select a preset number of NE meta-tasks (the value of NE is usually 50-200 to ensure that the evaluation results are statistically significant). For each meta-task, add synthetic multimodal data to the query set of that meta-task to form an enhanced query set.

[0105] Then, the meta-learner parameters are quickly adapted on the support set of the meta-task to obtain the adapted parameters, and the difference between the classification loss of the adapted parameters on the augmented query set and the classification loss on the original query set is calculated. The number of quick adaptation steps is usually 1-3 steps, and the inner loop learning rate is set to 0.01-0.05 to simulate the real few-shot learning scenario.

[0106] Finally, the average of the differences calculated on all NE meta-tasks is taken as the performance improvement score of the synthetic multimodal data sample. A negative performance improvement score indicates a performance improvement, and the larger the negative value, the more significant the improvement.

[0107] Subsequently, the data generation and refinement module compares the performance improvement score with a preset threshold, which is set to a negative value between -0.05 and -0.2, with the specific value adjusted according to the task difficulty and requirements. The more negative the threshold, the stricter the selection criteria, and a selection operation is performed to retain only synthetic multimodal data samples with a performance improvement score greater than the preset threshold. The synthetic multimodal data samples retained after this selection operation constitute the refined synthetic multimodal sample set. This refined synthetic multimodal sample set, as the final output of the data generation and refinement module, will be input into the adaptive diagnostic module as augmentation training data. This selection mechanism based on meta-learning performance ensures that the final retained synthetic samples are all "high-quality" samples that can effectively improve the model's small-sample learning ability, avoiding interference from invalid data on model training, and significantly improving the efficiency and effectiveness of data augmentation.

[0108] In this embodiment, it is specifically necessary to explain the process by which the refined synthetic multimodal sample set and the target rare disease real sample set are combined to form the enhanced support set in the adaptive diagnosis module, including:

[0109] First, the adaptive diagnosis module receives a target rare disease real sample set from an external source. This target rare disease real sample set contains a small number of real samples, typically 1 to 5, to simulate the extremely small sample scenario of rare disease diagnosis. The selection of 1 to 5 real samples is based on the actual clinical rare disease diagnosis scenario, because many rare pathological subtypes have very few diagnosed cases per year in a single medical center, and this range can effectively simulate the real-world diagnostic challenges. Simultaneously, it receives a refined synthetic multimodal sample set output by the data generation and refinement module. This synthetic multimodal sample set contains multiple synthetic samples, typically 10 to 100 times the number of real samples, to provide sufficient data diversity while avoiding synthetic data dominating the training process.

[0110] Then, the adaptive diagnostic module assigns weight values ​​to each real sample and each synthetic sample. For each real sample, the weight value is calculated based on the sample quality confidence score, which is obtained by evaluating the image sharpness, staining quality, and clinical text report completeness of the real sample. The confidence score value is between 0 and 1. Image sharpness is quantified by calculating the sharpness and resolution of the pathological slide image. Staining quality is measured by evaluating the normalization degree of hematoxylin-eosin staining contrast. Clinical text report completeness is assessed by checking the completeness of key fields such as medical history summary, microscopic description, and preliminary diagnosis. The weight value of the real sample is 1 plus a preset real sample weight coefficient multiplied by the sample quality confidence score of the real sample. This weight calculation method ensures that even poor quality real samples (confidence close to 0) still have a base weight of 1, while high-quality real samples (confidence close to 1) can obtain a weight of up to 1.3, highlighting the contribution of high-quality real samples.

[0111] For each synthetic sample, its weight is calculated based on the sample quality score. The sample quality score is obtained by the data generation and refining module through the evaluation of meta-learner parameters during the generation process. The evaluation method is to calculate the average performance improvement score after adding the synthetic sample to the query set of multiple meta-tasks. The quality score is between 0 and 1. The calculation of the performance improvement score involves adding the synthetic sample to the query set of 50-200 meta-tasks and statistically analyzing the average improvement of the model performance. The larger the improvement, the higher the quality of the synthetic sample. The weight of the synthetic sample is the product of a preset synthetic sample weight coefficient and the sample quality score of the synthetic sample. The weight of the synthetic sample is completely determined by the quality score, with a maximum weight of 0.7, reflecting the supplementary role of synthetic data to real data.

[0112] The weight coefficients of real samples and synthetic samples are hyperparameters, set through cross-validation, with typical values ​​of 0.3 and 0.7, respectively, used to balance the relative importance of real samples and synthetic samples in the augmentation support set. The weight coefficients are optimized and determined on the validation meta-task set through five-fold cross-validation, with the optimization objective being the macro F1 score of the model on the rare disease diagnosis task.

[0113] Finally, the adaptive diagnostic module fuses all weighted real samples with all weighted synthetic samples to form an enhanced support set. Each element in this enhanced support set contains four parts: image feature vector, text feature vector, class label, and weight value. The image and text feature vectors originate from the unified semantic space defined in the meta-task construction module. The image feature vector has a dimension of 1024, and the text feature vector has a dimension of 768. The choice of dimension for the unified semantic space has been verified through extensive experiments. 1024 and 768 dimensions can better balance feature representation ability and model complexity. Too low a dimension will lead to information loss, while too high a dimension will increase the risk of overfitting. As the input data for subsequent fast gradient descent adaptation, the enhanced support set, through its construction, can achieve intelligent weighted fusion of the authority of real data and the diversity of synthetic data, providing an optimal data foundation for few-shot learning.

[0114] The adaptive diagnostic module utilizes meta-learner parameters to perform fast gradient descent adaptation on the diagnostic model. The specific process includes:

[0115] First, the adaptive diagnosis module retrieves the meta-learner parameters obtained from the meta-knowledge learning module and uses these meta-learner parameters to initialize the network parameters of the diagnosis model. That is, the initial parameters of the diagnosis model are set as the meta-learner parameters. This initialization method ensures that the diagnosis model starts from an initial point that is highly sensitive to multimodal few-shot learning problems, which greatly accelerates the convergence speed of the subsequent adaptation process.

[0116] The diagnostic model is a multimodal neural network whose architecture consists of three parts: an image branch, a text branch, and a fusion module. The image branch is composed of multiple residual convolutional blocks connected in series, which are used to extract diagnostic-related features from the input image feature vector. The input dimension of the image feature vector is 1024-dimensional. The residual convolutional blocks adopt a pre-activation structure, which contains a sequence of batch normalization, ReLU activation, and 3x3 convolution. A total of 8-16 such blocks are stacked to gradually extract multi-level morphological features.

[0117] The text branch consists of a Transformer encoding layer, which is used to extract semantic features from the input text feature vector. The input dimension of the text feature vector is 768-dimensional. The Transformer encoding layer contains a multi-head self-attention mechanism and a feedforward neural network, using 12 attention heads to capture long-distance dependencies in clinical text.

[0118] The fusion module uses a cross-modal attention mechanism to fuse the output features of the image branch and the text branch, and outputs the class probability through a softmax layer. The cross-modal attention mechanism adopts a query-key-value model, in which image features are used as queries and text features are used as keys and values, to achieve adaptive focusing of image information on text information.

[0119] Then, the adaptive diagnostic module performs fast gradient descent adaptation on the enhanced support set. The adaptation process adopts a meta-learning inner loop optimization strategy. The optimization objective is to minimize the multi-task loss function on the enhanced support set. This multi-task loss function is composed of the sum of the classification loss and the alignment loss. The classification loss adopts the cross-entropy loss function to ensure that the diagnostic model correctly classifies the samples in the enhanced support set. The alignment loss adopts the same function as the cross-modal alignment loss function defined in the meta-task construction module to maintain the consistency between image features and text features in the unified semantic space. The design of the dual-loss joint optimization mechanism ensures that the diagnostic model does not destroy the learned multimodal semantic alignment when quickly adapting to new tasks. This is the core of improving the robustness of diagnosis.

[0120] During the adaptation process, the update rule for the diagnostic model parameters is as follows: calculate the gradient of the multi-task loss function with respect to the current parameters of the diagnostic model, use a preset fast adaptation learning rate multiplied by this gradient as the parameter update amount, and subtract this update amount from the current parameters of the diagnostic model to obtain the new parameters; the fast adaptation learning rate ranges from 0.01 to 0.05, and the weight coefficient of the alignment loss function in the multi-task loss function ranges from 0.1 to 0.5. The learning rate is determined through linear search; too large a value will lead to unstable training, while too small a value will significantly reduce adaptation efficiency; the setting of the alignment loss weight balances classification accuracy and modality consistency; too high a weight will weaken classification performance, while too low a weight may lead to modality alignment degradation.

[0121] By repeating the above parameter update process until the performance of the diagnostic model on the enhanced support set reaches the convergence condition, the parameters of the adapted diagnostic model are obtained. The convergence condition is set to the loss function decreasing by less than 0.0001 for 10 consecutive iterations, or the maximum number of iterations reaches 1000.

[0122] Finally, when diagnosing the queried cases, the adaptive diagnosis module operates according to the following steps:

[0123] The Monte Carlo Dropout method is used to sample the parameters of the adapted diagnostic model 10 to 20 times, generating multiple perturbation versions of the parameters. The dropout rate of Monte Carlo Dropout is set to 0.1-0.3, which can effectively introduce uncertainty without excessively impairing performance.

[0124] For each parameter perturbation version, the probability distribution of the query case belonging to each pathological category is calculated; the probability distributions generated by all parameter perturbation versions are arithmetically averaged to obtain the calibrated probability distribution; from the calibrated probability distribution, the category with the highest probability value is selected as the primary diagnosis result, and the difference between multiplying the highest probability value by one and subtracting the entropy of the probability distribution is used as the confidence score. All categories and their probability values ​​are listed in descending order of probability value as a differential diagnosis list. The confidence score formula combines the maximum value of the predicted probability and the overall uncertainty. When the probability distribution is concentrated, the entropy is small and the confidence is high; when the distribution is dispersed, the entropy is large and the confidence is low, providing doctors with a reliable decision reference.

[0125] Finally, the output includes the primary diagnosis, confidence score, and differential diagnosis list. This report format conforms to clinical pathology diagnostic standards. The primary diagnosis provides clear direction, the confidence score helps assess reliability, and the differential diagnosis list ensures comprehensive consideration, greatly enhancing the clinical practical value of the system.

[0126] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0127] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0131] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0132] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A pathological slide intelligent auxiliary differential diagnosis system based on multimodal fusion, characterized in that, Specifically, it includes: The module comprises a meta-task construction module, a meta-knowledge learning module, a data generation and refinement module, and an adaptive diagnosis module. Meta-task construction module: This module receives multimodal data of common diseases from a database. The multimodal data includes digital pathological slide images and corresponding clinical text reports and their pathological classification labels. Based on pathological classification standards, it constructs a set of meta-tasks simulating small-sample diagnostic tasks. Each meta-task includes a support set and a query set. The meta-task construction module also uses a cross-modal alignment loss function to perform semantic space alignment processing on the image features extracted from the digital pathological slide images and the text features extracted from the clinical text reports, and outputs a set of meta-tasks composed of multiple feature-aligned meta-tasks. Meta-knowledge learning module: It is used to receive the meta-task set output by the meta-task construction module and perform cross-task training using an improved model-independent meta-learning framework. The inner loop gradient update process of this framework integrates the classification loss based on the support set in the meta-task and the alignment loss calculated by the cross-modal alignment loss function. It learns and obtains a set of meta-learner parameters through the cross-task training. The data generation and refinement module receives conditional information about the target rare disease from external input and meta-learner parameters obtained by the meta-knowledge learning module. It then uses a conditional generative adversarial network (GAN) to generate synthetic multimodal data. The conditional input of the generator in the GAN includes prototype features obtained by processing the target rare disease information with the meta-learner parameters. The generation process is constrained by a differentiable pathological knowledge constraint function. The data generation and refinement module also performs performance evaluation and screening on the generated synthetic multimodal data using the meta-learner parameters and outputs a refined set of synthetic multimodal samples. The adaptive diagnosis module receives a small number of real samples of the target rare disease from external input, forming a real sample set of the target rare disease. It then combines the refined synthetic multimodal sample set output by the data generation and refinement module with the real sample set of the target rare disease to form an enhanced support set. Subsequently, it uses the meta-learner parameters obtained by the meta-knowledge learning module to perform fast gradient descent adaptation on a diagnostic model. The diagnostic model is a neural network model initialized by the meta-learner parameters. Finally, the adapted diagnostic model outputs the differential diagnosis result of the query case.

2. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 1, characterized in that: The specific process of constructing a set of meta-tasks simulating small-sample diagnostic tasks based on pathological classification criteria in the meta-task construction module includes: First, the meta-task construction module classifies the common disease multimodal data according to the World Health Organization's tumor classification standards, and groups samples with the same fine-grained pathological classification labels into the same pathological category. Next, the meta-task construction module performs meta-task sampling, randomly selects one or more categories from the classified pathological categories as the target category of the current meta-task, and then randomly extracts a first preset number of samples from each target category to form the support set of the meta-task, and randomly extracts a second preset number of samples from the target category or other categories that have differential diagnostic significance to form the query set of the meta-task, wherein the second preset number is greater than the first preset number. Finally, by repeating the above meta-task sampling process, a meta-task set consisting of multiple independent meta-tasks is generated, where the support set of each meta-task simulates the limited reference cases that can be obtained when diagnosing rare diseases, and the query set simulates new cases that need to be diagnosed.

3. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 2, characterized in that: The specific process of semantic space alignment processing of image features extracted from pathological slide image data and text features extracted from clinical text reports using the cross-modal alignment loss function in the meta-task construction module includes: First, the meta-task construction module uses a pre-trained deep convolutional neural network model to extract high-dimensional image feature vectors from pathological slide image data, and uses a pre-trained text encoder model to extract high-dimensional text feature vectors from clinical text reports. Subsequently, the meta-task construction module optimizes two learnable mapping functions by minimizing the cross-modal alignment loss function, which is defined based on optimal transport theory. The image feature vectors constitute an empirical distribution of image features, and the text feature vectors constitute an empirical distribution of text features. The cross-modal alignment loss function achieves semantic alignment by minimizing a total objective value, which consists of two parts: The first part is the total expected cost between the image feature empirical distribution and the text feature empirical distribution under the optimal joint transport coupling. This total expected cost is defined by the distance metric function between each pair of image feature vectors and text feature vectors after they are mapped to a unified semantic space by the mapping function. The second part is a regularization term, which is characterized by calculating the entropy of the optimal joint transmission coupling and multiplying it by a regularization coefficient; the total objective value is defined as the sum of the total expected cost and the regularization term; By minimizing the cross-modal alignment loss function, the two learnable mapping functions are optimized such that the vector obtained by transforming the image feature vector from the same sample through one mapping function and the vector obtained by transforming the text feature vector from the same sample through the other mapping function are close to each other in the unified semantic space, thereby achieving semantic space alignment of features. After semantic space alignment processing, the meta-task construction module outputs a set of feature-aligned meta-tasks, where the samples in the support set and query set of each meta-task are represented as aligned feature representations in the unified semantic space.

4. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 3, characterized in that: In the meta-knowledge learning module, the specific process of cross-task training using an improved model-independent meta-learning framework includes: The meta-knowledge learning module receives a set of feature-aligned meta-tasks output by the meta-task construction module. For each meta-task in the set, the meta-knowledge learning module first calculates the multi-task loss function on the support set of the meta-task. The multi-task loss function is formed by adding a first part and a second part. The first part is the classification loss based on the support set, which is used to measure the classification accuracy of the current meta-learner parameters on the support set samples. The second part is the alignment loss calculated by the cross-modal alignment loss function, which is used to measure the degree of alignment between the image feature vectors extracted from digital pathology slide images and the text feature vectors extracted from clinical text reports in the unified semantic space. The multi-task loss function is the sum of the classification loss and the alignment loss multiplied by a preset alignment loss weight coefficient.

5. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 4, characterized in that: The processing steps of the meta-knowledge learning module after calculating the multi-task loss function include: The meta-knowledge learning module performs adaptive update of the inner loop parameters, which means subtracting the product of the gradient of the multi-task loss function with respect to the meta-learner parameters and an inner loop learning rate hyperparameter from the current meta-learner parameters, thereby obtaining the updated meta-learner parameters for the current meta-task. Next, the meta-knowledge learning module performs outer loop meta-knowledge optimization. The outer loop meta-knowledge optimization includes defining the outer loop loss function as the mathematical expectation of the query set classification loss on all meta-tasks. That is, the outer loop loss function is the average value of the task adaptation parameters of each meta-task on the query set of that meta-task, and updating the original meta-learner parameters by minimizing the outer loop loss function. The update rule for the meta-learner parameters is as follows: subtract the outer loop learning rate hyperparameter from the current meta-learner parameters, and multiply the gradient of the outer loop loss function with respect to the meta-learner parameters.

6. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 5, characterized in that: The process by which the meta-knowledge learning module learns and obtains meta-learner parameters through cross-task training includes: The meta-knowledge learning module repeatedly executes the process of multi-task loss function calculation, inner loop parameter adaptive update and outer loop meta-knowledge optimization for each meta-task, traversing a large number of meta-tasks in the feature-aligned meta-task set until the meta-learner parameters converge to a stable state. The final meta-learner parameters are a set of meta-learner parameters that are highly sensitive to multimodal few-shot learning problems.

7. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 6, characterized in that: The specific process of generating synthetic multimodal data using conditional generative adversarial networks in the data generation and refinement module includes: First, the data generation and refinement module receives conditional information about the target rare disease from an external source and meta-learner parameters obtained by the meta-knowledge learning module. The meta-learner parameters are used to define and parameterize a feature extraction function. The data generation and refinement module uses the feature extraction function to process the conditional information and maps the conditional information into prototype features representing the core features of the rare disease through forward propagation calculation. Then, the data generation and refinement module uses the prototype features and random noise vectors as conditional inputs to the generator in the conditional generative adversarial network (GAN). Through the forward propagation of the generator, it calculates and synthesizes multimodal data samples, which include image feature vectors and text feature vectors. During this generation process, the data generation and refinement module introduces a differentiable pathological knowledge constraint function to restrict the generation process. The calculation of the pathological knowledge constraint function includes: First, a pre-trained pathological image feature extractor is used to map the image feature vectors in the generated data to the morphological feature space, and the square of the Euclidean distance between the mapped features and the ideal morphological feature vector corresponding to the rare disease retrieved from the pathological knowledge base is calculated. Secondly, calculate the KL divergence between the semantic distribution of the text feature vectors in the generated data and the distribution of the standard clinical text description of the rare disease; Finally, the square of the Euclidean distance is multiplied by the KL divergence by a preset weighting coefficient and then added together to obtain the pathological knowledge constraint function value. This generates synthetic multimodal data.

8. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 7, characterized in that: The specific process of performance evaluation and screening of the generated synthetic multimodal data using meta-learner parameters includes: For each generated synthetic multimodal data, the data generation and refining module calculates its performance improvement score, and the calculation method for the performance improvement score is as follows: First, select a preset number of NE meta-tasks. For each meta-task, add the synthetic multimodal data to the query set of that meta-task to form an enhanced query set. Then, the meta-learner parameters are quickly adapted on the support set of the meta-task to obtain the adapted parameters, and the difference between the classification loss of the adapted parameters on the augmented query set and the classification loss on the original query set is calculated. Finally, the average of the differences calculated on all NE meta-tasks is used as the performance improvement score for the synthetic multimodal data sample. Subsequently, the data generation and refinement module compares the performance improvement score with a preset threshold and performs a filtering operation to retain only synthetic multimodal data samples whose performance improvement score is greater than the preset threshold; the synthetic multimodal data samples retained after this filtering operation constitute the refined synthetic multimodal sample set.

9. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 8, characterized in that: In the adaptive diagnosis module, the specific process of combining the refined synthetic multimodal sample set with the target rare disease real sample set to form an enhanced support set includes: First, the adaptive diagnosis module receives a target rare disease real sample set from an external source. This target rare disease real sample set contains a small number of real samples to simulate the extreme small sample scenario of rare disease diagnosis. At the same time, it receives a refined synthetic multimodal sample set output by the data generation and refinement module. This synthetic multimodal sample set contains multiple synthetic samples. Then, the adaptive diagnostic module assigns weight values ​​to each real sample and each synthetic sample respectively. For each real sample, its weight value is calculated based on the sample quality confidence score, which is obtained by evaluating the image clarity, staining quality and clinical text report integrity of the real sample. The confidence score value is between 0 and 1. The weight value of the real sample is 1 plus a preset real sample weight coefficient and the sample quality confidence score of the real sample. For each synthetic sample, its weight value is calculated based on the sample quality score, which is obtained by the data generation and refining module through the evaluation of meta-learner parameters during the generation process. The evaluation method is to calculate the average of the performance improvement scores brought by the synthetic sample after adding it to the query set of multiple meta-tasks. The quality score value is between 0 and 1. The weight value of the synthetic sample is the product of a preset synthetic sample weight coefficient and the sample quality score of the synthetic sample. The weight coefficients of the real samples and the weight coefficients of the synthetic samples are hyperparameters, which are set through cross-validation. Finally, the adaptive diagnostic module fuses all real samples with all weight values ​​assigned with all synthetic samples with all weight values ​​assigned to form an enhanced support set. Each element in the enhanced support set contains four parts of information: image feature vector, text feature vector, class label, and weight value of a sample.

10. The intelligent assisted differential diagnosis system for pathological slides based on multimodal fusion according to claim 9, characterized in that: The specific process by which the adaptive diagnostic module uses meta-learner parameters to adapt the diagnostic model using fast gradient descent includes: First, the adaptive diagnosis module retrieves the meta-learner parameters obtained by the meta-knowledge learning module and uses the meta-learner parameters to initialize the network parameters of the diagnosis model, that is, the initial parameters of the diagnosis model are set to the meta-learner parameters. The diagnostic model is a multimodal neural network, whose architecture includes three parts: an image branch, a text branch, and a fusion module. The image branch is composed of multiple residual convolutional blocks connected in series, which are used to extract diagnostic-related features from the input image feature vector. The text branch consists of a Transformer encoding layer, used to extract semantic features from the input text feature vector; The fusion module uses a cross-modal attention mechanism to fuse the output features of the image branch and the text branch, and outputs the class probability through a softmax layer; Then, the adaptive diagnostic module performs fast gradient descent adaptation on the enhanced support set. The adaptation process adopts a meta-learning inner loop optimization strategy. The optimization objective is to minimize the multi-task loss function on the enhanced support set. This multi-task loss function is composed of the sum of the classification loss and the alignment loss. The classification loss adopts the cross-entropy loss function, and the alignment loss adopts the same function as the cross-modal alignment loss function defined in the meta-task construction module. During the adaptation process, the update rule for the diagnostic model parameters is as follows: calculate the gradient of the multi-task loss function with respect to the current parameters of the diagnostic model, use a preset fast adaptation learning rate and the product of the gradient as the parameter update amount, and subtract the update amount from the current parameters of the diagnostic model to obtain the new parameters; the value range of the fast adaptation learning rate is 0.01 to 0.05, and the value range of the weight coefficient of the alignment loss function in the multi-task loss function is 0.1 to 0.5; By repeating the above parameter update process until the performance of the diagnostic model on the enhanced support set reaches the convergence condition, the adapted diagnostic model parameters are obtained. Finally, when diagnosing the queried cases, the adaptive diagnosis module operates according to the following steps: The Monte Carlo Dropout method is used to sample the parameters of the adapted diagnostic model 10 to 20 times to generate multiple perturbation versions of the parameters. For each parameter perturbation version, calculate the probability distribution of the query case belonging to each pathological category; take the arithmetic mean of the probability distributions generated by all parameter perturbation versions to obtain the calibrated probability distribution; select the category with the highest probability value from the calibrated probability distribution as the primary diagnosis result, multiply the highest probability value by one minus the entropy of the probability distribution as the confidence score, and list all categories and their probability values ​​in descending order of probability value as the differential diagnosis list. Finally, the output includes the differential diagnosis result, which contains the primary diagnosis, confidence score, and a list of differential diagnoses.

Images