Image-text cross-modal alignment method and device based on adaptive fusion

Abstract

The invention relates to the technical field of artificial intelligence vision-language training, and provides an image-text cross-modal alignment method and device based on adaptive fusion in order to solve the problems of global feature redundancy and local semantic fuzziness in an existing vision-language pre-training model. The method comprises the following steps: extracting global and local features of a medical image, a chief complaint and a diagnosis text through an image and text encoder; a dimension perception regularization mechanism is introduced, features are purified through variance constraint and weighted covariance penalty, and discrimination of global representation is enhanced; by taking the chief complaint as a semantic bridge, the local features of the image and the diagnosis text are respectively enhanced through a two-way cross attention mechanism, and accurate comparison is carried out under the guidance of the chief complaint context. According to the method, through global and local collaborative and fused end-to-end training, accurate and robust alignment of the image and the text report is realized, the pre-trained model has excellent performance in downstream tasks such as medical image classification and cross-modal retrieval, and the method has important application value.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence vision-language training technology, specifically relating to a cross-modal alignment method and apparatus for text and images based on adaptive fusion. Background Technology

[0002] Medical vision-language pre-training is a current research hotspot in the field of medical artificial intelligence. Its core goal is to use large-scale, easily accessible paired medical images (such as X-rays) and radiology reports to obtain transferable robust feature representations through self-supervised learning, thereby effectively alleviating the bottleneck problem of scarce labeled data in medical deep learning models.

[0003] Existing medical vision-language pre-training model methods mainly follow the technical approach in the field of natural image processing, and are adapted accordingly. Based on their technical approaches to achieving cross-modal alignment, they can be broadly categorized into three types: (1) Global Feature Contrast Learning-Based Methods: These methods, represented by CONVIRT, MedCLIP, and M-FLAG, employ contrastive learning loss to bring the global feature representations of matching image-report pairs closer together in the feature space, while simultaneously pushing away the representations of mismatched pairs. These methods are simple in architecture and computationally efficient, serving as a fundamental paradigm for cross-modal alignment. However, they tend to compress the entire image and report into a single feature vector for matching, ignoring the inherent sparsity and local complexity of medical data.

[0004] (2) Introducing local fine-grained alignment methods: In order to overcome the limitations of global alignment, subsequent studies such as GLoRIA and MGCA have attempted to introduce fine-grained similarity calculations between local regions of the image and text description fragments in addition to global alignment. These methods aim to establish more refined modal correspondences and improve the model's ability to capture key diagnostic signals.

[0005] (3) Methods to promote cross-modal fusion and alignment of image and text features through masked content modeling: Representative techniques include MRM, MaCo, ALTA, etc., which mainly optimize the model through two types of tasks: masked image modeling and masked text modeling. However, the above-mentioned existing techniques often ignore the key chief complaint information in the report text, while the patient's chief complaint is often the key reference and important source of evidence for clinicians to make a diagnosis. Therefore, the above methods are still insufficient in terms of image-text semantic dependency mining. Recently, the MCG-Net method has further recognized the importance of the relationship between image, chief complaint and diagnostic text, and uses generative pre-training objectives of mask reconstruction to model this ternary relationship.

[0006] Despite the continuous progress made by the aforementioned technical approaches, they have not yet fully addressed the clinical challenges unique to the medical field. Existing technologies either neglect the crucial bridging role of the chief complaint in the report, directly aligning images with diagnostic text, leading to semantic ambiguity; or the masked modeling generative pre-training targets employed are not optimal for cross-modal retrieval tasks that require explicit computation of feature similarity.

[0007] Therefore, how to effectively integrate global and local information within a contrastive learning framework, and introduce prior clinical knowledge to model the complete logical chain of "image-chief complaint-diagnosis," has become a key issue that current technological development urgently needs to address. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a method and apparatus for cross-modal alignment of text and images based on adaptive fusion. The method systematically optimizes key aspects such as the selection of basic indicators, data normalization processing, and the fusion of multiple heterogeneous indicators. Experiments demonstrate that this method maintains ease of operation and strong versatility while having low time overhead.

[0009] This invention provides a cross-modal alignment method for images and text based on adaptive fusion, the method comprising: Step 110: Obtain the training dataset, which includes at least: medical images, and corresponding chief complaint text and diagnostic text; Step 120: Perform multimodal feature extraction: use an image encoder to extract global and local features of the medical image; use a text encoder to extract global and local features of the chief complaint text and the diagnostic text, respectively. Step 130, perform global adaptive alignment: calculate the contrastive learning loss based on the global features of the medical image and the global features of the diagnostic text; introduce a dimension-aware regularization mechanism, constrain the feature dimension variance through variance regularization, and dynamically penalize the feature dimension covariance through a lightweight network to obtain the regularization loss; sum the contrastive learning loss and the regularization loss by weight to form the global alignment loss. Step 140, perform local adaptive alignment: using the local features of the complaint text as a bridge, a dual-path cross-attention mechanism is adopted to generate the complaint context-aware representation of image patches and diagnostic text word units respectively, and calculate the local contrast loss; Step 150: Combine the global alignment loss and local contrast loss to construct the total loss function for training the medical vision-language pre-trained model, optimize the parameters of the image encoder and text encoder, and obtain the trained medical vision-language pre-trained model; use the image encoder and text encoder of the trained medical vision-language pre-trained model for downstream medical tasks.

[0010] On the other hand, the present invention also provides a cross-modal alignment device for text and images based on adaptive fusion, comprising: The first module is used to obtain the training dataset, which includes at least: medical images, and corresponding chief complaint text and diagnostic text; The second module is used for multimodal feature extraction: it uses an image encoder to extract global and local features of medical images; and it uses a text encoder to extract global and local features of the chief complaint text and the diagnostic text, respectively. The third module is used for global adaptive alignment: the contrastive learning loss is calculated based on the global features of the medical image and the global features of the diagnostic text; a dimension-aware regularization mechanism is introduced, which constrains the feature dimension variance through variance regularization and dynamically penalizes the feature dimension covariance through a lightweight network to obtain the regularization loss; the contrastive learning loss and the regularization loss are weighted and summed to form the global alignment loss. The fourth module is used for local adaptive alignment: using the local features of the main complaint text as a bridge, a dual-path cross-attention mechanism is adopted to generate the main complaint context-aware representation of image patches and diagnostic text word units respectively, and calculate the local contrast loss. The fifth module is used to combine the global alignment loss and the local contrast loss to construct the total loss function for training the medical vision-language pre-trained model, optimize the parameters of the image encoder and text encoder, and obtain the trained medical vision-language pre-trained model; the image encoder and text encoder of the trained medical vision-language pre-trained model are then used for downstream medical tasks.

[0011] Compared with the prior art, the beneficial effects of the present invention include: This invention achieves significant results through an adaptive fusion mechanism that combines global and local collaboration. Specifically, at the global level, the innovative dimension-aware regularization mechanism effectively cleanses the feature space, significantly alleviates feature redundancy, and enhances the model's sensitivity to key diagnostic signals and feature discrimination, resulting in more accurate global alignment. At the local level, the complaint-guided dual-path cross-attention mechanism successfully resolves semantic ambiguity, establishing fine-grained semantic correspondences that conform to clinical logic, and greatly improving the accuracy and interpretability of local alignment. Ultimately, this framework demonstrates excellent generalization performance in downstream tasks, achieving high accuracy with only minimal annotation in tasks such as medical image classification and cross-modal retrieval, providing a more reliable and robust tool for clinical auxiliary diagnosis. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0013] Figure 1 This is a flowchart illustrating the steps of an adaptive fusion-based cross-modal alignment method for images and text in one embodiment of the present invention. Figure 2 This is a schematic diagram of the algorithm framework of an image-text cross-modal alignment method based on adaptive fusion in one embodiment of the present invention, wherein, No. A chest X-ray image, showing frontal and side views. This is the corresponding chief complaint text. It is the corresponding number A diagnostic text, ; It is an image encoder. It is a text editor; Indicates the first Local features of a medical image ; Indicates the first Global features of a medical image ; They are the first Local features of the chief complaint text and the diagnostic text They are the first Global features of the chief complaint text and the diagnostic text; Represents global features of medical images. Global features representing diagnostic text; These represent sets of local features from medical images, diagnostic text, and chief complaint text, respectively; GAP represents global average pooling. and These are, respectively, the visual block representation of the subject's context and the lexical representation of the diagnostic text; and , respectively represent the visual block representation of the main complaint context awareness and the lexical representation of the diagnostic text after aggregation through layer normalization and global average pooling; Figure 3 This is a schematic diagram illustrating chest X-ray images and corresponding report texts from the MIMIC-CXR JPG dataset used in one embodiment of the present invention; Figure 4This is a schematic diagram illustrating the difference between traditional VICReg covariance regularization loss and dimension-aware covariance loss in one embodiment of the present invention. Figure 4 (a) is a schematic diagram illustrating the differences in how off-diagonal elements of all covariance matrices are decorated to the same degree. Figure 4 (b) is a schematic diagram illustrating the difference in weighted decorrelation of the off-diagonal elements of the covariance matrix according to dimensional importance; Figure 5 This is a schematic diagram illustrating the comparison of the global feature covariance matrix based on 1024 samples of the MIMIC-CXR validation set in one embodiment of the present invention. Figure 5 (a) is a schematic diagram of the covariance matrix of the baseline model without DAR. Figure 5 (a) is a schematic diagram of the covariance matrix of the ALICE framework model based on DAR used in this invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0015] To improve the cross-modal alignment quality and downstream task performance of medical vision-language pre-trained models, this invention proposes an adaptive medical vision-language pre-training framework that integrates global and local clinical information. The performance limitations of existing methods are mainly due to two factors: first, feature redundancy exists at the global alignment level, where sparse key diagnostic signals are diluted by a large number of anatomical features from normal tissues; second, semantic ambiguity exists at the local alignment level, neglecting the crucial bridging role of the chief complaint in connecting images and diagnostic text, and traditional contrastive learning disrupts the "one-to-many" clinical logical relationship between the chief complaint and the disease.

[0016] Based on this, the present invention designs a systematic solution that achieves accurate and robust alignment of medical images and text reports through collaborative optimization of global and local dual-path alignment.

[0017] In one embodiment, such as Figure 1 As shown, this invention provides a cross-modal alignment method for images and text based on adaptive fusion, comprising: Step 110: Obtain the training dataset, which includes at least: medical images, and corresponding chief complaint text and diagnostic text; Step 120: Perform multimodal feature extraction: use an image encoder to extract global and local features of the medical image; use a text encoder to extract global and local features of the chief complaint text and the diagnostic text, respectively. Step 130, perform global adaptive alignment: calculate the contrastive learning loss based on the global features of the medical image and the global features of the diagnostic text; introduce a dimension-aware regularization mechanism, constrain the feature dimension variance through variance regularization, and dynamically penalize the feature dimension covariance through a lightweight network to obtain the regularization loss; sum the contrastive learning loss and the regularization loss by weight to form the global alignment loss. Step 140, perform local adaptive alignment: using the local features of the complaint text as a bridge, a dual-path cross-attention mechanism is adopted to generate the complaint context-aware representation of image patches and diagnostic text word units respectively, and calculate the local contrast loss; Step 150: Combine the global alignment loss and local contrast loss to construct the total loss function for training the medical vision-language pre-trained model, optimize the parameters of the image encoder and text encoder, and obtain the trained medical vision-language pre-trained model; use the image encoder and text encoder of the trained medical vision-language pre-trained model for downstream medical tasks.

[0018] Specifically, in step 110, the input terminal obtains the chief complaint text (e.g., "chest pain", "cough and sputum") as a query condition corresponding to the medical image to guide attention; the diagnostic text is the diagnostic report text corresponding to the medical image, used to provide diagnostic semantics (e.g., "normal", "suggests pneumonia", "pneumonia cannot be ruled out").

[0019] In one embodiment, the medical image includes at least one of the following images: two-dimensional medical X-ray image, digital pathology whole slice image, skin lesion photograph, retinal fundus photograph, computed tomography (CT), magnetic resonance imaging, echocardiography, mammography, positron emission tomography or CT fusion image (e.g. PET-CT and SPECT-CT).

[0020] A medical vision-language pre-trained model is constructed based on a neural network to achieve cross-modal alignment of images and text based on adaptive fusion. The medical vision-language pre-trained model includes at least: a multi-encoder feature extraction layer consisting of an image encoder and a text encoder; a global feature adaptive fusion alignment module; and a local feature adaptive fusion alignment module. The global feature adaptive fusion alignment module introduces a novel dimension-aware regularization mechanism based on traditional contrastive learning loss to address the global feature redundancy problem. The local adaptive fusion alignment module is used to perform local adaptive alignment using local features of the chief complaint text as a bridge. Specifically, it implements three core functions: cross-attention fusion of chief complaint and local image features; cross-attention fusion of chief complaint and local diagnostic text features; and comparative alignment of local features (local features of the medical image and local features of the diagnostic text) that have been fused with the contextual information of the chief complaint.

[0021] This invention proposes a framework for training medical vision-language pre-training models: the Adaptive Local and Global Information Fusion with Clinical Embeddings (ALICE) framework, which guides (medical) image encoders and text encoders (neural network models) through parameter optimization-based pre-training, achieving efficient cross-modal alignment of global and local features between images and text. The ALICE framework guides global feature alignment between images and text through Dimension-Aware Regularization (DAR) to reduce feature redundancy; and it uses Local Alignment with Chief Complaint-guided Fusion (LACCF) guided by key chief complaint information.

[0022] The training process of the medical vision-language pre-trained model mainly includes: multimodal feature extraction; and pre-training of the image encoder based on the ALICE framework.

[0023] Furthermore, in step 120, multimodal feature extraction is performed using an image encoder (such as Vision Transformer) and a text encoder (such as BioClinical BERT), including: The medical image is input into the image encoder, which extracts global and local features from the medical image and outputs the global and local features in vector form. The local features are derived from the sequence of image patches into which the image is segmented. All the local features of the medical image constitute a vector sequence, and each local feature in the sequence corresponds to a specific image region (patch) in the original medical image.

[0024] The chief complaint text and the diagnostic text are input independently into the same text encoder, and each outputs global and local features in vector form.

[0025] First, the complaint / diagnosis text is segmented into a token sequence using a word segmenter. Then, a text encoder encodes the token sequence, outputting global and local features in vector form for each complaint / diagnosis text. This provides multi-level feature representations for subsequent global and local alignment. The local features of the complaint / diagnosis text originate from the token sequence after word segmentation. All local features of the complaint / diagnosis text constitute a vector sequence, where each local feature corresponds to a specific token in the text.

[0026] Specifically, in step 130, the dimension-aware regularization mechanism includes a variance regularization mechanism and a dimension-aware regularization mechanism; the global adaptive alignment includes: Step 131: The global features of the medical image and the global features of the diagnostic text are converted into normalized embedding vectors through a nonlinear projection layer, respectively, to obtain the image embedding and the text embedding, both with dimension 131. .

[0027] Step 132: On the batch data, align the image embeddings and text embeddings using symmetric InfoNCE loss; The symmetric InfoNCE loss is an evaluation metric calculated based on the Information Noise-Contrastive Estimation (InfoNCE) loss, using the contrast loss in both the image-to-text and text-to-image directions. The average of the contrast losses in both directions is taken as the overall contrastive learning loss. To facilitate cross-modal feature matching.

[0028] Step 133: Calculate the variance regularization loss using a variance regularization mechanism; the variance regularization mechanism calculates the standard deviation of each feature dimension within a batch and constrains these standard deviations to be higher than a set threshold through the loss term. If the standard deviation of a certain feature dimension is lower than the threshold... If a penalty is imposed, it ensures that the model fully activates all feature dimensions and maintains feature diversity.

[0029] The variance regularization mechanism is used to solve the problem of global feature redundancy and prevent feature dimension collapse.

[0030] Step 134, using the dimension-aware regularization mechanism, calculate the dimension-aware covariance regularization loss, including: Dynamic weight generation: A lightweight multilayer perceptron (MLP) network is used to dynamically generate weight vectors related to feature importance. For each feature dimension, there is a corresponding weight (a component in the weight vector). Adaptive covariance penalty: Based on the generated weight vector, the dimension-aware covariance regularization loss is calculated; the correlation between feature dimensions is adaptively penalized using the weight vector, specifically for each feature dimension, the correlation with other dimensions is adaptively penalized according to its weight; higher decorrelation pressure is applied to dimensions with high weight (corresponding to features with strong discriminative power), thereby purifying global features, improving discriminative power, and preserving natural correlation.

[0031] The dimensionality-aware regularization mechanism can exert higher decorrelation pressure on feature dimensions that are related to abnormal signals and have strong discriminative power, thereby effectively purifying global features and improving their feature diversity and discriminative power.

[0032] The DAR (Difference-Invariance Regularization) enforces invariance through variance preservation, covariance decorrelation, and enhanced Euclidean distance minimization between image views, demonstrating its effectiveness in existing visual self-supervised learning techniques. While standard Variance-Invariance-Covariance Regularization (VICReg) was initially used for unimodal self-supervised learning of images, applying the same degree of covariance regularization to all features, this is not optimal for medical data where pathological features (sparse but crucial) compete with dominant normal anatomical structures. To address this issue, VICReg is extended with dynamic dimensionality weighting, and a dimensionality-aware covariance regularization loss is computed.

[0033] Step 135: Through loss integration, the contrastive learning loss, variance regularization loss, and dimension-aware covariance regularization loss are multiplied by weight coefficients respectively. , and The summation of these values ​​yields the global alignment loss, which together drive the medical vision-language pre-trained model to learn diverse and discriminative feature representations.

[0034] The entire process of step 130 above dynamically optimizes feature distribution through a regularization mechanism, ensuring that key pathological features in medical data are not dominated by normal anatomical structures, thereby improving the robustness and effectiveness of global alignment.

[0035] Furthermore, in step 140, local adaptive alignment is performed: using the local features of the complaint text as a bridge, a dual-path cross-attention mechanism is adopted to generate the complaint context-aware representations of image patches and diagnostic text word units, and the local contrast loss is calculated, including: Step 141: Project the local features of the medical image (based on the local features of the image patch), the local features of the diagnostic text, and the local features of the chief complaint text into a low-dimensional normalized embedding space to obtain a unified low-dimensional representation of all local features, so as to reduce computational complexity and promote cross-modal interaction.

[0036] Step 142: Using the local features of the main complaint text as a bridge, a dual-path cross-attention mechanism is employed to perform cross-attention fusion guided by the main complaint: Using local features of the chief complaint text as keys and values, local features of medical images are queried and enhanced. A multi-head attention mechanism is used to generate a context-aware visual representation of the chief complaint. This allows local features based on image patches to be recalibrated under the guidance of the clinical context of the chief complaint text, highlighting image regions related to symptoms.

[0037] Using local features of the chief complaint text as keys and values, the local features of the diagnostic text are queried and enhanced. A multi-head attention mechanism is used to generate a text representation of the chief complaint context, which forces the local features based on diagnostic text lexical units to be enhanced in the semantic context related to the chief complaint text, thereby reducing ambiguity.

[0038] Step 143: The enhanced visual representation and text representation of the subject's context are subjected to layer normalization to stabilize the training. Then, they are aggregated into compact visual vectors and text vectors through global average pooling operation to facilitate subsequent comparative learning.

[0039] Step 144 involves contrastive learning optimization, calculating the local contrast loss: the aggregated visual vectors and text vectors are contrastively learned using symmetric InfoNCE loss, forcing the model to establish a more precise, fine-grained correspondence between image regions and diagnostic text terms under the guidance of the complaint. This local contrast loss encourages positive sample pairs (matched image-text pairs) to be close together in the embedding space, while negative sample pairs are kept far apart, thereby improving the robustness and clinical rationality of local feature alignment.

[0040] Therefore, step 140 uses the chief complaint as a semantic bridge to dynamically integrate multimodal information, effectively alleviating the semantic ambiguity caused by the "one-to-many" mapping between the chief complaint and the disease, and ensuring that the local alignment is more in line with clinical logic.

[0041] In step 150, the global alignment loss and local contrast loss are combined to construct the total loss function for training the medical vision-language pre-trained model. The parameters of the image encoder and text encoder are optimized to obtain the trained medical vision-language pre-trained model, enabling the image encoder and text encoder to extract general features of images and text.

[0042] Specifically, the total loss function is obtained by summing the global alignment loss and the local contrast loss. Through the synergistic optimization of global and local techniques, the total loss function controls the pre-training of the medical vision-language pre-trained model, optimizing the parameters of the image encoder and text encoder.

[0043] The pre-trained image encoder and text encoder can be directly transferred to downstream tasks such as medical image classification and cross-modal retrieval, achieving excellent performance without the need for extensive annotation, thus demonstrating their practical value in clinical auxiliary diagnosis.

[0044] Finally, the image-text cross-modal alignment method based on adaptive fusion provided by this invention, in addition to using the above-mentioned ALICE framework for image-text cross-modal alignment and medical vision-language pre-training model pre-training process, also includes a vision-language model testing process, such as a testing process for downstream medical tasks: image classification tasks or image-text cross-modal retrieval tasks.

[0045] Step 160: Test the trained medical vision-language pre-trained model on a downstream medical task.

[0046] The testing process includes at least two scenarios: For image classification tasks, the image encoder weights in the ALICE framework are loaded and a classification head is added to fine-tune the classification of the downstream fine-tuning classification dataset. For cross-modal image-text retrieval tasks, the image encoder weights and text encoder weights of ALICE are loaded. Cross-modal feature similarity is calculated based on the query image / text features and the candidate text / image features. The results are then sorted from largest to smallest based on the calculated image-text similarity to obtain the retrieval results.

[0047] In one embodiment, the medical image acquired at the input is a two-dimensional medical X-ray image obtained from an existing medical dataset.

[0048] like Figure 2 As shown, the image-text pairing data obtained from the MIMIC-CXR JPG dataset consists of: chest X-ray image (a two-dimensional medical X-ray image of the chest) - report text (chief complaint text and diagnosis text). Figure 2In fact, a schematic diagram of the ALICE framework for image-text multi-level cross-modal alignment with subject-led fusion and dimension-aware regularization is given.

[0049] In step 110, input the training dataset: paired 2D medical X-ray images - report text. ,in, It is the first Medical images, including front / side view X-ray images, ; It is the corresponding number The main complaint text, It is its corresponding number A diagnostic text.

[0050] like Figure 3 As shown in the example illustration of chest X-ray images and corresponding report text in the MIMIC-CXR JPG dataset, the text on the right is typical report text content, including the chief complaint and diagnostic text. The "HISTORY" information is typically collected as the chief complaint in clinical practice, listing the patient's medical history and symptoms. "FINDINGS" and "IMPRESSIONS" represent the diagnostic text, which are descriptions of abnormal / normal phenomena observed by the physician during image interpretation and preliminary diagnostic results, respectively. In this embodiment, the chief complaint is used as an intermediate bridge to perform local contextual understanding alignment modeling between the chest X-ray image and the diagnostic text.

[0051] Specifically, in step 120, multimodal feature extraction includes: Using an image encoder Extracting features from medical images: ; in, Represents the first image patch based on the image patch Local features of a medical image ; Indicates the first Global features of a medical image .use These represent sets of local features from medical images, diagnostic text, and chief complaint text, respectively.

[0052] Using a text encoder Process the main complaint text separately With diagnostic text Features: ; ; in, Indicates the main complaint text Local features (based on word sequence), Diagnostic text Local features (based on word sequence), Indicates the main complaint text global features Diagnostic text The global features. Among them, Represents the length of the word sequence. Represents the feature dimension.

[0053] In step 130, global image features and diagnostic text global features Normalized embedding is achieved through nonlinear layer projection. and These are embedded in a size of batch Alignment is performed using symmetric InfoNCE loss: .in and This represents two symmetric InfoNCE losses.

[0054] By extending VICReg with dynamic dimension weighting and calculating the dimension-aware covariance regularization loss, the global alignment loss is obtained. ; Among them, variance regularization loss , or , Represents batch size; by requiring the standard deviation of each feature to exceed a small threshold. To force dimensional activation and prevent the collapse of the medical feature subspace. Diagonalization of the dimensionality-aware covariance matrix. The weight vector is generated using a multilayer perceptron. Correlation from adaptive penalty features: ; in, Represents the feature dimension; for batches Its covariance matrix The elements are: , , is the Batch No. Features Use its mean Centralization characteristics after decentralization. The loss function focuses the decorrelation pressure on diagnostically relevant dimensions while preserving natural anatomical relevance. The weighting coefficients are as follows: , .

[0055] Multilayer perceptron (MLP) for generating dynamic weights employs methods such as Figure 2 The bottleneck architecture shown handles batch average features. This generates dimension-specific weights. The ALICE framework first calculates the channel-direction average of the global features, then processes them sequentially: linear projection from 128 to 64 dimensions, ReLU activation, and random dropout (dropout probability). The linear projection from 64 dimensions to 128 dimensions is performed, and finally Sigmoid normalization is applied.

[0056] Ultimately, the global alignment loss is a weighted sum of the contrastive learning loss, variance regularization loss, and dimension-aware covariance regularization loss.

[0057] Furthermore, in step 140, in the clinical practice of this embodiment, the chief complaint text represents the patient's self-reported symptoms. It serves as both the starting point for clinical reasoning and a key reference for radiologists when generating diagnostic reports. However, the "one-to-many" mapping relationship between the chief complaint text and disease categories presents a semantic ambiguity challenge to contrastive learning in medical vision-language pre-training. To address this issue, a chief complaint-guided fusion method is proposed, which utilizes local features of the chief complaint. A cross-attention mechanism that serves as a semantic anchor for local alignment. For the... Image-text pairs The local features of the generated main complaint text Diagnostic text local features and local features of medical images Projected as normalized low-dimensional embeddings .

[0058] Next, the two complaint-guided cross-attention modules employ a multi-head attention mechanism to fuse the complaint context into the image and diagnostic text modalities, respectively, generating a visual block representation of the complaint context. and diagnostic text lexical representation The context-aware visual block representation of the subject is obtained through layer normalization and global average pooling (GAP). and diagnostic text lexical representation Aggregate into and Characterization after aggregation and Through comparative learning, the InfoNCE loss is used for optimization to obtain the local alignment loss. .

[0059] In the final step 150, the model is pre-trained using the ALICE framework based on a loss function that jointly optimizes global and local cross-modal alignment. This enables the network to learn general representations applicable to downstream tasks. The overall pre-training loss function is composed of the following formulas: ; in, This represents the global alignment loss, which includes dimension-aware regularization. This represents the local alignment loss that includes the subject-guided fusion.

[0060] Output the trained medical vision-language pre-trained model .

[0061] In one embodiment, the downstream medical task of step 160 is a cross-modal image-text retrieval task, and the testing process includes: Input: The query image and candidate text to be retrieved. Load the pre-trained image encoder and text encoder weights, use the pre-trained model to extract global features of the query image and candidate text respectively, and calculate the global similarity.

[0062] Output: Sort the query image and candidate texts by similarity from highest to lowest, output the retrieved text, and calculate the precision. K and Hit K.

[0063] The Precision K represents the precision of the top K items, which measures the proportion of truly relevant items among the top K results returned by the retrieval system.

[0064] The Hit K represents the hit rate of the first K items. It is a binary indicator (0 or 1) that measures whether at least one relevant item is included in the first K results returned by the system.

[0065] In one embodiment, the downstream medical task of step 160 is an image classification task, and the testing process includes: Input: A 2D chest X-ray image to be classified, pre-trained image encoder weights loaded, and a classification head set according to the number of categories. Fine-tuning is performed using a certain amount of labeled training data, with the cross-entropy loss function.

[0066] Output: Disease category of the image, and calculate AUROC (Area Under the Receiver Operating Characteristic curve, i.e., the area under the ROC curve) and F1 score (harmonic mean of precision and recall, an important metric for evaluating the performance of classification models) based on the model classification results and the true labels.

[0067] In summary, this invention aims to learn highly transferable medical visual-linguistic representations through the synergistic integration of global and local fusion techniques. Its goal is to enhance the performance of downstream tasks (such as image classification and cross-modal retrieval) under limited annotation conditions by achieving a superior form of cross-modal alignment. The key points of this invention are: A dimension-aware regularization mechanism is proposed, a novel two-pronged approach for global alignment to address feature redundancy. This mechanism modulates global features from two complementary angles: first, by enforcing a minimum variance threshold with a small threshold, representation collapse is prevented, ensuring sufficient representational capacity in the feature space; second, adaptive weights are used to selectively decorrelate feature dimensions to enhance discriminative power. This dual mechanism ensures that the global feature space not only possesses diversity but also high discriminativity.

[0068] A chief complaint-guided fusion method is proposed to address the local semantic ambiguity problem in local alignment. Specifically, to achieve more accurate and clinically relevant local alignment between image regions and corresponding diagnostic terms, a contrastive learning method combining a cross-modal dual-path cross-attention mechanism is designed to fuse the contextual information of the chief complaint into the local features of the image and diagnostic text.

[0069] The aforementioned multi-level cross-modal image-text alignment method, which combines dimension-aware regularization and subject-directed fusion, can alleviate feature redundancy and semantic ambiguity. Compared with existing image-text cross-modal alignment methods based on contrastive learning, this invention achieves superior results in both image classification and cross-modal retrieval benchmarks, with higher AUROC and F1 scores and greater precision. K and Hit K is higher.

[0070] Furthermore, this invention conducts experiments specifically on chest X-ray images as medical images, according to... Figure 4 The experimental results clearly demonstrate that the adaptive fusion-based cross-modal alignment method for images and text provided in this invention differs fundamentally from the traditional VICReg covariance regularization by introducing dynamic feature weighting. The generated weights achieve two key objectives through dimension-wise modulation: Higher weights are assigned to pathological features (such as lesion indicators) to enhance their decorrelation. Anatomical structures (such as ribs) are given lower weights to preserve their natural relevance.

[0071] Furthermore, ALICE addresses the global feature redundancy problem through Dimensionality-Aware Regularization (DAR), an approach that adaptively decorrelates global features. This is achieved through methods such as... Figure 5 The visualization results based on covariance matrix analysis shown confirm that DAR effectively suppresses off-diagonal correlation (redundancy). Figure 5 As shown in (a), the covariance matrix of the baseline model without DAR exhibits significant off-diagonal correlation, a clear indication of feature redundancy. In contrast, the covariance matrix of the ALICE model (a model based on the ALICE framework) in this invention, as shown... Figure 5 As shown in (b), good diagonalization is achieved, with a large number of off-diagonal values ​​suppressed to near zero. This provides direct evidence that the DAR method successfully cleanses the feature space through decorrelation operations based on adaptive weights.

[0072] Figure 5 (a) indicates that the baseline model shows high redundancy (significant off-diagonal correlation); Figure 5 (b) Using the DAR-based ALICE model proposed in this invention, the suppressed off-diagonal covariance is observed, achieving approximate diagonalization. Color bars represent covariance values. For fair comparison, all matrices use a uniform color scale. The coordinate axes in this figure use a logarithmic scale.

[0073] In one embodiment, the present invention also provides a cross-modal alignment device for text and images based on adaptive fusion, comprising: The first module is used to obtain the training dataset, which includes at least: medical images, and corresponding chief complaint text and diagnostic text; The second module is used for multimodal feature extraction: it uses an image encoder to extract global and local features of medical images; and it uses a text encoder to extract global and local features of the chief complaint text and the diagnostic text, respectively. The third module is used for global adaptive alignment: the contrastive learning loss is calculated based on the global features of the medical image and the global features of the diagnostic text; a dimension-aware regularization mechanism is introduced, which constrains the feature dimension variance through variance regularization and dynamically penalizes the feature dimension covariance through a lightweight network to obtain the regularization loss; the contrastive learning loss and the regularization loss are weighted and summed to form the global alignment loss. The fourth module is used for local adaptive alignment: using the local features of the main complaint text as a bridge, a dual-path cross-attention mechanism is adopted to generate the main complaint context-aware representation of image patches and diagnostic text word units respectively, and calculate the local contrast loss. The fifth module is used to combine the global alignment loss and the local contrast loss to construct the total loss function for training the medical vision-language pre-trained model, optimize the parameters of the image encoder and text encoder, and obtain the trained medical vision-language pre-trained model; the image encoder and text encoder of the trained medical vision-language pre-trained model are then used for downstream medical tasks.

[0074] On the other hand, the present invention provides a computer device including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the adaptive fusion-based cross-modal alignment method for graphics and text provided in any of the above embodiments. The computer device may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device stores sample data. The network interface of the computer device is used for communication with external terminals via a network connection.

[0075] On the other hand, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the adaptive fusion-based cross-modal alignment method for graphics and text provided in any of the above embodiments.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] Matters not covered in this invention are common knowledge.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for adaptive fusion-based cross-modal alignment of text and images, characterized in that, The method comprises: Step 110, obtaining a training data set, at least comprising: a medical image, and corresponding chief complaint text and diagnosis text; Step 120, performing multi-modal feature extraction: using an image encoder to extract global features and local features of the medical image; using a text encoder to extract global features and local features of the chief complaint text and the diagnosis text respectively; Step 130, performing global adaptive alignment: calculating a contrast learning loss based on the global features of the medical image and the global features of the diagnosis text; introducing a dimension-aware regularization mechanism, constraining feature dimension variance through variance regularization, and dynamically weighting and penalizing feature dimension covariance through a lightweight network to obtain a regularization loss; and weighting and summing the contrast learning loss and the regularization loss to form a global alignment loss; Step 140, performing local adaptive alignment: using the local features of the chief complaint text as a bridge, adopting a double-path cross-attention mechanism to generate chief complaint context-aware representations of image blocks and diagnosis text tokens respectively, and calculating a local contrast loss; Step 150, combining the global alignment loss and the local contrast loss to construct a total loss function for training a medical visual-linguistic pre-training model, optimizing parameters of the image encoder and the text encoder, and obtaining a trained medical visual-linguistic pre-training model; and using the image encoder and the text encoder of the trained medical visual-linguistic pre-training model for medical downstream tasks.

2. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, The medical image comprises at least one of the following images: a two-dimensional medical X-ray image, a digital pathology whole slide image, a skin lesion photo, a retinal fundus photo, a computed tomography, a magnetic resonance imaging, an echocardiogram, a mammography, a positron emission tomography, or a fusion image of a computed tomography.

3. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, The medical visual-linguistic pre-training model is used to realize cross-modal alignment of images and texts based on adaptive fusion. The medical visual-linguistic pre-training model is constructed using a neural network and comprises at least a multi-encoder feature extraction layer composed of an image encoder and a text encoder, a global feature adaptive fusion alignment module, and a local feature adaptive fusion alignment module. The global feature adaptive fusion alignment module introduces a dimension-aware regularization mechanism based on the contrast learning loss to solve the global feature redundancy problem. The local adaptive fusion alignment module is used to perform local adaptive alignment with the local features of the chief complaint text as a bridge.

4. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, The medical visual-linguistic pre-training model is trained using an adaptive local and global information fusion framework based on clinical embedding, which includes guiding the image encoder and the text encoder to perform pre-training based on parameter optimization.

5. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, The step 120 comprises: The medical image is input into the image encoder to extract global features and local features of the medical image, and output the global features and the local features in vector form; the local features of the medical image are derived from a sequence of image blocks into which the image is divided; the local features of the medical image form a vector sequence, and each local feature in the sequence corresponds to a specific image region in the medical image; The chief complaint text and the diagnosis text are independently input into the same text encoder; The chief complaint text / diagnosis text is converted into a word sequence through a word segmentation process by a word segmenter; The word sequence is encoded by a text encoder, and global features and local features in vector form are output for the chief complaint text / diagnosis text; The local features of the chief complaint text / diagnosis text are derived from the word sequence after the text is segmented; the local features of the chief complaint text / diagnosis text constitute a vector sequence, and each local feature in the sequence corresponds to a specific word in the chief complaint text / diagnosis text.

6. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, The step 130 comprises: Step 131, converting the global features of the medical image and the global features of the diagnosis text into normalized embedding vectors respectively through a nonlinear projection layer to obtain image embedding and text embedding; Step 132, using symmetric InfoNCE loss to align image embedding and text embedding on batch data; The symmetric InfoNCE loss is an evaluation index obtained by calculating the contrast loss in two directions of image to text and text to image based on the information noise contrast estimation loss; the average of the contrast losses in the two directions is taken as the overall contrast learning loss; Step 133, calculating the variance regularization loss by using the variance regularization mechanism; The variance regularization mechanism calculates the standard deviation of each feature dimension within a batch and constrains the standard deviation to be higher than a set threshold through a loss term If the standard deviation of at least one feature dimension is lower than the threshold A penalty is applied Step 134, calculating the dimension-aware covariance regularization loss by using the dimension-aware regularization mechanism, comprising: using a lightweight multi-layer perceptron network to dynamically generate a weight vector related to feature importance; based on the generated weight vector, calculating the dimension-aware covariance regularization loss; Step 135, by loss integration, multiply the contrastive learning loss, the variance regularization loss and the dimension-aware covariance regularization loss by weight coefficients respectively 、 and After summing up, the global alignment loss is calculated.

7. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, The step 140 comprises: Step 141, projecting the local features of the medical image, the local features of the diagnosis text and the local features of the chief complaint text into a low-dimensional normalized embedding space to obtain a unified low-dimensional representation of the local features; Step 142, using the local features of the chief complaint text as a bridge, adopting a double-path cross-attention mechanism to perform chief complaint-guided cross-attention fusion: using the local features of the chief complaint text as keys and values to query and enhance the local features of the medical image, and generating chief complaint context-aware visual representation through a multi-head attention mechanism; using the local features of the chief complaint text as keys and values to query and enhance the local features of the diagnosis text, and generating chief complaint context-aware text representation through a multi-head attention mechanism; Step 143, performing layer normalization on the enhanced chief complaint context-aware visual representation and text representation respectively to stabilize the training, and respectively aggregating them into compact visual vectors and text vectors through global average pooling operation; Step 144, performing contrast learning on the aggregated visual vectors and text vectors through symmetric InfoNCE loss to calculate local contrast loss.

8. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, In step 150, the total loss function is obtained by summing the global alignment loss and the local contrast loss.

9. The adaptive fusion-based text-image cross-modal alignment method according to claim 1, characterized in that, The method further comprises: Step 160, testing the trained medical visual-linguistic pre-training model on a medical downstream task; the medical downstream task includes: image classification task and / or cross-modal retrieval task of image and text.

10. An apparatus for adaptive fusion-based image-text cross-modal alignment, comprising: Comprise: A first module for obtaining a training data set, comprising at least: medical images, and corresponding chief complaint text and diagnosis text; The second module is configured to perform multi-modal feature extraction, wherein a global feature and a local feature of a medical image are extracted by using an image encoder; global features and local features of chief complaint text and diagnosis text are extracted by using a text encoder; The third module is configured to perform global adaptive alignment, wherein a contrastive learning loss is calculated based on the global feature of the medical image and the global feature of the diagnosis text; a dimension-aware regularization mechanism is introduced, a feature dimension variance is constrained by variance regularization, and a feature dimension covariance is dynamically weighted and penalized by a lightweight network to obtain a regularization loss; and the contrastive learning loss and the regularization loss are weighted and summed to form a global alignment loss; The fourth module is configured to perform local adaptive alignment, wherein a double-path cross-attention mechanism is adopted to generate chief complaint context-aware representations of an image block and a diagnosis text token respectively by taking the local feature of the chief complaint text as a bridge, and a local contrastive loss is calculated; The fifth module is configured to combine the global alignment loss and the local contrastive loss to construct a total loss function for training a medical vision-language pre-training model, optimize parameters of the image encoder and the text encoder, and obtain a trained medical vision-language pre-training model; and the image encoder and the text encoder of the trained medical vision-language pre-training model are used for medical downstream tasks.

Images