Medical image information alignment method and device based on graph network guidance and medium

Abstract

The invention provides a medical image information alignment method and device based on graph network guidance and a medium, and belongs to the technical field of medical image processing. Converting the single-mode feature graph into a complete graph, and utilizing information transmission and aggregation characteristics of a graph neural network; updating edge weights and removing low-quality edges and nodes to obtain a plurality of sub-graphs containing important objects; constructing a cross-modal diagram; a cross attention mechanism is configured based on the patch level, patch level information matching is achieved, and differentiation between different medical images is found; defining an internal learning kernel function and an external learning kernel function; and calculating the distance between the two sample pairs in the high-dimensional space, and carrying out clustering analysis by using the distance. According to the method, attention of the model to important objects and a multi-mode matching mode are more in line with a human thinking mode, and meanwhile, a cross attention mechanism is introduced to achieve alignment of finer-grained information.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a medical image information alignment method, device and medium based on graph network guidance. Background Art

[0002] At present, the introduction of the self-attention mechanism solves the problem that convolution is difficult to parallelize, and has greatly promoted the development of large-scale pre-trained models. The introduction of large language models such as BERT and GPT has shown excellent performance in natural language processing tasks. With the further deepening of research, large language models are gradually turning into multimodal large models. Recent studies have shown that building a pre-trained multimodal large model in the field can integrate different types of data, provide more comprehensive and accurate feature representation, and show great potential in various single-modal downstream tasks, so it has received widespread attention.

[0003] The pre-training process of large multimodal models is affected by many factors, including alignment between modalities and the size of the dataset. Multimodal data usually involves correlations between different modalities, such as the association between objects in an image and the corresponding text description. Accurate modality alignment helps the model learn more accurate and consistent feature representations. However, alignment between modalities is not always intuitive and simple, especially when there is noise in the dataset and complex relationships between modalities. Therefore, how to design a better alignment strategy is a technical problem that needs to be solved urgently. In addition, in the medical field, due to the difficulty of obtaining medical data and the requirement of privacy protection, the size of multimodal datasets is relatively limited. Therefore, how to use more effective pre-training methods on limited datasets to improve the capabilities of multimodal models has become an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a medical image information alignment method based on graph network guidance. The method proposes a sample pair clustering method in a high-dimensional space to reduce the unrelated erroneous expressions of the same semantics in a large multimodal model.

[0005] The method includes: S101: extracting multimodal representation based on medical images; S102: Convert the feature graph of a single modality into a complete graph and use the information transfer and aggregation characteristics of the graph neural network; then obtain multiple subgraphs containing important objects by updating edge weights and removing low-quality edges and nodes; construct a cross-modal graph between different modalities and use the graph neural network to capture cross-modal associations and dependencies; S103: Based on the patch level configuration, a cross attention mechanism is implemented to achieve patch level information matching and find the differences between different medical images; S104: defining an internal learning kernel function and an external learning kernel function based on a graph convolutional network; The internal learning kernel function is used for operations between different modalities within a single sample pair, fusing the global features of different modalities; The external learning kernel function acts on the sample pairs, calculates the distance between the two sample pairs in the high-dimensional space, and then uses the distance to perform cluster analysis.

[0006] It should be further explained that step S101 also includes: Define the input of the multimodal model as S samples ; in Indicates image input, Indicates the corresponding text input; Using Image Encoder and text encoder Extract features from the input to obtain multiple modal features ,in , .

[0007] It should be further explained that the image encoder Vit and its variant models are used as feature extractors to output [CLS] tokens containing global features. And the patch block containing the local features of the image ; in, Indicates the number of patch blocks.

[0008] It should be further explained that step S102 also includes: Image encoder based Output Constructing the complete graph ; in, Represents the adjacency matrix of the complete graph, and the calculation process is ; Using graph convolutional networks to enhance shallow important nodes ; in, Represents a graph convolution operation.

[0009] It should be further explained that in the method, the edge weights are updated , and define the preset conditions as the update rules of edge weights; Definition 1: Random noise detection node importance: , (1) in for The nodes after adding random noise, represents the calculation of information entropy, Indicates the role played by the current node in the entire graph. Indicates the calculation result of node importance; Definition 2: The importance of an edge is determined by observing the effect of random noise on the cross entropy between two nodes connected by the edge: (2) in, represents the cross entropy calculation, Indicates the edge importance calculation result; Definition condition 3: The distance between two nodes is determined by calculating the Euclidean distance between the two nodes. The specific calculation process is as follows: (3) in, Indicates The node vector Number, Represents the distance between two nodes; Condition 1 is used to distinguish high-quality and low-quality nodes, condition 2 is used to distinguish high-quality and low-quality relationships, and condition 3 is used to distinguish whether the two nodes connected by the edge belong to the same important object; Adjacency Matrix The update process is as follows: , using the updated adjacency matrix and Perform graph convolution operations to achieve deep modeling of important objects .

[0010] It should be further explained that step S102 further includes: designing a matching formula based on KL divergence to remove the second case: (4) in, express is the node subscript belonging to the image in the cross-modal graph, and Respectively represent the nodes corresponding to the image and text in the cross-modal graph, represents the number of nodes generated by the image, is the length of the node vector; Configure the contrast loss function to implement local matching training. The calculation process is as follows: (5) in and They represent the image-to-text matching loss and the text-to-image matching loss respectively. represents the temperature coefficient, Represents the size of batch_size; the total loss function for local matching is: (6) in, Represents the total sample size.

[0011] It should be further explained that step S103 also includes: The calculation of the cross-attention mechanism is centered on medical images and texts respectively; For Medical image The query vector is obtained through linear transformation. The key vector and value vector are obtained through linear transformation of all important patch blocks in the text. The cross attention mechanism completes the The image The embedding of important patches and all important patches of the text is calculated as follows: (7) in , and represents the learnable parameter matrix, Indicates the length of the patch block vector, Indicates the number of text patches. Represents the calculation result of cross attention; In contrastive learning, an alignment loss function is set to bring patches with the same meaning between different modalities closer together and patches with different meanings farther apart. The loss function is calculated as follows: , (8) in, represents the number of sample pairs, Indicates the number of image patches. Indicates the number of text patches.

[0012] It should be further explained that step S104 also includes: In the internal learning kernel function, the following asymmetric kernel function is designed for fusion: , (9) in, is the learnable parameter matrix, and Represents the standard deviation of the global feature vector of image and text respectively; In the external learning kernel function, the following dynamic weight kernel function is designed based on the graph neural network to map sample pairs to a higher dimensional space for classification: , (10) in, yes The standard deviation of is the learnable parameter matrix, represents the distance between two sample pairs, Represents dynamic weight calculation based on graph convolutional network; Pair the samples Perform the above distance calculation with all remaining sample pairs and select the top The sample pairs are similar sample pairs; Pull in similar pairs and away from other pairs.

[0013] According to another embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the graph network-guided medical image information alignment method when executing the program.

[0014] According to another embodiment of the present application, a storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the medical image information alignment method guided by a graph network are implemented.

[0015] It can be seen from the above technical solutions that the present invention has the following advantages: This application nodes and subgraphs the features of multimodal data, thereby transforming the cross-modal matching method from global matching to subgraph matching, reducing the interference of low-quality information. At the same time, cross-attention is used to perform subgraph-level differentiation within a single modality, so that it can establish more detailed associations and semantic understanding in cross-modal learning. This application proposes a sample pair clustering method in high-dimensional space to reduce the unrelated erroneous expressions of the same semantics in large multimodal models.

[0016] This application also proposes a sub-graph matching strategy, which replaces global matching with matching between important objects, reduces the interference of low-quality information, and provides more fine-grained supervision for the learning of multi-dimensional information. This application configures an internal association strategy between objects, realizes soft matching of important objects between images and texts, and differentiated processing of important objects within the same modality. It also realizes the unified expression of the same semantics between different sample pairs, enhancing the model's ability to understand the real world. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution of the present invention, the accompanying drawings required for use in the description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0018] Figure 1 This is a flowchart of the medical image information alignment method based on graph network guidance; Figure 2 It is a schematic diagram of a multi-modal large model; Figure 3 Schematic diagram of the image feature composition process; Figure 4 It is a schematic diagram of the internal learning process; Figure 5 To perform the RSNA dataset and Comparison chart of different values ​​of ; Figure 6 Attention maps for different matching methods; Figure 7 Schematic diagram of matching accuracy of different alignment methods; Figure 8 A visualization diagram of the output features of the visual encoder; Fig. 9 It is a schematic diagram of sensitivity analysis of parameter K; Fig.10 Schematic diagram of an electronic device. DETAILED DESCRIPTION

[0019] In the medical image information alignment method provided by this application, based on the message passing mechanism of graph neural network, multimodal data features are noded and sub-graphed, thereby changing the cross-modal matching method from global matching to sub-graph matching, reducing the interference of low-quality information. At the same time, this application uses cross-attention to perform sub-graph-level differentiation within a single modality, so that it can establish more detailed associations and semantic understanding in cross-modal learning.

[0020] This application also proposes a sample pair clustering method in high-dimensional space to reduce the irrelevant erroneous expressions of the same semantics in multimodal large models. A large number of experiments were conducted on seven medical image datasets covering image classification, lesion area target detection and semantic segmentation tasks, proving that the model proposed in this paper has superior performance.

[0021] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0022] It should be understood that the expressions such as images, samples, feature maps, etc. described in the embodiments of the present application can all refer to medical image information, that is, medical images. The term "comprising" indicates the presence of the described features, entities, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, entities, steps, operations, elements, components and / or their collections. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] See also Figure 1 FIG. 1 is a flowchart of a medical image information alignment method based on graph network guidance in a specific embodiment, the method comprising: S101: Extracting multimodal representations based on medical images.

[0025] In some embodiments, the extracted medical images may be preprocessed to ensure the quality and consistency of the image data. Multimodal features are extracted from the preprocessed medical images using a deep learning model. The extracted features are also encoded and converted into a form suitable for subsequent processing, such as a vector or matrix.

[0026] S102: Convert the feature graph of a single modality into a complete graph and utilize the information transfer and aggregation characteristics of the graph neural network; then obtain multiple subgraphs containing important objects by updating edge weights and removing low-quality edges and nodes; construct a cross-modal graph between different modalities, and use the graph neural network to capture cross-modal associations and dependencies.

[0027] In some embodiments, constructing a complete graph is to convert a single-modality feature graph into a complete graph, where nodes represent features and edges represent relationships between features, and then use a graph neural network to transfer and aggregate information to update the feature representation of the nodes.

[0028] The edge weight update and node removal in this embodiment is to update the edge weight and remove low-quality edges and nodes according to the importance of the nodes and edges, so as to obtain multiple subgraphs containing important objects.

[0029] A cross-modal graph is constructed between different modalities, and GNN is used to capture cross-modal associations and dependencies. It can be seen that this embodiment can utilize the complex relationship between features and improve the accuracy of information alignment through the processing of graph neural networks. Removing low-quality edges and nodes can reduce noise interference and improve the robustness of the algorithm.

[0030] S103: Based on the patch level configuration, a cross-attention mechanism is implemented to achieve patch level information matching and find the differences between different medical images.

[0031] The patch division in this embodiment is to divide the medical image into multiple small areas, that is, patch blocks.

[0032] Cross-attention mechanism: A cross-attention mechanism is configured at each patch level to achieve patch-level information matching between different modalities.

[0033] This embodiment uses the cross-attention mechanism to find the difference information between different medical images, providing a basis for defining the internal learning kernel function and the external learning kernel function operation. In addition, the cross-attention mechanism can accurately match the information between different modalities and improve the accuracy of information alignment.

[0034] S104: Define internal learning kernel functions and external learning kernel functions based on the graph convolutional network; the internal learning kernel function is used for operations between different modes within a single sample pair to fuse the global features of different modes; the external learning kernel function acts between sample pairs to calculate the distance between two sample pairs in high-dimensional space, and then use the distance to perform clustering analysis.

[0035] In some embodiments, the internal learning kernel function uses the internal learning kernel function to fuse the global features of different modalities to obtain a fused feature representation. The external learning kernel function can use the external learning kernel function to calculate the distance between two sample pairs in a high-dimensional space.

[0036] Based on the above method, this embodiment constructs Figure 2 The multimodal large model shown.

[0037] In order to reduce the low-quality information pollution caused by global matching, this embodiment node-izes the single-modal features in the pre-training stage, and obtains the connection relationship between each node through similarity calculation. At the same time, the update conditions of the node and edge weights are designed based on the graph convolutional network to achieve the modeling of important objects and important relationships, and on this basis, a subgraph centered on itself is constructed for each key feature point, thereby achieving cross-modal local matching.

[0038] In order to solve the problem that large models are difficult to capture the internal correlation of important objects during pre-training, this embodiment introduces a cross-attention mechanism to differentiate the various important objects within a single modality, thereby enabling the model to correctly express the semantic information of key features during local matching. Finally, in order to avoid the model's disagreement on the expression of the same semantics between different sample pairs, resulting in false negatives. This embodiment designs internal and external learning kernel functions based on a graph convolutional network, which respectively realizes high-dimensional fusion of the same sample pair and high-dimensional spatial clustering analysis between different sample pairs, making the model closer to the way humans understand the real world.

[0039] Furthermore, as a refinement and extension of the specific implementation methods of the above-mentioned embodiments, in order to fully illustrate the specific implementation process in this embodiment, the following will more specifically illustrate the specific steps of the medical image information alignment method based on graph network guidance involved in this application.

[0040] This embodiment plays a role in the modeling, matching and clustering of important subgraphs, thereby obtaining a more robust and highly generalized pre-trained model. At the same time, the overall architecture of the model in this embodiment, such as Figure 2 As shown, Figure 2 The four parts in are: (a) multimodal representation extraction; (b) local matching; (c) cross-modal fine-grained alignment; (d) sample pair clustering. That is, the model consists of four components: multimodal representation extraction, local matching, cross-modal fine-grained alignment, and multimodal sample pair clustering.

[0041] S201: Extraction of multimodal representations.

[0042] In this embodiment, the multimodal representation extraction model is designed in accordance with CLIP, and different modalities have corresponding feature extractors, so as to obtain independent features between modalities.

[0043] Taking the image and text modalities as an example, the input of the model is S sample pairs .

[0044] in Indicates image input, Represents the corresponding text input. Then, the image encoder is used and text encoder Perform independent feature extraction on the input to obtain multiple modal features ,in , .

[0045] The image encoder of this embodiment Vit and its variant models are used as feature extractors, whose output consists of two parts: [CLS] tokens containing global features, And the patch block containing the local features of the image .

[0046] in, Indicates the number of patch blocks. . Text Encoder Bert and its variant models are used as feature extractors, and their outputs are similarly split into global features. and local features .

[0047] in, is the number of patches corresponding to the input text. .

[0048] Although single-modality encoders (such as Vit and Bert) have powerful feature extraction capabilities in their respective modalities, the independent operation cannot fully utilize the correlation and complementarity between different modalities. The purpose of this embodiment is to achieve federated learning between different modalities, thereby improving the model's processing capabilities for single-modality tasks.

[0049] S201: Local matching.

[0050] Considering that existing multimodal models are usually pre-trained using a global matching approach, they are easily disturbed by low-quality information, especially in medical multimodal data.

[0051] To solve this problem, this embodiment uses a graph neural network to model important objects and important relationships. First, the feature graph of a single modality is converted into a complete graph and the information transfer and aggregation characteristics of the graph neural network are used to enhance the representation of important objects. Secondly, by updating the edge weights and removing low-quality edges and nodes, multiple subgraphs containing important objects are obtained. Finally, a cross-modal graph is constructed between different modalities, and a graph neural network is used to capture complex cross-modal associations and dependencies.

[0052] Specifically, if Figure 3 As shown, the image encoder Output Construct a complete graph ,in Represents the adjacency matrix of the complete graph, and the calculation process is .

[0053] Using graph convolutional networks to enhance shallow important nodes ,in Represents the graph convolution operation. Then the edge weights are For this purpose, this embodiment formulates three conditions as the rules for updating edge weights: Condition 1: Considering that the two nodes connected by the edge have strong anti-interference ability, this embodiment adds random noise to detect the importance of the node: , (1) in for The node representation after adding random noise is: represents the calculation of information entropy, Indicates the role played by the current node in the entire graph. Indicates the calculation result of the importance of the node. The smaller the value, the stronger the anti-interference ability of the node.

[0054] Condition 2: Based on the strong anti-interference ability of the edge itself, the importance of the edge is determined by observing the impact of random noise on the cross entropy between the two nodes connected by the edge: , (2) in represents the cross entropy calculation, Indicates the calculation result of the importance of the edge. The smaller the value, the stronger the anti-interference ability of the edge.

[0055] Condition 3: The distance between two nodes connected by an edge is close. The distance between the two nodes is determined by calculating the Euclidean distance between the two nodes. The specific calculation process is as follows: , (3) in, Indicates The node vector Number, Represents the distance between two nodes. The smaller the result, the closer the distance between the two nodes is.

[0056] Condition 1 is used to distinguish high-quality and low-quality nodes, condition 2 is used to distinguish high-quality and low-quality relationships, and condition 3 is used to distinguish whether the two nodes connected by the edge belong to the same important object.

[0057] Adjacency Matrix The update process is as follows: , using the updated adjacency matrix and Perform graph convolution operations to achieve deep modeling of important objects . Each node in the image not only focuses on the nodes that are closely related to itself, but also retains the weak connection with other nodes to a certain extent, which increases the encoder's field of view when extracting local features. The text is processed in the same way to obtain .

[0058] In some specific embodiments, for A certain threshold is set to remove low-quality edges and nodes. The threshold is set in percentage form, and six different parameters are compared in the ablation experiment. At this time, the remaining nodes can form a subgraph centered on themselves, the central node of the subgraph is one of the nodes of an important object, and the remaining nodes are the remaining nodes of the important object.

[0059] This embodiment can also convert a single modality feature graph into multiple subgraphs containing important objects to complete the modeling of important relationships. At the same time, this paper constructs a cross-modality graph to achieve cross-modality information interaction.

[0060] Specifically, there are two conditions for constructing a cross-modal graph: 1) The node is the central node of the subgraph after graph convolution.

[0061] 2) Edge connections only occur between different modes, that is, the two nodes connected by each edge must be from different modes.

[0062] Two things happen at this point: 1) The nodes at both ends of the edge point to the same important object and express the same meaning.

[0063] 2) The nodes at both ends of the edge point to different important objects and express different meanings. To complete the matching of important objects between modalities, it is necessary to retain the first case and remove the second case. At the same time, different modal expressions of the same object can be regarded as different distributions generated by the same object, which meets the calculation conditions of KL divergence.

[0064] Therefore, this paper designs a matching formula based on KL divergence to remove the second case: , (4) in express is the node subscript belonging to the image in the cross-modal graph, and Respectively represent the nodes corresponding to the image and text in the cross-modal graph, represents the number of nodes generated by the image, is the length of the node vector. Similarly, the matching of text to image is recorded as .

[0065] In this embodiment, the pre-training of a large multimodal model requires the support of a large amount of data. Although the data set in the medical field is several orders of magnitude smaller than the general image and text data set, it is still difficult to meet the requirements of supervision. Contrastive learning is the most commonly used method for pre-training large models. This paper designs a contrast loss function to implement local matching training. The calculation process is as follows: , (5) in and They represent the image-to-text matching loss and the text-to-image matching loss respectively. represents the temperature coefficient, represents the size of batch_size. Therefore, the total loss function for local matching is: , (6) in, Represents the total sample size.

[0066] S203: Cross-modal fine-grained alignment.

[0067] This step uses the method of aligning important objects to achieve local matching, thereby reducing the interference caused by low-quality information. Although patch-level information is used, the overall distance between sample pairs is still considered, and patch-level alignment interaction is not performed. This will lead to homogenization at the patch level to a certain extent, that is, homogenization between different important objects, resulting in information loss. To this end, this embodiment introduces a cross-attention mechanism at the patch level to complete patch-level information matching and achieve differentiation between different important objects.

[0068] Specifically, the cross attention mechanism is calculated with the image and text as the center. When the image is the center, for the The image The query vector is obtained through linear transformation. The key vector and value vector are obtained through linear transformation of all important patch blocks in the text. The cross attention mechanism completes the The image The embedding of important patches and all important patches of the text is calculated as follows: , (7) in, , and represents the learnable parameter matrix, Indicates the length of the patch block vector, Indicates the number of text patches. Represents the calculation result of cross attention. The same process is applied to text.

[0069] In contrastive learning, the alignment loss function is designed to bring patches with the same meaning between different modalities closer together and patches with different meanings farther apart. The loss function is calculated as follows: , (8) in, represents the number of sample pairs, Indicates the number of image patches. Indicates the number of text patches.

[0070] S204: Clustering of multimodal sample pairs.

[0071] The local features are analyzed in detail, but the global features are not analyzed. and No constraints are added. The global feature contains the expression of the overall information of the sample pair, and there may be similar semantics between different sample pairs. For example, in medical multimodal data, images and reports from different patients may have similarities. Classifying images and texts in the same sample pair into one category and other sample pairs into another category will cause false negatives and make it difficult to correctly express the real world. To this end, this embodiment retrieves the most similar sample pair for each sample pair. sample pairs, so that the sample pair is consistent with this This operation avoids sample pairs with similar semantic information from being pushed apart in the embedding space, which is more consistent with the definition of objects in the real world.

[0072] In order to achieve the above goals, this paper designs two kernel functions based on graph convolutional networks: 1) Configure internal learning kernel function: The internal learning kernel function is used for operations between different modalities within a single sample pair, aiming to fuse the global features of different modalities. In this way, information from different modalities can influence and complement each other, thereby improving the overall understanding of the sample pair.

[0073] 2) Configure external learning kernel function: The external learning kernel function acts on sample pairs. The purpose is to calculate the distance between two sample pairs in high-dimensional space and use the distance for cluster analysis. Calculating distance in high-dimensional space can more accurately measure the similarity or difference between samples and achieve better data separation.

[0074] Specifically, if Figure 4 As shown, in internal learning, for the The global features of sample pairs and , considering that the two have different weights in the fusion result, this paper designs the following asymmetric kernel function for fusion: , (9) in is the learnable parameter matrix, and Represent the standard deviation of the global feature vector of image and text respectively.

[0075] In external learning, for the two sample pairs after fusion and , this paper designs the following dynamic weight kernel function based on graph neural network to map sample pairs to a higher dimensional space for classification: , (10) in yes The standard deviation of is the learnable parameter matrix, represents the distance between two sample pairs, Represents dynamic weight calculation based on graph convolutional network.

[0076] Pair the samples Perform the above distance calculation with all remaining sample pairs and select the top The sample pairs are Similar sample pairs. Pull in similar pairs and away from other pairs.

[0077] In contrastive learning, the corresponding loss function is as follows: , (11) in, represents the number of sample pairs, Representation and The closest sample pair, Representation and The remaining sample pairs with different distances. Finally, the overall model is trained by jointly optimizing the three loss functions to encourage the model to learn a more generalized multimodal information representation. The total loss function can be expressed as: , (12) This example also verifies the implementation process of the medical image information alignment method. The model in this paper is pre-trained using the JPG version of the MIMIC-CXR2.0.0 dataset. This dataset is a large publicly available JPG format chest X-ray dataset with free text radiology reports.

[0078] The dataset of this embodiment contains 377,110 JPG format images and structured labels of 227,827 free-text radiology reports associated with these images.

[0079] In terms of model composition, BioClinicalBERT is used as the text encoder, and the Base version of Vit is used as the image encoder. Since the structure adopted is in the form of CLIP, the choice of text encoder and image encoder is not fixed, and various language models and visual models can be freely matched as the encoder part. In the pre-training process, this paper uses 4 A100 40G graphics cards for training, batch_size is set to 128, the total number of rounds is set to 50, the optimizer is AdamW, the learning rate is 2e−5, and the weight decay is 0.05. In terms of hyperparameter settings, this paper compares various values ​​of the temperature coefficient. The experimental results are as follows Figure 5 As shown, through experimental comparison, it was finally determined to be , .

[0080] This example conducts experimental comparisons on the following three downstream tasks: (1) Medical image classification: Freeze the pre-trained model parameters and add a fully connected layer to the image encoder to complete the image classification task. It includes three commonly used experimental datasets: The CheXpert dataset is a large dataset containing 65,240 patients and 224,316 chest X-ray images. The information in the dataset is obtained by extracting 14 observations from radiology reports by the labeler. The observations are divided into positive samples, negative samples, and uncertain samples.

[0081] The validation set consists of 200 chest X-ray images, and the labels are annotated by 3 experts. The test set consists of 500 chest X-ray images, and the labels are annotated by 5 experts. The RSNA dataset contains approximately 29,700 frontal chest radiographs. The labels are divided into two categories: normal or pneumothorax positive. In the experiment, the dataset was divided into training set, validation set, and test set at a ratio of 70%, 15%, and 15%. COVIDx contains more than 30,000 CXR image data, which are sampled from 16,600 patients in different countries, including 16,490 COVID-19 images. All data are divided into three categories (COVID-19, non-COVID-19, and normal). 1%, 10%, and 100% of the training data are used to evaluate the model on each classification dataset. Among them, CheXpert and RSNA use the area under the ROC curve (AUC) as the evaluation indicator, and Covidx uses the accuracy (ACC) as the evaluation indicator.

[0082] (2) Target detection in lesion area: For the target detection task, the backbone part of YOLO is replaced with the pre-trained model of this paper, and the parameters of this part are frozen at the same time. The target detection task is completed with the help of the YOLO model. The experimental data sets include two: RSNA pneumonia data set contains 29,700 samples. The goal is to detect the pneumonia area. During the experiment, the data set is randomly divided into training set (16,010 samples), validation set (5,337 samples) and test set (5,337 samples). Object CXR contains 9,000 X-ray images. The goal is to detect abnormal areas, such as the identification of foreign bodies inside and outside the lungs. The training set contains 6,400 samples, the validation set contains 1,600 samples, and the test set contains 1,000 samples. The model is also evaluated using 1%, 10%, and 100% of the training data.

[0083] (3) Medical image semantic segmentation: For the semantic segmentation task, the main part of the U-Net model was replaced with the pre-trained model of this paper, and the parameters of this part were frozen. The semantic segmentation task was completed with the help of the U-Net model. There are two data sets: the SIIM data set contains 12047 chest radiographs, with the training set, validation set and test set accounting for 70%, 15% and 15% respectively. RSNA is processed in the same way as the target detection task, converting the true label of the target detection into a mask for semantic segmentation. The model is also evaluated using 1%, 10% and 100% of the training data.

[0084] Table 1 Comparison of medical image classification tasks

[0085] For the results of the experiment and verification process of this embodiment, the model of this paper is compared with ConVIRT, MedCLIP, MGCA-ResNet, MGCA-ViT, MRM, RECLF and MLIP. These methods all use the CLIP form of contrastive learning method.

[0086] Among them, ConVIRT, as a predecessor of CLIP, has basically the same processing method as CLIP. RECLF models the relationship between local matches through a relation-enhanced contrastive learning framework. MLIP uses domain-specific medical knowledge as a guiding signal and integrates language information into the visual domain through image-text contrastive learning.

[0087] In order to make a fair comparison, the pre-training datasets of the above methods are set to the same as those of this paper. The experimental results are shown in Table 1. From the experimental results, it can be seen that the model in this paper achieved the highest results on all datasets. This shows that the model in this paper can use the design of graph neural network and dynamic weight kernel function to extract richer and more discriminative feature representations from medical images. Compared with other methods, the model in this paper has stronger performance in medical image classification tasks.

[0088] Table 2 Comparison of medical object detection tasks

[0089] In the task of lesion area target detection, the proposed model was compared with ConVIRT, GLoRIA, MGCA, GLoRIA-MIMIC, MedKLIP and MLIP. Among them, MedKLIP is based on the Transformer fusion model to achieve spatial alignment of entity description and visual signal at the image block level. The experimental results are shown in Table 2. According to the experimental results, the proposed model showed excellent performance in the task of lesion area target detection and showed obvious advantages. Especially when the training data accounted for 1%, the proposed model had a significant improvement compared with other models. It can be seen that the proposed model uses the method of important object matching instead of global matching, which can better capture the abnormal target features in medical images and provide more accurate detection results.

[0090] Table 3 Comparison of medical semantic segmentation tasks

[0091] In the task of semantic segmentation of medical images, the proposed model was compared with ConVIRT, GLoRIA, MGCA, GLoRIA-MIMIC, MedKLIP and MLIP. The experimental results are shown in Table 3. Overall, the proposed model achieved five first places in six comparison cases. Edge information in medical images is crucial for accurate semantic segmentation. The results show that the proposed model can accurately capture the complex structures and lesion areas in medical images. This makes the segmentation results more accurate, can better reveal the boundaries and morphological characteristics of lesions, and has strong generalization ability.

[0092] In terms of model parameters, this paper only adds 4 graph convolution layers and fewer learnable parameters on the basis of CLIP, so that the increase in the number of model parameters is very limited. In addition, this paper also experiments on image encoders, replacing Vit-base with Deit-base with fewer parameters. The visual model is trained using the training strategy proposed in this paper and compared with the MedCLIP and MGCA methods.

[0093] The experimental results are shown in Table 4. The experimental results show that the visual model obtained by the training strategy proposed in this paper has achieved better results in the classification task. Even if the Deit-base with smaller parameters is used as the image encoder, the training method in this paper can still fully capture the important information in the image, thereby improving the performance of the model. Compared with the MedCLIP and MGCA methods, the training method in this paper can better utilize the features of the input image and achieve more accurate classification. This result further verifies the effectiveness of the training method proposed in this paper.

[0094] Table 4 Comparison of different visual encoders

[0095] In order to analyze the role of each module in the model in this paper during pre-training, this embodiment conducts ablation experiments on local matching, cross-modal fine-grained information alignment, and multimodal sample clustering.

[0096] In order to verify the role of local matching, this paper conducts quantitative and qualitative analysis of the module. In the quantitative analysis, this paper makes three settings for the local matching module: no matching module, global matching, and local matching. Experiments are conducted for these three cases, and the results are listed in Table 5. The experimental results show that if the local matching module is discarded, the classification accuracy will drop significantly. Whether using global matching or local matching, the purpose is to improve the model's ability to understand the correlation between images and texts. The loss of this ability will make it difficult to interact with multimodal information, resulting in a decrease in accuracy. Comparing the experimental results of global matching and local matching, it can be observed that local matching performs better in classification accuracy. This shows that local matching can better capture the fine-grained semantic relationship between images and texts, thereby improving the performance of the model in terms of detail information and local correlation.

[0097] In terms of qualitative aspects, this paper draws attention maps in three cases, and the experimental results are as follows Figure 6 As shown in the figure, it can be seen that in the absence of matching, the image encoder can still roughly identify the key areas (for example, a small part of the red area and the green area are concentrated in the lungs) because it has been pre-trained on data sets in other fields, but the sensitivity to the key objects is low, and there are even errors. Using global matching will prompt the model to pay more attention to the key areas, but it will be disturbed by low-quality information, causing the model to have a certain deviation in its focus on the key objects. Local matching can pay more attention to the important objects in the modality and reduce the interference of low-quality information such as background. Table 5 Local matching ablation experiment

[0098] When conducting ablation experiments on the three edge weight update rules used in important relationship modeling, this paper adopts different combinations to evaluate their impact on the semantic segmentation task. The following is a detailed description of the experiment: First, a baseline model is built, that is, a model without any edge weight update rule. Then the three edge weight update rules are applied independently to model important relationships. Finally, these three rules are combined in pairs to obtain three combined models. Table 6 shows the results of different models in the semantic segmentation task. The experimental results show that each edge weight update rule has a positive impact on the semantic segmentation task.

[0099] Applying any one of the rules improves performance compared to the baseline model. This suggests that each rule can effectively capture important relationships and guide the model to better understand the dependencies between multimodal data. In addition, when the two rules are combined, the performance is further improved. This suggests that these rules are complementary in terms of synergy, providing more comprehensive modeling capabilities. The results of the ablation experiment further verify the effectiveness of these edge weight update rules in modeling important relationships. It can help the model better understand the associations and dependencies in multimodal data, thereby improving the performance of the model.

[0100] Table 6 Ablation experiment of weight update rule

[0101] In the process of sub-graph construction, this paper uses a percentage threshold to sparse the sub-graph to remove low-quality edges and low-quality relations. The experiment explores the impact of different threshold selections on model performance. The experimental results are shown in Table 7. The experimental results show that when no sparseness is performed, the performance of the model is the worst. This is because there are a large number of low-quality edges and low-quality relations in the sub-graph, which interferes with the learning process of the model. This paper tried six different threshold selection conditions. The results show that the model obtained by discarding 30% of the edges has the best performance. This means that retaining 70% of the edges can remove low-quality information to a certain extent, and retain more important edges and relations, thereby improving the performance of the model. Through experimental analysis, this paper verifies the effectiveness of using a percentage threshold for sub-graph sparseness. Selecting an appropriate threshold can help remove low-quality information, so that the model can learn and understand the key information in the image more accurately. Table 7 Threshold sensitivity analysis

[0102] Cross-modal fine-grained information alignment: In order to verify that the cross-attention mechanism can achieve more fine-grained alignment between important objects, this paper also conducts quantitative and qualitative analysis. In terms of quantitative, this paper compares the results of discarding the module and retaining the module. The experimental results are shown in Table 8. It can be seen that the introduction of the cross-attention mechanism can improve the performance of downstream tasks. At the same time, in order to more intuitively observe that the cross-attention mechanism can effectively distinguish the important objects in the same modality, it can help the model better match between modalities.

[0103] Table 8 Cross-modal fine-grained information alignment ablation experiment

[0104] Qualitatively, this paper tracks the cross-modal matching accuracy of the model during pre-training in two cases. The experimental results are as follows Figure 6 As shown in the figure, it can be seen that when the alignment is discarded, the training process of the model is relatively unstable, with obvious jumps in the early stage, and the matching accuracy rate in the later stage is also slowly improved. When the alignment operation is added, the training process of the model is more stable. It can be seen from the horizontal axis that the model at this time has achieved advanced performance with only 80k steps of training process. Keeping the alignment can generate independent differentiation information between each important object, which enables the model to enhance the recognition and expression ability of important objects.

[0105] The ablation experiment for this module consists of two parts. The first part is to verify the effectiveness of this module by discarding and retaining the module. The experimental results are shown in Table 9. The experimental results show that retaining this module can significantly improve the effect of downstream tasks. This shows that multimodal samples play an effective role in the clustering module in the model. In addition, in order to more intuitively observe the impact of multimodal samples on the visual encoder in the clustering task, this paper conducts a visual analysis and uses the t-SNE algorithm to reduce the dimension of the visual encoder output results in the two cases of discarding clusters and retaining clusters. The experimental results are shown in Figure xx. By observing Figure 8 , we can clearly see the impact of adding clustering. When clustering is discarded, the output of the visual encoder shows a certain degree of confusion, and samples of different categories are not clearly distinguished in the dimensionality reduction space. However, when clustering is retained, the output of the visual encoder shows a clearer clustering effect. Samples of different categories form a clear cluster structure in the dimensionality reduction space, samples of the same category are closer, and samples of different categories are clearly distinguished. This experimental result shows that adding clustering has a positive effect and can improve the clustering effect of the visual encoder.

[0106] Table 9 Clustering ablation experiment on multimodal samples

[0107] Second, we conducted a sensitivity analysis on the adjustable parameter K, which is the number of similar samples selected during clustering. We conducted experiments on classification tasks on a single graphics card with batch_size equal to 32 and 24, respectively, while controlling the proportion of a single category to be at most about half of the total number of sample pairs. The experimental results are shown in Figure 2. Fig. 9 As shown in the figure, the two cases have the same trend. When K is 1 / 6 of batch_size, the effect of downstream tasks is the best. Using too many samples will make all samples the same, and it is difficult to accurately distinguish each sample pair. Using too few samples will result in insufficient clustering effect, and it is difficult to learn the semantic expression of important objects themselves.

[0108] Based on the above-mentioned medical image information alignment method, in order to solve the problem that the global matching method used in the pre-training process of multimodal large models will be disturbed by low-quality information, especially in the medical field, where key information is often concentrated in only a small part, this paper proposes a local matching method based on graph convolutional networks and a sample pair clustering method, so that the model's attention to important objects and multimodal matching methods are more in line with human thinking patterns. At the same time, the cross-attention mechanism is introduced to achieve alignment of more fine-grained information.

[0109] Combined with the above medical image information alignment method, Fig.10 As shown, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the medical image information alignment method guided by a graph network are implemented.

[0110] Of course, according to actual needs, a display unit may also be involved to display the execution process, implementation result information and verification information of the medical image information alignment method.

[0111] In the embodiment of the present application, the processor 101 can be implemented by using at least one of an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, and an electronic unit designed to perform the functions described herein. In some cases, such an implementation can be implemented in a controller. For software implementation, implementations such as processes or functions can be implemented with separate software modules that allow execution of at least one function or operation. The software code can be implemented by a software application (or program) written in any appropriate programming language, and the software code can be stored in a memory and executed by a controller.

[0112] The display unit 103 is used to display information input by the user or information provided to the user. The display unit 506 may include a display panel, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0113] The memory 102 may be used to store software programs and various data. The memory 102 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0114] In addition, the electronic device includes some functional modules not shown, which will not be described in detail here.

[0115] The present application also provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a medical image information alignment method guided by a graph network.

[0116] In the storage medium, a program product capable of implementing the above method of this specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of this specification.

[0117] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A medical image information alignment method based on graph network guidance, characterized in that: The method includes: S101: extracting multimodal representation based on medical images; S102: Convert the feature graph of a single modality into a complete graph and use the information transfer and aggregation characteristics of the graph neural network; then obtain multiple subgraphs containing important objects by updating edge weights and removing low-quality edges and nodes; construct a cross-modal graph between different modalities and use the graph neural network to capture cross-modal associations and dependencies; S103: Based on the patch level configuration, a cross attention mechanism is implemented to achieve patch level information matching and find the differences between different medical images; S104: defining an internal learning kernel function and an external learning kernel function based on a graph convolutional network; The internal learning kernel function is used for operations between different modalities within a single sample pair, fusing the global features of different modalities; The external learning kernel function acts on the sample pairs, calculates the distance between the two sample pairs in the high-dimensional space, and then uses the distance to perform cluster analysis.

2. The medical image information alignment method based on graph network guidance according to claim 1, characterized in that: Step S101 also includes: Define the input of the multimodal model as S samples ; in Indicates image input, Indicates the corresponding text input; Using Image Encoder and text encoder Extract features from the input to obtain multiple modal features ,in , .

3. The method for medical image information alignment based on graph network guidance according to claim 2, characterized in that: Image Encoder Vit and its variant models are used as feature extractors to output [CLS] tokens containing global features. And the patch block containing the local features of the image ; in, Indicates the number of patch blocks.

4. The medical image information alignment method based on graph network guidance according to claim 2, characterized in that: Step S102 also includes: Image encoder based Output Constructing the complete graph ; in, Represents the adjacency matrix of the complete graph, and the calculation process is ; Using graph convolutional networks to enhance shallow important nodes ; in, Represents a graph convolution operation.

5. The method for medical image information alignment based on graph network guidance according to claim 4, characterized in that: In the method, update the edge weights , and define the preset conditions as the update rules of edge weights; Definition 1: Random noise detection node importance: ， (1) in for The nodes after adding random noise, represents the calculation of information entropy, Indicates the role played by the current node in the entire graph. Indicates the calculation result of node importance; Definition 2: The importance of an edge is determined by observing the effect of random noise on the cross entropy between two nodes connected by the edge: (2) in, represents the cross entropy calculation, Indicates the edge importance calculation result; Definition condition 3: The distance between two nodes is determined by calculating the Euclidean distance between the two nodes. The specific calculation process is as follows: (3) in, Indicates The node vector Number, Represents the distance between two nodes; Condition 1 is used to distinguish high-quality and low-quality nodes, condition 2 is used to distinguish high-quality and low-quality relationships, and condition 3 is used to distinguish whether the two nodes connected by the edge belong to the same important object; Adjacency Matrix The update process is as follows: , using the updated adjacency matrix and Perform graph convolution operations to achieve deep modeling of important objects .

6. The medical image information alignment method based on graph network guidance according to claim 1, characterized in that: Step S102 also includes: designing a matching formula based on KL divergence to remove the second case: (4) in, express is the node subscript belonging to the image in the cross-modal graph, and Respectively represent the nodes corresponding to the image and text in the cross-modal graph, represents the number of nodes generated by the image, is the length of the node vector; Configure the contrast loss function to implement local matching training. The calculation process is as follows: (5) in and They represent the image-to-text matching loss and the text-to-image matching loss respectively. represents the temperature coefficient, Represents the size of batch_size; the total loss function for local matching is: (6) in, Represents the total sample size.

7. The medical image information alignment method based on graph network guidance according to claim 1, characterized in that: Step S103 also includes: The calculation of the cross-attention mechanism is centered on medical images and texts respectively; For Medical image The query vector is obtained through linear transformation. The key vector and value vector are obtained through linear transformation of all important patch blocks in the text. The cross attention mechanism completes the The image The embedding of important patches and all important patches of the text is calculated as follows: (7) in , and represents the learnable parameter matrix, Indicates the length of the patch block vector, Indicates the number of text patches. Represents the calculation result of cross attention; In contrastive learning, an alignment loss function is set to bring patches with the same meaning between different modalities closer together and patches with different meanings farther apart. The loss function is calculated as follows: ， (8) in, represents the number of sample pairs, Indicates the number of image patches. Indicates the number of text patches.

8. The method for medical image information alignment based on graph network guidance according to claim 1, characterized in that: Step S104 also includes: In the internal learning kernel function, the following asymmetric kernel function is designed for fusion: ，(9) in, is the learnable parameter matrix, and Represents the standard deviation of the global feature vector of image and text respectively; In the external learning kernel function, the following dynamic weight kernel function is designed based on the graph neural network to map sample pairs to a higher dimensional space for classification: ， (10) in, yes The standard deviation of is the learnable parameter matrix, represents the distance between two sample pairs, Represents dynamic weight calculation based on graph convolutional network; Pair the samples Perform the above distance calculation with all remaining sample pairs and select the top The sample pairs are similar sample pairs; Pull in similar pairs and away from other pairs.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the medical image information alignment method based on graph network guidance as described in any one of claims 1 to 8 are implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the medical image information alignment method based on graph network guidance as described in any one of claims 1 to 8 are implemented.

Images