Medical anomaly detection method based on graph language feature guidance

Abstract

The invention discloses a medical anomaly detection method based on graph language feature guidance. The method comprises the following steps: S1, acquiring medical image data and associated text description data; s2, extracting visual features of the medical image through a visual encoder of the CLIP model; extracting text features of the text description through a text encoder of the CLIP model; s3, segmenting the medical image into a plurality of image blocks, and constructing an image structure; graph structure features are extracted through a Vision GNN graph neural network, and local and global features are layered and aggregated by adopting a pyramid architecture; s4, the visual features and the graph structure features are aligned through a multi-stage feature adapter, and fusion features are generated; respectively processing the classification features and the segmentation features by adopting a double-branch architecture; and S5, in a zero sample mode and a few sample mode, a final anomaly detection result is obtained through weighted fusion.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical and health data processing, and particularly relates to a medical anomaly detection method guided by graph language features. Background Art

[0002] Anomaly detection (AD) aims to detect abnormal samples that deviate from the expected normal pattern and plays an important role in the identification of rare diseases and health screening in the medical field. Although medical anomaly detection plays a crucial role in disease diagnosis and early intervention, this field still faces many challenges due to the complexity of anatomical structures and the scarcity of labeled abnormal samples. With the growing demand for accurate and generalizable anomaly detection methods and the increasing complexity of medical imaging data, there is an urgent need for innovative computational solutions that can effectively model anatomical relationships while maintaining clinical relevance.

[0003] Existing methods for anomaly detection in medical images primarily include techniques based on statistical analysis, deep learning, and graph neural networks (GNNs). Traditional methods often rely on statistical models to identify abnormal patterns, but these methods have limited performance in complex medical data environments. In recent years, deep learning methods have achieved breakthroughs in this field. Among them, reconstruction methods based on autoencoders (AEs) attempt to learn the distribution of normal samples and use reconstruction errors as a criterion for abnormality discrimination. However, these methods struggle to accurately capture anatomical details and are easily limited by the training data, resulting in poor generalization to unseen abnormalities. Furthermore, knowledge distillation methods extract knowledge from a large-scale pre-trained teacher model and transfer it to a smaller student model to achieve anomaly detection. However, due to the high-dimensional complexity of medical imaging data, the application of these methods remains limited. Generative adversarial networks (GANs) have also been recently used for anomaly detection, identifying abnormal regions by learning the distribution of normal samples and generating similar images. However, GANs can suffer from pattern collapse or artifact generation in medical image generation tasks, which can affect model stability and detection accuracy. Graph neural networks (GNNs) can capture spatial relationships and anatomical structural information in medical images, thereby more accurately representing the associations between different regions. However, existing GNNs methods still have some obvious flaws. On the one hand, most graph neural network methods use a fixed graph structure, which makes it difficult for the model to adapt to the complexity of different anatomical regions and imaging modalities, thereby limiting the applicability of the model. On the other hand, many methods ignore the importance of different anatomical regions and fail to fully utilize clinical knowledge to guide feature extraction. In addition, current GNNs usually model medical images at a single scale and cannot capture local details and global anatomical structures at the same time, thus affecting the accuracy of abnormality detection. Existing methods often lack targeted optimization for the specificity of different imaging modalities and anatomical regions, resulting in poor performance when migrating across datasets, making it difficult to meet the diverse needs of medical imaging.

[0004] Therefore, although deep learning and graph neural networks have made some progress in medical anomaly detection, there are still many problems that need to be solved. First, many existing methods lack interpretability in actual clinical applications and cannot fully integrate medical knowledge, making the model's decision-making process difficult for doctors to trust. Second, current medical image anomaly detection methods perform poorly in zero-shot and few-shot learning tasks and cannot effectively adapt to new imaging modalities or anatomical regions, limiting their scope of application in clinical settings. In addition, since medical imaging data often has large inter-domain differences, how to achieve more accurate anomaly detection while maintaining global anatomical consistency remains a major challenge. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a medical anomaly detection method guided by graph language features.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A medical anomaly detection method guided by graph language features, comprising the following steps: S1. Multimodal data input: Obtain medical image data and associated text description data; S2. Visual-language feature extraction: Extract visual features of medical images through the visual encoder of the CLIP model ; Extract text features of text descriptions through the text encoder of the CLIP model ; S3. Visual-graph feature extraction: Segment the medical image into multiple image patches and construct a graph structure , where the nodes ∈ represent image patches; the edges ∈ represent spatial adjacency relationships; Extract graph structure features through the Vision GNN graph neural network , and adopt a pyramid architecture to hierarchically aggregate local and global features; S4. Multilevel feature fusion and adaptation: Align visual features and graph structure features through a multilevel feature adapter to generate fused features ; Adopt a bipartite architecture to separately process classification features and segmentation features ; S5. Anomaly detection and output: In the zero-shot mode and the few-shot mode, generate the anomaly classification scores and anomaly segmentation maps in their respective modes, and then obtain the final anomaly detection result through weighted fusion; The specific process of step S5 is as follows: S51. Process the test image , and generate multilevel adaptive features; S52. Compare the generated multilevel adaptive features with the text features , and calculate the zero-shot using the average softmax scores of four levels. The calculation formula is: , , where is the anomaly classification score generated in the zero-shot mode; max G (•) is the maximum value function, G is the number of grids; is the normalization exponential function; is the transpose of the text features; The abnormal segmentation map generated in the zero-shot mode; Is a bilinear interpolation operation; S53. Construct a multi-level feature memory bank to store normal sample features; S54. Calculate the minimum distance between the test feature and the memory bank through the cosine distance. The calculation formula is: , , where Is the abnormal classification score generated in the few-shot mode; Is to take the minimum distance between the normal sample features stored in the feature memory bank; Is the cosine distance; Is the object for comparison between the current layer feature and the normal sample features in the memory bank; Is the abnormal segmentation map generated in the few-shot mode; S55. Add weight factors to fuse the abnormal detection results of the zero-shot mode and the few-shot mode. The calculation formula is: , , where Is the fused abnormal classification score; Is the fused abnormal segmentation map; And Are both weight factors.

[0007] Preferably, the process of text feature extraction in step S2 includes: constructing a state-level prompt template, embedding the organ name into the text description; generating dynamic text input through a two-layer text prompt method, and outputting 2D text features , corresponding to the normal state and the abnormal state respectively; where d is the feature dimension.

[0008] Preferably, the visual feature extraction in step S2 adopts the four encoding stages of the CLIP model, and the intermediate stage features Are used for multi-level feature fusion; where Represents the feature level of the three intermediate stage features; G represents the number of grids.

[0009] Preferably, the construction of the graph structure in step S3 specifically includes: S31. Node feature generation: Encode the image patch into a feature vector , and add absolute position encoding ; S32. Edge connection rule: Establish connections between nodes based on the K-nearest neighbor algorithm, and calculate the edge weight through the Gaussian kernel function. The calculation formula is: , where Is the edge weight; And Are the spatial coordinates of two nodes respectively; is an adjustable parameter.

[0010] Preferably, in step S3, the feature extraction of the Vision GNN graph neural network adopts maximum relative graph convolution, specifically including: S33. Aggregation operation: By aggregating the features of neighbor nodes, calculate the representation of the node. The calculation formula for the aggregated features is: , where is the aggregated feature; represents the node and all neighbor nodes The feature difference; represents selecting the maximum value in the difference vector and retaining the most significant local difference features; represents the feature vector The set of neighbor nodes; S34. Update operation: Update the features separately through the multi-head weight matrix And splice after the feature update.

[0011] Preferably, the implementation steps of the multi-level feature adapter in step S4 include: S41. Linear transformation: The feature adapter is used to adapt the intermediate-stage features at each feature level It consists of two linear transformation layers. The feature transformation is expressed as: , where is the intermediate-stage feature The transformed representation; and are both learnable parameters of the linear transformation; is the activation function; S42. Residual connection: The residual connection is used to retain the original knowledge of the pre-trained CLIP encoding. The calculation formula is: , where is the fused feature; γ is the learnable residual coefficient.

[0012] Preferably, the processing of the bipartite architecture in step S4 includes: S43. The features at each level are divided into classification features and segmentation features Two parallel branches, which are respectively used for the classification task and the segmentation task, and can capture the overall anomalies and local detail anomalies of the image at the same time; The projection head Projects the visual features through a linear layer to obtain the global feature for classification and the local feature for segmentation; S44. Multilevel feature alignment is achieved by combining the loss functions of different components. The loss function adopts a weighted combination of Dice loss, focal loss, and cross-entropy loss.

[0013] After adopting the above technical solutions, the present invention has the following beneficial effects: The present invention first introduces an anatomy-aware graph construction mechanism, which can dynamically adapt to different levels of anatomical complexity, thereby achieving a flexible representation of fine details and global structures. Secondly, a graph-based multilevel visual feature adaptation framework is adopted. By integrating lightweight adapters at multiple feature levels, the domain gap between natural images and medical images is effectively bridged. Finally, a hierarchical semantic integration method is adopted to effectively integrate clinical knowledge into the graph structure, making the decision-making for abnormal classification and segmentation tasks more intelligent, enabling the model to focus on the diagnosis-related regions while maintaining global anatomical consistency. Moreover, this method supports zero-shot and few-shot learning scenarios, demonstrating strong generalization ability in different medical imaging modalities and having good application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] In order to make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0016] As Figure 1 shown, a medical anomaly detection method guided by graph language features includes the following steps: S1. Multimodal data input: Obtain medical image data and associated text description data; S2. Visual-language feature extraction: Extract the visual features of medical images through the visual encoder of the CLIP model ; extract the text features of text descriptions through the text encoder of the CLIP model ; The process of text feature extraction in step S2 includes: constructing a state-level prompt template, embedding organ names into text descriptions; generating dynamic text inputs through a two-layer text prompting method, and outputting 2D text features , corresponding to the normal state and the abnormal state respectively; where d is the feature dimension; The visual feature extraction in step S2 adopts four encoding stages of the CLIP model, and the intermediate stage features are used for multilevel feature fusion; where represents the feature levels of the three intermediate stage features; G represents the number of grids; S3. Visual - Graph Feature Extraction: Segment the medical image into multiple image patches and construct a graph structure , where the nodes ∈ represent the image patches; the edges ∈ represent the spatial adjacency relationship; extract the graph structure features through the Vision GNN graph neural network , and adopt a pyramid architecture to hierarchically aggregate local and global features; The specific construction of the graph structure in step S3 includes: S31. Node Feature Generation: Encode the image patches into feature vectors , and add absolute position encoding ; S32. Edge Connection Rule: Establish connections between nodes based on the K - nearest neighbor algorithm, and calculate the edge weights through the Gaussian kernel function. The calculation formula is: , where is the edge weight; and are the spatial coordinates of two nodes respectively; is an adjustable parameter; The feature extraction of the Vision GNN graph neural network in step S3 adopts maximum relative graph convolution, which specifically includes: S33. Aggregation Operation: Calculate the representation of the node by aggregating the features of neighbor nodes. The calculation formula for the aggregated features is: , where is the aggregated feature; represents the difference between the feature of node and all neighbor nodes ; represents selecting the maximum value in the difference vector and retaining the most significant local difference features; represents the neighbor node set of the feature vector ; S34. Update Operation: Update the features separately through the multi - head weight matrix and splice them after updating; S4. Multi - level Feature Fusion and Adaptation: Align the visual features with the graph structure features through the multi - level feature adapter to generate the fused feature ; adopt a two - branch architecture to process the classification feature and the segmentation feature respectively; The implementation steps of the multi - level feature adapter in step S4 include: S41. Linear Transformation: The feature adapter is used to perform a linear transformation at each feature level Adapt to the intermediate-stage features which consists of two linear transformation layers, and the feature transformation is expressed as: , where is the intermediate-stage feature The transformed representation; and are both learnable parameters of the linear transformation; is the activation function; S42. Residual connection: The residual connection is used to retain the original knowledge of the pre-trained CLIP encoding, and the calculation formula is: , where is the fused feature; γ is the learnable residual coefficient; The processing of the two-branch structure in step S4 includes: S43. The features at each level are divided into classification features and segmentation features Two parallel branches, which are used for classification tasks and segmentation tasks respectively, and can capture the overall anomalies and local detail anomalies of the image simultaneously; the projection head projects the visual feature through a linear layer to obtain the global feature for classification and the local feature for segmentation; S44. Multilevel feature alignment is achieved by combining the loss functions of different components, and the loss function adopts a weighted combination of Dice loss, focal loss and cross-entropy loss; S5. Anomaly detection and output: In the zero-shot mode and few-shot mode, the anomaly classification scores and anomaly segmentation maps in their respective modes are generated respectively, and then the final anomaly detection result is obtained through weighted fusion; The specific process of step S5 is: S51. Process the test image to generate multilevel adapted features; S52. Compare the generated multilevel adapted features with the text feature , and calculate the zero-shot using the average softmax scores at four levels. The calculation formula is: , , where is the anomaly classification score generated in the zero-shot mode; max G (•) is the maximum function, G is the number of grids; is the normalization exponential function; is the transpose of the text feature; is the anomaly segmentation map generated in the zero-shot mode; is the bilinear interpolation operation; S53. Construct a multi-level feature memory bank to store normal sample features; S54. Calculate the minimum distance between the test feature and the memory bank through cosine distance. The calculation formula is: , , where is the anomaly classification score generated in the few-shot mode; is to take the minimum distance between the normal sample features stored in the feature memory bank; is the cosine distance; is the object for comparison between the current layer feature and the normal sample features in the memory bank; is the anomaly segmentation map generated in the few-shot mode; S55. Add weight factors to fuse the anomaly detection results of the zero-shot mode and the few-shot mode. The calculation formula is: , , where is the fused anomaly classification score; is the fused anomaly segmentation map; and are both weight factors.

[0017] As mentioned above, it is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A medical anomaly detection method guided by graph language features, characterized in that, It includes the following steps: S1. Multi-modal data input: Obtain medical image data and associated text description data; S2. Visual-Language Feature Extraction: Extract visual features of medical images through the visual encoder of the CLIP model ; Extract text features of text descriptions through the text encoder of the CLIP model ; S3. Visual - Graph Feature Extraction: Segment the medical image into multiple image patches and construct a graph structure , where the nodes ∈ represent the image patches; the edges ∈ represent the spatial adjacency relationship; extract the graph structure features through the Vision GNN graph neural network , and adopt a pyramid architecture to hierarchically aggregate local and global features; S4. Multi-level Feature Fusion and Adaptation: Through a multi-level feature adapter Align visual features with graph structure features to generate fused features ; Use a two-branch architecture to process classification features separately and segmentation features ; S5. Anomaly detection and output: In the zero-shot mode and few-shot mode, generate the anomaly classification scores and anomaly segmentation maps in their respective modes, and then obtain the final anomaly detection result through weighted fusion; The specific process of step S5 is as follows: S51. Process the test image to generate multi-level adaptive features; S52. Compare the generated multi-level adaptation features with the text features and calculate the zero-shot using the average softmax scores of four levels. The calculation formula is: , where is the anomaly classification score generated in the zero-shot mode; max G (•) is the maximum value function, and G is the number of grids; is the normalization exponential function; is the transpose of the text feature; is the anomaly segmentation map generated in the zero-shot mode; is the bilinear interpolation operation; S53. Construct a multi-level feature memory bank to store normal sample features; S54. Calculate the minimum distance between the test feature and the memory bank through the cosine distance. The calculation formula is: , , where is the anomaly classification score generated in the few-shot mode; is the minimum distance between the normal sample features stored in the feature memory bank; is the cosine distance; is the object for comparing the current layer feature with the normal sample features in the memory bank; is the anomaly segmentation map generated in the few-shot mode. S55. Add a weight factor to fuse the anomaly detection results of the zero-shot mode and the few-shot mode. The calculation formula is as follows: , , where is the fused anomaly classification score; is the fused anomaly segmentation map; and are both weight factors.

2. The medical anomaly detection method based on graph language feature guidance according to claim 1, wherein, The process of text feature extraction described in step S2 includes: constructing a state-level prompt template, embedding organ names into text descriptions; generating dynamic text inputs through a two-layer text prompting method, and outputting two-dimensional text features , corresponding to the normal state and the abnormal state respectively; where d is the feature dimension.

3. The medical anomaly detection method guided by graph language features as described in claim 1, wherein The visual feature extraction described in step S2 uses the four encoding stages of the CLIP model, and the intermediate stage features are used for multi-level feature fusion; among them, represents the feature levels of the three intermediate stage features; G represents the number of grids.

4. The medical anomaly detection method based on graph language feature guidance according to claim 1, wherein The specific construction of the graph structure in step S3 includes: S31. Node feature generation: Encode the image patch into a feature vector , and add absolute position encoding ; S32. Edge connection rule: Establish connections between nodes based on the K-nearest neighbor algorithm. The edge weights are calculated using the Gaussian kernel function, and the calculation formula is: , where is the edge weight; and are the spatial coordinates of two nodes respectively; is an adjustable parameter.

5. The medical anomaly detection method based on graph language feature guidance according to claim 4, wherein In step S3, the feature extraction of the Vision GNN graph neural network adopts maximum relative graph convolution, which specifically includes: S33. Aggregation operation: Calculate the representation of a node by aggregating the features of neighboring nodes. The calculation formula for aggregating features is: , where is the aggregated feature; represents the node and all neighboring nodes of the feature difference; represents selecting the maximum value in the difference vector and retaining the most significant local difference features; represents the feature vector of the neighboring node set; S34. Update operation: After separately updating features through the multi-head weight matrix they are concatenated.

6. The medical anomaly detection method based on graph language feature guidance according to claim 1, wherein In step S4, the multi-level feature adapter The implementation steps include: S41. Linear transformation: The feature adapter is used to adapt the intermediate-stage features at each feature level and consists of two linear transformation layers. The feature transformation is expressed as: where is the intermediate-stage feature and is the transformed representation; and are both learnable parameters of the linear transformation; is the activation function. S42. Residual connection: The residual connection is adopted to retain the original knowledge of the pre-trained CLIP encoding. The calculation formula is: , where is the fused feature; γ is the learnable residual coefficient.

7. The medical anomaly detection method based on graph language feature guidance according to claim 1, wherein The processing of the bipartite architecture in step S4 includes: S43. The features at each level are divided into categorical features and segmentation features Two parallel branches, which are respectively used for the classification task and the segmentation task, can simultaneously capture the overall anomalies and local detail anomalies of the image; the projection head Projects the visual features Through a linear layer to obtain the global features for classification And the local features for segmentation ; S44. Achieve multi-level feature alignment by combining the loss functions of different components. The loss function adopts a weighted combination of Dice loss, focal loss, and cross-entropy loss.

Images