A multi-modal medical image anomaly detection method, medium, and device

Abstract

The application discloses a multimodal medical image anomaly detection method, medium and equipment, wherein during training of a neural network model, a large language model and a template are used to generate a text description dataset, the interval boundary is implicitly optimized by learning the potential distance and relationship between texts, the normal text description prompt interval and the abnormal prompt text interval are distinguished, and a more accurate text description dataset is screened out, so that the semantic ambiguity problem is relieved. In addition, the application also introduces an attention mechanism, the local features of image visual features are obtained without changing the original architecture of the pre-training model, and a multi-layer adapter is used to fine-tune the visual language model to adapt to the needs of medical image anomaly detection. The visual features and text features obtained through calculation are used to obtain a global-based anomaly score and a local anomaly map to realize anomaly detection and segmentation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and specifically to a multimodal method, medium, and device for detecting anomalies in medical images. Background Technology

[0002] Medical anomaly detection is a crucial task in computer vision, focusing on identifying discrepancies between medical images and other medical images, and pinpointing the location of these lesions. This provides healthcare professionals with valuable insights for decision-making, improving their efficiency. However, in real-world applications, medical data collection is often challenging and requires extensive annotation by professionals. This often necessitates training existing methods with thousands or even more data points, increasing development costs and resource consumption. Therefore, it is essential to explore how to train models using fewer samples to achieve good performance while avoiding the costs associated with collecting and annotating medical data. Furthermore, the significant differences in medical images across various medical fields make it difficult to find a universal model that performs well on all data; most models require training or adjustment on specific datasets.

[0003] Currently, large-scale pre-trained visual language models have demonstrated strong generalization capabilities by mapping natural images and raw text into a unified space, exhibiting excellent performance across various downstream tasks. However, existing methods rely on fixed textual descriptions of images, and the design of manual prompts not only requires significant domain knowledge and time investment but also faces semantic ambiguity issues. Furthermore, while visual language models can capture global anomalies by aligning image-level visual features with anomaly prompts, they fall short in precise anomaly localization. Therefore, how to leverage visual language models to capture local anomalies has become a crucial issue. Summary of the Invention

[0004] In view of the above problems, the present invention provides a multimodal medical image anomaly detection technical solution to solve the problem that existing neural network models are not accurate enough in text description and anomaly region localization when the number of samples is small when performing anomaly detection on medical images.

[0005] To achieve the above objectives, in a first aspect, this application provides a multimodal medical image anomaly detection method, comprising the following steps:

[0006] Receive medical images;

[0007] The medical image is input into the trained neural network model, and the abnormality detection result corresponding to the medical image is output. The abnormality detection result includes textual description information on whether the image is abnormal and an abnormal feature map.

[0008] The neural network model is trained by including the following steps:

[0009] S1: Generate different categories of text description datasets based on templates and large language models. The text description datasets include normal text description sets, abnormal text description sets, normal text generation sets, and abnormal text generation sets.

[0010] S2: Receive medical sample images, calculate the cosine similarity between the medical sample images and the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set, respectively, and calculate the relevance score based on the cosine similarity;

[0011] If the calculated relevance score is greater than 0, the corresponding text description is retained; otherwise, the corresponding text description is deleted. This process continues until all data in the text description dataset is traversed, resulting in the filtered, precise text description dataset that is most relevant to the currently input medical sample image. The precise text description dataset includes a precise normal text description set and a precise abnormal text description set.

[0012] S3: Extract key features of the medical sample image through an attention mechanism, input the key features into a multi-layer feature adapter for dynamic learning to obtain deep features, and perform cosine similarity calculation between the deep features and the data in the precise text description dataset to obtain the text description information and abnormal feature map corresponding to the current medical sample image.

[0013] Optionally, the normal text description set and the abnormal text description set are generated based on templates, where the templates are textual descriptions of a certain disease type and health status. The normal text generation set and the abnormal text generation set are generated based on the feedback results of the large language model for the question information, where the question information includes a certain disease type and health status.

[0014] If the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set are labeled as Tn, Ta, Gn, and Ga respectively, then the calculation of cosine similarity includes:

[0015] d a (x)= <f(x),g(t a )>,t a ∈{T a G a};

[0016] d n (x)= <f(x),g(t n )>,t n ∈{T n G n};

[0017] Where, d a (x) represents the cosine similarity between the input medical sample image x and the abnormal text dataset, d n (x) represents the cosine similarity between the input medical sample image x and the normal text dataset.

[0018] f(x)∈R d f(x) refers to the visual embedding features obtained by the pre-trained visual encoder in the neural network model, d is the dimension of the latent space, and R d Let g(t) represent a one-dimensional real vector space. a ) refers to the anomalous text embedding features obtained by the pre-trained text encoder, g(t) n ) refers to the normal text embedding features obtained by the pre-trained text encoder, and <> represents the cosine similarity function, which is expressed as follows:

[0019]

[0020] Where f(x)·g(t) represents the inner product of f(x) and g(t), ||f(x)|| represents the norm of f(x), and ||g(t)|| is the norm of g(t).

[0021] Optionally, the relevance score based on the cosine similarity is calculated using the following formula:

[0022]

[0023] S(t) represents the correlation score, with a value range of [0,1), where L represents the following:

[0024] L=||D(d x (t),d n (x))-D(d x (t),d a (x))||;

[0025] Where, d x (t) refers to the cosine similarity between the text description t and the medical sample image x, k is a hyperparameter controlling the slope of the function, and the function D() is used to calculate d. x (t) and d n (t), or d x (t) and d a The distance (x) is expressed as follows:

[0026] D(d x (t),d n (x))=max(0,max(min{d n (x)}-d x(t),d x (t))-max{d n (x)}));

[0027] D(d x (t),da(x))=max(0,max(min{d a (x}-d x (t),d x (t)-max{d a (x})).

[0028] Optionally, key features of the medical sample image are extracted using an attention mechanism and calculated according to the following formula:

[0029] Attn(Q,K,V)=softmax(Q·K·scale)·V;

[0030] Q, K, and V are vector representations that have undergone specific transformations or been extracted from the input data. They are used to calculate attention weights and obtain the weighted output. softmax represents the activation function, which is used to convert the input values ​​into a probability distribution. scale represents the scaling factor.

[0031] Optionally, the key features are input into a multi-layer feature adapter for dynamic learning to obtain deep features, including:

[0032]

[0033] Z l-1 =[t′ cls ;t′1;t′2,…;t′ T ];

[0034]

[0035] Z l =Proj. l (Attn(V l V l V l ))+z l-1 .

[0036] Among them, Z represents the output of the L-1 layer of the visual encoder. l-1 This represents the local feature output of layer L-1, QKV_Proj. l And Proj. l These represent the QKV projection and the output projection, respectively. The last layer outputs the original output Z. o And local output Z, categorical features Z o [0]∈R dFor image-level anomaly detection, the local feature map Z[1:]∈R Txd Used for pixel-level anomaly detection;

[0037] For the input medical sample image x∈R hxwx3 Transform the medical sample image x into the feature space Z. l ∈R Gxd , l∈{1,2,3,4}, where h and w represent the image size, G represents the number of grids, and d represents the feature dimension;

[0038] The multi-layer feature adapter includes four adapters, denoted as A. l (·), l∈{1,2,3,4}, in each stage S l For l∈{1,2,3,4}, the feature adapter is integrated into feature Z through two linear transformation layers. l In Chinese, it is represented as follows:

[0039]

[0040] Where ReLU represents the activation function and Linear represents the linear transformation layer;

[0041] After obtaining the features through multiple feature adapters, residual connections are used to preserve the original pre-trained features. The adapted features of the Lth layer can then be represented as:

[0042] Z′ l =αA l (Z l ) T +(1-α)Z l ,l∈{1,2,3,4} (11)

[0043] Among them, Z′ l As the next stage S l+1 The input is a constant value α, which is used as the residual ratio to adjust the degree to which the original features are preserved.

[0044] The feature adapter is a dual-branch feature adapter, generating two parallel features F at each layer. cls，l and F seg，l , of which F cls，l F represents the features generated at layer L through the bi-branch feature adapter for a classification task, used to indicate whether the input medical sample image x is abnormal. seg，l This indicates that the features generated by the dual-branch feature adapter at the 1st layer are used for segmentation tasks, specifically for segmenting out abnormal regions.

[0045] The deep features are compared with the data in the precise text description dataset using cosine similarity calculation to obtain the text description information and anomaly feature map corresponding to the current medical sample image, including:

[0046]

[0047] Where F∈{F cls，l ,F seg，l}, τ is a temperature hyperparameter used to adjust the output distribution of the softmax function, F T F represents the text description feature data in the precise text description dataset. n F represents the feature data of normal text description. a This represents the feature data describing abnormal text, and exp<> represents the exponential function;

[0048]

[0049] Where C1 represents the classification index of the anomaly type, S1 represents the segmentation index of the anomaly feature map, and BI() represents the image processing function used to reshape the anomaly map into... It is then restored to the resolution of the original input medical sample image using bilinear interpolation, where G represents the number of grids;

[0050]

[0051]

[0052] Where m represents the minimum distance between features at each level recorded in the multi-level feature memory G, and the function Dist() represents the cosine distance. The final predicted classification and segmentation results are as follows:

[0053] C pre =βC1+(1-β)C2;

[0054] S pre =βS1+(1-β)S2;

[0055] Among them, C pre For the finally determined image anomaly type, S pre The final anomaly feature map is defined by β, which represents the weight value.

[0056] Optionally, the loss function of the neural network model is obtained in the following way:

[0057] L1 = θ1Dice(softmax(F) seg，l F T ), S gt +θ2Focal(softmax(F seg，l FT ), S gt )+θ3BCE(max(softmax(F cls，l F T C gt );

[0058] The Dice loss, used to measure the similarity between two samples, is defined as follows:

[0059]

[0060] A higher Dice coefficient indicates a higher degree of similarity;

[0061] Focal(softmax(F seg，l F T ), S gt )=-αS gt (1-softmax(F seg，l F T )) γ log(softmax(F seg，l F T ))-α(1-S gt )S gt (1-softmax(F seg，l F T )) γ log(softmax(F seg，l F T ));

[0062] Where α is the class balance factor, used to reduce the impact of class imbalance; γ is the focus factor, used to adjust the weights of easy and difficult samples; S gt Indicates the true label;

[0063] BCE(max(softmax(F cls，l F T C gt )=-[C gt log(max(softmax(F cls，l F T )))+(1-C gt )log(1-max(softmax(F cls，l F T )))];

[0064] BCE loss is used to measure the difference between the predicted probability distribution and the true label, where C gt These are the actual labels, with values ​​of 0 or 1, and max(softmax(F)). cls，l FT The loss is the probability predicted by the model; the smaller the loss, the better the model performs.

[0065] The total loss of the model is expressed as follows:

[0066] Where L1 represents the total layer loss, and θ1, θ2, and θ3 represent hyperparameters that are set to different proportions to optimize the neural network model.

[0067] Optionally, the neural network model, during training, further includes the following steps:

[0068] The accuracy of a neural network model is determined by AUC, and the calculation formula is as follows:

[0069]

[0070] Where TPR represents the true positive rate, FPR represents the false positive rate, TP represents the number of pixels correctly detected as abnormal in the input medical sample image, FN represents the number of abnormal pixels incorrectly detected as normal in the input medical sample image, FP represents the number of normal pixels incorrectly detected as abnormal in the medical sample image, and TN represents the number of pixels correctly detected as normal in the medical sample image.

[0071]

[0072] In a second aspect, this application provides a multimodal medical image anomaly detection system, comprising:

[0073] An image receiving module is used to receive medical images;

[0074] An anomaly detection module is used to input the medical image into a trained neural network model and output the anomaly detection result corresponding to the medical image. The anomaly detection result includes textual description information on whether the image is abnormal and an anomaly feature map.

[0075] The neural network model, during training, includes:

[0076] The text data generation module is used to generate different categories of text description datasets based on templates and large language models. The text description datasets include normal text description sets, abnormal text description sets, normal text generation sets, and abnormal text generation sets.

[0077] The correlation analysis module is used to receive medical sample images, calculate the cosine similarity between the medical sample images and the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set, and calculate the correlation score based on the cosine similarity.

[0078] If the calculated relevance score is greater than 0, the corresponding text description is retained; otherwise, the corresponding text description is deleted. This process continues until all data in the text description dataset is traversed, resulting in the filtered, precise text description dataset that is most relevant to the currently input medical sample image. The precise text description dataset includes a precise normal text description set and a precise abnormal text description set.

[0079] Attention mechanism module, used to extract key features of the medical sample image through attention mechanism;

[0080] A multi-layer feature adapter is used to dynamically learn the key features to obtain deep features, and to calculate the cosine similarity between the deep features and the data in the precise text description dataset to obtain the text description information and abnormal feature map corresponding to the current medical sample image.

[0081] In a third aspect, the present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method described in the first aspect.

[0082] In a fourth aspect, the present invention also provides an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0083] Unlike existing technologies, the above solution provides a multimodal medical image anomaly detection method, medium, and device. The method includes: receiving a medical image, inputting the medical image into a trained neural network model, and outputting the anomaly detection result corresponding to the medical image. During training, the neural network model generates a text description dataset based on a large language model and templates. By learning the latent distances and relationships between texts, it implicitly optimizes interval boundaries, distinguishes between normal text description prompt intervals and abnormal text prompt intervals, and filters out a more accurate text description dataset, thereby alleviating semantic ambiguity. Furthermore, this application introduces an attention mechanism to obtain local features of the image's visual features without changing the original architecture of the pre-trained model. A multi-layer adapter is used to fine-tune the visual language model to adapt to the needs of medical image anomaly detection. Anomaly detection and segmentation are achieved by calculating the visual and text features to obtain a global anomaly score and a local anomaly map.

[0084] The above description of the invention is merely an overview of the technical solution of the present invention. In order to enable those skilled in the art to better understand the technical solution of the present invention and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of the present invention easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of the present invention. Attached Figure Description

[0085] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on the present invention.

[0086] In the accompanying drawings of the instruction manual:

[0087] Figure 1 A flowchart illustrating a multimodal medical image anomaly detection method according to a first exemplary embodiment of the present invention;

[0088] Figure 2 A flowchart illustrating a multimodal medical image anomaly detection method according to a second exemplary embodiment of the present invention;

[0089] Figure 3 This is a schematic diagram of the modules of a multimodal medical image anomaly detection system according to a specific embodiment of the present invention;

[0090] Figure 4 This is a schematic diagram of the modules of a neural network model according to a specific embodiment of the present invention;

[0091] Figure 5 This is a schematic diagram of an electronic device according to a specific embodiment of the present invention;

[0092] Figure 6 This is a schematic diagram illustrating the principle of neural network model training according to a specific embodiment of the present invention;

[0093] Figure 7 This is a schematic diagram of the neural network model testing process according to a specific embodiment of the present invention;

[0094] Figure 8 This is a schematic diagram illustrating the principle of the attention mechanism involved in a specific embodiment of the present invention;

[0095] Figure 9 This is a schematic diagram illustrating the principle of a multi-layer feature adapter according to a specific embodiment of the present invention;

[0096] Figure 10 This is a detailed flowchart of text boundary optimization according to a specific embodiment of the present invention.

[0097] The reference numerals used in the above figures are explained as follows:

[0098] 10. Electronic devices;

[0099] 101. Processor;

[0100] 102. Storage medium.

[0101] 30. Multimodal medical image anomaly detection system;

[0102] 301. Image receiving module;

[0103] 302. Anomaly Detection Module;

[0104] 40. Neural network model;

[0105] 401. Text Data Generation Module;

[0106] 402. Correlation Analysis Module;

[0107] 403. Attention Mechanism Module;

[0108] 404. Multi-layer feature adapter. Detailed Implementation

[0109] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this invention in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this invention and are therefore intended only as examples, not as limiting the scope of protection of this invention.

[0110] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this invention, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0111] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit the invention.

[0112] In the description of this invention, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " generally indicates that the preceding and following objects have an "or" logical relationship.

[0113] In this invention, terms such as “first” and “second” are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy, or order between these entities or operations.

[0114] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this invention is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0115] In this invention, expressions such as "greater than", "less than", and "exceeding" are understood to exclude the stated number; expressions such as "above", "below", and "within" are understood to include the stated number. Furthermore, in the description of the embodiments of this invention, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times", unless otherwise explicitly specified.

[0116] In the first aspect, such as Figure 1 As shown, this application provides a multimodal medical image anomaly detection method, including the following steps:

[0117] Step S201: Receive medical images;

[0118] Step S202: Input the medical image into the trained neural network model and output the anomaly detection result corresponding to the medical image. The anomaly detection result includes textual description information on whether the image is abnormal and an anomaly feature map.

[0119] In short, medical images are medical images to be analyzed and processed. The abnormal text description information includes two parts: first, whether there are any abnormalities in the current medical image; and second, when the medical image is an abnormal image, it will explain the type of abnormality, the abnormal area, and other details.

[0120] To improve the accuracy of anomaly detection results identified by neural network models, such as Figure 2As shown, in this application, the neural network model includes the following steps during training:

[0121] S1: Generate different categories of text description datasets based on templates and large language models. The text description datasets include normal text description sets, abnormal text description sets, normal text generation sets, and abnormal text generation sets.

[0122] In step S1, the normal text description set and the abnormal text description set are generated based on a template, where the template is textual description information describing a certain disease type and health state. Medical image categories refer to the classification of different disease types, body parts, or pathological states involved in medical images. For example, this may include categories of brain diseases (such as brain tumors, cerebral hemorrhage, etc.), chest diseases (such as lung cancer, pneumonia, etc.), and bone diseases (such as fractures, osteoporosis, etc.). In this application, when generating the text description set based on the template, the text description template can be "a healthy image of [category]" or "a lesion image of [category]". The [category] in this range refers to the specific medical image category. By clearly defining the category, text information related to that category can be constructed specifically, so that it can be better associated and analyzed with medical images of the corresponding category, thereby achieving anomaly detection and segmentation of different types of medical images.

[0123] The normal and abnormal text generation sets are generated based on the feedback results of the large language model in response to the query information, which includes a specific disease type and health status. Specifically, the normal and abnormal text generation sets can be obtained by asking questions to the large language model, such as "What abnormalities might occur in [category]?" or "Describe the normal / abnormal image situation regarding [category]". The large language model generates corresponding text descriptions based on its learned knowledge and language patterns. A normal text generation set Gn can be constructed based on answers in a normal state, and an abnormal text generation set Ga can be constructed based on answers in an abnormal state. Utilizing the powerful language generation capabilities of the large language model, more diverse and richer text descriptions can be obtained, overcoming the limitations of template-based generation and further expanding the sources of text description data for analyzing medical images. This helps improve the adaptability and accuracy of medical image anomaly analysis in different application scenarios.

[0124] S2: Receive a medical sample image, calculate the cosine similarity between the medical sample image and the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set, respectively, and calculate the relevance score based on the cosine similarity. If the calculated relevance score is greater than 0, the corresponding text description is retained; otherwise, the corresponding text description is deleted. This process continues until all data in the text description datasets has been traversed, resulting in the filtered accurate text description dataset that is most relevant to the currently input medical sample image. The accurate text description dataset includes the accurate normal text description set and the accurate abnormal text description set.

[0125] In step S2, the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set are labeled as Tn, Ta, Gn, and Ga, respectively. The calculation of cosine similarity includes:

[0126] d a (x)= <f(x),g(t a )>,t a ∈{T a G a};

[0127] d n (x)= <f(x),g(t n )>,t n ∈{T n G n};

[0128] Where, d a (x) represents the cosine similarity between the input medical sample image x and the abnormal text dataset, d n (x) represents the cosine similarity between the input medical sample image x and the normal text dataset.

[0129] f(x)∈R d f(x) refers to the visual embedding features obtained by the pre-trained visual encoder in the neural network model, d is the dimension of the latent space, and R d Let g(t) represent a one-dimensional real vector space. a ) refers to the anomalous text embedding features obtained by the pre-trained text encoder, g(t) n ) refers to the normal text embedding features obtained by the pre-trained text encoder, and <> represents the cosine similarity function, which is expressed as follows:

[0130]

[0131] Where f(x)·g(t) represents the inner product of f(x) and g(t), and ||f(x)|| represents the norm of f(x), calculated by the following formula: ||g(t)|| is the norm of g(t), and its calculation formula is similar to that of the norm of f(x).

[0132] A pre-trained visual encoder is a neural network model pre-trained on large-scale image data. Its main function is to convert input visual image data into a feature representation with semantic information. For example, it can take a medical image as input and output a fixed-dimensional vector representing the features of that image.

[0133] A pre-trained text encoder is a neural network model pre-trained on text data. It can convert input text into feature vectors containing semantic information. For example, it can take a medical image as input and output a vector indicating whether the medical image is abnormal, and use disease descriptions in the text to better identify abnormal regions in the image.

[0134] Visual embedding features are the output of a pre-trained visual encoder after processing an input medical image x. These features are high-dimensional vectors, and their purpose is to represent the information of the original medical image in a form that is easier to process and analyze later.

[0135] By calculating cosine similarity, the correlation between medical sample images and text descriptions can be measured. Ideally, the distances between normal text descriptions and medical images, as well as between abnormal text descriptions and medical images, should lie within two independent intervals. However, due to the uncertainty in the text description prompts generated by large language models, some abnormal prompts have features closer to normal prompts, causing overlap between the normal and abnormal text description prompt intervals. These prompts negatively impact the performance of neural network models.

[0136] Based on this, this application quantifies the impact of anomalies in normal prompts using the logits function. Specifically, the relevance score is calculated based on the cosine similarity using the following formula:

[0137]

[0138] S(t) represents the correlation score, with a value range of [0,1), where L represents the following:

[0139] L=|D(d x (t),d n (x))-D(d x (t),d a (x))||;

[0140] Where, d x(t) refers to the cosine similarity between the text description t and the medical sample image x, k is a hyperparameter controlling the slope of the function (usually k is set to 1), and the function D() is used to calculate d. x (t) and d n (x), or d x (t) and d a The distance (x) is expressed as follows:

[0141] D(d x (t),d n (x))=max(0,max(min{d n (x)}-d x (t),d x (t)-max{d m (x}));

[0142] D(d x (t),d a (x))=max(0,max(min{d a (x)}-d x (t),d x (t))-max{d a (x)})).

[0143] Typically, if the distance between the text description prompt and the input medical image spans both positive and negative abnormal intervals, and the score stabilizes at 0, it indicates a poor correlation between the text description prompt and the input medical image. Therefore, in practical applications, this application filters out intervals with correlation scores less than or equal to 0, retaining only text description prompts in non-overlapping intervals, i.e., T = {t′} a ,t′ n}, where t′ a t′ represents the filtered anomaly description text (i.e., the precise anomaly text description set). n This represents the filtered normal descriptive text (i.e., the precise normal text description set). Based on this filtering process, a unique set of text features best suited to the input image can be adaptively selected for each medical image, thereby improving the effectiveness of the model.

[0144] S3: Extract key features of the medical sample image through an attention mechanism, input the key features into a multi-layer feature adapter for dynamic learning to obtain deep features, and perform cosine similarity calculation between the deep features and the data in the precise text description dataset to obtain the text description information and abnormal feature map corresponding to the current medical sample image.

[0145] Some visual language models (such as CLIP) perform poorly in guided image localization tasks because Q-K self-attention ignores the importance of local features in its global features, which is a fatal flaw in pixel-level detection and segmentation. This application introduces a VV local attention mechanism to enhance the influence of local features without affecting the network structure of the original model. The QK self-attention calculation formula is as follows:

[0146] Attn(Q,K,V)=softmax(Q·K·scale)·V;

[0147] Q, K, and V are vector representations that have undergone specific transformations or been extracted from the input data. They are used to calculate attention weights and obtain the weighted output. softmax represents the activation function, which is used to convert the input values ​​into a probability distribution. scale represents the scaling factor.

[0148] In some embodiments, the key features are input into a multi-layer feature adapter for dynamic learning to obtain deep features, including:

[0149]

[0150] Z l-1 =[t′ cls ;t′1;t′2,…;t′ T ];

[0151]

[0152] Z l =Proj. l (Attn(V l V l V l ))+Z l-1 ;

[0153] Among them, Z represents the output of the L-1 layer of the visual encoder. l-1 This represents the local feature output of layer L-1, QKV_Proj. l And Proj. l These represent the QKV projection and the output projection, respectively. The last layer outputs the original output Z. o And local output Z, categorical features Z o [0]∈R d For image-level anomaly detection, the local feature map Z[1:]∈R Txd Used for pixel-level anomaly detection;

[0154] For the input medical sample image x∈R hxwx3Transform the medical sample image x into the feature space Z. l ∈R Gxd , l∈{1,2,3,4}, where h and w represent the image size, G represents the number of grids, and d represents the feature dimension;

[0155] The multi-layer feature adapter includes four adapters, denoted as A. l (·), l∈{1,2,3,4}, in each stage S l For l∈{1,2,3,4}, the feature adapter is integrated into feature Z through two linear transformation layers. l In Chinese, it is represented as follows:

[0156]

[0157] Where ReLU represents the activation function and Linear represents the linear transformation layer;

[0158] After obtaining the features through multiple feature adapters, residual connections are used to preserve the original pre-trained features. The adapted features of the Lth layer can then be represented as:

[0159] Z' l =αA l (Z l ) T +(1-α)Z l ,l∈{1,2,3,4} (11)

[0160] Among them, Z′ l As the next stage S l+1 The input is a constant value α, which is used as the residual ratio to adjust the degree to which the original features are preserved.

[0161] The feature adapter is a dual-branch feature adapter, generating two parallel features F at each layer. cls，l and F seg，l , of which F cls，l F represents the features generated at layer L through the bi-branch feature adapter for a classification task, used to indicate whether the input medical sample image x is abnormal. seg,l This indicates that the features generated by the dual-branch feature adapter at the 1st layer are used for segmentation tasks, specifically for segmenting out abnormal regions.

[0162] The deep features are compared with the data in the precise text description dataset using cosine similarity calculation to obtain the text description information and anomaly feature map corresponding to the current medical sample image, including:

[0163]

[0164] Where F∈{F cls,l ,Fseg,l}, τ is a temperature hyperparameter used to adjust the output distribution of the softmax function, F T F represents the text description feature data in the precise text description dataset. n F represents the feature data of normal text description. a This represents the feature data describing abnormal text, and exp<> represents the exponential function;

[0165]

[0166] Where C1 represents the classification index of the anomaly type, S1 represents the segmentation index of the anomaly feature map, and BI() represents the image processing function used to reshape the anomaly map into... It is then restored to the resolution of the original input medical sample image using bilinear interpolation, where G represents the number of grids;

[0167]

[0168] Where m represents the minimum distance between features at each level recorded in the multi-level feature memory G, and the function Dist() represents the cosine distance. The final predicted classification and segmentation results are as follows:

[0169] C pre =βC1+(1-β)C2;

[0170] S pre =βS1+(1-β)S2;

[0171] Among them, C pre For the finally determined image anomaly type, S pre The final anomaly feature map is defined by β, which represents the weight value.

[0172] The above scheme constructs a multi-level feature memory G using all multi-level visual features from several normal medical images to facilitate feature comparison with normal features. A nearest neighbor search process is used to compare the minimum distance between test features and features in each level of the memory, thereby improving anomaly detection and segmentation performance.

[0173] In some embodiments, the loss function of the neural network model is obtained as follows:

[0174] L1 = θ1Dice(softmax(F) seg，l F T ),S gt )+θ2Focal(softmax(F seg，l F T ),S gt )+θ3BCE(max(softmax(F cls，l FT C gt );

[0175] The Dice loss, used to measure the similarity between two samples, is defined as follows:

[0176]

[0177] A higher Dice coefficient indicates a higher degree of similarity;

[0178] Focal(softmax(F seg，l F T ),S gt )=-αS gt (1-softmax(F seg，l F T )) γ log(softmax(F seg，l F T ))-α(1-S gt )S gt (1-softmax(F seg，l F T )) γ log(softmax(F seg，l F T ));

[0179] Where α is the class balance factor, used to reduce the impact of class imbalance; γ is the focus factor, used to adjust the weights of easy and difficult samples; S gt Indicates the true label;

[0180] BCE(max(softmax(F cls，l F T )),C gt )=-[C gt log(max(softmax(F cls，l F T )))+(1-C gt )log(1-max(softmax(F cls，l F T )))];

[0181] BCE loss is used to measure the difference between the predicted probability distribution and the true label, where C gt These are the actual labels, with values ​​of 0 or 1, and max(softmax(F)). cls,l ,F T The loss is the probability predicted by the model; the smaller the loss, the better the model performs.

[0182] The total loss of the model is expressed as follows:

[0183] Where L1 represents the total layer loss, and θ1, θ2, and θ3 represent hyperparameters that are set to different proportions to optimize the neural network model.

[0184] After the above training, a trained neural network model can be obtained. When the neural network model of this application is applied to medical image anomaly analysis, it has the following advantages:

[0185] (1) It can effectively utilize text description information, learn the relationship with medical images, and improve the accuracy of recognition;

[0186] (2) Only a small number of samples (e.g., 2, 4, 8) are needed to obtain high accuracy, which makes up for the lack of medical image data and the resulting low accuracy of the training model, thus providing assistance to medical staff.

[0187] (3) The use of lightweight multi-layer feature adapters consumes fewer resources and has a fast inference speed, which can be widely used in practice.

[0188] like Figures 6-10 As shown, this application provides a method for iteratively training a neural network based on a small number of medical abnormality image samples, specifically including the following steps:

[0189] Will Figure 6 The 12-layer image encoder of the multimodal pre-trained model shown is used as the basic feature extraction module. Based on this, without changing the original neural network architecture, a VV attention mechanism is introduced. The features obtained after the input medical sample image passes through 3 layers of VV attention mechanism are input to the corresponding layer adapter for dynamic learning. As the number of layers increases, more and more deep feature details are preserved. Then, the features output by the multi-layer adapter (i.e., deep features) are compared with the accurate text description feature set to calculate cosine similarity.

[0190] Meanwhile, the loss function described above is used as the loss for image classification and segmentation, and the SGD optimization algorithm is used to perform reverse correction on the neural network. The above steps are repeated iteratively until the loss no longer decreases significantly or the maximum number of training iterations is reached, at which point training stops, resulting in a fully trained neural network model.

[0191] For a trained neural network model, upon receiving any medical image x (x∈X), the model can predict an anomaly score and a pixel-level anomaly feature map. Then, based on the minimum distance between the medical image x and the features in each level of memory, a normal anomaly score and a pixel-level anomaly feature map are obtained. This anomaly score and pixel-level anomaly feature map are combined with the two parameters obtained from the model to obtain the final anomaly score and anomaly feature map. The existence of the current medical image x is determined by whether the final anomaly score is greater than a certain threshold. Furthermore, by comparing the input medical image with the real image pixel by pixel, abnormal regions are identified and segmented for output.

[0192] In some embodiments, the neural network model, during training, further includes the following steps:

[0193] The accuracy of a neural network model is determined by AUC, and the calculation formula is as follows:

[0194]

[0195] Where TPR represents the true positive rate, FPR represents the false positive rate, TP represents the number of pixels correctly detected as abnormal in the input medical sample image, FN represents the number of abnormal pixels incorrectly detected as normal in the input medical sample image, FP represents the number of normal pixels incorrectly detected as abnormal in the medical sample image, and TN represents the number of pixels correctly detected as normal in the medical sample image.

[0196]

[0197] In the second aspect, such as Figure 3 As shown, this application provides a multimodal medical image anomaly detection system 30, comprising:

[0198] Image receiving module 301 is used to receive medical images;

[0199] Anomaly detection module 302 is used to input the medical image into a trained neural network model and output the anomaly detection result corresponding to the medical image. The anomaly detection result includes textual description information on whether the image is abnormal and an anomaly feature map.

[0200] like Figure 4 As shown, the neural network model 40 includes the following during training:

[0201] The text data generation module 401 is used to generate different categories of text description datasets based on templates and large language models. The text description datasets include normal text description sets, abnormal text description sets, normal text generation sets, and abnormal text generation sets.

[0202] The correlation analysis module 402 is used to receive medical sample images, calculate the cosine similarity between the medical sample images and the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set, and calculate the correlation score based on the cosine similarity.

[0203] If the calculated relevance score is greater than 0, the corresponding text description is retained; otherwise, the corresponding text description is deleted. This process continues until all data in the text description dataset is traversed, resulting in the filtered, precise text description dataset that is most relevant to the currently input medical sample image. The precise text description dataset includes a precise normal text description set and a precise abnormal text description set.

[0204] Attention mechanism module 403 is used to extract key features of the medical sample image through an attention mechanism;

[0205] The multi-layer feature adapter 404 is used to dynamically learn the key features to obtain deep features, and to perform cosine similarity calculation between the deep features and the data in the precise text description dataset to obtain the text description information and abnormal feature map corresponding to the current medical sample image.

[0206] In a second aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal medical image anomaly detection method as described in the first aspect of the present invention.

[0207] The computer-readable storage medium may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0208] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory may be a disk storage device or a magnetic tape storage device.

[0209] The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The computer-readable storage media described in the embodiments of the present invention are intended to include these and any other suitable types of memory.

[0210] like Figure 5 As shown, in a third aspect, the present invention provides an electronic device 10, including a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and the computer program, when executed by the processor, implements the multimodal medical image anomaly detection method as described in the first aspect of the present invention.

[0211] In some embodiments, the processor may be implemented by software, hardware, firmware, or a combination thereof, and may use at least one of the following: circuit, single or multiple application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), central processing units (CPUs), controllers, microcontrollers, and microprocessors, thereby enabling the processor to execute some or all of the steps or any combination thereof in the multimodal medical image anomaly detection method described in the various embodiments of this application.

[0212] Finally, it should be noted that although the above embodiments have been described in the description and drawings of this invention, this should not limit the scope of patent protection of this invention. Any technical solutions that are based on the essential concept of this invention, utilize the content described in the description and drawings of this invention to make equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this invention.

Claims

1. A multimodal medical image anomaly detection method, characterized in that, Includes the following steps: Receive medical images; The medical image is input into the trained neural network model, and the abnormality detection result corresponding to the medical image is output. The abnormality detection result includes textual description information on whether the image is abnormal and an abnormal feature map. The neural network model is trained by including the following steps: S1: Generate different categories of text description datasets based on templates and large language models. The text description datasets include normal text description sets, abnormal text description sets, normal text generation sets, and abnormal text generation sets. S2: Receive a medical sample image, calculate the cosine similarity between the medical sample image and the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set, respectively, and calculate the relevance score based on the cosine similarity. If the calculated relevance score is greater than 0, the corresponding text description is retained; otherwise, the corresponding text description is deleted. This process continues until all data in the text description datasets has been traversed, resulting in the filtered accurate text description dataset that is most relevant to the currently input medical sample image. The accurate text description dataset includes the accurate normal text description set and the accurate abnormal text description set. S3: Extract key features of the medical sample image through an attention mechanism, input the key features into a multi-layer feature adapter for dynamic learning to obtain deep features, and perform cosine similarity calculation between the deep features and the data in the precise text description dataset to obtain the text description information and abnormal feature map corresponding to the current medical sample image; The feature adapter is a dual-branch feature adapter, generating two parallel features at each layer. and ,in, This represents the features generated at layer L through the bi-branch feature adapter for the classification task, used to indicate whether the input medical sample image x is abnormal. This indicates that the features generated by the dual-branch feature adapter at the 1st layer are used for segmentation tasks, specifically for segmenting out abnormal regions. The deep features are compared with the data in the precise text description dataset using cosine similarity calculation to obtain the text description information and anomaly feature map corresponding to the current medical sample image, including: ； Where F∈{ , }, It is a temperature hyperparameter used for adjustment. The output distribution of the function, F T F represents the text description feature data in the precise text description dataset. n F represents the feature data of normal text description. a This represents the feature data describing abnormal text, and exp<> represents the exponential function; ； ； Where C1 represents the classification index of the anomaly type, S1 represents the segmentation index of the anomaly feature map, and BI() represents the image processing function used to reshape the anomaly map into... × It then uses bilinear interpolation to restore the resolution of the original input medical sample image to that of the original input image, where G represents the number of grids. ； ； Where m represents the minimum distance between features at each level recorded in the multi-level feature memory G, and the function Dist() represents the cosine distance. The final predicted classification and segmentation results are as follows: ； ； in, To determine the final type of image anomaly, This is the final determined anomaly feature map. This represents the weight value.

2. The multimodal medical image anomaly detection method as described in claim 1, characterized in that, The normal text description set and the abnormal text description set are generated based on templates. The templates are textual descriptions of a certain disease type and health status. The normal text generation set and the abnormal text generation set are generated based on the feedback results of the large language model on the question information. The question information includes a certain disease type and health status. If the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set are labeled as Tn, Ta, Gn, and Ga respectively, then the calculation of cosine similarity includes: ； ； in, This represents the cosine similarity between the input medical sample image x and the anomalous text dataset. This represents the cosine similarity between the input medical sample image x and the normal text dataset. f(x)∈R d f(x) refers to the visual embedding features obtained by the pre-trained visual encoder in the neural network model, d is the dimension of the latent space, and R d Let g represent a one-dimensional real vector space. ) refers to the anomalous text embedding features obtained by the pre-trained text encoder, g( The ) refers to the normal text embedding features obtained by the pre-trained text encoder, and <> represents the cosine similarity function, which is expressed as follows: Cosine Similarity( )= ; in, express (x) and The inner product of (t), || (x)|| represents The norm of (x), yes The norm of (t).

3. The multimodal medical image anomaly detection method as described in claim 2, characterized in that, The correlation score based on the cosine similarity is calculated using the following formula: ； S(t) represents the correlation score, with a value range of [0,1), where L represents the following: ； Where, d x (t) refers to the cosine similarity between the text description t and the medical sample image x, k is a hyperparameter controlling the slope of the function, and the function D() is used to calculate and ,or and The distance is expressed as follows: ； 。 4. The multimodal medical image anomaly detection method as described in claim 1, characterized in that, Key features of the medical sample images are extracted using an attention mechanism and calculated according to the following formula: ； in, The vector representation, after specific transformation or extracted from the input data, is used to calculate attention weights and obtain the weighted output. This represents the activation function, used to transform the input numerical value into a probability distribution. This represents the scaling factor.

5. The multimodal medical image anomaly detection method as described in claim 4, characterized in that, The key features are input into a multi-layer feature adapter for dynamic learning to obtain deep features, including: ； ； ； ； Among them, This represents the output of the L-1 layer of the visual encoder. This represents the local feature output of layer L-1. and These represent QKV projection and output projection, respectively, with the last layer outputting the original output. And local output Z, categorical features For image-level anomaly detection, local feature map Z[1:]∈ Used for pixel-level anomaly detection; For the input medical sample image x∈ Transform the medical sample image x into a feature space , Where h and w represent the image size, G represents the number of grids, and d represents the feature dimension; The multi-layer feature adapter includes four adapters, denoted as follows: In each stage S l , The feature adapter is integrated into the feature through two linear transformation layers. In Chinese, it is represented as follows: ； in, This represents the activation function. Indicates a linear transformation layer; After obtaining the features through multiple feature adapters, residual connections are used to preserve the original pre-trained features. The adapted features of the Lth layer are then represented as follows: （11） in, As the next stage The input is a constant value α, which is used as the residual ratio to adjust the degree to which the original features are preserved.

6. The multimodal medical image anomaly detection method as described in claim 5, characterized in that, The loss function of the neural network model is obtained as follows: ； The Dice loss, used to measure the similarity between two samples, is defined as follows: ； A higher Dice coefficient indicates a higher degree of similarity; ； Where α is the class balance factor, used to reduce the impact of class imbalance; γ is the focus factor, used to adjust the weights of easy and difficult samples. Indicates the true label; ； BCE loss is used to measure the difference between the predicted probability distribution and the true label, where... These are real tags, with values ​​of 0 or 1. It represents the probability predicted by the model; the smaller the loss, the better the model's performance. The total loss of the model is expressed as follows: ; Where L1 represents the total hierarchical loss, , This represents hyperparameters, which are set to different proportions to optimize the neural network model.

7. The multimodal medical image anomaly detection method as described in claim 6, characterized in that, The neural network model, during training, also includes the following steps: The accuracy of a neural network model is determined by AUC, and the calculation formula is as follows: ； ； Where TPR represents the true positive rate, FPR represents the false positive rate, TP represents the number of pixels correctly detected as abnormal in the input medical sample image, FN represents the number of abnormal pixels incorrectly detected as normal in the input medical sample image, FP represents the number of normal pixels incorrectly detected as abnormal in the medical sample image, and TN represents the number of pixels correctly detected as normal in the medical sample image. 。 8. A multimodal medical image anomaly detection system, characterized in that, include: An image receiving module is used to receive medical images; An anomaly detection module is used to input the medical image into a trained neural network model and output the anomaly detection result corresponding to the medical image. The anomaly detection result includes textual description information on whether the image is abnormal and an anomaly feature map. The neural network model, during training, includes: The text data generation module is used to generate different categories of text description datasets based on templates and large language models. The text description datasets include normal text description sets, abnormal text description sets, normal text generation sets, and abnormal text generation sets. The correlation analysis module is used to receive medical sample images, calculate the cosine similarity between the medical sample images and the normal text description set, the abnormal text description set, the normal text generation set, and the abnormal text generation set, and calculate the correlation score based on the cosine similarity. If the calculated relevance score is greater than 0, the corresponding text description is retained; otherwise, the corresponding text description is deleted. This process continues until all data in the text description dataset is traversed, resulting in the filtered, precise text description dataset that is most relevant to the currently input medical sample image. The precise text description dataset includes a precise normal text description set and a precise abnormal text description set. Attention mechanism module, used to extract key features of the medical sample image through attention mechanism; A multi-layer feature adapter is used to dynamically learn the key features to obtain deep features, and to perform cosine similarity calculation between the deep features and the data in the precise text description dataset to obtain the text description information and abnormal feature map corresponding to the current medical sample image. The feature adapter is a dual-branch feature adapter, generating two parallel features at each layer. and ,in, This represents the features generated at layer L through the bi-branch feature adapter for the classification task, used to indicate whether the input medical sample image x is abnormal. This indicates that the features generated by the dual-branch feature adapter at the 1st layer are used for segmentation tasks, specifically for segmenting out abnormal regions. The deep features are compared with the data in the precise text description dataset using cosine similarity calculation to obtain the text description information and anomaly feature map corresponding to the current medical sample image, including: ； Where F∈{ , }, It is a temperature hyperparameter used for adjustment. The output distribution of the function, F T F represents the text description feature data in the precise text description dataset. n F represents the feature data of normal text description. a This represents the feature data describing abnormal text, and exp<> represents the exponential function; ； ； Where C1 represents the classification index of the anomaly type, S1 represents the segmentation index of the anomaly feature map, and BI() represents the image processing function used to reshape the anomaly map into... × It then uses bilinear interpolation to restore the resolution of the original input medical sample image to that of the original input image, where G represents the number of grids. ； ； Where m represents the minimum distance between features at each level recorded in the multi-level feature memory G, and the function Dist() represents the cosine distance. The final predicted classification and segmentation results are as follows: ； ； in, To determine the final type of image anomaly, This is the final determined anomaly feature map. This represents the weight value.

9. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the method as described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1 to 7.

Images