Interactive medical image annotation and model optimization method based on artificial feedback

Abstract

The invention discloses an interactive medical image annotation and model optimization method based on artificial feedback, and belongs to the technical field of medical image analysis. The interactive medical image annotation and model optimization method based on artificial feedback combines the advantages of artificial feedback and a deep learning model; a disease tag and a heat map positioning tag which are manually audited and corrected by medical professionals serve as training supervision signals, and model parameters are optimized through back propagation, so that the disease diagnosis and positioning capability of a model is improved, and compared with the prior art, the disease diagnosis and positioning capability of the model is improved by improving deep cognition and a complex model learning strategy. And the model can better capture and understand detailed information of a disease area, so that the classification and positioning capabilities of medical images are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to an interactive medical image annotation and model optimization method based on manual feedback. Background Art

[0002] In the traditional medical image analysis process, diagnosis and lesion localization mainly rely on the experience of medical professionals, which is not only time-consuming, but the accuracy of the results is also limited by the experience and available time of the experts. In recent years, deep learning technology has been widely used in medical image analysis, improving the accuracy and efficiency of automated diagnosis. However, although deep learning models have made progress in automatic image annotation and disease detection, there are still challenges in the precise positioning of details and the fine classification of specific diseases.

[0003] At present, the medical image automatic labeling and model updating algorithms on the market mainly include the following solutions: a method for multi-stage fine-grained disease classification based on deep neural networks, as shown in patent CN116740714A; a model self-labeling method based on knowledge distillation, as shown in patent CN116662881A.

[0004] The solution proposed by patent CN116740714A adopts a two-stage approach to optimize disease diagnosis and identification. However, this method has certain limitations. Error transmission and accumulation may occur between the joint recognition model in the first stage and the feature extraction model in the second stage. If the early feature extraction fails to effectively capture key information, this may have an adverse effect on the subsequent K-means clustering analysis, thereby weakening the accuracy of the final disease diagnosis to a certain extent. The solution proposed by patent CN116662881A adopts a knowledge distillation model self-labeling method. By utilizing the powerful visual positioning and diagnostic capabilities of the pre-trained model, a parameter reduction method is used to design a small model for automatic labeling, which improves the efficiency of labelers to a certain extent and reduces the labeling cost. Although the scheme enhances the performance of the model by gradually iterating label correction and model updating, the method is useful in improving the overall diagnostic ability of the model. However, it faces limitations in actual medical application scenarios. This method simulates the process of data updating in a natural environment, but it has limited ability to improve the detection and precise positioning of fine and complex local lesion areas in medical images. This is because it relies on a simple annotation correction mechanism and cannot fully capture and understand the detailed information of the disease area without deeper cognition and complex model learning strategies.

[0005] To this end, we propose an interactive medical image annotation and model optimization method based on manual feedback to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to solve the problem of limited classification and positioning capabilities in the prior art, and to propose an interactive medical image annotation and model optimization method based on manual feedback.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] An interactive medical image annotation and model optimization method based on manual feedback includes the following steps:

[0009] S1: Process the disease description related information into a disease description vector through a text encoder;

[0010] S2: Convert medical images into high-dimensional visual embedding vectors through visual encoders;

[0011] S3: Deeply interact the disease description vector with the visual embedding vector through a multi-head attention module to obtain visually weighted disease features and a visual attention map of disease response;

[0012] S4: The visually weighted disease features are passed through the fully connected layer (FC) to obtain the disease classification results;

[0013] S5: The visual attention map of the disease response is restored to the original image through thresholding and image scaling technology to obtain the disease heat map positioning result;

[0014] S6: Through manual review and correction, the disease classification results and disease heat map positioning results are reviewed and corrected to obtain optimized disease labels and heat map positioning labels;

[0015] S7: Use the audit-corrected disease labels and heatmap location labels as training supervision signals and back-propagate to optimize model parameters.

[0016] Preferably, in step S1, the text encoder f text Convert the text description of the disease into a vector representation with high-dimensional semantic features to obtain the disease description vector Q i , the text feature extraction formula is

[0017] Q i ={f text (Descript(S m ))|m∈[0,...,75]}

[0018] Where m represents the disease number, S m Indicates the corresponding disease name, Descript(S m ) represents the text description of the disease, Q iThe size is (B, 75, D), B represents the batch size, D represents the number of feature dimensions, and 75 represents the type of disease.

[0019] Preferably, in step S2, the visual encoder f vision Extract visual features from the input image and obtain the visual embedding vector V k , the image feature extraction formula is

[0020] V k ={f vision (I∈R h×w×d )}

[0021] where I∈R h×w×d Represents the size of the input image, which consists of height h, width w, and number of image channels d. k The size is (B, N, D), where B represents the batch size, N represents the number of visual tokens, and D represents the number of feature dimensions.

[0022] Preferably, in step S3, the disease description vector Q i As Query, the visual embedding vector V k As Key and Value, it is input into multiple multi-head attention layer modules. The formula for the interaction of multi-head attention layer modules is:

[0023] A i ,AttenionMap i =Multihead_attn(Query,Key,Value)

[0024] After multiple multi-head attention modules, the visually weighted disease representation A is obtained. i AttentionMap corresponding to the disease i , the sizes are (B,75,D) and (B,75,N) respectively.

[0025] Preferably, in step S4, the visually weighted disease representation A i After the fully connected FC layer, the disease classification result P is obtained i , the size is (B,75,2).

[0026] Preferably, in step S5, the visual attention map AttenionMap corresponding to the disease is i After image scaling and thresholding technology, the heat map positioning result H on the original image is obtained. i , with a size of (B, 75, H, W), indicating the degree of association between each type of disease and each image pixel.

[0027] Preferably, in step S6, the preliminary disease diagnosis results and heat map positioning are manually reviewed and corrected to obtain an optimized disease label L disease and heatmap positioning label L Heatmap , used to calculate the classification loss Loss classify and positioning loss Loss grounding , so as to optimize the model parameters through back propagation. Classification loss Loss classify , Positioning loss Loss grounding and the total loss Loss total The function formula is as follows:

[0028] Loss classify =CE(P i ,L disease )

[0029] Loss grounding =DiceLoss(H i ,L Heatmap )

[0030] Loss total =Loss classify +Loss grounding

[0031] To sum up, the technical effects and advantages of the present invention are as follows: compared with existing methods, the interactive medical image annotation and model optimization method based on manual feedback combines the advantages of manual feedback and deep learning models, uses the disease labels and heat map positioning labels that are manually reviewed and corrected by medical professionals as training supervision signals, and back-propagates to optimize model parameters, thereby improving the model's ability to diagnose and locate diseases. Compared with existing technologies, by improving deep-level cognition and complex model learning strategies, the model can better capture and understand detailed information about the disease area, thereby improving the classification and positioning capabilities of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0034] Reference Figure 1 , an interactive medical image annotation and model optimization method based on manual feedback, comprising the following steps:

[0035] S1: Process the disease description related information into a disease description vector through a text encoder;

[0036] S2: Convert medical images into high-dimensional visual embedding vectors through visual encoders;

[0037] S3: Deeply interact the disease description vector with the visual embedding vector through a multi-head attention module to obtain visually weighted disease features and a visual attention map of disease response;

[0038] S4: The visually weighted disease features are passed through the fully connected layer (FC) to obtain the disease classification results;

[0039] S5: The visual attention map of the disease response is restored to the original image through thresholding and image scaling technology to obtain the disease heat map positioning result;

[0040] S6: Through manual review and correction, the disease classification results and disease heat map positioning results are reviewed and corrected to obtain optimized disease labels and heat map positioning labels;

[0041] S7: Use the audit-corrected disease labels and heatmap location labels as training supervision signals and back-propagate to optimize model parameters.

[0042] This interactive medical image annotation and model optimization method based on manual feedback uses the manually reviewed and corrected disease labels and heat map location labels as training supervision signals, and back-propagates to optimize model parameters, thereby improving the model's ability to diagnose and locate diseases.

[0043] In step S1, the text encoder f text Convert the text description of the disease into a vector representation with high-dimensional semantic features to obtain the disease description vector Q i , the text feature extraction formula is

[0044] Q i ={f text (Descript(S m ))|m∈[0,...,75]}

[0045] Where m represents the disease number, S m Indicates the corresponding disease name, Descript(S m ) represents the text description of the disease, Q i The size is (B, 75, D), where B represents the batch size, D represents the number of feature dimensions, and 75 represents the type of disease. Since the number of diseases that an image may contain in actual use is generally less than 75, defining m∈[0,...,75] can satisfy most cases.

[0046] In step S2, the visual encoder f vision Extract visual features from the input image and obtain the visual embedding vector Vk , the image feature extraction formula is

[0047] V k ={f vision (I∈R h×w×d )}

[0048] where I∈R h×w×d Represents the size of the input image, which consists of height h, width w, and number of image channels d. k The size is (B, N, D), where B represents the batch size, N represents the number of visual tokens, and D represents the number of feature dimensions.

[0049] In step S3, the disease description vector Q i As Query, the visual embedding vector V k As Key and Value, it is input into multiple multi-head attention layer modules. The formula for the interaction of multi-head attention layer modules is:

[0050] A i ,AttenionMap i =Multihead_attn(Query,Key,Value)

[0051] After multiple multi-head attention modules, the visually weighted disease representation A is obtained. i AttentionMap corresponding to the disease i , the sizes are (B,75,D) and (B,75,N) respectively.

[0052] In step S4, the visually weighted disease representation A i After the fully connected FC layer, the disease classification result P is obtained i , the size is (B,75,2), 2 represents binary classification: 0 represents no disease, 1 represents disease.

[0053] In step S5, the visual attention map corresponding to the disease is i After image scaling and thresholding technology, the heat map positioning result H on the original image is obtained. i , with a size of (B,75,H,W), indicating the degree of association between each type of disease and each image pixel (H,W).

[0054] In step S6, the preliminary disease diagnosis results and heat map positioning are manually reviewed and corrected to obtain the optimized disease label L disease and heatmap positioning label L Heatmap , used to calculate the classification loss Loss classify and positioning loss Loss grounding, so as to optimize the model parameters through back propagation. Classification loss Loss classify , Positioning loss Loss grounding and the total loss Loss total The function formula is as follows:

[0055] Loss classify =CE(P i ,L disease )

[0056] Loss grounding =DiceLoss(H i ,L Heatmap )

[0057] Loss total =Loss classify +Loss grounding

[0058] Back propagation in step S7 is an algorithm based on the chain rule, which is used to efficiently calculate the gradient of the loss function to the model parameters. Since the gradient of the total loss to the parameters is the sum of the classification loss and the positioning loss, in the classification and positioning tasks, back propagation optimizes the classification loss and positioning loss respectively, thereby updating all relevant parameters of the model, gradually improving the classification accuracy and positioning accuracy, and gradually reducing the total loss, achieving a coordinated improvement in disease classification and heat map positioning.

[0059] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An interactive medical image annotation and model optimization method based on manual feedback, characterized in that: The following steps are involved: S1: Process the disease description related information into a disease description vector through a text encoder; S2: Convert medical images into high-dimensional visual embedding vectors through visual encoders; S3: Deeply interact the disease description vector with the visual embedding vector through a multi-head attention module to obtain visually weighted disease features and a visual attention map of disease response; S4: The visually weighted disease features are passed through the fully connected layer to obtain the disease classification results; S5: The visual attention map of the disease response is restored to the original image through thresholding and image scaling technology to obtain the disease heat map positioning result; S6: Through manual review and correction, the disease classification results and disease heat map positioning results are reviewed and corrected to obtain optimized disease labels and heat map positioning labels; S7: Use the audit-corrected disease labels and heatmap location labels as training supervision signals and back-propagate to optimize model parameters.

2. The interactive medical image annotation and model optimization method based on manual feedback according to claim 1, characterized in that: In step S1, the text encoder f text Convert the text description of the disease into a vector representation with high-dimensional semantic features to obtain the disease description vector Q i , the text feature extraction formula is Q i ={f text (Descript(S m ))|m∈[0,...,75]} Where m represents the disease number, S m Indicates the corresponding disease name, Descript(S m ) represents the text description of the disease, Q i The size is (B, 75, D), B represents the batch size, D represents the number of feature dimensions, and 75 represents the type of disease.

3. The interactive medical image annotation and model optimization method based on manual feedback according to claim 1, characterized in that: In step S2, the visual encoder f vision Extract visual features from the input image and obtain the visual embedding vector V k , the image feature extraction formula is V k ={f vision (I∈R h×w×d )} where I∈R h×w×d Represents the size of the input image, which consists of height h, width w, and number of image channels d. k The size is (B, N, D), where B represents the batch size, N represents the number of visual tokens, and D represents the number of feature dimensions.

4. The interactive medical image annotation and model optimization method based on manual feedback according to claim 1, characterized in that: In step S3, the disease description vector Q i As Query, the visual embedding vector V k As Key and Value, it is input into multiple multi-head attention layer modules. The formula for the interaction of multi-head attention layer modules is: A i ,AttenionMap i =Multihead_attn(Query,Key,Value) After multiple multi-head attention modules, the visually weighted disease representation A is obtained. i AttentionMap corresponding to the disease i , the sizes are (B,75,D) and (B,75,N) respectively.

5. The interactive medical image annotation and model optimization method based on manual feedback according to claim 1, characterized in that: In step S4, the visually weighted disease representation A i After the fully connected FC layer, the disease classification result P is obtained i , the size is (B,75,2).

6. The interactive medical image annotation and model optimization method based on manual feedback according to claim 1, characterized in that: In step S5, the visual attention map AttenionMap corresponding to the disease is i After image scaling and thresholding technology, the heat map positioning result H on the original image is obtained. i , with a size of (B, 75, H, W), indicating the degree of association between each type of disease and each image pixel.

7. The interactive medical image annotation and model optimization method based on manual feedback according to claim 1, characterized in that: In step S6, the preliminary disease diagnosis results and heat map positioning are manually reviewed and corrected to obtain an optimized disease label L disease and heatmap positioning label L Heatmap , used to calculate the classification loss Loss classify and positioning loss Loss grounding , so as to optimize the model parameters through back propagation. Classification loss Loss classify , Positioning loss Loss grounding and the total loss Loss total The function formula is as follows: Loss classify =CE(P i ,IT disease ) Loss grounding =DiceLoss(H i ,L Heatmap ) Loss total =Loss classify +Loss grounding。

Images