Prediction method, system, equipment and program product for benign and malignant breast nodules

Abstract

The invention provides a method, a system, equipment and a program product for predicting benign and malignant breast nodules, and the method comprises the steps: collecting an ultrasonic image of a to-be-detected breast nodule; based on the ultrasonic image, a gaze graph generator of the attention guiding module generates a virtual gaze graph corresponding to the ultrasonic image; based on the ultrasonic image and the virtual staring image, a staring image classifier of the attention guiding module outputs the malignant probability of the breast nodule to be detected corresponding to the virtual staring image; wherein the staring image generator and the staring image classifier are obtained based on training of the training ultrasonic image and the real staring image; the real staring map comprises attention information of the expert during ultrasonic diagnosis; and based on the ultrasonic image, the uncertainty calculation module generates an evidence vector corresponding to the ultrasonic image, and generates credibility corresponding to the malignancy probability based on the evidence vector. The interpretability of benign and malignant prediction of breast nodules can be improved, and automation bias is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to a method, system, device, and program product for predicting benign or malignant breast nodules. Background Art

[0002] Breast cancer is the most common cancer and the second leading cause of cancer death among women worldwide. The incidence of breast cancer is increasing annually, with a cumulative global case count of 1.6 million in 2010 and projected to reach 2.1 million by 2030. Early detection of breast cancer can facilitate early clinical intervention, improve patient prognosis, and reduce cancer mortality. Ultrasound examination, with its advantages of real-time availability, low cost, wide availability, and lack of ionizing radiation, has become an important tool for breast cancer screening. The Breast Imaging Reporting and Data System (BI-RADS), published by the American College of Radiology, is widely used to standardize image interpretation and reporting by radiologists. However, subjective interpretation of medical images and intra- and inter-observer variability still limit BI-RADS grading.

[0003] In recent years, artificial intelligence (AI) has been widely used in the medical field. Deep neural networks (DNNs), a key branch of AI technology, can automatically extract high-dimensional features from medical images and directly assist clinical diagnosis and treatment in an end-to-end manner. In the early diagnosis of breast cancer, many studies have proposed AI models that can assist clinicians in accurately calculating the probability of malignancy of breast nodules, with sensitivity and specificity reaching the level of advanced experts. At the same time, commercially available AI technologies such as S-Detect are gradually being applied in clinical practice. They can objectively analyze breast lumps based on shape, orientation, edges, posterior features, and echogenicity. Doctors can refer to and combine the malignancy probability derived from AI to accurately and objectively classify breast lumps. However, even with its excellent expert-level performance, AI still faces two key challenges when assisting clinical decision-making: unexplainability and automation bias.

[0004] Unexplainability refers to the difficulty in understanding AI models' internal logic and reasoning, due to their massive number of parameters and complex architecture. This problem is particularly critical in high-risk healthcare settings, where the model's unexplainability directly impacts physicians' trust in it. A study on pneumonia detection revealed that an AI model had high accuracy on chest X-rays taken from internal hospitals, but significantly decreased performance in external hospitals. The model had inadvertently learned to detect hospital-specific metal markers in the corners of the scan field and indirectly used them for disease diagnosis, resulting in unreliable output. Therefore, improving the interpretability of AI models is crucial for their clinical application.

[0005] Currently, research on eye tracking is gaining momentum. In-depth exploration of eye movement data can help understand the important perceptual and cognitive processes of doctors in medical decision-making. Inputting eye movement data containing rich information about doctors' attention as an additional data source into DNN may become a key step in the future development of explainable and reliable AI models. Many studies have shown that injecting doctors' eye gaze information into DNN can improve model interpretability while further improving model performance. Jiang et al. input ophthalmologists' eye movement data into DNN to develop an early diabetic retinopathy detection model, significantly improving the model's classification accuracy and interpretability. Mall et al. injected radiologists' eye movement data on breast cancer into DNN. The AI ​​model they developed can more efficiently detect missed lesions in breast X-rays while improving interpretability. However, research on eye movements in breast cancer ultrasound images is still blank.

[0006] Automation bias, which refers to bias or error caused by overreliance on AI recommendations, neglecting other information sources, and losing self-judgment, is another major challenge facing medical AI. When using AI tools, physicians often make independent diagnoses while also considering AI recommendations and potentially revising their initial judgments accordingly. Ideally, physicians should be able to integrate AI recommendations critically, benefiting from correct AI recommendations while identifying and ignoring AI errors, thereby making better clinical decisions. However, some studies suggest that when interacting with AI, physicians may cease to critically consider AI recommendations and instead begin to blindly follow them. A controlled study of 320,000 mammograms showed that the average sensitivity and specificity for breast cancer detection among 271 radiologists without AI assistance was 87.3% and 91.4%, respectively, while with AI assistance, the sensitivity and specificity were 85.3% and 91.6%, respectively. These results suggest that AI-assisted detection does not significantly improve physicians' diagnostic performance, but rather reduces sensitivity. Another prospective controlled study showed that AI-assisted mammography grading performance among 27 physicians was significantly affected by AI-assisted results. When faced with correct AI predictions, the diagnostic accuracy rates for junior, middle, and senior doctors were 79.7%, 81.3%, and 82.3%, respectively. However, when faced with incorrect AI predictions, the diagnostic accuracy rates for junior, middle, and senior doctors were only 19.8%, 24.8%, and 45.5%, respectively, indicating a significant degree of automation bias. This suggests that automation bias can pose significant risks to the clinical application of AI, and optimizing the interaction between AI and doctors is a pressing issue.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0008] In view of this, the present disclosure provides a method, system, device and program product for predicting benign and malignant breast nodules, so as to at least solve the problems of unexplainability and automation bias in the existing artificial intelligence models in the process of predicting benign and malignant breast nodules.

[0009] In one aspect, embodiments of the present disclosure provide a method for predicting benign or malignant breast nodules, comprising:

[0010] Acquire ultrasound images of breast nodules to be detected;

[0011] Based on the ultrasound image, a gaze map generator of the attention guidance module generates a virtual gaze map corresponding to the ultrasound image; wherein the gaze map generator includes a first neural network and a second neural network, and is trained based on multiple sets of corresponding training ultrasound images and real gaze maps; the real gaze map includes attention information of the expert when performing ultrasound diagnosis;

[0012] Based on the ultrasound image and the virtual gaze map, the gaze map classifier of the attention guidance module outputs the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map; wherein the gaze map classifier includes a third neural network and is trained based on the training ultrasound image and the real gaze map;

[0013] Based on the ultrasound image, the uncertainty calculation module generates an evidence vector corresponding to the ultrasound image, and generates a credibility corresponding to the malignancy probability based on the evidence vector; wherein the uncertainty calculation module includes a fourth neural network, and is trained based on the training ultrasound image.

[0014] In some embodiments, the gaze map generator includes: an encoder and a decoder;

[0015] The encoder includes a graph construction layer based on feature distance and a first neural network for extracting global features of the ultrasound image; wherein the first neural network is a graph neural network;

[0016] The decoder includes a second neural network for extracting local features of the ultrasound image; the second neural network is a convolutional neural network.

[0017] In some embodiments, based on the ultrasound image, the gaze map generator of the attention guidance module generates a virtual gaze map corresponding to the ultrasound image, including:

[0018] In the encoder, the features of each position of the ultrasound image are converted into a node to form a node set;

[0019] For each node in the node set, the K nearest neighbor algorithm is used to search for the node's neighboring nodes based on the feature distance, and the node and the neighboring nodes are connected with edges to construct the first graph structure corresponding to the node set;

[0020] After using graph neural networks to perform feature fusion and feature transformation of interconnected nodes, a virtual gaze map is output through the decoder and generation head.

[0021] In some embodiments, based on the ultrasound image and the virtual gaze map, the gaze map classifier of the attention guidance module outputs the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map, including:

[0022] The gaze graph classifier constructs a second graph structure by fusing the feature distance of the ultrasound image and the gaze distance of the virtual gaze graph, and eliminates node connections that are irrelevant to the classification task;

[0023] The gaze map classifier uses a third neural network to make the nodes in the virtual gaze map area more closely connected, thereby strengthening the information aggregation in the virtual gaze map area; wherein the third neural network is a graph neural network;

[0024] The gaze map classifier outputs the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map.

[0025] In some embodiments, the gaze map generator uses mean squared error loss as the first loss; the gaze map classifier uses cross entropy loss as the second loss; and the attention guidance module uses the sum of the first loss and the second loss as the total loss.

[0026] In some embodiments, based on the ultrasound image, the uncertainty calculation module generates an evidence vector corresponding to the ultrasound image, and generates a credibility corresponding to the malignancy probability based on the evidence vector, including:

[0027] Based on the input ultrasound image, the hybrid encoder and decoder architecture outputs the unnormalized classification score, and the activation function is used to make the unnormalized classification score non-negative, and obtain the evidence vector;

[0028] Convert evidence vectors into uncertainties;

[0029] The credibility corresponding to the malignancy probability is obtained based on uncertainty.

[0030] In some embodiments, the training step of the uncertainty calculation module includes:

[0031] Based on the input ultrasound image, the hybrid encoder and decoder architecture outputs the unnormalized classification score, and the activation function is used to make the unnormalized classification score non-negative, and obtain the evidence vector;

[0032] Convert evidence vectors into strength of evidence and uncertainty;

[0033] Parameterize the strength of evidence and uncertainty as Dirichlet distributions;

[0034] Calculate the expected value of the cross entropy loss through Dirichlet distribution;

[0035] Calculate the KL divergence loss value through Dirichlet distribution;

[0036] Calculate the expected value of the cross entropy loss and the KL divergence loss value as the total loss value of the uncertainty calculation module.

[0037] On the other hand, an embodiment of the present disclosure further provides a system for predicting benign or malignant breast nodules, comprising:

[0038] A data acquisition module, used for acquiring ultrasound images of breast nodules to be detected;

[0039] An attention guidance module, the attention guidance module includes a gaze map generator and a gaze map classifier; the gaze map generator is used to generate a virtual gaze map corresponding to the ultrasound image based on the ultrasound image; wherein the gaze map generator includes a first neural network and a second neural network, and is trained based on multiple sets of training ultrasound images and real gaze maps with corresponding relationships; the real gaze map includes the attention information of the expert when performing ultrasound diagnosis; the gaze map classifier is used to output the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map based on the ultrasound image and the virtual gaze map; wherein the gaze map classifier includes a third neural network, and is trained based on the real gaze map;

[0040] The uncertainty calculation module is used to generate an evidence vector corresponding to the ultrasound image based on the ultrasound image, and generate a credibility corresponding to the malignancy probability based on the evidence vector; wherein the uncertainty calculation module includes a fourth neural network and is trained based on the training ultrasound image.

[0041] In another aspect, an embodiment of the present disclosure further provides a device for predicting benign or malignant breast nodules, comprising:

[0042] processor;

[0043] a memory storing computer-readable instructions;

[0044] The processor is configured to execute the above-mentioned method for predicting benign or malignant breast nodules by executing computer-readable instructions.

[0045] On the other hand, an embodiment of the present disclosure further provides a computer program product, comprising computer-readable instructions, which, when executed by a processor, implement the above-mentioned method for predicting benign or malignant breast nodules.

[0046] Compared with the prior art, the present disclosure has at least the following technical effects:

[0047] The method, system, device and program product for predicting benign and malignant breast nodules disclosed in the present invention collect the attention information of experts when performing ultrasound diagnosis, and use the attention information for training a gaze map generator and a gaze map classifier, thereby guiding the gaze map generator and the gaze map classifier to focus on the classification-related target areas of breast nodules, thereby improving the interpretability of the method for predicting benign and malignant breast nodules; by integrating the uncertainty assessment algorithm into the method for predicting benign and malignant breast nodules, the credibility of the malignancy probability is calculated while outputting the malignancy probability of the breast nodule to be detected, thereby reducing the problem of automation bias; by generating a virtual gaze map through the gaze map generator, there is no need to input an additional real gaze map, that is, real attention information, when applying it, which can save time and cost and avoid the problem of reduced prediction accuracy caused by differences in different real attention information. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0049] Figure 1 This is a flowchart of the steps of a method for predicting benign or malignant breast nodules provided by an embodiment of the present disclosure;

[0050] Figure 2 yes Figure 1 Specific step flow chart of step S120;

[0051] Figure 3 yes Figure 1 Specific steps of step S130 are shown in the flowchart;

[0052] Figure 4 yes Figure 1 Specific steps of step S140 in the flowchart;

[0053] Figure 5 is a flow chart of the training steps of an uncertainty calculation module provided by an embodiment of the present disclosure;

[0054] Figure 6 This is a comparison chart of the attention of the AI ​​model disclosed in this disclosure and the traditional AI model;

[0055] Figure 7 Receiver operating characteristic curves of the disclosed AI model and the traditional AI model in the internal validation set (A) and the external validation set (B);

[0056] Figure 8is the decision curve of the AI ​​model of the present disclosure and the traditional AI model in the internal validation set (A) and the external validation set (B);

[0057] Figure 9 This is a module structure diagram of a system for predicting benign and malignant breast nodules provided by an embodiment of the present disclosure;

[0058] Figure 10 This is a schematic diagram of the structure of a device for predicting benign or malignant breast nodules provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Identical reference numerals in the figures represent identical or similar structures, and thus a repeated description thereof will be omitted.

[0060] The terms "first," "second," and similar terms used in the specific description do not denote any order, quantity, or importance, but are simply used to distinguish different components. Furthermore, in the description of this disclosure, the terms "upper," "lower," and the like indicate positions or locations based on those shown in the accompanying drawings. These terms are intended solely for ease of description and do not indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation. Therefore, they should not be construed as limitations on this disclosure.

[0061] It should be noted that, in the absence of conflict, the embodiments of the present disclosure and features in different embodiments may be combined with each other.

[0062] One aspect, such as Figure 1 As shown, the embodiment of the present disclosure provides a method for predicting benign or malignant breast nodules, comprising:

[0063] S110, acquiring an ultrasound image of the breast nodule to be detected;

[0064] S120. Based on the ultrasound image, a gaze map generator generates a virtual gaze map corresponding to the ultrasound image; wherein the gaze map generator includes a first neural network and a second neural network, and is trained based on multiple sets of training ultrasound images and real gaze maps having corresponding relationships; the real gaze map includes attention information of the expert when performing ultrasound diagnosis;

[0065] S130. Based on the ultrasound image and the virtual gaze map, a gaze map classifier outputs a malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map; wherein the gaze map classifier includes a third neural network and is trained based on the training ultrasound image and the real gaze map;

[0066] S140. Based on the ultrasound image, the uncertainty calculation module generates an evidence vector corresponding to the ultrasound image, and generates a credibility corresponding to the malignancy probability based on the evidence vector; wherein the uncertainty calculation module includes a fourth neural network, and is trained based on the training ultrasound image.

[0067] It is worth noting that the above-mentioned S110 to S140 are merely step numbers, which are used for ease of reference and to avoid duplication. Unless otherwise specified, the above-mentioned and subsequent step numbers do not limit the order in which the steps of this method are to be performed. In other embodiments, the above-mentioned steps of this method may be written and performed in an interchangeable order, and the order is not limited thereto.

[0068] Specifically, the gaze map generator (GMG) needs to use multiple sets of training ultrasound images and real gaze maps with corresponding relationships during training, so that the gaze map generator can learn the ability to generate gaze maps through ultrasound images. Then, during the testing and application process, the ultrasound image of the breast nodule to be detected can be input to output a virtual gaze map corresponding to the ultrasound image of the breast nodule to be detected. The gaze map is a heat map of the same size as the observed ultrasound image, and the different attention levels of the expert are represented by color differences. The real gaze map can be obtained by using an eye tracker device (such as a Tobii eye tracker device, etc.) to capture the attention information (i.e., eye movement data) of the expert during ultrasound diagnosis and encoding it into a gaze map. The virtual gaze map is the gaze map predicted and generated by the gaze map generator.

[0069] Specifically, during training, a gaze-directed classifier (GDC) needs to use multiple sets of corresponding training ultrasound images and real gaze maps, so that the gaze map classifier can learn the ability to calculate the malignancy probability of the breast nodule to be detected corresponding to the ultrasound image and the gaze map. Then, during testing and application, the malignancy probability of the breast nodule to be detected can be output by inputting the ultrasound image of the breast nodule to be detected and the virtual gaze map of the ultrasound image generated by the gaze map generator.

[0070] Specifically, the uncertainty calculation module uses an uncertainty estimation (UE) algorithm to calculate the confidence level corresponding to the malignancy probability. The UE algorithm evaluates the uncertainty of the predicted malignancy probability output by the AI ​​model. Since the sum of uncertainty and confidence is 1, the confidence level corresponding to the malignancy probability can be calculated, providing doctors with reliable predictions and confidence levels, and improving the interpretability of the prediction method.

[0071] In this embodiment, by collecting the attention information of experts when performing ultrasound diagnosis and using the attention information for training the gaze map generator and the gaze map classifier, the gaze map generator and the gaze map classifier are guided to focus on the classification-related target areas of breast nodules, thereby improving the interpretability of the method for predicting benign and malignant breast nodules; by integrating the uncertainty assessment algorithm into the method for predicting benign and malignant breast nodules, the credibility of the malignancy probability is calculated while outputting the malignancy probability of the breast nodule to be detected, thereby reducing the problem of automation bias; by generating a virtual gaze map through the gaze map generator, there is no need to input an additional real gaze map, that is, real attention information, when applying it, which can save time cost and avoid the problem of reduced prediction accuracy caused by differences in different real attention information.

[0072] In some embodiments, the gaze map generator includes an encoder and a decoder. The encoder includes a graph construction layer based on feature distance and a first neural network, which is used to extract global features of the ultrasound image; wherein the first neural network is a graph neural network (GNN). The decoder includes a second neural network, which is used to extract local features of the ultrasound image; the second neural network is a convolutional neural network (CNN). Specifically, GNN is a type of deep learning model specifically designed for processing graph-structured data. Its core concept is to achieve feature propagation and aggregation between nodes by modeling the nodes and their adjacency relationships in the graph, thereby extracting high-dimensional features with structural information. GNN is particularly suitable for complex data modeling scenarios with spatial or semantic associations and can learn contextual information in irregular structures. CNN is a typical feedforward neural network widely used in the field of image processing. It automatically extracts local spatial features in images through convolution operations, has advantages such as parameter sharing and local connections, and can efficiently capture low-level and high-level features such as texture and edges in images.

[0073] In some embodiments, as Figure 2 As shown, step S120 includes:

[0074] S121. In the encoder, convert the feature of each position of the ultrasound image into a node to form a node set;

[0075] S122. For each node in the node set, use the K-nearest neighbor algorithm to search for neighboring nodes of the node based on feature distance, and connect the node and the neighboring nodes with edges to construct a first graph structure corresponding to the node set;

[0076] S123. After using a graph neural network to perform feature fusion and feature transformation of interconnected nodes, a virtual gaze map is output through a decoder and a generation head.

[0077] This embodiment uses GNN to fuse and propagate features between nodes to achieve "context-aware" information flow within the first graph structure, enabling the model to learn the doctor's focusing behavior on key areas during diagnosis from the overall image, thereby improving the consistency of the generated "virtual gaze map" with the real expert gaze map; the virtual gaze map output by the decoder is a heat map that simulates the expert's attention area, which not only retains the information of the key areas of the image, but can also be used by the subsequent gaze map classifier to guide the model to focus on the simulated expert's attention area, thereby enhancing the interpretability and accuracy of the prediction.

[0078] In some embodiments, as Figure 3 As shown, S130, including:

[0079] S131, the gaze graph classifier constructs a second graph structure by fusing the feature distance of the ultrasound image and the gaze distance of the virtual gaze graph, and eliminates node connections that are not related to the classification task;

[0080] S132. The gaze graph classifier uses a third neural network to make the nodes in the virtual gaze graph area more closely connected, so as to strengthen the information aggregation in the virtual gaze graph area; wherein the third neural network is a graph neural network;

[0081] S133. The gaze graph classifier outputs the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze graph.

[0082] This embodiment constructs a second graph structure that is closer to the doctor's actual cognitive process by fusing image feature distance (representing the similarity of ultrasound image content) and gaze distance (representing the area of ​​attention that the expert focuses on). The second graph structure can automatically focus on the diagnostic attention area while eliminating connections between irrelevant areas, effectively reducing the interference of background, artifacts, and non-lesion areas on the classification results, and improving interpretability; the graph neural network strengthens the information connection between nodes in the virtual gaze graph area, so that this part of the nodes forms a subgraph structure with dense feature interactions. This enhanced subgraph helps to focus on the flow of diagnosis-related information and avoid the model being affected by non-lesion features such as texture and boundaries, thereby improving the accuracy of the prediction results.

[0083] In some embodiments, the gaze map generator uses mean squared error loss as the first loss; the gaze map classifier uses cross entropy loss as the second loss; and the attention guidance module uses the sum of the first loss and the second loss as the total loss.

[0084] Specifically, the formula of the first loss of the gaze map generator is as follows:

[0085]

[0086] in, It is a true gaze picture, is the virtual gaze map, H is the number of column pixels of the real gaze map, W is the number of row pixels of the real gaze map, i and j are index variables, L GMG It can be understood that the number of row pixels and column pixels of the ultrasound image, the real gaze map, and the virtual gaze map are all the same.

[0087] The formula for the second loss of the gaze map classifier is as follows:

[0088]

[0089] in, is the true label value of the gaze classifier, y is the predicted value of the gaze classifier, c represents the number of classification categories of the gaze classifier, i is the index variable, L GDC It is the first loss.

[0090] Finally, the loss function of the entire attention guidance module is formulated as follows:

[0091] L total =L GMG +L GDC

[0092] Among them, L total is the total loss of the entire attention guidance module.

[0093] In some embodiments, as Figure 4 As shown, S140, including:

[0094] S141. Based on the input ultrasound image, output an unnormalized classification score through a hybrid encoder and decoder architecture, and make the unnormalized classification score non-negative through an activation function, and obtain an evidence vector;

[0095] S142, convert evidence vector into uncertainty;

[0096] S143. Obtain the credibility corresponding to the malignancy probability based on the uncertainty.

[0097] This embodiment integrates Dempster-Shafer theory (DST) and subjective logic (SL) to establish an uncertainty estimation method based on evidence theory. This method can effectively quantify the strength of evidence and uncertainty (i.e., credibility) of the prediction, helping the model to handle difficult-to-judge cases in image classification.

[0098] In some embodiments, as Figure 5 As shown in Figure 2, the training steps of the uncertainty calculation module include:

[0099] S151. Based on the input ultrasound image, output an unnormalized classification score through a hybrid encoder and decoder architecture, and make the unnormalized classification score non-negative through an activation function, and obtain an evidence vector;

[0100] S152, convert evidence vector into evidence strength and uncertainty;

[0101] S153, parameterize the strength of evidence and uncertainty as a Dirichlet distribution;

[0102] S154. Calculate the expected value of the cross entropy loss through Dirichlet distribution;

[0103] S155. Calculate the KL divergence loss value through Dirichlet distribution;

[0104] S156. Calculate the expected value of the cross entropy loss and the KL divergence loss value as the total loss value of the uncertainty calculation module.

[0105] For the above step S151, specifically, the model input x i For the real gaze map, we use a hybrid encoder-decoder architecture f θ , generate evidence vector e i :

[0106] e i =B(f θ (x i ))

[0107] Among them, f θ (x i ) is the output logits (unnormalized classification score) of the model, B(·): ReLU is the activation function, which ensures that logits are non-negative, thereby converting them into evidence vectors e i .

[0108] Regarding the above step S152, specifically, the evidence vector e is transformed into i Converted to strength of evidenceb n and uncertainty u:

[0109]

[0110] Among them, e n is the evidence vector for category n, is the strength of evidence (normalized Dirichlet parameter), and N is the total number of categories. For category n: b n Indicates the strength of evidence that the pixel belongs to category n, and u indicates uncertainty. n=0), u = 1; and when the evidence tends to infinity (e n →∞), u=0.

[0111] For the above step S153, specifically, the strength of evidence b n and uncertainty u parameterized as a Dirichlet distribution:

[0112] α n =e n +1

[0113] Where α=[α 1 ,α 2 ,…,α N ] is the parameter of the Dirichlet distribution.

[0114] The probability density function of the Dirichlet distribution is:

[0115]

[0116] Where p is the probability vector, N is the total number of categories, for category n: p n is the probability vector of category n, B(α j,k ) is the Beta function of the multinomial distribution, which is used to ensure that the integral of the probability density function is 1.

[0117] For the above step S154, specifically, the expected value L of the cross entropy loss is calculated by Dirichlet distribution CE , which is used to evaluate the difference between the probability distribution p output by the model and the true label y:

[0118]

[0119] Among them, y represents the true label of category n, α n represents the parameters of the Dirichlet distribution, S N Represents the total evidence strength S of the Dirichlet distribution N , the calculation formula is:

[0120]

[0121] Among them, ψ(·) is the Digamma function, which is the derivative formula of the gamma function. By using the Digamma function, the probability of model prediction can be accurately optimized, so that the model can better calibrate uncertainty.

[0122] Regarding the above step S155, specifically, to ensure that the model can express high uncertainty when there is insufficient evidence, the KL divergence loss is introduced:

[0123] L KL=KL[D(p j,k |α j,k )||D(p j,k |1)]

[0124] By expanding the calculation:

[0125]

[0126] Where D(p|1) represents a uniform Dirichlet distribution (indicating complete uncertainty), Γ(·) represents the gamma function, ψ(·) represents the digamma function, and N represents the total number of classes. The KL divergence measures the difference between the current predicted distribution D(p|α) and the uniform distribution. This encourages the model to output high uncertainty when there is insufficient evidence, thus reducing the automation bias of this example.

[0127] For the above step S156, specifically, based on the above calculations, the total loss function L of the evidence calibration model is defined as Ecml for:

[0128] L Ecml =L CE +λL KL

[0129] Among them, L CE represents the cross entropy loss, ensuring that the predicted distribution is close to the true label, L KL represents the KL divergence loss to prevent evidence collapse, and λ represents the weight factor used to balance the impact of different losses.

[0130] like Figure 6 As shown, Figure 6 This is a comparison chart of the attention of the AI ​​model disclosed in this invention and the traditional AI model. Among them, the IRBU (Interpretable and Reliable classification of lesions on Breast Ultrasound, interpretable and reliable breast ultrasound nodule classification) model is the AI ​​model disclosed in this invention, and ResNet 34, ResNet 50, DenseNet and VIT are traditional AI models for breast ultrasound nodule classification. It can be seen that the IRBU model disclosed in this invention collects the attention information of experts during ultrasound diagnosis, and uses the attention information for the training of the gaze graph generator and the gaze graph classifier, thereby guiding the gaze graph generator and the gaze graph classifier to focus on the classification-related target areas of breast nodules, thereby improving the interpretability of the prediction method for benign and malignant breast nodules.

[0131] like Figure 7 As shown, Figure 7: The receiver operating characteristic curves of the AI ​​model of the present disclosure and the traditional AI model in the internal validation set (A) and the external validation set (B). The present disclosure uses the Youden index to calculate the optimal cutoff value of credibility, and divides the prediction results into a high credibility group and a low credibility group according to the optimal cutoff value. In the internal validation set consisting of 1136 lesions, the area under the receiver operating curve (AUC) of the high credibility group was 0.962, which was significantly higher than that of the low credibility group (AUC = 0.586, P < 0.05); similarly, in the multicenter external validation set consisting of breast cancer patients from four hospitals (a total of 1286 lesions), the AUC of the high credibility group was also significantly higher than that of the low credibility group (0.943 vs. 0.676, P < 0.05).

[0132] like Figure 8 As shown, Figure 8 Figure 2 shows the decision curves for the disclosed AI model and traditional AI models on the internal validation set (A) and external validation set (B). Decision curve analysis shows that across a wide range of threshold probabilities, the high-confidence group achieves superior net network benefits compared to both the low-confidence group and the traditional AI model.

[0133] As shown in Table 1, the accuracy, specificity, sensitivity, positive predictive value, and negative predictive value of the IRBU-high confidence group were generally superior to those of the BI-RADS classification and various traditional AI models in the internal validation set and external validation cohort.

[0134] Table 1:

[0135] Model Accuracy Specificity Sensitivity Positive predictive value Negative predictive value Internal validation set BIRADS 0.843 0.687 0.966 0.796 0.942 ResNet34 0.895 0.859 0.924 0.893 0.899 ResNet50 0.904 0.865 0.934 0.898 0.912 DenseNet 0.886 0.822 0.937 0.869 0.911 VIT 0.877 0.887 0.870 0.907 0.843 IRBU-low confidence group 0.682 0.578 0.763 0.701 0.652 IRBU-High Trust Group 0.962 0.927 0.990 0.945 0.986 External validation set BIRADS 0.850 0.847 0.852 0.924 0.724 ResNet34 0.812 0.890 0.789 0.960 0.559 ResNet50 0.810 0.938 0.777 0.980 0.517 DenseNet 0.807 0.897 0.782 0.964 0.538 VIT 0.850 0.826 0.863 0.908 0.752 IRBU-low confidence group 0.603 0.737 0.457 0.615 0.596 IRBU-High Trust Group 0.950 0.936 0.956 0.973 0.898

[0136] Among them, BIRADS, ResNet34, ResNet50, DenseNet and VIT are traditional AI models for breast ultrasound nodule classification, IRBU-low confidence group is the low confidence group of the AI ​​model disclosed in the present invention, IRBU-high confidence group is the high confidence group of the AI ​​model disclosed in the present invention, accuracy refers to the proportion of correct predictions among all predictions of the model, specificity refers to the proportion of patients who are actually not sick (benign) and are correctly identified as benign by the model, sensitivity refers to the proportion of patients who are actually sick (malignant) and are correctly identified by the model, positive predictive value refers to the proportion of all patients predicted by the model to be malignant who are actually malignant, and negative predictive value refers to the proportion of all patients predicted by the model to be benign who are actually benign.

[0137] As shown in Table 2, in order to verify the effect of IRBU in reducing automation bias, the present disclosure randomly selected 20 ultrasound images that were accurately judged by the AI ​​model of the present disclosure and 20 ultrasound images that were incorrectly judged, and each image contained only one lesion. 49 ultrasound doctors were prospectively recruited to perform benign and malignant assessments on the 40 lesions. First, the doctors were informed that they needed to evaluate the effectiveness of the AI ​​model product of the present disclosure. Each doctor was first shown an ultrasound image with AI prediction probability, and was then classified into benign and malignant types. The same ultrasound image with prediction probability and credibility was then shown and classified again. The benign and malignant classification results of each doctor were recorded and sorted, and the average accuracy of each doctor in the ordinary AI group and the AI ​​group with credibility output (referred to as: credible AI group) was calculated and compared in both correct and incorrect AI judgments.

[0138] Table 2:

[0139]

[0140] Among them, the P value is the analysis result of the difference in accuracy of doctors of different seniority in the ordinary AI and trusted AI groups when AI prediction errors occur.

[0141] It can be seen that when faced with incorrect AI suggestions, the diagnostic skills of doctors of all seniority dropped significantly, and the problem of automation bias is serious. However, when faced with the same incorrect AI suggestions, the accuracy of the trusted AI group was significantly improved compared to the ordinary AI: the accuracy of junior doctors increased from 48.3% to 68.0%; the accuracy of middle-aged doctors increased from 62.5% to 74.4%; and the accuracy of senior doctors increased from 71.4% to 77.5%, with P values ​​all less than 0.05. Therefore, the AI ​​model disclosed in this paper can provide a powerful aid for flexible interaction between doctors and AI, reducing automation bias.

[0142] On the other hand, Figure 9 As shown, an embodiment of the present disclosure further provides a system 50 for predicting benign and malignant breast nodules, comprising:

[0143] A data acquisition module 51 is used to acquire ultrasound images of breast nodules to be detected;

[0144] Attention guidance module 52, attention guidance module 52 includes a gaze map generator 521 and a gaze map classifier 522; the gaze map generator is used to generate a virtual gaze map corresponding to the ultrasound image based on the ultrasound image; wherein the gaze map generator includes a first neural network and a second neural network, and is trained based on multiple sets of training ultrasound images and real gaze maps with corresponding relationships; the real gaze map includes the attention information of the expert when performing ultrasound diagnosis; the gaze map classifier is used to output the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map based on the ultrasound image and the virtual gaze map; wherein the gaze map classifier includes a third neural network, and is trained based on the real gaze map;

[0145] The uncertainty calculation module 53 is used to generate an evidence vector corresponding to the ultrasound image based on the ultrasound image, and generate a credibility corresponding to the malignancy probability based on the evidence vector; wherein the uncertainty calculation module includes a fourth neural network and is trained based on the training ultrasound image.

[0146] The specific technical solutions and technical effects of the system for predicting benign and malignant breast nodules disclosed in the present invention can be referred to the aforementioned embodiment of the method for predicting benign and malignant breast nodules, and will not be repeated here.

[0147] On the other hand, Figure 10 As shown, an embodiment of the present disclosure further provides a device for predicting benign or malignant breast nodules, comprising: a processor; and a memory storing computer-readable instructions, wherein the processor is configured to execute a method for predicting benign or malignant breast nodules by executing the computer-readable instructions.

[0148] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "platforms."

[0149] Refer to the following Figure 9 hereinafter, an electronic device 600 according to this embodiment of the present disclosure is described. Figure 9 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0150] like Figure 9 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including storage unit 620 and processing unit 610), and a display unit 640.

[0151] The storage unit 620 stores computer-readable instructions, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps described in the above method section of this specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 610 can perform the following steps: Figure 1 Follow the steps shown in .

[0152] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0153] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0154] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0155] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0156] The specific technical solutions and technical effects of the device for predicting benign and malignant breast nodules disclosed in the present invention can be referred to the aforementioned embodiment of the method for predicting benign and malignant breast nodules, and will not be repeated here.

[0157] In yet another aspect, embodiments of the present disclosure further provide a computer program product comprising computer-readable instructions stored in a computer-readable storage medium. A processor of a computing device can read the computer-readable instructions from the computer-readable storage medium and execute the computer-readable instructions, causing the computing device to perform the methods for predicting benign or malignant breast nodules described in the various embodiments above.

[0158] The computer-readable storage medium includes computer-readable instructions that can be written in any combination of one or more programming languages. Programming languages ​​include object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0159] The specific technical solutions and technical effects of the computer program product disclosed herein can be referred to the aforementioned embodiment of the method for predicting benign or malignant breast nodules, and will not be described in detail here.

[0160] In summary, the method, system, device and program product for predicting benign and malignant breast nodules disclosed in the present invention collects the attention information of experts when performing ultrasound diagnosis, and uses the attention information for training the gaze map generator and the gaze map classifier, thereby guiding the gaze map generator and the gaze map classifier to focus on the classification-related target areas of breast nodules, thereby improving the interpretability of the method for predicting benign and malignant breast nodules; by integrating the uncertainty assessment algorithm into the method for predicting benign and malignant breast nodules, the credibility of the malignancy probability is calculated while outputting the malignancy probability of the breast nodule to be detected, thereby reducing the problem of automation bias; by generating a virtual gaze map through the gaze map generator, there is no need to input an additional real gaze map, that is, real attention information, when applying it, which can save time and cost and avoid the problem of reduced prediction accuracy caused by differences in different real attention information.

[0161] The above content is a further detailed description of the present disclosure in conjunction with specific optional implementation methods, and the specific implementation of the present disclosure should not be considered to be limited to these descriptions. For those skilled in the art of the present disclosure, without departing from the concept of the present disclosure, they can also make several simple deductions or substitutions, which should be considered to fall within the scope of protection of the present disclosure.

Claims

1. A method for predicting benign or malignant breast nodules, characterized in that: include: Acquire ultrasound images of breast nodules to be detected; Based on the ultrasound image, a gaze map generator of the attention guidance module generates a virtual gaze map corresponding to the ultrasound image; wherein the gaze map generator includes a first neural network and a second neural network, and is trained based on multiple sets of training ultrasound images and real gaze maps having corresponding relationships; the real gaze map includes attention information of the expert when performing ultrasound diagnosis; Based on the ultrasound image and the virtual gaze map, the gaze map classifier of the attention guidance module outputs the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map; wherein the gaze map classifier includes a third neural network and is trained based on the training ultrasound image and the real gaze map; Based on the ultrasound image, the uncertainty calculation module generates an evidence vector corresponding to the ultrasound image, and generates a credibility corresponding to the malignancy probability based on the evidence vector; wherein the uncertainty calculation module includes a fourth neural network, and is trained based on the training ultrasound image.

2. The method for predicting benign or malignant breast nodules according to claim 1, wherein: The gaze map generator includes: an encoder and a decoder; The encoder includes a graph construction layer based on feature distance and the first neural network, for extracting global features of the ultrasound image; wherein the first neural network is a graph neural network; The decoder includes the second neural network, which is used to extract local features of the ultrasound image; the second neural network is a convolutional neural network.

3. The method for predicting benign or malignant breast nodules according to claim 2, wherein: The gaze map generator of the attention guiding module generates a virtual gaze map corresponding to the ultrasound image based on the ultrasound image, including: In the encoder, the feature of each position of the ultrasound image is converted into a node to form a node set; For each of the nodes in the node set, a K-nearest neighbor algorithm is used to search for neighboring nodes of the node based on the feature distance, and the node and the neighboring nodes are connected with edges to construct a first graph structure corresponding to the node set; After using a graph neural network to perform feature fusion and feature transformation of the interconnected nodes, the virtual gaze graph is output through the decoder and the generation head.

4. The method for predicting benign or malignant breast nodules according to claim 2, wherein: The gaze map classifier of the attention guiding module outputting the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map based on the ultrasound image and the virtual gaze map includes: The gaze graph classifier constructs a second graph structure by fusing the feature distance of the ultrasound image and the gaze distance of the virtual gaze graph, and eliminates node connections that are not related to the classification task; The gaze map classifier makes the nodes in the virtual gaze map area more tightly connected through the third neural network to strengthen the information aggregation of the virtual gaze map area; wherein the third neural network is a graph neural network; The gaze map classifier outputs the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map.

5. The method for predicting benign or malignant breast nodules according to claim 1, wherein: The gaze map generator uses mean square error loss as the first loss; the gaze map classifier uses cross entropy loss as the second loss; and the attention guidance module uses the sum of the first loss and the second loss as the total loss.

6. The method for predicting benign or malignant breast nodules according to claim 1, wherein: The uncertainty calculation module generates, based on the ultrasound image, an evidence vector corresponding to the ultrasound image, and generates a credibility corresponding to the malignancy probability based on the evidence vector, including: Based on the input ultrasound image, output an unnormalized classification score through a hybrid encoder and decoder architecture, and make the unnormalized classification score non-negative through an activation function, and obtain an evidence vector; converting the evidence vector into uncertainty; A credibility corresponding to the malignancy probability is obtained based on the uncertainty.

7. The method for predicting benign or malignant breast nodules according to claim 1, wherein: The training steps of the uncertainty calculation module include: Based on the input ultrasound image, output an unnormalized classification score through a hybrid encoder and decoder architecture, and make the unnormalized classification score non-negative through an activation function, and obtain an evidence vector; Converting the evidence vector into evidence strength and uncertainty; Parameterizing the strength of evidence and the uncertainty as a Dirichlet distribution; Calculate the expected value of the cross entropy loss through the Dirichlet distribution; Calculate the KL divergence loss value through the Dirichlet distribution; The expected value of the cross entropy loss and the KL divergence loss value are calculated as the total loss value of the uncertainty calculation module.

8. A system for predicting benign or malignant breast nodules, characterized by: include: A data acquisition module, used for acquiring ultrasound images of breast nodules to be detected; An attention guidance module, the attention guidance module comprising a gaze map generator and a gaze map classifier; the gaze map generator is used to generate a virtual gaze map corresponding to the ultrasound image based on the ultrasound image; wherein the gaze map generator comprises a first neural network and a second neural network, and is trained based on multiple sets of training ultrasound images and real gaze maps having corresponding relationships; the real gaze map includes attention information of an expert when performing ultrasound diagnosis; the gaze map classifier is used to output the malignancy probability of the breast nodule to be detected corresponding to the virtual gaze map based on the ultrasound image and the virtual gaze map; wherein the gaze map classifier comprises a third neural network, and is trained based on the real gaze map; An uncertainty calculation module is used to generate an evidence vector corresponding to the ultrasound image based on the ultrasound image, and to generate a credibility corresponding to the malignancy probability based on the evidence vector; wherein the uncertainty calculation module includes a fourth neural network and is trained based on the training ultrasound image.

9. A device for predicting benign or malignant breast nodules, characterized by: include: processor; a memory storing computer-readable instructions; The processor is configured to execute the method for predicting benign or malignant breast nodules according to any one of claims 1 to 7 by executing the computer-readable instructions.

10. A computer program product comprising computer-readable instructions, characterized in that When the computer-readable instructions are executed by a processor, the method for predicting benign or malignant breast nodules according to any one of claims 1 to 7 is implemented.

Images