A multi-task bi-rads risk assessment and lesion localization method for mammography

Abstract

The present application relates to a kind of breast molybdenum target examination multitask BI-RADS risk assessment and lesion marking method.The method provides risk assessment and lesion marking suggestion using multitask framework (including BI-RADS assessment and auxiliary task based on BI-RADS dictionary).The pipeline can explain BI-RADS assessment results, making it easier for clinicians to accept.Compared with traditional semi-automatic methods, it does not require segmentation or radiologist intervention, and can provide an end-to-end solution for breast X-ray analysis.To achieve these advantages, we propose a low-level feature space attention mechanism to enhance the detection capability of small object lesions in low-level features.Experimental results on two open-source breast X-ray datasets, CBIS-DDSM and CCD-CESM, show that performance in BI-RADS assessment and auxiliary tasks has reached the state-of-the-art level.The improvement of the proposed pipeline effectively bridges clinical practice and the development of breast CAD systems, marking a major advance in early risk assessment and lesion marking for breast cancer.

Description

Technical Field

[0001] This invention belongs to the field of image processing, specifically relating to a multi-task BI-RADS risk assessment and lesion identification method for mammography. Background Technology

[0002] Mammography is one of the most recommended methods for early breast cancer screening because it can detect small lesions at an early stage. In clinical practice, radiologists typically use the Breast Imaging Reporting and Data System (BI-RADS) glossary as a standard for evaluating mammograms. This glossary was developed by the American College of Radiology (ACR). The glossary aims to standardize reporting language between radiologists and clinicians and organizes reports according to BI-RADS guidelines, where clinicians use the terminology provided by the glossary to describe the different features found on a mammogram. Breast tissue density indicates the relative likelihood that a lesion is obscured by normal tissue; higher breast density is associated with an increased risk of breast cancer. Lumps and calcifications are the most common findings indicating the presence of cancer in the tissue. Clinicians then perform a BI-RADS assessment, classifying the lesion from 0 to 6 according to the probability of malignancy and providing appropriate clinical recommendations.

[0003] Mammogram analysis is a time-consuming and error-prone task for radiologists, with documented missed rates of 10-30% for radiological abnormalities. Computer-aided diagnostic (CAD) systems typically treat mammogram analysis as a single classification task, focusing on different aspects of mammogram classification. Early breast cancer lesions can be as small as 2% on a mammogram and are easily overlooked or obscured in neural network analysis, leading to reduced CAD effectiveness and potential false negatives. To address this issue, most methods choose to annotate training data with bounding boxes and segmentation labels, or employ semi-automatic techniques to improve the performance of CAD systems. However, several issues need to be addressed before these methods can be applied clinically. First, using bounding boxes and manual segmentation labels typically requires significant resource investment. Furthermore, adopting semi-automatic methods may inadvertently increase the workload of clinicians, thus conflicting with the purpose of the CAD system. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-task BI-RADS risk assessment and lesion identification method for mammography. This method reduces the workload of manual marking while ensuring segmentation accuracy, thereby improving work efficiency.

[0005] To achieve the above objectives, the technical solution of the present invention is: a multi-task BI-RADS risk assessment and lesion identification method for mammography, comprising the following steps:

[0006] Step 1: Acquire mammograms and preprocess them;

[0007] Step 2: Using the preprocessed image obtained in Step 1, input it into the Swin Transformer network framework to generate a multi-scale feature set, including low-frequency features and high-frequency features; where low-frequency features refer to regions in the image that change relatively slowly and smoothly, and high-frequency features refer to regions in the image that change rapidly.

[0008] Step 3: Enhance the network's feature extraction capability for small-resolution objects by fusing the multi-scale features obtained in Step 2. This includes proposing a low-frequency feature spatial attention module to dynamically detect and enhance task-related regions in low-level feature maps using high-frequency information.

[0009] Step 4: Complete 5 tasks using the features obtained in Step 3, including the main task of multi-task breast imaging reporting and data system BI-RADS assessment, supplemented by 4 auxiliary tasks.

[0010] Step 5: Integrate the results of multiple tasks obtained in Step 4 into risk assessment and lesion identification results;

[0011] Step 6: Optimize using a multi-objective optimization method.

[0012] In one embodiment of the present invention, the preprocessing process includes image cropping, image scaling, and image enhancement.

[0013] In one embodiment of the present invention, the four auxiliary tasks are mass, calcification, pathology, and breast density classification.

[0014] In one embodiment of the present invention, the risk assessment and lesion identification results include breast density, examination result description and BI-RADS assessment, corresponding to the three components of a BI-RADS report.

[0015] In one embodiment of the present invention, step 1 is implemented as follows:

[0016] Read a mammogram and use morphological operations to remove the black background. Specifically, first, the Sobel operator is used to calculate the gradient of the image to detect the edges. Based on the edge detection results, the contours are identified and the boundary rectangles are extracted to crop the region of interest (ROI) of the complete breast. Next, a non-square input size is selected by calculating the average aspect ratio of the mammogram dataset to minimize the distortion usually caused by scaling. Subsequently, contrast-limited adaptive histogram equalization is used for image enhancement. The pixel values ​​in the image are redistributed through grayscale mapping to make the histogram distribution more uniform.

[0017] In one embodiment of the present invention, step 2 is implemented as follows:

[0018] Image features are extracted using the Swing Transformer network framework, which includes a linear embedding module and a block merging module. The latter merges adjacent blocks and maps the channels of the feature map to a higher dimension. Information transfer and feature extraction are performed by several cascaded Swing Transformer modules and block merging modules. In the merging module, the feature hierarchy is calculated at multiple scales with a scale step of 2. The sizes of these multi-scale feature sets correspond to the pixel dimensions of the input image in the range of [1 / 4, 1 / 8, 1 / 16, 1 / 32].

[0019] In one embodiment of the present invention, step 3 is implemented as follows:

[0020] Before fusing multi-scale features, to reduce the significant semantic differences between features of different scales that hinder the effectiveness of multi-scale feature fusion, a low-frequency feature spatial attention module is proposed to enhance small-resolution lesion regions and reduce semantic differences during the fusion process. The fused multi-scale features are represented as follows:

[0021]

[0022] Where norm(·) represents layer normalization. These are attention weights, and pool(·) represents global average pooling, which converts enhanced multi-scale features into semantic features.

[0023] The low-frequency feature spatial attention module maps the sampling results through a group of convolutions, including a 1×1 convolutional layer to match the number of channels of low-level features. The convolutional group includes two sets of residual convolutions to enhance feature extraction capabilities, and an upsampling layer aligns the spatial dimensions. Then, a sigmoid layer generates the attention weights W. l .

[0024] In one embodiment of the present invention, step 4 is implemented as follows:

[0025] The fully connected classifier outputs results for five tasks. The main task is to output five BI-RADS assessment categories, namely B1 to B5, which correspond to the "BI-RADS Assessment" section in the BI-RADS report. The auxiliary tasks include mass, calcification, and pathological classification, which correspond to the "Examination Result Description" section in the BI-RADS report. In addition, it also includes breast density classification, which is consistent with the "Breast Density" section in the BI-RADS report.

[0026] In one embodiment of the present invention, step 5 is implemented as follows:

[0027] The structure of risk assessment and lesion identification results includes three key elements consistent with the BI-RADS report: breast density, examination result description, and BI-RADS assessment; recommendations related to breast density and BI-RADS assessment are directly derived from their respective tasks, while the examination result description integrates the results of multiple tasks.

[0028] The examination result description includes "Result Type" and "Preliminary Assessment". "Result Type" has five categories: None, Calcification, Mass, Calcification and Mass, and Other; the assignment follows specific rules: "None" comes from the pathology prediction task; "Calcification", "Mass", and "Calcification and Mass" are determined by two binary classification tasks for Mass and Calcification; the result type is categorized as "Other"; when both "Mass" and "Calcification" tasks yield "Negative" results, but the pathology prediction result is not "Normal", it is classified as "Other". The preliminary assessment includes five categories: "Normal", "One or More Benign Results", "Multiple Benign Results", "One or More Results Including Malignancy", and "Multiple Results Including Malignancy"; these categories are determined as follows: "Normal" comes from the pathology prediction task; a single BI-RADS assessment category result is classified as "One or More Benign Results" or "One or More Results Including Malignancy" based on the pathology prediction result; conversely, multiple BI-RADS assessment categories result in "Multiple Benign Results" or "Multiple Results Including Malignancy".

[0029] In one embodiment of the present invention, step 6 is implemented as follows:

[0030] For optimization across multiple tasks, the following multi-objective loss is employed:

[0031]

[0032] Among them, w i L represents the weights of each objective optimization function. i It represents the loss for each objective, allowing the priority of optimizing different tasks to be determined by manually adjusting the scalar weights.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] First, this method provides recommendations for risk assessment and lesion identification, including breast density, finding description, and BI-RADS assessment, all of which correspond to the three components of a BI-RADS report. The method's auxiliary tasks further enhance the comprehensiveness and interpretability of clinical reference aids by detecting and assessing lesions—an aspect that previous methods could not effectively achieve.

[0035] Second, this method leverages high-level features to detect and enhance small-object lesions within low-level features, allowing the fusion process to focus more on the semantic aspects of the lesions. This enhances the model's ability to distinguish representations and improves the success rate of early breast cancer detection. Importantly, the proposed module enables the system to operate without segmentation annotation or radiologist intervention, providing an end-to-end solution for mammography image analysis. Attached Figure Description

[0036] Figure 1 This is a schematic diagram illustrating the principle of the present invention;

[0037] Figure 2 This is a schematic diagram of the feature extraction method of the present invention;

[0038] Figure 3 This is a visualization of some of the features of the present invention. Detailed Implementation

[0039] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0040] This invention provides a multi-task BI-RADS (Breast Imaging-Reporting and Data System) method for risk assessment and lesion identification during mammography. (See appendix.) Figure 1 It mainly includes the following steps:

[0041] 1. Acquire and preprocess mammograms, including image cropping, scaling, and enhancement. This method reads a mammogram and uses morphological operations to remove the black background. Specifically, the system first uses the Sobel operator to calculate the image gradient to detect edges. Based on the edge detection results, the system identifies contours and extracts boundary rectangles, cropping the ROI of the complete breast. Next, the method selects an appropriate non-square input size by calculating the average aspect ratio of the dataset, thereby minimizing distortion typically caused by scaling. Subsequently, the method utilizes contrast-limited adaptive histogram equalization for image enhancement, which redistributes pixel values ​​in the image through grayscale mapping, making the histogram distribution more uniform. Applying this method can effectively alleviate data skew and make the training data more balanced. This method not only alleviates data skew and makes the training data more balanced, but also helps the model learn feature differences more effectively without bias towards a particular class (see...). Figure 1 a).

[0042] 2. Using the preprocessed image obtained in step 1, input it into the Swin Transformer network framework to generate a multi-scale feature set, including low-frequency features and high-frequency features. Low-frequency features typically refer to regions in the image that change gradually and smoothly, such as the background and large areas of the same color. High-frequency features refer to regions in the image that change rapidly, such as edges, textures, and details. These features correspond to higher frequencies in the frequency domain.

[0043] Image features are extracted using the Swing Transformer network framework, which includes a linear embedding module and a block merging module. The latter merges adjacent blocks and maps the channels of the feature map to a higher dimension. Information transfer and feature extraction are performed by several cascaded Swing Transformer modules and the block merging module. In the merging module, feature levels are computed at multiple scales with a scale step of 2, and the sizes of these multi-scale feature sets correspond to [1 / 4, 1 / 8, 1 / 16, 1 / 32] of the input image pixel dimension, respectively. (See...) Figure 1 b and Figure 2 ).

[0044] 3. The network's feature extraction capability for small-resolution objects is enhanced by fusing multi-scale features obtained in step 2. This includes our proposed low-level feature space attention module, which dynamically detects and enhances task-relevant regions in the low-level feature map using high-frequency information. Before fusing multi-scale features, reducing significant semantic differences between features of different scales hinders the effectiveness of multi-scale feature fusion. This method uses the proposed low-frequency feature space attention module to enhance small-resolution lesion regions and reduces semantic differences during the fusion process, thereby improving the performance. The fused multi-scale features are represented as follows:

[0045]

[0046] Where norm(·) represents layer normalization. These are attention weights, and pool(·) represents global average pooling, which converts enhanced multi-scale features into semantic features.

[0047] The low-frequency feature spatial attention module maps the sampling results through a group of convolutions, including a 1×1 convolutional layer that matches the number of channels of the low-level features. This design also helps to make the model more lightweight. The convolutional group also includes two sets of residual convolutions to enhance feature extraction capabilities and aligns the spatial dimensions through an upsampling layer. Attention weights W are then generated through a sigmoid layer. l .(See Figure 2 ).

[0048] 4. The proposed system completes five tasks using the features obtained in step 3, with BI-RADS assessment as the primary task, supplemented by mass, calcification, pathology, and ACR density classification. These auxiliary tasks help the system to further detect and evaluate lesions.

[0049] The fully connected classifier outputs results for five tasks. The primary task is to output five BI-RADS assessment categories (B1 to B5), corresponding to the "BI-RADS Assessment" section of the report. Ancillary tasks include mass, calcification, and pathological classification, corresponding to the "Description of Examination Results" section of the report. Multiple classification tasks provide a more comprehensive and easily interpretable risk assessment and lesion localization reference, which is not possible with a single classification task. Additionally, a breast density classification task is included, consistent with the "Breast Density" section. (See...) Figure 1 c).

[0050] 5. The results from multiple tasks obtained in section 4 are integrated into risk assessment and lesion identification recommendations, including breast density, examination result description, and BI-RADS assessment, all of which correspond to the three components of a BI-RADS report.

[0051] The results of the risk assessment and lesion identification described above include three key elements consistent with the BI-RADS report: breast density, examination result description, and BI-RADS assessment. Recommendations related to breast density and BI-RADS assessment are directly derived from their respective tasks, while the result description synthesizes results from multiple tasks.

[0052] The result description includes "Result Type" and "Preliminary Assessment". "Result Type" has five categories: None, Calcification, Mass, Calcification and Mass, and Other. Assignment follows specific rules: "None" comes from the pathology prediction task; "Calcification", "Mass", and "Calcification and Mass" are determined by a two-category task of Mass and Calcification; the result type is categorized as "Other". When both the "Mass" and "Calcification" tasks yield "Negative" results, but the pathology prediction result is not "Normal", it is classified as "Other" (e.g., structural deformity, asymmetry, intramammary lymph nodes, etc.). "Preliminary Assessment" includes five categories: Normal, One or More Benign Results, Multiple Benign Results, One or More Results Including Malignancy, and Multiple Results Including Malignancy. These categories are determined as follows: "Normal" comes from the results of the pathology prediction task; a single BI-RADS assessment category result is categorized as "one or more benign results" or "one or more results including malignant results" based on the pathology prediction result. Conversely, multiple BI-RADS assessment categories result in "multiple benign results" or "multiple results including malignant results." (See...) Figure 1 d).

[0053] 6. A multi-objective optimization method is used to optimize the proposed deep learning model. The method optimizes the model for multiple tasks, employing the following multi-objective loss:

[0054]

[0055] Among them, w i L represents the weights of each objective optimization function. i It represents the loss for each objective, allowing the priority of optimizing different tasks to be determined by manually adjusting the scalar weights.

[0056] To demonstrate the proposed method's ability to help the network locate lesion regions, we validated this hypothesis through comparison using visual attention heatmaps. Figure 3 (b) and Figure 3 In the comparison in (d), the model without attention focuses roughly on the entire breast area. Adding the attention module allows the model to highlight only areas with potential lesions, avoiding unnecessary focus on normal tissue. Figure 3 (c) and Figure 3 In the comparison in (e), the model using attention can focus on each key region, i.e., accurately locate small-resolution lesions in multi-scale feature maps, a capability lacking in the model not using attention. (See...) Figure 3 ).

[0057] To gain a clearer understanding of the system's performance, it is necessary to compare it with existing methods. However, this is a challenging task because few publications focus on BI-RADS evaluation of mammograms, and there is no unified dataset or standard evaluation metric in this field. Table 1 compares the task and performance of this invention with previous mammogram classification studies, highlighting the contributions of this invention.

[0058] Table 1

[0059]

[0060]

[0061] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A multi-task BI-RADS risk assessment and lesion identification method for mammography, characterized in that, Includes the following steps: Step 1: Acquire mammograms and preprocess them; Step 2: Using the preprocessed image obtained in Step 1, input it into the Swin Transformer network framework to generate a multi-scale feature set, including low-frequency features and high-frequency features; in Low-frequency features refer to regions in an image that change relatively slowly and smoothly, while high-frequency features refer to regions in an image that change rapidly. Step 3: Enhance the network's feature extraction capability for small-resolution objects by fusing the multi-scale features obtained in Step 2. This includes proposing a low-frequency feature spatial attention module to dynamically detect and enhance task-related regions in low-level feature maps using high-frequency information. Step 4: Complete 5 tasks using the features obtained in Step 3, including BI-RADS assessment as the main task and 4 auxiliary tasks. Step 5: Integrate the results of multiple tasks obtained in Step 4 into risk assessment and lesion identification results; Step 6: Optimize using a multi-objective optimization method.

2. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 1, characterized in that, The preprocessing steps include image cropping, image scaling, and image enhancement.

3. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 1, characterized in that, The four auxiliary tasks are mass, calcification, pathology, and breast density classification.

4. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 1, characterized in that, Risk assessment and lesion identification results include breast density, examination result description, and BI-RADS assessment, corresponding to the three components of a BI-RADS report.

5. A multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 1 or 2, characterized in that, The specific implementation method of step 1 is as follows: Read a mammogram and use morphological operations to remove the black background. Specifically, first, the Sobel operator is used to calculate the gradient of the image to detect the edges. Based on the edge detection results, the contours are identified and the boundary rectangles are extracted to crop the region of interest (ROI) of the complete breast. Next, a non-square input size is selected by calculating the average aspect ratio of the mammogram dataset to minimize the distortion usually caused by scaling. Subsequently, contrast-limited adaptive histogram equalization is used for image enhancement. The pixel values ​​in the image are redistributed through grayscale mapping to make the histogram distribution more uniform.

6. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 1, characterized in that, The specific implementation method of step 2 is as follows: Image features are extracted using the Swing Transformer network framework, which includes a linear embedding module and a block merging module. The latter merges adjacent blocks and maps the channels of the feature map to a higher dimension. Information transfer and feature extraction are performed by several cascaded Swing Transformer modules and block merging modules. In the merging module, the feature hierarchy is calculated at multiple scales with a scale step of 2. The sizes of these multi-scale feature sets correspond to the pixel dimensions of the input image in the range of [1 / 4, 1 / 8, 1 / 16, 1 / 32].

7. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 1, characterized in that, The specific implementation method of step 3 is as follows: Before fusing multi-scale features, to reduce the significant semantic differences between features of different scales that hinder the effectiveness of multi-scale feature fusion, a low-frequency feature spatial attention module is proposed to enhance small-resolution lesion regions and reduce semantic differences during the fusion process. The fused multi-scale features are represented as follows: Where l represents the l-th layer, and norm(·) represents layer normalization. These are attention weights, and pool(·) represents global average pooling, which converts enhanced multi-scale features into semantic features. The low-frequency feature spatial attention module maps the sampling results through a group of convolutions, including a 1×1 convolutional layer to match the number of channels of low-level features. The convolutional group includes two sets of residual convolutions to enhance feature extraction capabilities, and an upsampling layer aligns the spatial dimensions. Then, a sigmoid layer generates the attention weights W. l .

8. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 4, characterized in that, The specific implementation method of step 4 is as follows: The fully connected classifier outputs results for five tasks. The main task is to output five BI-RADS assessment categories, namely B1 to B5, which correspond to the "BI-RADS Assessment" section in the BI-RADS report. The auxiliary tasks include mass, calcification, and pathological classification, which correspond to the "Examination Result Description" section in the BI-RADS report. In addition, it also includes breast density classification, which is consistent with the "Breast Density" section in the BI-RADS report.

9. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 8, characterized in that, The specific implementation method of step 5 is as follows: The structure of risk assessment and lesion identification results includes three key elements consistent with the BI-RADS report: breast density, examination result description, and BI-RADS assessment; recommendations related to breast density and BI-RADS assessment are directly derived from their respective tasks, while the examination result description integrates the results of multiple tasks. The examination results description includes "Result Type" and "Preliminary Assessment". The "Result Type" has five categories: None, Calcification, Mass, Calcification and Mass and Others; the assignment follows specific rules: "None" comes from the results of the pathology prediction task; "Calcification", "Mass" and "Calcification and Mass" are determined by two binary classification tasks for Mass and Calcification. The result type is categorized as "Other"; when both the "Mass" and "Calcification" tasks yield "Negative" results, but the pathological prediction result is not "Normal," it will be classified as "Other." The initial assessment includes five categories: "Normal," "One or more benign results," "Multiple benign results," "One or more results including malignancy," and "Multiple results including malignancy." These categories are determined as follows: "Normal" comes from the results of the pathological prediction task; a single BI-RADS assessment category result is classified as "One or more benign results" or "One or more results including malignancy" based on the pathological prediction result; conversely, multiple BI-RADS assessment categories will result in "Multiple benign results" or "Multiple results including malignancy." 10. The multi-task BI-RADS risk assessment and lesion identification method for mammography according to claim 1, characterized in that, The specific implementation method of step 6 is as follows: For optimization across multiple tasks, the following multi-objective loss is employed: Among them, w i L represents the weights of each objective optimization function. i It represents the loss for each objective, allowing the priority of optimizing different tasks to be determined by manually adjusting the scalar weights.

Images