Method and apparatus for mass identification in multi-view mammography

Abstract

A method applied to an apparatus for multi-view mammography mass identification includes receiving a main image, a first auxiliary image, and a second auxiliary image. The main image and the first auxiliary image are images of one breast of a person, and the second auxiliary image is an image of the other breast of a person. The method further includes detecting nipple location based on the main image and the first auxiliary image; generating a first probability map of the main image based on the main image, the first auxiliary image, and the nipple location; generating a second probability map of the main image based on the main image, the second auxiliary image, and the nipple location; and generating and outputting a fused probability map based on the first and second probability maps.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 072,379, filed August 31, 2020. This application also claims priority to U.S. Patent Application No. 17 / 165,087, filed February 2, 2021, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This invention relates to the field of medical device technology, and in particular to a method and apparatus for identifying lumps in mammography using multi-view imaging. Background Technology

[0004] Mammography is widely used as a cost-effective method for early detection of breast cancer. Computer-aided diagnosis (CAD) systems hold promise for detecting abnormal areas on digitized mammogram images. Most deep neural network-based (DNN) mammogram analysis methods are designed for single-view analysis. Significant progress has recently been made in the performance of CAD systems, particularly with improvements in DNN-based methods. Nevertheless, mammogram anomaly detection remains challenging, largely due to the high accuracy requirements set by clinical practice.

[0005] Standard mammographic screening procedures obtain two low-dose X-ray projection views for each breast: a craniocaudal (CC) view and a mediolateral oblique (MLO) view. Radiologists typically use both views in breast cancer diagnosis. Ipsilateral analysis refers to diagnosis based on both CC and MLO views of the same breast, while bilateral analysis combines the results of identical views from both breasts (i.e., two CC views or two MLO views). For example, radiologists can cross-examine lesion locations using ipsilateral analysis and utilize the symmetry information from bilateral analysis to improve decision-making accuracy.

[0006] Many existing mammography lesion detection methods focus on a single view, thus failing to capture rich information from multi-view analysis. Several DNN-based dual-view methods have recently been proposed, performing ipsilateral or bilateral analysis. However, existing DNN-based architectures still require further improvement to provide the necessary performance for multi-view analysis. The disclosed method and apparatus for multi-view mammography mass identification aim to address one or more of the aforementioned problems, as well as other issues in the art. Summary of the Invention

[0007] A first aspect of this disclosure provides a method for use in an apparatus for multi-view mammography to identify lumps. The method includes: receiving a main image, a first auxiliary image, and a second auxiliary image. The main image and the first auxiliary image are images of one breast of a person, and the second auxiliary image is an image of the other breast of the person. The method further includes: detecting a nipple location based on the main image and the first auxiliary image; generating a first probability map of the main image based on the main image, the first auxiliary image, and the nipple location; generating a second probability map of the main image based on the main image, the second auxiliary image, and the nipple location; and generating and outputting a fused probability map based on the first probability map and the second probability map.

[0008] Another aspect of this disclosure provides an apparatus for multi-view mammography mass identification. The apparatus includes: a nipple detector configured to receive a main image and a first auxiliary image of one breast of a person, and to detect a nipple position based on the main image and the first auxiliary image; a ipsilateral analyzer configured to receive the main image, the first auxiliary image, and the nipple position obtained by the nipple detector, and to generate and output a first probability map of the main image; and a bilateral analyzer configured to receive the main image, the nipple position, and a second auxiliary image, and to generate and output a second probability map of the main image. The second auxiliary image is an image of the other breast of the person. The apparatus also includes an integrated fusion network device configured to receive the main image, the first probability map, and the second probability map, and to generate and output a fused probability map.

[0009] Other aspects of this disclosure will be understood by those skilled in the art based on the specification, claims and drawings.

[0010] Brief description of the attached figures

[0011] The following figures are merely examples for illustrative purposes based on various disclosed embodiments and are not intended to limit the scope of this disclosure.

[0012] Figure 1 A schematic diagram of the structure of an exemplary device for multi-view breast mass identification according to various embodiments of the present disclosure is shown;

[0013] Figure 2 A data flow diagram illustrating the operation of an exemplary apparatus for multi-view breast mass identification according to various embodiments of the present disclosure is shown.

[0014] Figure 3 It shows Figure 1 A schematic diagram of the structure of an exemplary bilateral analyzer in a device for multi-view breast mass identification is shown.

[0015] Figure 4 It shows Figure 1 A schematic diagram of the structure of an exemplary ipsilateral analyzer in a device for multi-view breast mass identification is shown.

[0016] Figure 5 Examples of similarity in RoI-to-nipple distances are shown in cephalothorax and mediolateral oblique images;

[0017] Figure 6 It shows Figure 1 A schematic diagram of the structure of an exemplary integrated fusion network device in a apparatus for multi-view breast mass recognition is shown.

[0018] Figure 7 A flowchart illustrating the operation process of a nipple detector according to various embodiments of the present disclosure is shown;

[0019] Figure 8 A flowchart illustrating the operation process of an image registration device according to various embodiments of the present disclosure is shown;

[0020] Figure 9 A flowchart illustrating the operation process of a same-side analyzer according to various embodiments of the present disclosure is shown;

[0021] Figure 10 Flowcharts illustrating the operation of a two-sided analyzer according to various embodiments of the present disclosure are shown; and

[0022] Figure 11 A flowchart illustrating the operation process of an integrated converged network device according to various embodiments of the present disclosure is shown. Detailed Implementation

[0023] Reference will now be made in detail to exemplary embodiments of the invention illustrated in the accompanying drawings. The same reference numerals will be used wherever possible to refer to the same or similar parts in the drawings.

[0024] The importance of information fusion from multi-view mammograms has been previously recognized. Many existing multi-view information fusion methods rely on handcrafted features and fusion rules. With the development of deep learning technology, DNN-based methods have achieved widespread success in medical image processing and computer vision.

[0025] Deep learning has been applied to mammographic mass detection, with most work focusing on single-view-based methods. Recently, multi-view-based methods have attracted increasing interest, and various DNN-based approaches have been proposed for ipsilateral analysis of multi-view mammograms. However, most existing methods do not explicitly model the geometric relationships across views. For example, in existing methods, a cross-view relationship network is added to Siamese Networks for mass detection, but the geometric features and embeddings used for the relationship network are the same as those used in methods for single-view object detection. In another example, a DNN-based method has been proposed for bilateral analysis without simultaneously providing ipsilateral analysis. Other exemplary multi-view-based methods include RNN-based multi-view methods for mass classification, DNN-based multi-view methods for breast cancer screening, and so on.

[0026] According to the disclosed apparatus and method for multi-view mammography mass identification, a Faster-RCNN network module with conjoined input and a DeepLab network module with conjoined input operate in parallel to simultaneously perform ipsilateral and bilateral analyses. The relational network of this disclosure is explicitly designed to encode the mass-to-nipple distance along with a DNN-based nipple detector for ipsilateral analysis. The method of this disclosure explicitly embeds the mass-to-nipple distance into a DNN architecture for mammographic lesion detection.

[0027] This disclosure provides an apparatus for identifying lumps in a multi-view mammogram. Figure 1 A schematic diagram of the structure of an exemplary apparatus for multi-view mammography mass identification according to various embodiments of the present disclosure is shown. Reference Figure 1 The device may include a nipple detector, an image registration unit, a same-side analyzer, a two-side analyzer, and an integrated fusion network device.

[0028] The nipple detector can be configured to receive a main image of a human breast and a first auxiliary image of the breast, and to detect the nipple location based on the main image and the first auxiliary image. The detected nipple location may help determine the distance from the region of interest (RoI) to the nipple in subsequent processes. In one embodiment, the nipple detector may be a DNN-based nipple detector.

[0029] In one embodiment, the primary image may be a right craniocaudal (RCC) image, and the first auxiliary image may be an ipsilateral image, such as a right mediolateral oblique (RMLO) image. That is, the primary image may be a CCC image of the right breast, and the first auxiliary image may be an MLO image of the right breast. It should be noted that the primary image and the first auxiliary image may be ipsilateral images of the same breast. In many cases, four types of mammographic images may be available: RCC image, RMLO image, left craniocaudal (LCC) image, and left mediolateral oblique (LMLO) image. Therefore, the combination of the primary image and the first auxiliary image can have four different options: the primary image is an RCC image, and the first auxiliary image is an RMLO image; the primary image is an RMLO image, and the first auxiliary image is an RCC image; the primary image is an LCC image, and the first auxiliary image is an LMLO image; the primary image is an LMLO image, and the first auxiliary image is an LCC image.

[0030] The image registration unit can be configured to receive the main image and an initial auxiliary image, flip the initial auxiliary image, and then distort the flipped initial auxiliary image toward the main image according to the breast contour to generate a second auxiliary image. The main image received by the image registration unit can be the same as the main image received by the nipple detector. In one embodiment, the initial auxiliary image received by the image registration unit can be a bilateral image, such as an LCC image. After flipping and distorting the initial auxiliary image, the obtained second auxiliary image can be sent to the bilateral analyzer to facilitate DNN-based bilateral image symmetry constraint learning. For example, the input is a pair of identical view images (e.g., the two CC view images or the two MLO view images), including the main image and the initial auxiliary image, which can be registered, and the image serving as the initial auxiliary image can be horizontally flipped and distorted toward the main image according to the breast contour to obtain the second auxiliary image. The main image can be an image selected from two pairs of images, including a pair of CC images and a pair of MLO images. Once the main image is selected, the initial auxiliary image can be another image from the same pair, and the first auxiliary image can be an image from the other pair. For example, when the main image is a CC image of one breast, the initial auxiliary image can also be a CC image, but for the corresponding breast, the first auxiliary image can be an MLO image of the same breast. Therefore, the first auxiliary image and the main image can be taken from the same breast, making them a pair of images from the same side. For example, when the main image is an LCC image, the first auxiliary image can be an LMLO image, and the initial auxiliary image can be an RCC image; when the main image is an RMLO image, the first auxiliary image can be an RCC image, and the initial auxiliary image can be an LMLO image.

[0031] The ipsi-probability analyzer can be configured to receive the main image, the first auxiliary image, and the nipple position obtained by the nipple detector. The ipsi-probability analyzer can also be configured to generate and output a first probability map (e.g., an ipsi-prob map) of the main image based on the main image, the first auxiliary image, and the nipple position.

[0032] The two-sided analyzer can be configured to receive the main image along with the nipple location and the second auxiliary image obtained by image registration. The two-sided analyzer can also be configured to generate and output a second probability map (e.g., a bi-prob map) of the main image based on the main image, the nipple location, and the distorted and flipped two-sided images.

[0033] The first and second probability maps can be points of interest in the combined information of the ipsilateral and bilateral images. For example, the first and second probability maps can respectively include tumor information of pixels in the main image generated by the ipsilateral analyzer and the bilateral analyzer. The tumor information of pixels in the main image can include texture density determined by the brightness of the pixels. Furthermore, adjacent pixels with similar brightness can be further identified as a region for further analysis. For example, each of the ipsilateral and bilateral analyzers can be further configured to calculate the similarity between adjacent pixels based on two input images (e.g., the main image and the first auxiliary image for the ipsilateral analyzer, and the main image and the second auxiliary image for the bilateral analyzer) to determine each region containing pixels with similar brightness. Additionally, regions whose brightness, shape, area size, and / or position deviate from expected values ​​can be identified as RoIs. For example, a threshold can be preset for each of the parameters such as brightness, shape, area size, and relative position to the nipple position; when at least one of the parameters of a region exceeds the corresponding threshold, the region is identified as an RoI.

[0034] In one embodiment, the first probability map may further include the lesion probability of each pixel in the main image; the second probability map may further include the lesion probability of each pixel in the main image. Each of the same-side analyzer and the two-side analyzer can determine the lesion probability of each pixel based on the input image. Furthermore, pixel regions with a lesion probability exceeding a preset probability can be identified as RoIs.

[0035] It should be noted that both the first probability map and the second probability map are intermediate results of lesion analysis performed using the apparatus for breast multi-view mass identification disclosed herein. The first probability map and the second probability map can be further analyzed in the integrated fusion network device to generate a more accurate probability map of the main image.

[0036] The integrated fusion network device can be configured to receive the main image, a first probability map from the same-side analyzer, and a second probability map from the two-side analyzer. Furthermore, the integrated fusion network device can be configured to generate and output a fused probability map based on the main image and the two probability maps. The fused probability map can include the lesion probability of each pixel in the main image. By combining the first probability map and the second probability map... Figure 1The analysis to generate the fusion probability map allows for a more accurate determination of the lesion probability of each pixel in the main image. Therefore, the disclosed apparatus for multi-view mammography mass identification can improve the accuracy of identifying lesions in mammography. Furthermore, one or more RoIs can be determined based on the fusion probability map, and the location, size, and shape of each RoI can also be determined.

[0037] Figure 2 A schematic diagram illustrating the data flow of an exemplary apparatus for multi-view mammography mass identification according to various embodiments of the present disclosure is shown. The apparatus can be used with... Figure 1 The apparatus shown is identical. (Reference) Figure 2 As shown, firstly, an image of a breast can be selected as the main image, and its corresponding ipsilateral and bilateral views can be selected as auxiliary images. The main image and the auxiliary images can be input into the device together. For example, the main image can be an RCC view, and the auxiliary images can include the ipsilateral and bilateral views corresponding to the main view. The ipsilateral view can be an RMLO view, and the bilateral views can be LCC views. The RCC view and the RMLO view can be input together into the ipsilateral branch. In parallel, the RCC view and the LCC view can be input together into the bilateral branch. Each branch can generate a probability map of the main image (e.g., Figure 2 The Ipsi-prob and Bi-prob maps shown; or the first and second probability maps as described above) and the probability maps generated by the two branches can be input into the integrated fusion network device along with the main image (RCC view) to generate the final output, such as the fused probability map. In the ipsilateral branch, a DNN-based nipple detector can be added to extract nipple locations on both views (e.g., the RCC view and the RMLO view). These nipple locations can then be combined with the two views used for ipsilateral analysis. Figure 1 The image is input into the same-side analyzer. Within the bilateral branch, the bilateral view (LCC view) can be registered in the image register before being sent together with the main image (RCC view) to the bilateral analyzer. This combined same-side and bilateral analysis can be applied to any given image used as the main image.

[0038] According to the disclosed apparatus, both the image registration unit and the ipsilateral analyzer may need to use the nipple location extracted from the input image. In one embodiment, the nipple detector may include a Faster-RCNN-based keypoint detection framework, which enables the identification of the nipple location with satisfactory accuracy. For example, in an internal dataset containing a total of 11,228 images, only one nipple was incorrectly predicted.

[0039] Most women have breasts that are roughly symmetrical in terms of density and texture. Radiologists take full advantage of this property to identify abnormalities in mammograms. Relying on bilateral bi-view images, radiologists are able to locate masses based on their unique morphological appearance and their relative position to the corresponding area in the image on the other side.

[0040] To incorporate this diagnostic information and facilitate the learning of symmetric constraints, a two-sided analyzer was developed. Figure 3 It shows Figure 1 The illustrated architecture is an exemplary dual-sided analyzer in a device for multi-view mass identification in mammography. Figure 3 As shown, the bilateral analyzer may originate from the DeepLab v3+ architecture, enhancing the first Siamese input mode and the pixel-wise focal loss (PWFL) function. The first Siamese input mode may include two dilated convolutional modules. These two modules may share the same weights and extract feature maps from the bilateral images in the same manner. Each dilated convolutional module may consist of a five-level network formed by concatenating five 50-layer ResNet-50 residual networks, and a non-local (NL) block located between Stage 4 and Stage 5. The ResNet-50 in each dilated convolutional module can serve as the backbone. The dilated convolutional module receiving the main image may also include the output of low-level features from Stage 3. The feature maps extracted by the two dilated convolutional modules can be a primary feature map and an auxiliary feature map, respectively. The auxiliary feature map can then be assumed as a reference and concatenated with the primary feature map. Conversely, feature differences at the same location can highlight anomalies. For example, the outputs from the backbones of two dilated convolutional modules can perform channel concatenation, while a 1×1 convolution can downsample the number of channels in the concatenated tensor to half. The two-sided analyzer can also generate a segmentation map of the master image. During training, the PWFL function can be used to improve the performance of the two-sided analyzer.

[0041] Ipsilateral images provide information about the same breast from two different views. Therefore, masses in ipsilateral images often present similar brightness, shape, size, and distance to the nipple. This knowledge is crucial in helping radiologists make decisions. To incorporate this diagnostic knowledge, an ipsilateral analyzer was developed. Figure 4 It shows Figure 1 The architecture of an exemplary ipsilateral analyzer in a device for multi-view mass identification in mammography is shown. (Reference) Figure 4The same-side analyzer may be built on a Faster-RCNN detection architecture. The same-side analyzer may include a second Siamese input pattern, a feature pyramid network (FPN) module, and relation blocks. The second Siamese input pattern, together with the FPN module, allows the two input branches to share the same weights and extract features from the two same-side views in the same way. Therefore, the relation blocks can be used to compute appearance similarity and geometric constraints between RoIs from the two branches. Further, block information of pixels in the main image can be detected and converted into a probability map. Additionally, during training, focal loss (FL) and distance-intersection-over-union loss (DIoU) can be used to improve the performance of the same-side analyzer, and training with negative samples (normal cases) can be enabled. It should be noted that the same-side analyzer may include multiple relation blocks to improve the output. For example, as... Figure 4 As shown, the appearance similarity and geometric constraints between RoIs can be recalculated using a second relation block (details are not shown in the relation block), thereby improving the accuracy of the results.

[0042] like Figure 4 As shown, the relation block can model the attention-based relationship between two RoIs in a single image based on the similarity of appearance and geometric features, thereby improving detection accuracy. For example, the relation block can model the relationship between two RoIs in a single image based on the similarity of their brightness, shape, size, and position. In one embodiment, the relative position of the RoI can include the distance from the RoI to the nipple location determined by the nipple detector. Furthermore, it can be suggested that the relation block emphasize the appearance and geometric similarity of lesion RoIs in two ipsilateral images. In one embodiment, appearance and geometric similarity can be described as:

[0043]

[0044] Where ε(·,·) represents the geometric embedding operation. The geometric factor representing the i-th RoI of the main image. Representing the

[0045] The geometric factor of the j-th RoI in the auxiliary image. It is the distance from the i-th RoI of the main image to the nipple (RoI-to-nipple). It is the distance from the nipple to the j-th RoI of the auxiliary image. It is the width of the i-th RoI of the main image. It is the width of the j-th RoI of the auxiliary image. It is the height of the i-th RoI in the main image. It is the height of the j-th RoI in the auxiliary image.

[0046] To integrate the outputs of same-side learning and two-side learning, the ensemble fusion network is designed to accept three inputs: the main image and the two probability maps from the same-side analyzer and the two-side analyzer (see [link to ensemble network]). Figures 1 to 2 These two probability maps likely focus on combined information from bilateral and ipsilateral images. This strategy is also applicable to mammogram cancer screening. Figure 6 It shows Figure 1 The diagram shows a schematic representation of an exemplary integrated fusion network device in a apparatus for multi-view breast mass recognition. It should be noted that... Figure 6 The ResNet-50 backbone in the network is derived from the same-side analyzer and is frozen during training. The ensemble fusion network can be configured to process the input and generate predictions.

[0047] This disclosure also provides a method for identifying masses in multi-view mammography. This method can be applied to apparatus conforming to various embodiments of this disclosure. That is, the apparatus may include a nipple detector, an image registration unit, an ipsilateral analyzer, a bilateral analyzer, and an integrated fusion network device. Figure 7 A flowchart illustrating the operation of a nipple detector according to various embodiments of the present disclosure is shown.

[0048] like Figure 7 As shown, the method may include: in S101, the nipple detector receives a main image of a human breast and a first auxiliary image; in S102, the nipple detector detects the nipple position based on the main image and the first auxiliary image; in S103, the nipple detector outputs the detected nipple position. In one embodiment, the main image may be an RCC image, and the first auxiliary image may be an image of the same side, such as an RMLO image.

[0049] Figure 8 A flowchart illustrating the operation process of an image registration device according to various embodiments of the present disclosure is shown. (See also...) Figure 8 As shown, the method may further include: S201, the image registration device receives the main image and the initial auxiliary image; S202, the initial auxiliary image is flipped; S203, the flipped initial auxiliary image is distorted towards the main image according to the breast contour to obtain a second auxiliary image; in S204, the image registration device outputs the second auxiliary image. In one embodiment, the initial auxiliary image received by the image registration device may be a bilateral image, such as an LCC image.

[0050] Figure 9 A schematic flowchart illustrating the operation of a same-side analyzer according to various embodiments of the present disclosure is shown. (See also:) Figure 9 As shown, the method may include: in S301, the ipsilateral analyzer receives the main image, the first auxiliary image, and the nipple position; in S302, the ipsilateral analyzer generates and outputs a first probability map (e.g., an Ipsi-prob map) of the main image based on the main image, the first auxiliary image, and the nipple position.

[0051] Figure 10 A schematic flowchart illustrating the operation of a two-sided analyzer according to various embodiments of the present disclosure is shown. (Reference) Figure 10 As shown, the method may include: in S401, the two-sided analyzer receives the main image, the second auxiliary image, and the nipple position; in S402, the two-sided analyzer generates and outputs a second probability map (e.g., a Bi-prob map) of the main image based on the main image, the second auxiliary image, and the nipple position.

[0052] Figure 11 A schematic flowchart illustrating the operation of an integrated converged network device according to various embodiments of the present disclosure is shown. (Reference) Figure 11 As shown, the method may further include: in S501, the integrated converged network device receives the main image, the first probability map, and the second probability map; in S502, a converged probability map is generated and output based on the main image, the first probability map, and the second probability map.

[0053] The combined analysis can be applied to all views in the available dataset to generate lump detection on each view. In practice, the available datasets include public and internal datasets. The public dataset is the Digital Database of Mammary Screening (DDSM) dataset, which has been widely used for mammographic lesion detection. The DDSM dataset contains 2,620 patient cases, each with four views of mammograms (two views per breast: left CC and MLO, right CC and MLO). After excluding some defective / damaged cases, 2,578 cases (10,312 images in total) were applied to the published apparatus. All cases were randomly split into training, validation, and test sets in an approximately 8:1:1 ratio, generating 8,256, 1,020, and 1,036 images in their respective sets. The internal dataset was obtained from a hospital to validate the proposed method. The internal mammogram dataset contains 2,749 cases, including normal, cancerous, and benign cases, closely approximating the actual distribution. Lesion areas were first annotated by two radiologists and then reviewed by a senior radiologist. Similar to the splitting strategy on the public DDSM dataset, all cases were randomly split in an 8:1:1 ratio, with 8,988, 1,120, and 1,120 images in the training, validation, and test sets, respectively. It should be noted that some cases had mammograms taken on multiple dates.

[0054] To facilitate learning symmetry constraints from the bilateral images using a DNN, identical view image input pairs (e.g., two CC view images or two MLO view images) can be registered. In one embodiment, as shown in Figure 2, the input image pair can be two CC view images, such as an RCC image and an LCC image. It should be noted that in other embodiments, the input image pair can be two MLO view images, such as an RMLO and an LMLO. Further, the auxiliary image input to the image register can be horizontally flipped and then warped toward the main image according to the breast contour. Additionally, the nipple position can be used to roughly align the two MLO images before warping. An example of flipped and warped CC views is shown in... Figure 2 As shown in the image.

[0055] The model on each dataset is independently trained based on a pre-trained ImageNet model. A state-of-the-art (SOTA) free-response operating characteristic (FROC) can be chosen as the evaluation metric for comparison with known methods. A tumor can be assumed to have been successfully identified when the intersection-over-union (IoU) ratio between the predicted output and the correct annotation is greater than 0.2.

[0056] The bilateral analyzer, the ipsilateral analyzer, and a downgraded version of the ipsilateral analyzer, such as "IpsiDualNet w / o Relation Blocks" (two feature streams from the main and auxiliary images are directly concatenated after the RoI alignment stage), are trained and tested using both ipsilateral and bilateral images. Table 1 shows the results of DDSM (e.g., ablation study results), indicating that the bilateral analyzer consistently achieves the highest recall score on bilateral images, while the ipsilateral analyzer generally performs better on ipsilateral images. Furthermore, the ipsilateral analyzer outperforms the downgraded version on ipsilateral images, demonstrating that the designed relation module significantly improves the performance of the ipsilateral analyzer. Therefore, the bilateral analyzer and the ipsilateral analyzer can be applied, respectively, to bilateral and ipsilateral analysis in devices for multi-view mammography mass recognition. Thus, the segmentation network may be better suited to symmetric learning as it may preserve spatial information, while the detection network may be better suited to RoI-based relation learning.

[0057] Table 1

[0058]

[0059] Furthermore, the impact of different geometric features on the same-side analyzer was investigated, including the shape and location of the RoI, the virtual nipple, and the distance from the RoI to the nipple. Table 2 shows the impact of different geometric features on the prediction performance of the internal dataset. As shown in Table 2, the results indicate that geometric features based on the distance from the RoI to the nipple may produce the best performance for the same-side analyzer.

[0060] Table 2

[0061]

[0062] Table 3 compares the performance of various methods on the DDSM dataset. Different single-view and two-view methods were selected as competing methods, reporting evaluations of DDSM on normal patient data using the FROC metric. The disclosed three-view method demonstrates a higher recall score than any existing single-view or two-view method. In Table 3, CVR-RCNN represents a Cross-View Relation Region-based Convolutional Neural Network, and CBN represents a Contrasted Bilateral Network.

[0063] Table 3

[0064]

[0065]

[0066] As shown in Table 4, various methods were also tested on the internal dataset. The disclosed three-view method again achieved the highest recall score on all FPIs. Furthermore, due to the higher image quality, the proposed method achieved a significantly higher recall score on the internal dataset than on the DDSM dataset. Additionally, it should be noted that the disclosed device is capable of accepting mammograms with incomplete views.

[0067] Table 4

[0068]

[0069] Compared with existing mammography lesion detection systems and methods, the disclosed apparatus and methods demonstrate the following exemplary advantages.

[0070] Based on the disclosed apparatus and method, a first three-view DNN architecture is employed to fully aggregate information from all views, thereby performing common end-to-end ipsilateral and bilateral analyses. Furthermore, a novel relational network designed in tandem with a DNN-based nipple detector is developed to incorporate cross-view geometric constraints, thereby improving the accuracy of the analysis. Moreover, state-of-the-art (SOTA) FROC performance is achieved on both the DDSM dataset and internal datasets using the disclosed apparatus and method.

[0071] The embodiments provided in this invention are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and similar or identical components between embodiments can be referenced interchangeably. The apparatuses disclosed in the embodiments are described simply because they correspond to the disclosed methods. Details of the disclosed apparatuses can be found in the corresponding sections of the method description.

[0072] Those skilled in the art will also recognize that the units and algorithm steps described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the elements and steps of various examples have been generally described in terms of their functionality. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. For each specific application, those skilled in the art can use different methods to implement the described functions. However, such implementations should not be considered beyond the scope of this disclosure.

[0073] The steps of the methods or algorithms described in the embodiments disclosed herein can be implemented directly by hardware, processor-executable software modules, or a combination of both. The software modules can reside in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or other storage media, as is well known in the art.

[0074] The foregoing description of the disclosed embodiments enables those skilled in the art to make or use this disclosure. Various modifications to the described embodiments will be apparent to those skilled in the art. The general principles herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure should not be limited to the embodiments described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for mammographic multi-view mass identification, applied to an apparatus for mammographic multi-view mass identification, characterized in that, The method comprises: receiving a main image, a first auxiliary image and a second auxiliary image, wherein the main image and the first auxiliary image are images of one breast of a person, and the second auxiliary image is an image of another breast of the person; detecting a nipple position based on the main image and the first auxiliary image; generating a first probability map of the main image using a same-side analyzer based on the main image, the first auxiliary image and the nipple position, wherein the same-side analyzer is based on a Faster-RCNN detection architecture and comprises at least one relational block for modeling a relationship between a region of interest (ROI) in the main image and the first auxiliary image based on appearance similarity and geometric similarity, the geometric similarity comprises a distance from the ROI to the nipple, and during a training process, the same-side analyzer is optimized using a focal loss and a distance cross joint loss; generating a second probability map of the main image using a bilateral analyzer based on the main image, the second auxiliary image and the nipple position, wherein the bilateral analyzer is based on a DeepLab v3+ structure and comprises a first Siamese input mode and a pixel-level focal loss function, the first Siamese input mode comprises two weight-shared hollow convolution modules for extracting feature maps from the main image and the second auxiliary image, and during a training process, the bilateral analyzer is optimized using a pixel-level focal loss function; generating and outputting a fusion probability map based on the main image, the first probability map and the second probability map by an integrated fusion network device, wherein the integrated fusion network device uses a frozen ResNet-50 backbone network, and the fusion probability map comprises a lesion possibility of each pixel point on the main image.

2. The method of claim 1, wherein, The device for mammography multi-view mass identification comprises a nipple detector, a same-side analyzer, a bilateral analyzer and an integrated fusion network device, and the method comprises: receiving the main image and the first auxiliary image by the nipple detector; detecting a nipple position based on the main image and the first auxiliary image by the nipple detector; receiving the main image, the first auxiliary image and the nipple position obtained by the nipple detector by the same-side analyzer; generating and outputting the first probability map of the main image by the same-side analyzer; receiving the main image, the second auxiliary image and the nipple position by the bilateral analyzer; generating and outputting the second probability map of the main image by the bilateral analyzer; receiving the main image, the first probability map and the second probability map by the integrated fusion network device; and generating and outputting the fusion probability map by the integrated fusion network device. The device further comprises an image registrator, and the method further comprises:

3. The method of claim 1, wherein, receiving the main image and an initial auxiliary image by the image registrator; flipping the initial auxiliary image by the image registrator; and warping the flipped initial auxiliary image to the main image according to a breast contour to obtain the second auxiliary image by the image registrator. ​ ​ 4. The method of claim 3, wherein : The method selects the main image from a left cranial-caudal (LCC) image, a right cranial-caudal (RCC) image, a left mediolateral oblique (LMLO) image, and a right mediolateral oblique (RMLO) image; and When the main image is the LCC image, the first auxiliary image is the LMLO image, and the initial auxiliary image is the RCC image; when the main image is the RCC image, the first auxiliary image is the RMLO image, and the initial auxiliary image is the LCC image; when the main image is the LMLO image, the first auxiliary image is the LCC image, and the initial auxiliary image is the RMLO image; and when the main image is the RMLO image, the first auxiliary image is the RCC image, and the initial auxiliary image is the LMLO image.

5. The method of claim 2, wherein : The first probability map includes mass information of a pixel in the main image generated by the ipsilateral analyzer; The second probability map includes mass information of a pixel in the main image generated by the bilateral analyzer; and The mass information of the pixel in the main image includes a texture density determined by a brightness of the pixel.

6. The method of claim 5, wherein, The method further includes: calculating, by each of the ipsilateral analyzer and the bilateral analyzer, a similarity of adjacent pixels from an input image to determine a region including pixels with similar brightness; determining, by each of the ipsilateral analyzer and the bilateral analyzer, a shape, an area size, and a relative position of the determined region relative to the nipple position; and determining, by each of the ipsilateral analyzer and the bilateral analyzer, a region in which at least one of the brightness, the shape, the area size, and the relative position deviates from a preset value as a RoI. Each of the first probability map and the second probability map further includes a lesion possibility of each pixel on the main image, and the method further includes:

7. The method of claim 2, wherein : calculating, by each of the ipsilateral analyzer and the bilateral analyzer, the lesion possibility of each pixel on the main image from an input image; and determining, by each of the ipsilateral analyzer and the bilateral analyzer, a pixel region in which the lesion possibility exceeds a preset possibility as a RoI. The fusion probability map includes the lesion possibility of each pixel on the main image, and the method further includes: calculating, by the integrated fusion network device, the lesion possibility of each pixel on the main image based on the first probability map, the second probability map, and the main image.

8. The method of claim 2, wherein : The apparatus includes: a nipple detector configured to receive a main image and a first auxiliary image of a breast of a person, and to detect a nipple position based on the main image and the first auxiliary image; 9. A device for identifying lumps in multi-view mammography, characterized in that, ​ ​ The ipsilateral analyzer is configured to receive the main image, the first auxiliary image, and the nipple position obtained by the nipple detector, and generate and output a first probability map of the main image, wherein the ipsilateral analyzer is based on a Faster-RCNN detection architecture, and includes at least one relationship block for modeling a relationship between a region of interest (ROI) in the main image and the first auxiliary image based on appearance similarity and geometric similarity, the geometric similarity including a distance from the ROI to the nipple, and during a training process, the ipsilateral analyzer is optimized using a focal loss and a distance cross joint loss; The bilateral analyzer is configured to receive the main image, the nipple position, and a second auxiliary image, and generate and output a second probability map of the main image, wherein the bilateral analyzer is based on a DeepLab v3+ structure, and includes a first Siamese input mode and a pixel-level focal loss function, the first Siamese input mode including two weight-shared atrous convolution modules for extracting feature maps from the main image and the second auxiliary image, and during a training process, the bilateral analyzer is optimized using a pixel-level focal loss function; The integrated fusion network device is configured to receive the main image, the first probability map, and the second probability map, and generate and output a fusion probability map, wherein the integrated fusion network device uses a frozen ResNet-50 backbone network, and the fusion probability map includes a lesion likelihood of each pixel point on the main image.

10. The apparatus of claim 9, wherein : The main image is selected from a left cranial caudal (LCC) image, a right cranial caudal (RCC) image, a left mediolateral oblique (LMLO) image, and a right mediolateral oblique (RMLO) image; and When the main image is the LCC image, the first auxiliary image is the LMLO image; when the main image is the RCC image, the first auxiliary image is the RMLO image; when the main image is the LMLO image, the first auxiliary image is the LCC image; and when the main image is the RMLO image, the first auxiliary image is the RCC image.

11. The apparatus of claim 9, wherein, The apparatus further includes: An image registerer configured to receive the main image and an initial auxiliary image, flip the initial auxiliary image, and warp the flipped initial auxiliary image according to a breast contour to the main image to obtain a second auxiliary image.

12. The apparatus of claim 11, wherein : The main image is selected from a left cranial caudal (LCC) image, a right cranial caudal (RCC) image, a left mediolateral oblique (LMLO) image, and a right mediolateral oblique (RMLO) image; and When the main image is the LCC image, the first auxiliary image is the LMLO image, and the initial auxiliary image is the RCC image; when the main image is the RCC image, the first auxiliary image is the RMLO image, and the initial auxiliary image is the LCC image; when the main image is the LMLO image, the first auxiliary image is the LCC image, and the initial auxiliary image is the RMLO image; and when the main image is the RMLO image, the first auxiliary image is the RCC image, and the initial auxiliary image is the LMLO image.

13. The apparatus of claim 9, wherein : The nipple detector is a nipple detector based on a deep neural network (DNN).

14. The apparatus of claim 9, wherein : The first probability map includes mass information of a pixel point in the main image generated by the ipsilateral analyzer; and The second probability map includes mass information of a pixel point in the main image generated by the bilateral analyzer.

15. The apparatus of claim 14, wherein : The mass information of the pixel point in the main image includes a texture density determined by the brightness of the pixel point.

16. The apparatus of claim 15, wherein : Each of the ipsilateral analyzer and the bilateral analyzer is configured to calculate the similarity of adjacent pixel points according to an input image to determine a region including pixel points with similar brightness. Each of the ipsilateral analyzer and the bilateral analyzer is further configured to determine the shape, area size, and relative position of the determined region relative to the nipple position. And Each of the ipsilateral analyzer and the bilateral analyzer is further configured to determine a region in which at least one of the brightness, shape, area size, and relative position deviates from a preset value as a RoI.

17. The apparatus of claim 15, wherein : Each of the first probability map and the second probability map further includes a lesion possibility of each pixel point on the main image. Each of the ipsilateral analyzer and the bilateral analyzer is further configured to calculate the lesion possibility of each pixel point on the main image according to an input image. And Each of the ipsilateral analyzer and the bilateral analyzer is further configured to determine a region in which the pixel points have a lesion possibility exceeding a preset possibility as a RoI.

18. The apparatus of claim 9, wherein : The fusion probability map includes a lesion possibility of each pixel point on the main image; and The integrated fusion network device calculates the lesion possibility of each pixel point on the main image based on the first probability map, the second probability map, and the main image.

19. The apparatus of claim 9, wherein In the bilateral analyzer, each hole convolution module of the first Siamese input mode includes a 50-layer residual network (ResNet-50) as a backbone, and a non-local (NL) block is arranged between the fourth and fifth levels of the ResNet-50; and the NL blocks of the two hole convolution modules share weights, and the input dimension and output dimension of the NL block remain consistent.

20. The apparatus of claim 9, wherein : In the ipsilateral analyzer, the relationship block calculates geometric similarity based on the following formula: ； wherein, represents a geometric embedding operation, represents a geometric factor of the i-th RoI of the primary image, represents a geometric factor of the j-th RoI of the secondary image, is a RoI-to-nipple distance of the i-th RoI of the primary image, is a RoI-to-nipple distance of the j-th RoI of the secondary image, is a width of the i-th RoI of the primary image, is a width of the j-th RoI of the secondary image, is a height of the i-th RoI of the primary image, is a height of the j-th RoI of the secondary image.

Images