Systems and methods for analyzing two-dimensional and three-dimensional image data - Patents.com

Abstract

The present disclosure provides a computerized method for determining malignancy likelihood for breast tissue of a patient, the method including receiving, using a computer processor, an image of the breast tissue, providing the image of the breast tissue to a model including a trained neural network, receiving an indicator from the model, and outputting a report including the indicator to at least one of a memory or a display, wherein the trained neural network is pre-trained to train a first neural network, initialize a second neural network based on the first neural network, train the second neural network, and output the second neural network as the trained neural network.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 953,033, filed December 23, 2019, the entire contents of which are incorporated herein by reference.

[0002] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT: Not applicable. [Background technology]

[0003] Breast cancer remains a global challenge, resulting in over 600,000 deaths in 2018. To achieve early cancer detection, health organizations worldwide recommend screening mammography, which is estimated to reduce breast cancer mortality by 20% to 40%. Despite the value of screening mammography, significant false positive and false negative rates, as well as heterogeneity in the availability of skilled readers, leave opportunities to improve quality and access.

[0004] Despite technological improvements such as digital breast tomosynthesis (DBT), studies reviewing mammograms in which cancer was detected estimate that signs of cancer presence are visible 20% to 60% of the time on previous exams that were deemed normal. DBT provides three-dimensional (3D) imaging of the breast and can generate 10 to 150 or more two-dimensional (2D) images representing thin slices of the breast (i.e., approximately 1 mm thick) per view. Due to the large number of slices involved, DBT requires additional time compared to 2D imaging (e.g., 2D full-field digital mammography), thus forcing radiologists to read at high speeds. However, 2D imaging may not provide health care professionals with as much detail as 3D imaging for predicting the presence of cancer.

[0005] It would therefore be desirable to have a system and method for more efficiently and accurately analyzing 2D and / or 3D mammography data and for uniformly estimating malignancies and / or lesions using the 2D and / or 3D mammography data. Summary of the Invention

[0006] The present disclosure provides systems and methods for efficiently and accurately analyzing 2D and / or 3D mammography data and uniformly estimating malignancies and / or lesions using the 2D and / or 3D mammography data. In one non-limiting aspect, the present disclosure provides a computerized method for predicting the likelihood of malignancy in a patient's breast tissue. The method includes, using a computer processor, receiving an image of breast tissue, providing the image of breast tissue to a model including a trained neural network, receiving an indicator from the model, and outputting a report including the indicator to at least one of a memory or a display, where the trained neural network is pre-trained as follows: training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, each annotated patch including a patch-level label and a portion of the two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; and performing a step of ... The method includes: training a second neural network based on the first set of annotated images, where at least one annotated image in the first set of annotated images includes a two-dimensional image included in a second group of two-dimensional images, at least one bounding box, and at least one malignancy label associated with the at least one bounding box; training the second neural network based on a second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image included in a third group of two-dimensional images and an image-level malignancy likelihood score; and outputting the second neural network as a learned neural network.

[0007] In this method, the image of the breast tissue may be a two-dimensional digital mammogram.

[0008] In the method, the image of breast tissue may be a composite two-dimensional image generated based on a digital breast tomosynthesis image. In some configurations, the digital breast tomosynthesis image includes a plurality of slices, the composite two-dimensional image includes an array of pixels, and pixel intensity values ​​of at least some of the pixels in the array of pixels are pre-generated by providing at least some of the slices to a second trained neural network, receiving a plurality of indicators from the second trained neural network, each indicator including a likelihood score of malignancy, and determining a target pixel intensity for a target pixel in the array of pixels based on a first indicator in the plurality of indicators, the first indicator including a likelihood score of malignancy that is greater than a second indicator in the plurality of indicators. In some configurations, pixel intensity values ​​of at least some of the pixels in the array of pixels may be pre-generated by determining, for a second target pixel in the array of pixels, that none of the pixels in the plurality of pixels includes a pixel location associated with the second target pixel, and setting a pixel intensity of the second target pixel to be the same as a default pixel in a default slice in the plurality of slices, the pixel location associated with the target pixel. In some configurations, each slice in the plurality of slices may be associated with a position value, and a default slice may be associated with an intermediate position value. In some configurations, the digital breast tomosynthesis image may include a plurality of slices, and each slice in the plurality of slices may be associated with a position value, and the composite two-dimensional image may be generated based on a subset of the plurality of slices associated with position values ​​within a predetermined range of position values. In some configurations, the subset of the plurality of slices may comprise approximately 80 percent of the plurality of slices.

[0009] In the method, the second neural network may include a backbone portion, a regression portion, and a classification portion, and the method may further include training the regression portion based on the first set of annotated images without retraining the backbone portion and the classification portion. In some configurations, training the second neural network based on the second set of annotated images may include training the backbone portion and the classification portion without training the regression portion.

[0010] In the method, the first neural network may include a first plurality of weights, the second neural network may include a backbone portion including a second plurality of weights, and initializing the second neural network based on the first neural network may include setting weights included in the second plurality of weights to be the same as weights included in the first plurality of weights. In some configurations, the first neural network may include a deep residual network, and the second neural network may include a single-stage object detection network.

[0011] In this method, the indicator may include a patient malignancy likelihood score.

[0012] In this method, the indicator may include an area of ​​the image of the breast tissue.

[0013] In another non-limiting aspect, the present disclosure provides a system for assessing malignancy likelihood for breast tissue of a patient, the system comprising: a memory configured to store an image of the breast tissue; and a processor configured to access the memory, the processor configured to provide the image of the breast tissue to a model including a trained neural network, receive an indicator from the model, and output a report including the indicator to at least one of the memory or a display, the trained neural network being pre-trained as follows: training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, each annotated patch including a patch-level label and a portion of the two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; and annotating the annotated patch based on the set of annotated patches. the second neural network is pre-trained to: train a second neural network based on a first set of annotated images, where at least one annotated image in the first set of annotated images includes a two-dimensional image included in a second group of two-dimensional images, at least one bounding box and at least one malignancy label associated with the at least one bounding box; train the second neural network based on a second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image included in a third group of two-dimensional images and an image-level malignancy likelihood score; and output the second neural network as a learned neural network.

[0014] In this method, the image of the breast tissue may be a two-dimensional digital mammogram.

[0015] In the method, the image of breast tissue may be a composite two-dimensional image generated based on a digital breast tomosynthesis image. In some configurations, the digital breast tomosynthesis image may include a plurality of slices, and the composite two-dimensional image may include an array of pixels, and the pixel intensity values ​​of at least some of the pixels in the array of pixels have been pre-generated by providing at least some of the slices to a second trained neural network, receiving a plurality of indicators from the second trained neural network, each indicator including a likelihood score of malignancy, and determining a target pixel intensity for a target pixel in the array of pixels based on a first indicator in the plurality of indicators, the first indicator including a likelihood score of malignancy that is greater than a second indicator in the plurality of indicators. In some configurations, the pixel intensity values ​​of at least some of the pixels in the array of pixels may be pre-generated by determining, for a second target pixel in the array of pixels, that none of the pixels in the plurality of pixels includes a pixel location associated with the second target pixel, and setting a pixel intensity for the second target pixel to be the same as a default pixel in a default slice in the plurality of slices, the default pixel including a pixel location associated with the second target pixel. In some configurations, each slice in the plurality of slices may be associated with a position value, and a default slice may be associated with an intermediate position value. In some configurations, the digital breast tomosynthesis image may include a plurality of slices, and each slice in the plurality of slices is associated with a position value, and the composite two-dimensional image may be generated based on a subset of the plurality of slices associated with position values ​​within a predetermined range of position values. In some configurations, the subset of the plurality of slices may comprise approximately 80 percent of the plurality of slices.

[0016] In the method, the second neural network may include a backbone portion, a regression portion, and a classification portion, and the method may further include training the regression portion based on the first set of annotated images without retraining the backbone portion and the classification portion. In some configurations, training the second neural network based on the second set of annotated images may include training the backbone portion and the classification portion without training the regression portion.

[0017] In the method, the first neural network may include a first plurality of weights, the second neural network may include a backbone portion including a second plurality of weights, and initializing the second neural network based on the first neural network may include setting weights included in the second plurality of weights to be the same as weights included in the first plurality of weights.

[0018] In the method, the first neural network may include a deep residual network and the second neural network may include a single-stage object detection network.

[0019] In this method, the indicator may include a patient malignancy likelihood score.

[0020] In this method, the indicator may include an area of ​​the image of the breast tissue.

[0021] In yet another non-limiting aspect, the present disclosure provides a method for determining a malignancy likelihood score for a patient's breast tissue, the method comprising the steps of receiving an image of the breast tissue, providing the image of the breast tissue to a model including a trained neural network, receiving a malignancy likelihood score from the model, and outputting a report including the patient malignancy likelihood score to at least one of a memory or a display, where the trained neural network is pre-trained as follows: training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, each annotated patch including a malignancy likelihood score and a portion of a two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; and performing a malignancy likelihood score determination on the first set of annotated images, the first set of annotated images including the annotated patches. The pre-training method includes: training a second neural network based on the first set of annotated images, where at least one annotated image in the first set of annotated images includes a two-dimensional image in the second group of two-dimensional images, at least one bounding box and at least one malignancy likelihood score associated with the at least one bounding box; training the second neural network based on the second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image in the third group of two-dimensional images and an image-level malignancy likelihood score; and outputting the second neural network as a learned neural network.

[0022] In yet another non-limiting aspect, the present disclosure provides a system for generating a malignancy likelihood score for a patient's breast tissue, the system comprising: a memory configured to store an image of the breast tissue; and a processor configured to access the memory, the processor configured to provide the image of the breast tissue to a model including a trained neural network, receive a malignancy likelihood score from the model, and output a report including the malignancy likelihood score to at least one of the memory or a display, the trained neural network being pre-trained as follows: training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, each annotated patch including a malignancy likelihood score and a portion of the two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; and annotating the annotated images based on the set of annotated patches. the second neural network is pre-trained to: train a second neural network based on a first set of images, where at least one annotated image in the first set of annotated images includes a two-dimensional image included in a second group of two-dimensional images, at least one bounding box and at least one malignancy likelihood score associated with the at least one bounding box; train the second neural network based on a second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image included in a third group of two-dimensional images and an image-level malignancy likelihood score; and output the second neural network as a learned neural network. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a block diagram illustrating an exemplary X-ray imaging system. [Diagram 2]FIG. 2 is a block diagram illustrating an example embodiment of a model for generating a region of interest in a two-dimensional image. [Diagram 3] 1 is an exemplary process 300 for training a neural network. [Figure 4] 1 is an exemplary process for generating a malignancy indicator based on 2D breast images. [Diagram 5] 1 is an exemplary process for creating a composite image using a model trained to detect ROIs in 2D slices of 3D tomosynthesis data. [Figure 6] 4 is another exemplary process for training a neural network. [Figure 7] 1 is an exemplary process for generating a malignancy indicator based on 3D breast images. [Figure 8] Receiver operating characteristic (ROC) plot summarizing the results of the "index" component of the reader study. [Figure 9] 2 is a ROC plot summarizing the second component of the reader study with the "pre-index" examination from the same patients. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] As described herein, exemplary systems and methods are provided for efficiently and uniformly displaying relevant regions of two-dimensional (2D) mammography data and / or three-dimensional (3D) tomography data, and for determining malignancy in the 2D mammography data and / or 3D tomography data using machine learning techniques.

[0025] There may be approaches other than the systems and methods described below for analyzing relevant regions of 2D mammography data and / or 3D tomosynthesis data and / or determining malignancy of tumors and / or lesions in breast tissue, but all of them have drawbacks. One approach is for a human practitioner to label the relevant regions of all slices contained in a 3D mammogram. Human-based approaches can be costly and slow, especially for 3D mammograms, as it takes time for a human practitioner to label each 2D slice. Furthermore, human-based approaches can be inconsistent due to different preferences / expertise levels among human practitioners and different amounts of information (i.e., radiology reports, pathology reports, etc.) available to assist the human labeling.

[0026] Another approach is to train a machine learning model that receives the entire 3D tomography dataset (i.e., all slices contained in the 3D tomography dataset) and outputs a malignancy likelihood score indicative of the malignancy of tumors and / or lesions present in the breast tissue. This approach may be infeasible due to data size and memory limitations of data processing systems and may be prone to overfitting.

[0027] Yet another approach involves randomly selecting a slice from the set of 2D slices, or a default slice (i.e., the 25th slice or the middle slice) of the set of 2D slices, and training the machine learning model to output a malignancy likelihood score based on this slice. This approach may also be prone to overfitting, since the slice is likely to not include tumors and / or lesions, even if they are present.

[0028] Yet another approach is to train a machine learning model using only weakly or strongly labeled training data, which may not provide sufficiently accurate predictions of malignant and / or associated regions to be clinically useful. Training only on strongly labeled data may limit the amount of data on which the model can be trained, thereby limiting accuracy, while training only on weakly labeled data may lead to overfitting and poor generalization performance.

[0029] Referring to FIG. 1, an example of an X-ray imaging system 100, such as a 3D digital breast tomosynthesis (DBT) system, is shown. The X-ray imaging system 100 can include an X-ray source assembly 108 coupled to a first end 110 of an arm 102. An X-ray detector assembly 112 can be coupled proximate to an opposing end 114. The X-ray source assembly 108 can extend substantially perpendicular to the arm 102 and can be oriented toward the X-ray detector assembly 112. The X-ray detector assembly 112 also extends from the arm 102, and is configured to receive X-ray radiation generated by the X-ray source assembly 108, passing through the breast, and incident on the X-ray detector assembly 112. A breast support plate 116 and a breast compression plate 118 are disposed between the X-ray source assembly 108 and the X-ray detector assembly 112. The X-ray source assembly 108 can be fixed or movable. The X-ray imaging system 100 can generate a reconstructed image including the 3D DBT data. The 3D DBT data may include multiple 2D slices. In some configurations, the reconstructed image may include 3D tomography data including multiple 2D slices having a thickness of about 1 mm. The 2D slices may be used to create a composite 2D image, as described below. In some configurations, the x-ray imaging system 100 may generate intermediate two-dimensional "slabs" that represent maximum intensity projections of a subset of the 2D slices, for example. For example, a maximum intensity projection slab may be generated from 10 slices, multiple slabs may be generated from multiple slices, such as 10 slabs from 100 slices, etc. The reconstructed image may be input to a computer 126 that stores the image in a mass storage device 128, which may include a memory. The computer 126 may also provide instructions to the x-ray imaging system 100 to control the generation of the reconstructed image.

[0030] 2, an example embodiment of a model 200 for generating a region of interest (ROI) for an input two-dimensional (2D) image 204 is shown. The ROI may be referred to as an indicator. In some configurations, the 2D image 204 may be a native 2D mammogram image (e.g., an image generated by a 2D full-field digital mammography system), a synthetic 2D mammogram image (e.g., a 2D image generated based on 3D image data), and / or a 2D slice of 3D DBT data.

[0031] The model 200 can accept a 2D image 204 and output any number of ROIs, such as a first ROI including a first area 208A and a first score 208B, and a second ROI including a second area 212A and a second score 212B. In some configurations, each ROI can be associated with a slice number indicating the 2D slice from which the ROI was generated, such as the fourth slice of a set of 75 2D slices. As described below, the slice number can be used in selecting and / or combining the ROIs to create a composite image. Depending on the characteristics of the 2D image 204, such as when no regions in the 2D image 204 are deemed to be sufficiently indicative of possible malignancy, zero ROIs may be output. Each ROI can include areas that may be subregions of the 2D image 204. As described above, each 2D image can be formatted as an array of pixels. A subregion can be a subset of the array of pixels. In some configurations, the model 200 can include one or more neural networks configured to detect objects in the 2D image. The object can be a ROI.

[0032] In some configurations, the model 200 can output an ROI that follows a predetermined shape. For example, a rectangular bounding box can be used to enclose a potential candidate for a tumor or lesion. It is believed that an irregular shape (e.g., a "blob" of pixels) can be used to better outline a potential tumor or lesion. When creating a training database of ROIs, one or more human practitioners may find it more intuitive to use a rectangular bounding box than other shapes. A neural network that uses a segmentation mask-based approach to identify objects can be used to output a predicted ROI of irregular shape. The model 200 can then be trained to identify a rectangular shaped ROI that includes a subarray of pixels contained in the 2D image 204. The pixels of the ROI can include one or more color intensity values ​​(e.g., a white intensity value) and a location within the 2D image 204 (e.g., a pixel at a given (x,y) location within a 2000x1500 pixel slice). It is understood that while some mammography imaging systems generate grayscale images of breast tissue, the model can be used with colored 2D images.

[0033] In addition to the subarray of pixels, the ROI may include a relevance score indicating how relevant the subarray of pixels is to determining a malignancy likelihood score. As described in more detail below, the relevance score may be used to create a composite 2D image using one or more ROIs and / or to generate a malignancy likelihood score for a patient. The relevance score may be selected from a numerical range, such as from 0 to 1. When identifying ROIs for a training dataset, a human medical practitioner may assign a relevance score for each ROI within a range of values. The human medical practitioner may assign relevance scores using different scales, such as 0 to 100 (higher scores with a higher likelihood of malignancy), which may then be normalized to a relevance score range used by the model 200. In some configurations, the human medical practitioner may identify ROIs as benign to better train the model 200 to identify potentially malignant ROIs.

[0034] In some configurations, model 200 may include a neural network, such as a recurrent neural network. To train model 200, the model may be trained using a training data set that includes 2D data consisting of slices from a set of full-field digital mammography (FFDM) images and / or 3D tomosynthesis images, as well as pre-specified ROIs (e.g., by one or more medical personnel).

[0035] At least a portion of the 2D data may be strongly annotated data. The strongly annotated data may include a predefined ROI. A human medical practitioner may identify the ROI by examining a given 2D image (e.g., a 2D slice or a 2D FFDM image), outlining a potential region of interest using a predefined shape, such as a rectangular box, and assigning a relevance score to the predefined shape based on medical expertise and / or experience in evaluating tumors and / or lesions. Alternatively, the relevance score may be assigned based on a pathology result indicating whether the lesion is malignant or not.

[0036] At least a portion of the 2D data may be weakly annotated data. Weakly annotated data may include image-level relevance scores that do not include any bounding box or other contoured areas in the image. To generate the weakly annotated data, a medical professional may annotate a 2D image as malignant (e.g., "1") or non-malignant (e.g., "0"). For example, it may be determined through pathology results that a patient's right breast contains a malignant tumor, and then all mammography images of the right breast may be given a label of "1". A large training database may be generated by one or more medical professionals identifying (e.g., annotating) ROIs in 2D images obtained from slices of multiple FFDM images or 3D tomosynthesis images (e.g., images of multiple patients) and annotating the entire 2D image as malignant (e.g., "1") or non-malignant.

[0037] An advantage of using FFDM images in the training data is that annotated FFDM images are currently more publicly available than annotated 3D tomosynthesis images. Furthermore, 2D images are easier to annotate than 3D tomosynthesis images, which require annotating many individual slices in each 3D tomosynthesis image. Once trained, model 200 can receive input 2D slices and output one or more ROIs, where each ROI includes an estimated relevance score and a subarray of pixels from the input 2D slice.

[0038] Model 200 may include multiple layers, such as convolutional layers. It is understood that some configurations of model 200 may have a different number of layers, a different arrangement of layers, or other differences. However, in all configurations, model 200 may receive an input 2D input slice and output any regions of interest associated with the input 2D input slice. Model 200 may be a single-stage detection network that includes one or more sub-networks. In some configurations, model 200 may include RetinaNet.

[0039] The model 200 may include a primary sub-network 216. The primary sub-network 216 may be a feed-forward neural network having one or more layers 218A-C. In some configurations, the primary sub-network 216 may include a deep residual network ("ResNet"). In some configurations, the deep residual network may be a ResNet-50 neural network. As described below, the primary sub-network 216 may be first trained independently and then integrated into the model 200. The secondary sub-network 220 may be built on top of the primary sub-network 216 to effectively create a single neural network, using the primary sub-network 216 as the backbone of the network. The primary sub-network 216 may be referred to as the backbone portion of the model 200. The secondary sub-network 220 can include multiple layers, including a first layer 222A, a second layer 222B, and a third layer 222C, although other numbers of layers (i.e., five layers) can be used and three layers are shown here for simplicity. For example, a ResNet-50 neural network can include 50 layers.

[0040] Each of the first layer 222A, the second layer 222B, and the third layer 222C may be a convolutional layer. Each layer may be composed of a number of building blocks (not shown). Each building block may include a number of parameter layers, such as three parameter layers, each parameter layer including a number (e.g., 256) of filters having a predetermined filter size (e.g., 3×3). The first layer 222A, the second layer 222B, and the third layer 222C may have associated output sizes, such as 144×144, 72×72, and 36×36, respectively. The output sizes may vary between input slices based on pre-processing conditions and / or parameters. As the output size decreases between layers of the secondary sub-network 220, the number of filters in the parameter layers may increase proportionally, i.e., halving the output size will double the number of filters. The secondary sub-network may also include a global average pooling layer coupled to the final layer (i.e., the third layer 222C), a fully connected layer coupled to the global average pooling layer, and a softmax layer coupled to the fully connected layer and having an output size of 1×1 (i.e., a single value).

[0041] The model 200 may include multiple tertiary sub-networks, such as a first tertiary network 224A, a second tertiary network 224B, and a third tertiary network 224C. The tertiary networks 224A-C may each be coupled to a layer of the secondary sub-network 220. The first tertiary network 224A may be coupled to the first layer 222A, the second tertiary network 224B may be coupled to the second layer 222B, and the third tertiary network 224C may be coupled to the third layer 222C. Each tertiary network may receive features from a layer of the secondary sub-network 220 to detect tumors and / or lesions at different levels of scale.

[0042] Each tertiary network may include a box regression sub-network 226. The box regression sub-network 226 may include one or more convolutional layers 228A-B, each followed by a rectified linear (ReLU) activation, and a final convolutional layer 230 configured to output regression coordinates corresponding to anchors associated with a portion of one of the layers of the secondary sub-network 220 (and corresponding to an array of pixels of the input 2D image 204). The anchors may be predefined sub-arrays of the various layers of the secondary sub-network 220. The regression coordinates may represent predicted offsets between the anchors and predicted bounding boxes. For each bounding box included in the ROI, the set of regression coordinates (e.g., four regression coordinates) and the corresponding anchors may be used to calculate the bounding box coordinates. The model 200 may include a regression portion including box regression sub-networks included in the first tertiary network 224A, the second tertiary network 224B, and the third tertiary network 224C.

[0043] Each tertiary network may include a classification sub-network 232. The classification sub-network 232 may include one or more convolutional layers 234A-B, each followed by a ReLU activation, and a final convolutional layer 238 followed by a sigmoid activation that outputs a prediction of the presence of the object (i.e., the presence of malignancy and / or lesions). The classification sub-network 232 may be used to obtain one or more estimations of whether the patient has malignancy and / or lesions at various spatial locations of the 2D image 204. More specifically, each bounding box may be associated with an estimation score output by the classification sub-network. In some configurations, the value of each estimation score ranges from zero to one. One of the spatial locations may include an entire layer of the secondary sub-network 220, i.e., the first layer 222A. In this manner, the classification sub-network 232 may output an estimation of whether the patient has malignancy and / or lesions based on the 2D slices. It is contemplated that in a model trained to classify multiple types of malignant regions, such as multiple levels of malignancy (e.g., low-risk regions, high-risk regions, etc.), the final convolutional layer 238 may be followed by a softmax activation. The model 200 may include a classification portion including classification sub-networks contained in a first tertiary network 224A, a second tertiary network 224B, and a third tertiary network 224C.

[0044] The model 200 may include an output layer 250 to normalize data across different scales, calculate bounding box coordinates, and / or filter out low scoring bounding box predictions. The output layer 250 receives the output from the tertiary sub-networks 224A-C and outputs one or more ROIs, each ROI including an array of pixels scaled to the array size of the 2D image 204 and an associated score. The array of pixels may be a bounding box (e.g., a rectangular bounding box) calculated based on the regression coordinates and anchors. The output layer 250 may filter out any scores below a predefined threshold, e.g., 0.5. In some configurations, the output layer 250 may receive the output from the tertiary sub-networks 224A-C and output a single malignancy likelihood score. In some configurations, the single malignancy likelihood score may be selected to be the highest scoring bounding box core.

[0045] 2 and 3, a process 300 for training a neural network is shown. The neural network may be included in the model 200. The process 300 may be implemented as instructions on one or more memories included in a data processing system. The data processing system may further include one or more processors configured to communicate with and execute the instructions on the one or more memories. The one or more processors may be configured to access the memory, which may be included in the mass storage device 128, where the training data is stored.

[0046] At 304, the process 300 may receive a set of annotated patches. The annotated patches may be derived from a first group of two-dimensional images (e.g., native 2D mammograms). Each annotated patch may include a label and a portion of a two-dimensional image included in the first group of two-dimensional images. Each patch may include a portion of a two-dimensional image. In some configurations, to generate the set of annotated patches, the process 300 may randomly select raw patches of two-dimensional images and may randomly rotate the patches, randomly resize the patches by up to 20%, and / or randomly mirror the patches vertically. In some configurations, the process 300 may pre-process the raw patches by normalizing pixel values ​​to a predetermined range (e.g., the range [127.5, 127.5]). The randomization and / or pre-processing may be performed in advance. In some configurations, to generate patches containing lesions, the process 300 may select a random location within the lesion boundary generated by the medical professional as the center of the patch containing the lesion. Then, if the number of pixels in the resulting patch that contain the lesion mask is less than a predetermined number (e.g., 6 pixels), the patch can be discarded and a new patch can be sampled. In some configurations, the process 300 can remove patches that contain less than 10% breast foreground (e.g., as determined by Otsu's method and / or by a threshold method using the minimum pixel value in the image) from the set of annotated patches. In some embodiments, the label can indicate the type of lesion (e.g., mass, calcification, regional asymmetry, architectural distortion, or no lesion). In some embodiments, the label can indicate the type of malignancy (e.g., normal, benign, or malignant). The set of annotated patches can be referred to as strongly annotated. The process 300 can then proceed to 308.

[0047] At 308, the process 300 can train a first neural network based on the set of annotated patches. In some configurations, the primary sub-network 216 can include at least a portion of the first neural network. For example, the primary sub-network 216 can include at least some of the convolutional layers included in the first neural network, but not the classification layers included in the first neural network. The first neural network can be a ResNet-50 neural network. The first neural network can be initialized based on ImageNet pre-training weights. In some configurations, the process 300 can train the first neural network to classify a lesion type based on the patches included in the set of annotated patches that are labeled by the lesion type. In some configurations, the lesion type can be a mass, a calcification, a regional asymmetry, an architectural distortion, or no lesion. After the first neural network is trained to classify lesion types, the process 300 may train the first neural network to determine a malignancy type based on the patches in the set of annotated patches labeled with a malignancy type. In some configurations, the malignancy type may be normal, benign, or malignant. In some configurations, the first neural network may be trained using a batch size that is 16. In some configurations, the first neural network may be trained using an Adam optimizer with a learning rate of 1e5. The process 300 may then proceed to 312.

[0048] At 312, the process 300 can initialize a second neural network based on the first neural network. The second neural network can include one or more components of the model 200. In some configurations, the second neural network can be a single-stage object detection network (e.g., RetinaNet). In some configurations, the second neural network can include a backbone portion (e.g., the first sub-network 216), a regression portion (e.g., the box regression sub-networks included in the first tertiary network 224A, the second tertiary network 224B, and the third tertiary network 224C), and a classification portion (e.g., the classification sub-networks included in the first tertiary network 224A, the second tertiary network 224B, and the third tertiary network 224C). In some configurations, the process 300 can initialize the backbone portion based on the first neural network. The process 300 can initialize the weights included in the backbone portion based on the weights included in the first neural network. Thereafter, the process 300 can proceed to 316.

[0049] At 316, the process 300 may receive a first set of annotated images. The annotated images may be generated based on a second group of two-dimensional images (e.g., native 2D mammograms). Each annotated image may include one or more ROIs, each ROI including a label and a bounding box included in the annotated image. In some embodiments, the label may indicate a malignancy type (e.g., normal, benign, or malignant). In some configurations, the process 300 may pre-process the raw images by normalizing pixel values ​​to a predetermined range (e.g., in the range of [127.5, 127.5]) and / or normalizing the height of the images to a predetermined height (e.g., 1750 pixels). In some configurations, to generate the first set of annotated images, the process 300 may randomly resize the images by up to 15% and / or randomly mirror the images vertically. In some configurations, the process 300 may crop out background portions of the annotated images. The randomization and / or pre-processing can be done beforehand. The first set of annotated images can be referred to as strongly annotated. Then, process 300 can proceed to 320.

[0050] At 320, the process may train a second neural network based on the first set of annotated images. The process 300 may train the second neural network to generate an ROI including a malignancy type and a bounding box based on the first set of annotated images labeled with a malignancy type. In some configurations, the malignancy type may be normal, benign, or malignant. In some configurations, the first neural network may be trained using a batch size that is 1. In some configurations, the first neural network may be trained using an Adam optimizer with a learning rate of 1e5 and gradient norm clipping with a value of 0.001. The process 300 may then proceed to 324.

[0051] At 324, the process 300 may receive a second set of annotated images. The annotated images may be generated based on a third group of two-dimensional images (e.g., native 2D mammograms). Each annotated image in the second set of annotated images may include a 2D image and an image-level label. The image level may be a binary malignancy label (e.g., malignant or non-malignant). In some configurations, the process 300 may pre-process the raw images by normalizing pixel values ​​to a predetermined range (e.g., in the range [127.5, 127.5]) and / or normalizing image height to a predetermined height (e.g., 1750 pixels). In some configurations, to generate the second set of annotated images, the process 300 may randomly resize the images by up to 15% and / or randomly mirror the images vertically. In some configurations, the process 300 may crop out background portions of the annotated images. The randomization and / or pre-processing may be performed in advance. The second set of annotated images can be referred to as weakly annotated.

[0052] At 328, the process 300 can train a second neural network based on the second set of annotated images. The second neural network, previously used as an object detection model, can be converted to an image classification model by modifying the output layer 250 to output the maximum across all of the bounding box classification scores. In this way, the second neural network can be fully discriminative while allowing end-to-end training with binary labels. In some configurations, the process 300 can train the model using an Adam optimizer starting with a learning rate of 2.5e6, which is reduced by a factor of four at a given number of iterations (e.g., every 100,000 iterations, for a total of 300,000 iterations). In some configurations, the process 300 can determine that the second neural network is fully trained based on the AUC performance. For example, the process 300 can determine the AUC performance of the validation set of images every 4000 iterations, and if the AUC performance decreases or remains the same, the second neural network can be determined to be fully trained. In some configurations, the process 300 can fine-tune the fully trained model after determining that the model is fully trained. In some configurations, the process 300 can fine-tune the regression portion based on the first set of annotated images. More specifically, the process 300 can freeze the weights included in the backbone and classification portions of the trained model and train only the regression portion. In this way, the process 300 can adapt the regression portion to any changes in the weights of the backbone portion during previous training on the second set of annotated images where the regression portion is not updated.In some configurations, the process 300 can train the learned model for a predetermined number of iterations (e.g., 50,000) at a predetermined learning rate (e.g., 2.5e6) using the same preprocessing and data augmentation techniques described above. The process 300 can then proceed to 332.

[0053] At 332, the process 300 may output the trained neural network. The trained neural network may include a second trained neural network. The process 300 may store the trained neural network (e.g., in a memory). The process 300 may then terminate.

[0054] 1, 2, 3, and 4, a process 400 for generating a malignancy indicator based on 2D breast images is shown. Process 400 may be implemented as instructions on one or more memories included in a data processing system. The data processing system may further include one or more processors configured to communicate with and execute the instructions on the one or more memories. The one or more processors may be configured to access the memory, which may be included in mass storage device 128, in which the 2D breast images are stored.

[0055] At 404, the process 400 can receive a 2D breast image. The 2D breast image can be a native 2D mammogram image (e.g., an image generated by a 2D full-field digital mammography system). The 2D breast image can be associated with a patient. Thereafter, the process 400 can proceed to 408.

[0056] At 408, the process 400 can provide the 2D breast images to a trained model. The trained model can include a trained neural network generated using the process 300. The process 400 can then proceed to 412.

[0057] At 412, the process 400 may receive an indicator from the trained model. In some configurations, the indicator may include a patient likelihood score and an area of ​​the 2D breast image. The area of ​​the 2D breast image may be associated with a bounding box generated by the trained model, and the patient likelihood score may be associated with malignancy of an area of ​​the 2D breast image that may contain a lesion. In some configurations, the indicator may include a patient likelihood score associated with the entire 2D breast image (i.e., an image-level score). Components of the indicator may be modified based on a medical professional's preferences by modifying the output layer 250. In some configurations, the process 400 may receive multiple indicators (e.g., if the 2D breast image has multiple lesions). The process 400 may then proceed to 416.

[0058] At 416, the process 400 can output the indicator to a memory for storage and / or use by another process and / or to a display, such as a computer monitor, for viewing by a human medical practitioner. For example, the process 400 can output the indicator to a display in a medical facility so that a medical practitioner can view the indicator and possibly determine a patient diagnosis based on the indicator. The indicator may be stored in a medical record database for future analysis and / or research of breast cancer patients. In some configurations, the process 400 can also output one or more of the indicators received at 412 to a display, such as a computer monitor, for storage and / or use by another process and / or for viewing by a human medical practitioner. The process 400 can output the indicator as a report. The process 400 can then end.

[0059] 1, 2 and 5, a process 500 is shown for creating a composite image using a model trained to detect ROIs in 2D slices of 3D tomosynthesis data. The process 500 may include one or more steps for selecting one or more most relevant ROIs output from the model and creating a composite image using the ROIs. The process 500 may be implemented as instructions on one or more memories included in a data processing system. The data processing system may further include one or more processors configured to communicate with and execute the instructions from the one or more memories. The one or more processors may be configured to access the memory, which may be included in mass storage device 128, and in which 3D tomosynthesis data including multiple two-dimensional slices is stored.

[0060] At 504, the process 500 may receive 3D tomosynthesis data of the patient's breast tissue. The 3D tomosynthesis data may be generated by a 3D mammography imaging system, such as the X-ray imaging system 100. The 3D tomosynthesis data may include a number of 2D slices corresponding to a predetermined thickness of the breast tissue, such as 1 mm. The 3D tomosynthesis data may include approximately 10 to 150 or more 2D slices, depending on the patient and / or imaging system. Each 2D slice may be associated with a slice number indicating the position of the 2D slice in the number of 2D slices. Each 2D slice may be an array of pixels having a predetermined size, such as 2000 by 1500 pixels. The process 500 may then proceed to 508.

[0061] At 508, the process 500 can individually provide each of the multiple 2D slices to a trained model that can detect an ROI based on the 2D slices. The trained model can be model 200 described above. In some configurations, a subset of the 2D slices can be provided to the model, such as slices associated with a series of slice numbers. The series of slice numbers can include a middle percentage of the 2D slices. For example, the subset of slices can include the middle 80 percent of the 2D slices (e.g., the 10th to 90th slices of 100 slices). Using the middle range of the 2D slices can remove noisy images near the edge of the DBT stack. In some configurations, the trained model can be trained based on an image dataset that includes two-dimensional full-field digital mammography images annotated by a medical professional. In some configurations, the trained model can be trained using the process described in connection with FIG. 4. The process 500 can then proceed to 512.

[0062] At 512, the process 500 can receive a number of ROIs output by the model for each 2D slice provided to the model. The model may output zero ROIs, one ROI, or multiple ROIs. Depending on the 2D slice and / or model, most slices may not have any ROIs at all. As described above, the ROI can include a score and a subarray of pixels of the 2D slice, each pixel having an intensity value and a location within the 2D slice. Each ROI can be associated with a slice number that indicates which 2D slice of the 3D tomosynthesis data the ROI was generated from. The process 500 can then proceed to 516.

[0063] At 516, the process 500 can filter out ROIs from the number of ROIs that have a score below a predetermined threshold. The threshold can be selected to include more or fewer ROIs in the final composite image (by selecting a lower or higher threshold, respectively) and / or to reduce possible false negatives (by selecting a lower value) and / or to reduce possible false positives (by selecting a higher value). For example, a user can select a threshold of 0.5 where scores can range from 0 to 1. The process 500 can then filter out ROIs from the number of ROIs that have a score below 0.5. The process 500 can then proceed to 520. In some configurations, the process 500 can proceed from 512 to 520 without filtering out any ROIs (e.g., the threshold is zero).

[0064] At 520, the process 500 may determine whether the multiple ROIs include at least one ROI. The process 500 may then proceed to 524.

[0065] If, at 524, the process 500 determines that the multiple ROIs do not include at least one ROI ("NO" at 524), the process 500 may proceed to 528. Alternatively, if the process 500 determines that the multiple ROIs do include at least one ROI, the process 500 may proceed to 532.

[0066] At 528, the process 500 may generate a final composite image without using any ROI. In some configurations, the process may select a default slice of the 2D slices included in the 3D tomosynthesis data to use as the final composite image. The default slice may be a middle slice of the 2D slices (e.g., the 38th slice of the 75 2D slices). Pixels of the default slice may then be included as pixels of the final composite image. The process 500 may then end.

[0067] At 532, process 500 may populate a region of the pre-composite image with one or more ROIs contained in multiple ROIs. The pre-composite image may be an array of the same size as the final composite image initialized with null values ​​for each of the pixel intensity values. Process 500 may then add one or more ROIs to the pre-composite image. The ROIs may be added based on one or more criteria. In some configurations, the process may determine the ROI with the highest score for each pixel location. The intensity value of the pre-composite image at the pixel location may then be set equal to the intensity value of the ROI at the pixel location. In other words, process 532 may utilize a maximum suspicion technique to populate the pre-composite image. If multiple ROIs have the same score at a given pixel location, the ROI with the largest number of pixels, the ROI with the largest number of pixels surrounding the pixel location, and / or the ROI with the highest intensity value at the pixel location may be selected as the ROI for the pixel location. In some configurations, the pre-composite image can be populated using non-maximum suppression (NMS) based on an intersection-over-union (IoU) threshold (e.g., an IoU of 0.2). If multiple ROIs overlap at a given pixel, the process 500 can populate the pre-composite image by "suppressing" all ROIs except the highest scoring ROI by disabling the ROIs with lower scores. For example, the pre-composite image can be populated using ROIs with scores of 0.85 and 0.75. The ROI with a score of 0.85 can overlap (e.g., occupy at least one of the same pixel locations as) the ROI with a score of 0.65, and the ROI with a score of 0.75 can not overlap with the ROI with a score of 0.65. Also, each pixel of the pre-composite image can be associated with a slice number of the ROI used to populate a given intensity value of the given pixel. The process 500 can then proceed to 536.

[0068] At 536, the process 500 may fill the unfilled regions of the pre-composite image. In some configurations, the process 500 may fill the unfilled regions of the pre-composite image based on a default slice. In some configurations, the process may select a default slice of the 2D slices included in the 3D tomosynthesis data to use to fill the unfilled regions of the pre-composite slices. The default slice may be a middle slice of the 2D slices (e.g., the 38th slice of the 75 2D slices). The process 500 may then proceed to 540.

[0069] At 540, the process 500 can output the final composite image to a memory for storage and / or use by another process, and / or to a display, such as a computer monitor, for viewing by a human medical practitioner. The process 500 can then terminate.

[0070] 1-5 and 6, another process 600 for training a neural network is shown. In particular, the process 600 can train a neural network for detecting malignancy based on 3D images. The neural network can be included in the model 200. The process 600 can be implemented as instructions on one or more memories included in a data processing system. The data processing system can further include one or more processors configured to communicate with and execute the instructions from the one or more memories. The one or more processors can be configured to access the memory, which can be included in the mass storage device 128, where the training data is stored.

[0071] At 604, process 600 may receive a partially trained neural network. The partially trained neural network may be a second trained neural network after 320 and / or 328 in process 300. In other words, the partially trained neural network may be trained based on the first set of annotated images. Then, process 600 may proceed to 608.

[0072] At 608, the process 600 may receive a set of annotated 3D images. In some configurations, each 3D image in the set of annotated 3D images may be a DBT image labeled with an image-level label. The image level may be a binary malignancy label (e.g., malignant or non-malignant). The second set of annotated images may be referred to as weakly annotated. The process 600 may then proceed to 612.

[0073] At 612, the process 600 can generate a plurality of composite images. In some configurations, the process 600 can use the process 500 with a set of annotated 3D images to generate each composite image included in the plurality of composite images. For example, the process 500 can generate a maximum doubt 2D image for each 3D image included in the set of annotated 3D images. Each composite image can be associated with a label associated with the 3D image used to generate the composite image. The process 600 can then proceed to 616.

[0074] At 616, the process 600 can train the partially trained neural network based on the multiple synthetic images. The partially trained neural network, previously used as an object detection model, can be converted to an image classification model by modifying the output layer 250 to output the maximum across all of the bounding box classification scores. In this way, the partially trained neural network can be fully discriminative while allowing end-to-end training with binary labels. In some configurations, the process 600 can train the model using an Adam optimizer starting with a learning rate of 2.5e6, which is reduced by a factor of four at a given number of iterations. The process 600 can train the partially trained neural network for fewer iterations than the second trained neural network is trained in the process 600 due to less available DBT data. In some configurations, the process 600 can determine that the second neural network is fully trained based on AUC performance. For example, the process 600 can determine the AUC performance of the validation set of images every 4000 iterations, and if the AUC performance decreases or remains the same, the partially trained neural network can be determined to be fully trained. In some configurations, the process 600 can fine-tune the fully trained model after determining that the model is fully trained. In some configurations, the process 600 can fine-tune the regression part based on a 2D set of annotated images. More specifically, the process 600 can freeze the weights included in the backbone and classification parts of the trained model and train only the regression part. In this way, the process 600 can adapt the regression part to any changes in the weights of the backbone part during the previous training on the second set of annotated images where the regression part is not updated.In some configurations, the process 600 can train the learned model for a predetermined number of iterations (e.g., 50,000) at a predetermined learning rate (e.g., 2.5e6) using the same preprocessing and data augmentation techniques described above. The process 600 can then proceed to 620.

[0075] At 620, process 600 may output a trained neural network. The trained neural network may include the fully trained neural network generated at 616. Process 600 may store (e.g., in a memory) the trained neural network. Process 600 may then terminate.

[0076] 1, 2, 5, 6 and 7, a process 700 for generating a malignancy indicator based on a 3D breast image is shown. The process 600 may be implemented as instructions on one or more memories included in a data processing system. The data processing system may further include one or more processors configured to communicate with and execute the instructions on the one or more memories. The one or more processors may be configured to access the memory, which may be included in mass storage device 128, in which the 3D breast images are stored.

[0077] At 704, process 700 may receive a 3D breast image. The 3D breast image may be a DBT image. The 3D breast image may be associated with a patient. Thereafter, process 700 may proceed to 708.

[0078] At 708, the process 700 may generate a composite 2D image based on the 3D breast image. The process 700 may generate the composite 2D image using the process 500. The process 700 may then proceed to 712.

[0079] At 712, the process 700 can provide the synthetic 2D image to a trained model. The trained model can include a trained neural network generated using the process 600. The process 700 can then proceed to 716.

[0080] At 716, the process 700 may receive an indicator from the trained model. In some configurations, the indicator may include a patient likelihood score and an area of ​​the composite 2D image. In some configurations, the area and / or pixels of the composite image may be associated with one or more slice numbers indicating the original slice(s) used to generate the composite 2D image. In this way, the medical practitioner may see one or more slices that contain a lesion. In some configurations, the indicator may include a ranked ordering of the slices used to generate the area of ​​the composite 2D image (e.g., slices ordered by percentage of the area of ​​the composite 2D image generated based on a given slice). In this way, the medical practitioner may see the most relevant slices. The area of ​​the composite 2D image may be associated with a bounding box generated by the trained model, and the patient likelihood score may be associated with the malignancy of the area of ​​the composite 2D image that may contain a lesion. In some configurations, the indicator may include a patient likelihood score associated with the entire composite 2D image (i.e., an image level score). Components of the indicator may be modified based on the medical practitioner's preferences by modifying the output layer 250. In some configurations, process 700 may receive multiple indicators (e.g., if the composite 2D image has multiple lesions), after which process 700 may proceed to 720.

[0081] At 720, the process 700 can output the indicator to a memory for storage and / or use by another process and / or to a display, such as a computer monitor, for viewing by a human medical practitioner. For example, the process 700 can output the indicator to a display in a medical facility so that a medical practitioner can view the indicator and possibly determine a patient diagnosis based on the indicator. The indicator may be stored in a medical record database for future analysis and / or research of breast cancer patients. In some configurations, the process 700 can also output one or more of the indicators received at 716 to a display, such as a computer monitor, for storage and / or use by another process and / or for viewing by a human medical practitioner. The process 700 can output the indicator as a report. The process 700 can then end.

[0082] <Test> In testing, models trained with process 300 have been shown to outperform human health care workers and several alternative prediction approaches. Table 1 summarizes the data sources used to train and test the models. In addition to datasets OMI-DB and DDSM, datasets were collected from three U.S. clinical sites for training, denoted Sites A, B, and C. Data used for testing included the test partitions of OMI-DB and the "Site A-DM" dataset, in addition to three datasets that were never used for model training or selection. These test-only datasets included a screening DM (digital mammography, e.g., 2D mammography) dataset from a Massachusetts health system used for reader studies (Site D), a diagnostic DM dataset from an urban hospital in China (Site E), and a screening DBT dataset from a community hospital in Oregon (Site A-DBT). Here, screening mammograms were tested whenever possible, but for the Site E dataset, diagnostic tests (i.e., tests where women present with symptoms) had to be used due to low screening rates in China.

[0083] [Table 1]

[0084] A reader study was conducted to directly compare the expert radiologists of both regimens using both "index" and "pre-index" cancer tests. Specifically, the index test was defined as a mammogram obtained up to 3 months before the biopsy-diagnosed malignancy. A pre-index test was defined as one obtained 12 to 24 months before the index test and was considered negative in clinical practice. According to the BIRADS criteria, a BIRADS score of 1 or 2 was considered a negative test and a "confirmed negative" was further defined as a negative test followed by an additional BIRADS 1-2 screen. All negatives used in the reader study were confirmed negatives.

[0085] Figure 8 is a receiver operating characteristic (ROC) plot summarizing the results of the “index” component of the reader study. Five radiologists participated in the study, each fellowship-trained in breast imaging and practicing full-time in the field. Data consisting of screening DM cases were collected retrospectively from a regional health system in a different US state from any source of training data. Figure 8 is a ROC plot based on case-level performance, comparing the deep learning model trained using the disclosed technology to the readers on a set of 131 index cancer tests and 154 confirmed negatives. All points representing each reader are below the model’s ROC curve, indicating that the model outperformed all five radiologists. At the mean reader specificity, the model achieved an absolute sensitivity increase of 14.2% (95% confidence interval (CI): 9.2–18.5%; p<0.0001). At the mean reader sensitivity, the model achieved an absolute specificity increase of 24.0% (95% CI: 17.4-30.4%; p<0.0001). Reader ROC curves based on the continuous "likelihood of malignancy" score were also constructed and showed a similar superior performance by the model. Moreover, the model outperformed all simulated combinations of readers and compared favorably with other recently published models on this dataset.

[0086] Figure 9 is an ROC plot summarizing the second component of the reader study using "pre-index" tests from the same patients. Figure 9 was generated based on a dataset including 120 pre-index cancer tests (defined as mammograms judged negative 12 to 24 months before the index test in which the cancer was detected) and 154 confirmed negatives. Since studies estimate that breast cancer is typically present 3 or more years before mammographic detection, the pre-index tests can be considered mostly challenging false negatives. The deep learning model outperformed all five readers in the early detection and pre-index paradigms. The absolute performance of the readers and models was lower for the pre-index cancer tests than for the index cancer tests, which was expected given the difficulty of the latter. Nevertheless, the model still showed an absolute sensitivity increase of 17.5% at mean reader specificity (95% CI: 6.0-26.2%; p=0.0009) and an absolute specificity increase of 16.2% at mean reader sensitivity (95% CI: 7.3-24.6%; p=0.0008). At 90% specificity, the model would have flagged 45.8% (95% CI: 28.8-57.1%) of preindex (i.e., "missed") cancer cases for additional workup. The model further performed better than recently published models on the preindex dataset.

[0087] Given the judging localization output of the model, it is also possible to evaluate the sensitivity while requiring precise localization. For both left-right and quadrant level localization, the model again shows improved sensitivity for both index and pre-index cases in the reader study. Also, a trend towards better model performance is applicable when factors such as lesion type, cancer type, cancer size, and breast density are considered.

[0088] Based on the performance of the reader study, independent performance was evaluated on a larger and more diverse dataset across different populations, device manufacturers, and modalities. These results are summarized in Table 2 and were calculated using the index cancer test.

[0089] [Table 2]

[0090] In Table 2, all results correspond to using the "index" test for cancer cases and "confirmed" negatives for non-cancer cases, except for site E, where the negatives are unconfirmed. Columns 1 and 2 show the performance of the 2D deep learning model trained with process 300 on the held-out test sets of the OMI-DB (1205 cancers, 1538 negatives) and site A (254 cancers, 7697 negatives) datasets. Columns 3 and 4 show the performance on a dataset collected at a hospital in China (site E; 533 cancers, 1000 negatives). The dataset consists entirely of diagnostic tests, given the low penetration of screening mammography in China. Nevertheless, the model still achieves high performance when adjusting tumor size using bootstrap resampling to approximate the distribution of tumor sizes expected in a US screening population (column 7). Columns 5 to 7 show the performance on the DBT data (site A-DBT; 78 cancers, 518 negatives). Column 5 contains the results of a 2D model fine-tuned on synthetic 2D images generated by the manufacturer, created to augment / replace DM images in DBT studies (* denotes this fine-tuned model). Column 6 contains the results of a weakly supervised 3D model trained using process 600, showing strong performance when evaluated on synthetic images generated via process 500 calculated from DBT slices. Here, the strongly supervised 2D model used to create the synthetic images shows an AUC of 0.865±0.020 when scoring the DBT volume as the maximum bounding box core across all slices. Thus, fine-tuning this model on synthetic images created according to process 500 significantly improves its performance. Column 7 shows the results when combining predictions on the final 3D model trained according to process 600 with the 2D model trained according to process 300. The standard deviation of each AUC value was calculated by bootstrap.

[0091] Starting with the test partition of OMI-DB, which contains 1205 cancers and 1538 confirmed negatives, the model trained using the disclosed technology shows strong performance for DM tests from the UK screening population with an AUC of 0.963±0.003 (0.961±0.003 using all 1976 negatives - confirmed and unconfirmed). For the test partition of the Site A-DM dataset, which contains 254 cancers and 7697 confirmed negatives, the model achieves an AUC of 0.927±0.008 (0.931±0.008 using all 16,369 negatives), which is not statistically different from the results for the other tested US screening DM dataset (Site D; p=0.22). The Site A-DM dataset is composed of mammograms acquired using GE equipment, as opposed to the Hologic equipment used for the majority of the other datasets.

[0092] To further test the generalizability of the model trained according to process 400, we evaluated its performance on a DM dataset collected at an urban hospital in China (Site E). Testing generalization to this dataset is particularly meaningful given the low screening rates in China and the known (and sometimes unknown) biological differences found in mammograms between Western and Asian populations, such as the higher proportion of women with dense breasts in Asian populations. The model evaluated locally at the Chinese hospital generalized well to this population, achieving an AUC of 0.971 ± 0.005 (using all negatives, although "confirmation" is not possible given the lack of follow-up screening). Even when tumor size was adjusted to roughly match the statistics expected in a US screened population, the model achieved an AUC of 0.956 ± 0.020.

[0093] Finally, models trained according to the process 600 approach perform well when evaluated on sites not used for DBT model training. The model achieves an AUC of 0.947 ± 0.012 (78 cancers and 519 confirmed negatives; 0.950 ± 0.01 using all 11,609 negative tests). When the strongly supervised 2D model is instead fine-tuned on the manufacturer-generated synthetic 2D images that are generated by default in each DBT study, the resulting model achieves an AUC of 0.922 ± 0.016 on the test set (0.923 ± 0.015 using all negatives). Averaging predictions over the manufacturer-generated synthetic images and synthetic images generated according to process 500 results in an overall performance of 0.957 ± 0.010 (0.959 ± 0.008 using all negatives).

[0094] <Method> ·Description of the dataset Details of all utilized datasets listed in Table 1 are provided below. All private datasets were collected with IRB approval and were de-identified before model training and testing. Data splits were created at the patient level for each data source, meaning that all tests from a particular patient were included in the same split. Splits consisted of training, model selection, and testing, and the model selection split was used to choose the final model and to decide when to stop model training. Label assignment and case selection rules for training data differed slightly between datasets due to variability in collection duration and available metadata (described below). However, definitions of test sets and label criteria were standardized across datasets unless otherwise stated. The following definitions were used for label assignment: Index cancer - Mammogram taken within 3 months before cancer diagnosis. Preindex cancer - mammogram determined to be BIRADS category 1 or 2 and obtained 12 to 24 months prior to the index examination. Negative - A mammogram scored as BIRADS 1 or 2 from a patient with no past or future history of breast cancer. A confirmed negative-negative test was followed by an additional BIRADS 1 or 2 at the next screening test 9 to 39 months later (which represents a 1- to 3-year follow-up depending on the screening paradigm with a 3-month buffer). The time window was extended beyond 3 years to include triennial screening (e.g., UK). Throughout, "pre-index" cases were treated as positive because for the majority of these tests, cancer was likely present at the time of acquisition, although there is no guarantee that adequate follow-up would have led to a pathologically diagnosed cancer.

[0095] All datasets shared the same algorithm to create the test set, except for site D (described in detail in the corresponding section below). Studies were labeled as “index”, “preindex”, “confirmed negative”, “unconfirmed negative”, or “none” based on the criteria described above. For each patient in the test set, one study was selected in the order of preference from top to bottom: “index”, “preindex”, “confirmed negative”, or “unconfirmed negative”. If a patient had multiple tests with the selected label, one test was randomly sampled. If a patient had an index test, one preindex test was also included, if available. For all training and testing, only craniocaudal (CC) and mediolateral oblique (MLO) mammograms were used. All test sets included only screening tests (i.e., screening index cancer, screening negative), except for site E, where all tested tests were diagnostic tests given the low screening rate in China. Note that the proportion of confirmed and unconfirmed negatives varied widely across sites because of different time periods for test acquisition (i.e., not enough time may have passed for confirmation), different screening paradigms, and / or different scope of BIRADS information collection. Performance was reported using both confirmed and unconfirmed negatives when possible to evaluate the larger amount of data while still considering results with a strict definition of negativity.

[0096] For training, each training instance (e.g., image or bounding box) was labeled by assigning a label of “1” for cancer and a label of “0” for non-cancer. The primary decision to assign a label of “1” (cancer) to an image is made within the time window allowed between cancer confirmation (biopsy) and image acquisition. For the US dataset, this window is set to 15 months. This time window was chosen to balance the risk of overfitting while including some pre-index cancer tests for training. For the US dataset (except for DDSM, which has only index cancer), localization annotations were not available, so a wider time window could lead to overfitting for more insidious cancers. Nevertheless, the mix of annual and biennial screening in the US allows the inclusion of some pre-index cancers using a 15-month time window. For the UK OMI-DB, this window is extended by one year due to the high proportion of strongly labeled data in the dataset and the fact that the standard screening interval is longer in the UK. For non-cancer, screen-negative tests (BIRADS 1 or 2) from patients with no history of cancer were used unless otherwise stated, and pathologically confirmed benign cases from patients with no history of cancer were used when available. For pathologically confirmed benign cases, training was performed on both screening and diagnostic tests. For cancer, both screening and diagnostic tests were additionally included in the training. It was found that training on diagnostic tests can improve performance even when evaluated on screening tests alone (and vice versa). The only dataset where training was performed on screening tests alone was the Site A-DM dataset, and all diagnostic tests were excluded to avoid such bias, since in the absence of benign biopsy information, all diagnostic tests that would be included in the training would be cancer. All eligible tests for a given patient were used for training, in contrast to model testing that included only one test per patient.Further details of the dataset are provided below.

[0097] · Digital Database of Screening Mammography (DDSM) DDSM is a public database of US scanned film mammography studies that contain cases categorized as normal, benign, and malignant with verified pathology information. The dataset includes radiologist-drawn segmentation maps for each detected lesion. The data was split into a 90% / 10% training / model selection split, resulting in 732 cancer, 743 benign, and 807 normal studies for training. Data from DDSM was not used for testing, given that it is a scanned film dataset.

[0098] · Optimal Mammography Imaging Database (OMI-DB) OMI-DB is a UK public dataset that contains screening and diagnostic digital mammograms obtained primarily with Hologic equipment. The unique list of patients was split into a 60% / 20% / 20% training / model selection / test split. This resulted in a training set of 5233 cancer studies (2332 with bounding boxes), 1276 benign studies (296 with bounding boxes), and 16,887 negative studies. Here, although the ratio of positives to negatives in OMI-DB is much higher than would be expected in a screening population, the positives and negatives themselves were randomly sampled from the respective populations. Thus, given the invariance of the ROC curve to incidence, no bias is expected in the AUC of the test set in this population compared to the full population with natural incidence.

[0099] Site A Site A is a community hospital in Oregon. The dataset from Site A consisted primarily of screening mammograms, with DM data from 2010–2015 collected almost entirely from GE machines and DBT data from 2016–2017 collected almost entirely from Hologic machines. For the DM data, 40% of patients were used for training, 20% for model selection, and 40% for testing. DBT data was used solely for testing due to its higher proportion of screening exams compared to the other DBT datasets utilized. The local hospital cancer registry was used to obtain ground truth cancer status for both modalities. Radiology reports were also attached to each study and included BIRADS information. For the DBT data, a list of benign cases was additionally provided by the hospital, but no such information was available for the DM data. Given the extent of longitudinal data present in the DM dataset and the lack of confirmed benign pathology information for this data, non-cancers were more rigorously selected for training, specifically requiring negatives to have no records of non-screening procedures or non-normal determinations on the patient for 18 months before or after the exam. As a result, 466 cancer studies and 48,248 negative studies were obtained for training on the Site A-DM dataset.

[0100] Site B Site B consists of an inpatient medical center and associated imaging center in Rhode Island. Data from this site includes DBT mammograms from Hologic equipment and is a mix of screening and diagnostic tests collected retrospectively between 2016 and 2017. Cancer status, benign results, and BIRADS were determined using a local database. The list of unique patients was split into an 80% / 20% training / model selection split. Given the relatively small amount of DBT available for training and the desirability of testing on a dataset not used for training, Site B was used only for model development. The training split consisted of 13,767 negative, 379 benign, and 263 cancer cases. Manufacturer-generated synthetic 2D images were also included in the weakly supervised training for the final 2D model.

[0101] Site C Site C is a separate Oregon Health System from Site A. DBT cases from Site C were collected retrospectively between 2017 and 2018. The data consisted of a mix of screening and diagnosis cases acquired almost entirely using Hologic equipment. The list of unique patients was split into a 70% / 30% training / model selection split. Cancer status was determined using the regional cancer registry. Similar to Site B, Site C was used for model development only. Historical BIRADS information was not readily available for all cases in Site C, so cases from patients not registered in the regional cancer registry were used as non-cancer for training. Given the geographic proximity of Site C and Site A, a small number of overlapping patients from both sets were used when testing Site A. Here, synthetic 2D images generated by the manufacturer were also included in the weakly supervised training of the final 2D model.

[0102] Site D Data from site D were used for the reader study and consisted of 405 screening DM exams collected retrospectively from one Massachusetts health system with four different imaging collection centers. Data from this site have not been used previously for model training or selection. Exams included in the study were acquired between July 2011 and June 2014. Of the 405 studies, 154 were negative, 131 were index cancer exams, and 120 were preindex cancer exams. All negatives were confirmed negatives. Index cancer exams were screening mammograms that were deemed suspicious and pathologically confirmed to be malignant within 3 months of acquisition. Preindex exams were obtained from the same set of women as the index exam and consisted of screening exams that were classified as BIRADS 1 or 2 and were acquired 12 to 24 months prior to the index exam. All studies were acquired using Hologic equipment. Case selection was performed in multiple steps. Cancer patients included in the study were selected by first using the local cancer registry to retrieve all patients who had eligible index and preindex examinations over the specified time period. Some preindex cases could not be obtained due to PACS restrictions. Secondly, bucketing was used to select noncancer cases to have a similar distribution in patient age and breast density compared to cancer cases. In total, 154 noncancer mammograms, 131 index cancer mammograms, and 120 preindex cancer mammograms were collected from 285 women.

[0103] Site E Site E consists of a dataset from an urban hospital in China collected retrospectively from a continuous period between 2012 and 2017. Over this period, all pathologically proven cancers were collected along with a uniformly random sample of non-cancers, resulting in 533 cancers, 1000 negatives (BIRADS 1 or 2), and 100 pathologically proven benign cases. Due to low screening rates in China, the data were derived from diagnostic tests (i.e., tests where the patient presented with symptoms), and therefore the distribution of tumor sizes from the cancer cases included larger tumors (e.g., 64% >2 cm) than would be expected in a typical US screening population. To better compare with the US screening population, the results from Site E were also calculated to approximately match the distribution of tumor sizes from a US population according to the National Radiology Data Registry using a bootstrap resampling method. Using this approach, the average AUC was calculated for 5K bootstrapped populations. Site E was used only for testing and has not been used for model development so far. Moreover, the deep learning system was evaluated locally at the hospital and the data never left the site.

[0104] Model development and training In the tests described above, the first stage of training the model used in the tests consisted of patch-level classification (e.g., 308 in Figure 3). Patches of size 275x275 pixels were created from the DDSM and OMI-DB datasets after the original images were resized to a height of 1750 pixels. Data augmentation, including random 360 degree rotations, resizing up to 20%, and vertical mirroring, were also used in creating the patches. Preprocessing included normalizing pixel values ​​to the range [-127.5, 127.5]. When creating patches containing lesions, a random location within the lesion boundary was selected as the center of the patch. If the resulting patch contained less than 6 pixels containing the lesion mask, the patch was discarded and a new patch was sampled. For all patches, the patch was discarded if it contained less than 10% breast foreground, as determined by Otsu's method for DDSM and a thresholding method using the minimum pixel value in the image for OMI-DB. In total, 2 million patches were created, with an equal number of patches with and without lesions. For the patch classification model, a ResNet-50 network was used. The patch-based training stage itself consisted of two training sequences. First, starting from the ImageNet pre-trained weights, a ResNet-50 model was trained for a 5-class classification of lesion types: mass, calcification, local asymmetry, architectural distortion, or no lesion. Patches from DDSM and OMI-DB were sampled proportionally to the number of cancer cases in each dataset. The model was trained on 62,500 batches with a batch size of 16, sampling equally from all lesion types. The Adam optimizer was used with a learning rate of 1e-5. Next, a patch-level model was trained for a 3-class classification, with normal, benign, or malignant labels, again sampling equally from all three categories. The same training parameters were used for this stage of patch-level training.

[0105] After patch-level training, the backbone part of RetinaNet, a popular detection model, was initialized with ResNet-50 weights for the second stage of training, i.e., strongly supervised, image-level training (e.g., 320 in Figure 3). Image preprocessing included resizing to a height of 1750 pixels (preserving the original aspect ratio), cropping out the background using the thresholding method described above, and normalizing pixel values ​​to the range [-127.5, 127.5]. Data augmentation during training included random resizing up to 15% and random vertical mirroring. Given the high class imbalance in mammograms (far fewer positives than negatives), class balancing was performed during training by sampling malignant and non-malignant cases with equal probability. This class balancing was performed within the dataset to prevent the model from learning biases for the different proportions of cancer between the datasets. For this strongly supervised image-level training stage, bounding boxes in the OMI-DB and DDSM datasets were used. Three-class bounding box classification was performed using the labels normal, benign, or malignant. The RetinaNet model was trained for 100K iterations with a batch size of 1. The Adam optimizer was used with a learning rate of 1e-5 and gradient norm clipping at a value of 0.001. Default hyperparameters were used for the RetinaNet losses, except that the regression loss was given a weight of 0.5 and the classification loss was given a weight of 1.0.

[0106] For the weakly supervised training stage (e.g., 328 in Figure 3 and 616 in Figure 6), binary cancer / non-cancer classification was performed using binary cross-entropy loss. The same image input processing steps as for the strongly supervised training stage were used. The RetinaNet architecture was converted to a classification model by taking the maximum over all of the bounding box classification scores, resulting in a model that remained perfectly discriminative while allowing end-to-end training with binary labels. For 2D, training consisted of 300K iterations using the Adam optimizer, starting with a learning rate of 2.5e-6, which was reduced by a factor of 4 every 100K iterations. Final model weights were chosen by monitoring the AUC performance on the validation set every 4K iterations.

[0107] For DBT, the "maximum suspicion projection" (MSP) approach described in process 500 is motivated by the value of DBT in providing optimal views of lesions that may otherwise be obscured by overlapping tissue, and by the similarity between DBT and DM images, suggesting the applicability of transfer learning. Furthermore, the collective nature of 2D mammography is particularly believed to help reduce overfitting compared to end-to-end training on large DBT volumes. To this end, in the weakly supervised stage for the 3D model, a synthetic image ("MSP image") created through process 500 was created using the model obtained from the 2D strongly supervised training described above after 50K training iterations with an additional learning rate of 2.5e-6. To create the MSP image, the 2D model was evaluated for each slice of the DBT stack, excluding the first and last 10% of the slices, which are often noisy. The minimum bounding box core threshold was set to a level that achieved 99% sensitivity on the OMI-DB validation set. Bounding boxes across all evaluated slices were filtered using non-maximum suppression (NMS) with an IoU threshold of 0.2. Image patches defined by the filtered bounding boxes were then collapsed into a single 2D image array representing the optimized image for further model training. The "empty" pixels in the projection were filled with corresponding pixels from the central slice of the DBT stack to obtain the final MSP image. Overall, the MSP process is similar to maximum intensity projection (MIP), except that the maximum is not calculated on pixel-level intensities, but instead on the ROI suspected malignancy predicted by the AI ​​model. Training on the resulting MSP images was performed similarly to the 2D weakly supervised approach, except that the model was trained for 100K iterations. The input processing parameters used for the 2D images were reused for the DBT slices and MSP images.

[0108] After weakly supervised training on both the 2D and 3D models, the regression part of the RetinaNet architecture was fine-tuned on the strongly labeled data used in the second stage of training. Specifically, the backbone and classification parts of the network were frozen and only the regression part was updated during this fine-tuning. This allows the regression part to adapt to any changes in weights in the backbone part of the network during the weakly supervised training stage where the regression part is not updated. In this regression fine-tuning stage, the network was trained for 50K iterations with a learning rate of 2.5e-6 using the same preprocessing and data augmentation procedures as in the previous stage. Final model selection was based on the performance of the hold-out model selection data partition. The final models were an ensemble of three equally trained models starting from different random seeds. The prediction score for a given image was calculated by averaging the predictions of the three models for both horizontal directions of the image (i.e., the average of the six scores). The regression coordinates of the bounding box anchors were additionally averaged across the three models. Each breast was assigned a malignancy score by taking the average score over all of its views. Each study was assigned a score by taking the greater of its two breast levels.

[0109] Reader research A reader study was conducted to directly evaluate the performance of the described deep learning system compared to expert radiologists. Although a reader study is certainly an artificial setting, such a study avoids the "gatekeeper bias" inherent in retrospective performance comparisons, because the ground truth for each case is established a priori in the reader study. Recent evidence also suggests that a high positive rate in a reader study, per se, will have little impact on reader population ROC performance.

[0110] Reader Selection Five board-certified and MQSA-certified radiologists were recruited as readers for the reader study. All readers were fellowship trained in breast imaging with an average of 5 years of post-fellowship experience (range 2-12 years). Readers read an average of 6969 mammograms (range 2921-9260) in the year prior to the reader study, 60% of which were DM and 40% were DBT.

[0111] <Research design> Data for the reader study were obtained from site D, described above. The study was conducted in two sessions. In the first session, radiologists read 131 index cancer exams and 76 negative exams. In the second session, radiologists read 120 preindex exams and the remaining 78 negative exams. There was a washout period of at least 4 weeks between the two sessions for each reader. Readers were instructed to provide a mandatory BIRADS score (1 to 5) for each case. BIRADS scores of 1 and 2 were considered non-recalls, and BIRADS scores of 3, 4, and 5 were considered recalls. The radiologists did not have any information about the patients (e.g., past medical history, radiology reports, other patient records, etc.) and were informed that the study dataset contained an enrichment of cancer mammograms compared to the standard prevalence observed in screening, but were blinded to the proportion of case types. All radiologists viewed and judged the studies at dedicated mammography workstations in an environment similar to their clinical practice. Readers recorded their judgements in electronic case reports. In addition to the mandatory BIRADS, readers provided breast density classification and, for each detected lesion, lesion type, laterality, quadrant difference, and probable malignancy score from 0 to 100 (for up to four lesions). Reader binary recall decisions using BIRADS were used for analysis because this more closely reflects clinical practice.

[0112] Localization-based analysis While the main reader study results described above correspond to case-level classification performance, a localization-based analysis was also performed. In this study, readers reported breast laterality and quadrant difference for each determined lesion to justify recall. Ground truth laterality and quadrant difference for malignant lesions were provided by the clinical initiative of the reader study by examining the mammogram images together with the pathology and radiology reports. For pre-index cases, the ground truth location was set to the ground truth location of the corresponding index case, even if the lesion was considered not visible in the pre-index examination. The considered deep learning model provides localization in the form of bounding boxes. For comparison with readers and also to serve as an exercise in model output judgeability, an MQSA-certified radiologist from a different practice than the reader study site mapped the model output boxes to breast laterality and quadrant. The radiologist was blinded to the ground truth of the case and was instructed to estimate the location of the centroid for each given bounding box, restricting the estimation to one quadrant.

[0113] Both left-right and quadrant-based localization sensitivities were considered, requiring accurate localization at the corresponding level in addition to recall of cases. Because readers reported at most one lesion for the vast majority of cases (90%), and to avoid scenarios in which predictions involving many locations are rewarded, in the first analysis, we restricted the predicted location to the location corresponding to the highest scoring lesion. For the model, this corresponds to taking the bounding box of the best scoring left-right of the best scoring. For a reader, the best scoring location was selected using the malignancy likelihood score provided for each lesion. In cases with multiple malignant lesions, a true positive was assigned if the reader or model location matched the location of any malignant lesion. The sensitivity of each reader was compared to the model by choosing a model score threshold that matched the specificity of a given reader. The model was also compared to the reader mean in a similar manner.

[0114] ·Statistical analysis The primary evaluation method was the receiver operating characteristic (ROC) curve throughout the test, where ROC analysis is a standard method for evaluating diagnostic performance because it is invariant to the proportion of positive and negative cases and allows performance to be compared across the full range of possible recall rates (i.e., operating points). Confidence intervals and standard deviations of the AUC, as well as the mean reader sensitivity and specificity, were calculated via a bootstrap method with 10,000 random resamplings. p-values ​​for comparing the sensitivity and specificity of the possible models with the mean reader sensitivity and specificity were calculated by taking the proportion of times the difference between the model and the reader was less than 0 across the bootstrap resamplings. AUC performance between datasets was compared using a bootstrap method with 10,000 random resamplings.

[0115] While the present invention has been described in terms of one or more preferred configurations, it should be understood that, except as expressly stated, many equivalents, alternatives, variations, and modifications are possible and are within the scope of the invention.

Claims

1. 1. A computer-based method for determining likelihood of malignancy in breast tissue of a patient, comprising: receiving, using a computer processor, an image of the breast tissue; providing the image of the breast tissue to a model comprising a trained neural network; receiving indicators from the model; outputting a report including the indicator to at least one of a memory or a display; The trained neural network is training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, where each of the annotated patches includes a patch-level label and a portion of a two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; training the second neural network based on a first set of annotated images, where at least one annotated image in the first set of annotated images includes a two-dimensional image in a second group of two-dimensional images, at least one bounding box, and at least one malignant label associated with the at least one bounding box; training the second neural network based on a second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image in a third group of two-dimensional images and an image-level malignancy likelihood score; The method of claim 1, wherein the first neural network is pre-trained to output the second neural network as the trained neural network.

2. The method of claim 1 , wherein the image of the breast tissue is a two-dimensional digital mammogram.

3. The method of claim 1 , wherein the image of the breast tissue is a synthetic two-dimensional image generated based on a digital breast tomosynthesis image.

4. the digital breast tomosynthesis image includes a plurality of slices; the composite two-dimensional image includes an array of pixels; At least a portion of the pixels in the array of pixels have pixel intensity values ​​of: providing at least a portion of the plurality of slices to a second trained neural network; receiving a plurality of indicators from the second trained neural network, each indicator including a malignancy likelihood score; 4. The method of claim 3, wherein for a target pixel included in the array of pixels, a target pixel intensity is pre-generated by determining a target pixel intensity based on a first indicator included in the plurality of indicators, the first indicator including a greater malignancy likelihood score than a second indicator included in the plurality of indicators.

5. At least a portion of the pixels in the array of pixels have pixel intensity values ​​of: determining, for a second target pixel in the array of pixels, that the plurality of pixels does not include a pixel location associated with the second target pixel; 5. The method of claim 4, wherein the pixel intensity of the second target pixel is pre-generated by setting the pixel intensity of the second target pixel equal to a default pixel included in a default slice included in the plurality of slices, the default pixel including a pixel position associated with the target pixel.

6. The method of claim 5 , wherein each slice in the plurality of slices is associated with a position value, and the default slice is associated with an intermediate position value.

7. the digital breast tomosynthesis image includes a plurality of slices; Each slice in the plurality of slices is associated with a position value; The method of claim 3 , wherein the composite two-dimensional image is generated based on a subset of the plurality of slices associated with position values ​​that fall within a predetermined range of position values.

8. The method of claim 7 , wherein the subset of the plurality of slices comprises approximately 80 percent of the plurality of slices.

9. the second neural network includes a backbone portion, a regression portion, and a classification portion; 2. The method of claim 1, further comprising training the regression portion based on the first set of annotated images without retraining the backbone portion and the classification portion.

10. 10. The method of claim 9, wherein training the second neural network based on the second set of annotated images comprises training the backbone portion and the classification portion without training the regression portion.

11. the first neural network includes a first plurality of weights; the second neural network includes a backbone portion including a second plurality of weights; 2. The method of claim 1 , wherein initializing the second neural network based on the first neural network comprises setting weights in the second plurality of weights to be the same as weights in the first plurality of weights.

12. The method of claim 11 , wherein the first neural network comprises a deep residual network and the second neural network comprises a single-stage object detection network.

13. The method of claim 1 , wherein the indicator comprises a patient malignancy likelihood score.

14. The method of claim 1 , wherein the indicator comprises an area of ​​the image of the breast tissue.

15. 1. A system for assessing malignant potential of breast tissue of a patient, comprising: a memory configured to store the image of the breast tissue; a processor configured to access the memory; The processor, providing the image of the breast tissue to a model comprising a trained neural network; receiving indicators from the model; configured to output a report including the indicator to at least one of a memory or a display; The trained neural network is training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, where each of the annotated patches includes a patch-level label and a portion of a two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; training the second neural network based on a first set of annotated images, where at least one annotated image in the first set of annotated images includes a two-dimensional image in a second group of two-dimensional images, at least one bounding box, and at least one malignant label associated with the at least one bounding box; training the second neural network based on a second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image in a third group of two-dimensional images and an image-level malignancy likelihood score; The system is pre-trained to output the second neural network as the trained neural network.

16. The system of claim 15 , wherein the image of the breast tissue is a two-dimensional digital mammogram.

17. The system of claim 15 , wherein the image of the breast tissue is a synthetic two-dimensional image generated based on a digital breast tomosynthesis image.

18. the digital breast tomosynthesis image includes a plurality of slices; the composite two-dimensional image includes an array of pixels; At least a portion of the pixels in the array of pixels have pixel intensity values ​​of: providing at least a portion of the plurality of slices to a second trained neural network; receiving a plurality of indicators from the second trained neural network, each indicator including a malignancy likelihood score; 20. The system of claim 17, wherein the target pixel intensity is pre-generated by determining, for a target pixel in the array of pixels, a target pixel intensity based on a first indicator in the plurality of indicators, the first indicator including a greater likelihood of malignancy score than a second indicator in the plurality of indicators.

19. At least a portion of the pixels in the array of pixels have pixel intensity values ​​of: determining, for a second target pixel in the array of pixels, that the plurality of pixels does not include a pixel location associated with the second target pixel; 20. The system of claim 18, wherein the pixel intensity of the second target pixel is pre-generated by setting the pixel intensity of the second target pixel to be the same as a default pixel included in a default slice included in the plurality of slices, the default pixel including a pixel location associated with the second target pixel.

20. 20. The system of claim 19, wherein each slice in the plurality of slices is associated with a position value, and the default slice is associated with an intermediate position value.

21. the digital breast tomosynthesis image includes a plurality of slices; Each slice in the plurality of slices is associated with a position value; The system of claim 17 , wherein the composite two-dimensional image is generated based on a subset of the plurality of slices associated with position values ​​that fall within a predetermined range of position values.

22. 22. The system of claim 21, wherein the subset of the plurality of slices comprises approximately 80 percent of the plurality of slices.

23. the second neural network includes a backbone portion, a regression portion, and a classification portion; 16. The system of claim 15, wherein the trained neural network further comprises training the regression portion based on the first set of annotated images without retraining the backbone portion and the classification portion.

24. 24. The system of claim 23, wherein training the second neural network based on the second set of annotated images comprises training the backbone portion and the classification portion without training the regression portion.

25. the first neural network includes a first plurality of weights; the second neural network includes a backbone portion including a second plurality of weights; 16. The system of claim 15, wherein initializing the second neural network based on the first neural network includes setting weights in the second plurality of weights to be the same as weights in the first plurality of weights.

26. 16. The system of claim 15, wherein the first neural network comprises a deep residual network and the second neural network comprises a single-stage object detection network.

27. The system of claim 15 , wherein the indicator comprises a patient malignancy likelihood score.

28. The system of claim 15 , wherein the indicator comprises an area of ​​the image of the breast tissue.

29. 1. A computer-implemented method for generating a malignancy likelihood score for a patient's breast tissue, comprising: receiving, using a computer processor, an image of the breast tissue; providing the image of the breast tissue to a model comprising a trained neural network; receiving the malignancy likelihood score from the model; outputting a report including the malignancy likelihood score for the patient to at least one of a memory or a display; The trained neural network is training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, where each of the annotated patches includes a malignancy likelihood score and a portion of a two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; training the second neural network based on a first set of annotated images, where at least one annotated image in the first set of annotated images includes a two-dimensional image in a second group of two-dimensional images, at least one bounding box, and at least one malignancy likelihood score associated with the at least one bounding box; training the second neural network based on a second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image in a third group of two-dimensional images and an image-level malignancy likelihood score; The method of claim 1, wherein the first neural network is pre-trained to output the second neural network as the trained neural network.

30. 1. A system for generating a malignancy likelihood score for a patient's breast tissue, comprising: a memory configured to store the image of the breast tissue; a processor configured to access the memory; The processor, providing the image of the breast tissue to a model comprising a trained neural network; receiving the malignancy likelihood score from the model; and outputting a report including the malignancy likelihood score to at least one of a memory or a display; The trained neural network is training a first neural network based on a set of annotated patches derived from a first group of two-dimensional images, where each of the annotated patches includes a malignancy likelihood score and a portion of a two-dimensional image included in the first group of two-dimensional images; initializing a second neural network based on the first neural network; training the second neural network based on a first set of annotated images, where at least one annotated image in the first set of annotated images includes a two-dimensional image in a second group of two-dimensional images, at least one bounding box, and at least one malignancy likelihood score associated with the at least one bounding box; training the second neural network based on a second set of annotated images, where each annotated image in the second set of annotated images includes a two-dimensional image in a third group of two-dimensional images and an image-level malignancy likelihood score; The system is pre-trained to output the second neural network as the trained neural network.

Images