Systems and methods for analyzing three-dimensional image data

Abstract

The present disclosure provides a method for determining a malignancy likelihood score for a patient's breast tissue, the method comprising the steps of receiving a plurality of two-dimensional images of the breast tissue derived from a three-dimensional image of the breast tissue, inputting, for each two-dimensional image, the two-dimensional image into a first model including a trained first neural network, receiving from the first model a plurality of indicators associated with each two-dimensional image included in the plurality of two-dimensional images, generating a composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images, inputting the composite two-dimensional image into a second model including a trained second neural network, receiving a malignancy likelihood score from the second model, and outputting a report including the malignancy likelihood score to at least one of a memory or a display.

Description

[Technical field]

[0001] <CROSS REFERENCE TO RELATED APPLICATIONS> This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 926,088, filed October 25, 2019, the entire contents of which are incorporated herein by reference.

[0002] <Statement Regarding Federally Sponsored Research> none. [Background technology]

[0003] Digital breast tomosynthesis (DBT), also known as three-dimensional (3D) mammography, has been shown to have better clinical performance than traditional full-field digital mammography in detecting breast cancer. Full-field mammography can generate two-dimensional (2D) projection images representing the full depth of the breast for many views. In contrast, DBT can generate 2D images representing thin slices of the breast (i.e., slices approximately 1 mm thick), with 10 to 150 or more images per view.

[0004] Because DBT produces more data than full-field mammography, it can be difficult for a human physician to sift through the DBT-produced images and make a diagnosis regarding the presence or absence of malignant tumors and / or lesions. Furthermore, because physicians vary in what constitutes a potentially malignant region and what does not, even with the same set of images, there can be inconsistencies in determining whether a patient has a malignant tumor and / or lesion.

[0005] It would therefore be desirable to provide a system and method for more efficiently and accurately analyzing 3D mammography data and for consistently estimating malignant tumors and / or lesions using the 3D mammography data. Summary of the Invention

[0006] The present disclosure provides systems and methods for efficiently and accurately analyzing 3D mammography data and for uniformly assessing malignant tumors and / or lesions using the 3D mammography data. In one non-limiting aspect, the present disclosure provides a method for determining a malignancy likelihood score for a patient's breast tissue, the method comprising the steps of receiving a plurality of two-dimensional images of the breast tissue derived from a three-dimensional image of the breast tissue, inputting the two-dimensional image for each of the two-dimensional images into a first model including a trained first neural network, receiving a plurality of indicators from the first model associated with each of the two-dimensional images in the plurality of two-dimensional images, generating a composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images, inputting the composite two-dimensional image into a second model including a trained second neural network, receiving a malignancy likelihood score from the second model, and outputting a report including the malignancy likelihood score to at least one of a memory or a display.

[0007] In the method, the indicator can include a relevance score, and the generating step can include determining an overlap between the first indicator and the second indicator, determining that the first indicator has a higher relevance score than the second indicator, and including at least a portion of the first indicator in the composite two-dimensional image if the first indicator is determined to have the higher relevance score. The first indicator can be associated with a first two-dimensional image included in the plurality of two-dimensional images, and the second indicator can be associated with a second two-dimensional image included in the plurality of two-dimensional images.

[0008] In this method, each indicator may include an array of pixels each having an intensity value and a relevance score.

[0009] The generating step may include setting a first intensity value of a first pixel in the composite two-dimensional image equal to a second intensity value in the indicator.

[0010] In the method, the generating step may include setting a first intensity value of a first pixel included in the composite two-dimensional image equal to a second intensity value of a second pixel included in a second two-dimensional image included in the plurality of two-dimensional images, the second intensity value being not associated with any indicator.

[0011] In the method, the second two-dimensional image can be associated with at least one indicator.

[0012] The generating step may include determining a first coverage area associated with a first indicator of the plurality of indicators, the first coverage area including a second array of pixels, determining a second coverage area associated with a second indicator of the plurality of indicators, the second coverage area including a third array of pixels, and determining a first intensity value for a first pixel in the composite two-dimensional image based on a second intensity value in the first coverage area and a third intensity value in the second coverage area. The first coverage area may include at least one pixel not included in the first indicator, and the second coverage area may include at least one pixel not included in the second indicator. Determining the first intensity value for the first pixel may include setting the first intensity value to be equal to a sum of the second intensity value multiplied by a first weight and the third intensity value multiplied by a second weight.

[0013] In the method, the generating step may include the steps of determining that the first indicator and the second indicator overlap, determining that the first indicator has a higher relevance score than the second indicator, and not including any portion of the second indicator in the composite two-dimensional image if it is determined that the first indicator has the higher relevance score.

[0014] Three-dimensional images can be generated using digital breast tomosynthesis.

[0015] In the method, the trained first neural network can include a first sub-network and the trained second neural network can include a second sub-network, and the first sub-network and the second sub-network can have the same number of layers and filters. The trained second neural network can be trained using a set of weight values ​​of the first neural network as initial weight values.

[0016] The method may further include the steps of deriving at least one indicator from the plurality of indicators and generating a second composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images.

[0017] In the method, the first model can be trained based on an image dataset including two-dimensional full-field digital mammography images with physician annotations, where each of the multiple two-dimensional images of breast tissue can be a slice of a three-dimensional digital breast tomosynthesis volume.

[0018] In another non-limiting aspect, the present disclosure provides a method for generating a composite two-dimensional image of a patient's breast tissue, the method including receiving a plurality of two-dimensional images of the breast tissue derived from a three-dimensional image of the breast tissue, inputting, for each two-dimensional image, the two-dimensional image into a model including a trained neural network, receiving from the model a plurality of regions of interest each having a score, the plurality of regions of interest each associated with a two-dimensional image included in the plurality of two-dimensional images, determining that a target region of interest has a highest score of any of the plurality of regions of interest that at least partially overlap a two-dimensional location of the target region of interest, where the two-dimensional location is shared by each of the plurality of two-dimensional images, and generating a composite two-dimensional image based on the target region of interest and at least one of the plurality of two-dimensional images.

[0019] In this method, each region of interest may further include an array of pixels each having an intensity value.

[0020] In the method, the generating step may include setting a first intensity value of a first pixel in the composite two-dimensional image equal to a second intensity value in the region of interest.

[0021] In the method, the generating step may include setting a first intensity value of a first pixel included in the composite two-dimensional image equal to a second intensity value of a second pixel included in a second two-dimensional image included in the plurality of two-dimensional images, the second intensity value being not associated with any region of interest.

[0022] In the method, the generating step can include determining a first coverage area associated with a first region of interest in the plurality of regions of interest, the first coverage area including a second array of pixels, determining a second coverage area associated with a second region of interest in the plurality of regions of interest, the second coverage area including a third array of pixels, and determining a first intensity value for a first pixel in the composite two-dimensional image based on a second intensity value in the first coverage area and a third intensity value in the second coverage area, The first coverage area includes at least one pixel not included in the first region of interest and the second coverage area includes at least one pixel not included in the second region of interest. Determining a first intensity value for the first pixel can include setting the first intensity value equal to the sum of the second intensity value multiplied by the first weight and the third intensity value multiplied by the second weight.

[0023] In the method, the generating step may include determining that the first region of interest and the second region of interest overlap, determining that the first region of interest has a higher relevance score than the second region of interest, and not including any portion of the second region of interest in the composite two-dimensional image if it is determined that the first region of interest has the higher relevance score.

[0024] In this method, the three-dimensional image can be generated using digital breast tomosynthesis.

[0025] The method may further include extracting at least one region of interest from the plurality of regions of interest, and generating a second synthetic two-dimensional image based on the plurality of regions of interest and at least one of the plurality of two-dimensional images.

[0026] In yet another non-limiting aspect, the present disclosure provides a system for determining a malignancy likelihood score for breast tissue of a patient, the system comprising: a memory configured to store a plurality of two-dimensional images of the breast tissue derived from a three-dimensional image of the breast tissue; and a processor configured to access the memory, the processor configured to input each of the two-dimensional images to a first model comprising a trained first neural network, receive for each of the two-dimensional images a plurality of indicators from the first model each associated with the two-dimensional image in the plurality of two-dimensional images, generate a composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images, input the composite two-dimensional image to a second model comprising a trained second neural network, and determine a malignancy likelihood score using the second model, the system further comprising a display configured to display a report including the malignancy likelihood score.

[0027] In the method, the model can be trained based on an image dataset including two-dimensional full-field digital mammography images with physician annotations, and in the method, each of the multiple two-dimensional images of breast tissue can be a slice of a three-dimensional tomosynthesis volume.

[0028] In another non-limiting aspect, the present disclosure provides a system for generating a composite two-dimensional image of a patient's breast tissue, the system comprising: a memory configured to store a plurality of two-dimensional images of the breast tissue derived from a three-dimensional image of the breast tissue; and a processor configured to access the memory, the processor inputting each of the two-dimensional images into a model including a trained neural network, receiving from the model, for each of the two-dimensional images, a plurality of regions of interest each having a score, the plurality of regions of interest each associated with a two-dimensional image included in the plurality of two-dimensional images, determining that a target region of interest has a highest score of any of the plurality of regions of interest that at least partially overlaps with a two-dimensional location of the target region of interest, where the two-dimensional location is shared by each of the plurality of two-dimensional images, and generating a composite two-dimensional image based on the target region of interest and at least one of the plurality of two-dimensional images.

[0029] In another non-limiting aspect, the present disclosure provides a system for determining a malignancy potential score for breast tissue of a patient. The system comprises a memory configured to store a plurality of two-dimensional images of breast tissue derived from a three-dimensional image of the breast tissue, and a processor configured to access the memory, wherein the processor is configured to input each two-dimensional image to a first model including a trained first neural network, receive for each two-dimensional image a plurality of indicators from the first model each associated with a two-dimensional image included in the plurality of two-dimensional images, generate a first synthetic two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images, generate a second synthetic two-dimensional image based on the first synthetic two-dimensional image and at least one of the plurality of two-dimensional images, generate a third synthetic two-dimensional image based on the first synthetic two-dimensional image and at least one of the plurality of two-dimensional images, input the first synthetic two-dimensional image, the second synthetic two-dimensional image, and the third synthetic two-dimensional image to a second model including a trained second neural network, and determine a malignancy likelihood score using the second model, wherein the system further comprises a display configured to display a report including the malignancy likelihood score.

[0030] In yet another non-limiting aspect, the present disclosure provides a system for generating a composite two-dimensional image of a patient's breast tissue, the system comprising: a memory configured to store a plurality of two-dimensional images of the breast tissue derived from a three-dimensional image of the breast tissue, the plurality of two-dimensional images being associated with a slice number; and a processor configured to access the memory, the processor inputting a first two-dimensional image, a second two-dimensional image, and a third two-dimensional image of the plurality of two-dimensional images into a model including a trained neural network, where the slice number associated with the first two-dimensional image differs from the slice number associated with the second two-dimensional image by a predetermined offset value, and the slice number associated with the second two-dimensional image differs from the slice number associated with the third two-dimensional image by a predetermined offset value. a plurality of regions of interest each having a score for each of the plurality of two-dimensional images from the model, the plurality of regions of interest differing from a slice number associated with the two-dimensional image by a predetermined offset value, the plurality of regions of interest each associated with a two-dimensional image from the plurality of two-dimensional images; determining that the target region of interest has a highest score of any of the plurality of regions of interest that at least partially overlaps with a two-dimensional position of the target region of interest, wherein the two-dimensional position is shared by each of the plurality of two-dimensional images; and generating a composite two-dimensional image based on the target region of interest and at least one of the plurality of two-dimensional images. [Brief description of the drawings]

[0031] [Figure 1] FIG. 1 is a block diagram illustrating an example X-ray imaging system. [Diagram 2] FIG. 2 is a block diagram illustrating an example embodiment of a model for generating a region of interest for a two-dimensional slice of three-dimensional tomosynthesis data. [Diagram 3] 1 is a flowchart illustrating example steps in a process for creating a composite image using a model trained to detect regions of interest in two-dimensional slices of tomosynthesis data. [Figure 4]FIG. 2 is a schematic diagram illustrating a region of interest associated with an example temporary composite image. [Diagram 5] 5 is another schematic diagram illustrating regions of interest and coverage associated with the example provisional composite image of FIG. 4. [Figure 6] FIG. 13 is another block diagram illustrating an example quadratic model for generating a malignancy likelihood score. [Figure 7] 1 is a flow chart illustrating an example of steps in a process for creating a synthetic image using a model trained to estimate malignancy of tumors and / or lesions in breast tissue. [Figure 8] 8 is a collection of images showing the visual data flow path of the process of FIG. 7. [Figure 9] 1 illustrates an example embodiment of a model for generating a region of interest (ROI) for a second 2D slice of 3D tomosynthesis data based on a first 2D slice of 3D tomosynthesis data, the second 2D slice of 3D tomosynthesis data, and a third 2D slice of the 3D tomosynthesis data. [Figure 10] This is a process for generating a synthetic image using the model in FIG. [Figure 11] 13 is another exemplary quadratic model for generating a malignancy likelihood score. [Figure 12] A process for creating three composite images using a model trained to predict the malignancy of tumors and / or lesions in breast tissue and generating a malignancy likelihood score based on the composite images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] This application discloses example systems and methods for efficiently and consistently displaying relevant regions of three-dimensional (3D) tomography data using one or more synthesized two-dimensional (2D) images, and for efficiently and consistently determining malignancy using one or more synthesized images using machine learning techniques.

[0033] Other than the systems and methods described below, there may be other approaches for analyzing relevant regions of 3D tomosynthesis data of breast tissue and / or determining malignancy of tumors and / or lesions, but they all have drawbacks. One such approach is to provide human physician labels of relevant regions of all slices included in a 3D mammogram. Human-based approaches can be expensive and slow, as it takes a significant amount of time for a human physician to label each and every 2D slice. Moreover, human-based approaches can be inconsistent, as different human physicians have different tendencies and levels of expertise.

[0034] Another approach is to train a machine learning model that receives the entire 3D tomography dataset (i.e., all slices contained in the 3D tomography dataset) and outputs a malignancy likelihood score indicative of the malignancy of tumors and / or lesions present in the breast tissue. This approach may be infeasible and prone to overfitting due to the size of the data and memory limitations of data processing systems.

[0035] Yet another approach would be to randomly select a slice from the set of 2D slices, or sample or select a default slice from the set of 2D slices (e.g., the 25th slice or the middle slice, etc.) and train the machine learning model to output a malignancy likelihood score based on that slice, which may also be prone to overfitting since the slice may not include tumor and / or lesions if they are present.

[0036] Referring to FIG. 1, an example of an X-ray imaging system 100, such as a 3D digital breast tomosynthesis (DBT) system, is shown. The X-ray imaging system 100 can include an X-ray source assembly 108 coupled to a first end 110 of an arm 102. An X-ray detector assembly 112 can be coupled near an opposite end 114. The X-ray source assembly 108 can extend substantially perpendicular to the arm 102 and can be oriented toward the X-ray detector assembly 112. The X-ray detector assembly 112 also extends from the arm 102 to receive X-rays generated by the X-ray source assembly 108 that pass through the breast and impinge on the X-ray detector assembly 112. A breast table plate 116 and a breast compression plate 118 are disposed between the X-ray source assembly 108 and the X-ray detector assembly 112. The X-ray source assembly 108 can be stationary or movable. The X-ray imaging system 100 can generate a reconstructed image including 3D DBT data. The 3D DBT data can include a number of 2D slices. In some embodiments, the reconstructed image may include 3D tomographic data including multiple 2D slices having a thickness of about 1 mm. The 2D slices may be used to create a composite 2D image, as described below. In some embodiments, the x-ray imaging system 100 may generate intermediate two-dimensional "slabs" that represent, for example, the highest intensity projections of a subset of the 2D slices. For example, a single highest intensity projection slab may be generated from 10 slices, or multiple slabs may be generated from multiple slices, such as 10-100 slices. The reconstructed image may be used as an input to the computer 126, which stores the reconstructed image in a mass storage device 128, which may include a memory. The computer 126 may also provide instructions to the x-ray imaging system 100 to control the generation of the reconstructed image.

[0037] Referring to FIG. 2, an example embodiment of a model 200 for generating regions of interest (ROIs) for a two-dimensional (2D) slice 204 of 3D tomosynthesis data is shown. The ROIs may be referred to as indicators. The model 200 receives the 2D slice 204 and may receive any number of ROIs, such as a first ROI including a first region 208A and a first score 208B, and a second ROI including a second region 212A and a second score 212B. Each ROI may be associated with a slice number indicating the 2D slice from which it was generated, such as the slice number of the fourth slice in a set of 75 2D slices. The slice number may be used in selecting and / or combining ROIs to create a composite image, as described below. Depending on the characteristics of the 2D slice 204, a zero ROI may be output, such as a zero ROI if it is determined that the 2D slice 204 does not have a region sufficiently indicative of a potential malignancy. Each ROI may include a region that may be a sub-region of the 2D slice 204. As mentioned above, each 2D slice may be in the form of an array of pixels. A sub-region may be a subset of the pixel array. In some embodiments, the model 200 may include one or more neural networks configured to detect objects in the 2D image. The objects may be ROIs.

[0038] In some embodiments, the model 200 can output an ROI that follows a predetermined shape. For example, a rectangular bounding box can be used to enclose a possible tumor or lesion candidate. Irregular shapes (such as "blobs" of pixels) can also be used to better delineate possible tumors or lesions. When creating a training database of ROIs, a human physician may find rectangular bounding boxes more intuitive than bounding boxes of other shapes. A neural network that uses a segmentation mask-based approach to identify objects can also be used to output a predicted ROI of irregular shape. The model 200 can then be trained to identify a rectangular ROI that includes a sub-array of pixels contained in the 2D slice 204. Each pixel of the ROI can have one or more color intensity values ​​(such as a white intensity value) and a location in the 2D slice 204 (e.g., a pixel at a particular (x,y) location in a 2000x1500 pixel slice). Although some mammography imaging systems generate grayscale images of breast tissue, it is clear that the above model can be used with color 2D slices. Each 2D slice of the 3D tomosynthesis data may be the same size, for example 2000x1500 pixels.

[0039] In addition to the sub-array of pixels, the ROI may further have a relevance score, which indicates how relevant the sub-array of pixels is in determining the malignancy likelihood score (as described below with reference to FIG. 7). The relevance score may be used to create a composite 2D image using one or more ROIs, as described in more detail below. The relevance score may be selected from a numerical range, such as 0-1. When identifying ROIs for the training dataset, a human physician may assign a relevance score to each ROI within the numerical range. Alternatively, the human physician may assign a relevance score using a different scale, such as 0-100, with a higher score indicating a higher likelihood of malignancy, which may then be normalized to the relevance score range used by the model 200. In some embodiments, the human physician may perform the task of identifying ROIs as benign in order to better train the model 200 to identify potentially malignant ROIs.

[0040] In some embodiments, the model 200 may include a neural network, such as a convolutional neural network. To train the model 200, a training data set may be used that includes 2D data consisting of slices from a set of full-field digital mammography (FFDM) and / or 3D tomosynthesis images and pre-identified ROIs (e.g., by one or more physicians). A human physician may examine a given 2D image, identify ROIs by outlining any regions of potential interest using predefined shapes, such as rectangular boxes, and assign a relevance score to the predefined shapes based on his or her medical knowledge and / or experience in evaluating tumors and / or lesions. Alternatively, the relevance score may be assigned based on pathology results indicating whether the lesion is malignant or not. A large training database may be generated by having one or more physicians identify (e.g., annotate, etc.) ROIs in 2D or 3D tomosynthesis image slices taken from multiple FFDM images (e.g., images from multiple patients). One advantage of using FFDM images is that there are currently more annotated FFDM images publicly available than annotated 3D tomosynthesis images. Furthermore, annotating 2D images is easier than annotating 3D tomosynthesis images, which may require annotating a large number of individual slices in each 3D tomosynthesis image. After model 200 has been trained, it can receive an input 2D slice and output one or more ROIs, each of which includes an estimated relevance score and a subarray of pixels of the input 2D slice.

[0041] Model 200 may include multiple layers, such as convolutional layers. It is understood that some embodiments of model 200 may have a different number of layers, a different arrangement of layers, or other differences. However, in all embodiments, model 200 may receive an input 2D input slice and output any region of interest associated with the input 2D input slice. Model 200 may be a single stage detection network that includes one or more sub-networks.

[0042] The model 200 may include a first sub-network 216. The first sub-network 216 may be a feed-forward residual neural network ("ResNet") having one or more layers 218A-C. A second sub-network 220 may be formed on top of the first sub-network to effectively create a single neural network using the first sub-network 216 as the backbone of the single neural network. The second sub-network 220 may include multiple layers including a first layer 222A, a second layer 222B, and a third layer 222C, although the number of layers used may vary (e.g., five layers), with three layers shown for simplicity. Each of the first layer 222A, the second layer 222B, and the third layer 222C may be a convolutional layer. Each layer may consist of a number of building blocks (not shown). Each building block may include multiple parameter layers, such as three parameter layers, each with multiple filters (e.g., 256, etc.) of a particular filter size (e.g., 3×3, etc.). Each of the first layer 222A, second layer 222B, and third layer 222C may have an associated output size of 144×144, 72×72, 36×36, etc. The output size may vary for each input slice based on pre-processing conditions and / or parameters. As the output size decreases between each layer of the second sub-network 220, the number of filters in the parameter layers may be increased proportionally. That is, the number of filters doubles when the output size is halved. The second sub-network may also include a global average pooling layer connected to the final layer (i.e., the third layer 222C), a fully connected layer connected to the global average pooling layer, and a softmax layer with output size 1×1 (i.e., single-valued) connected to the fully connected layer.

[0043] The model 200 may include multiple tertiary sub-networks, such as a first tertiary network 224A, a second tertiary network 224B, and a third tertiary network 224C. Each tertiary network 224A-C may be connected to a layer of the second sub-network 220. The first tertiary network 224A may be connected to the first layer 222A, the second tertiary network 224B may be connected to the second layer 222B, and the third tertiary network 224C may be connected to the third layer 222C. Each tertiary network may receive features from a layer of the second sub-network 220 to detect tumors and / or lesions at different scale levels.

[0044] Each tertiary network may include a box regression sub-network 226. The box regression sub-network 226 may include one or more convolutional layers 228A-B, each followed by a rectified linear (ReLU) activation unit, and may also include a final convolutional layer 230 that outputs regression coordinates corresponding to anchors associated with a portion of one of the layers of the second sub-network 220 (the regression coordinates correspond to an array of pixels of the input 2D slice 204). The anchors may be pre-determined sub-arrays of the layers of the second sub-network 220. The regression coordinates may represent predicted offsets between the anchors and the predicted bounding box. For each bounding box included in the ROI, the bounding box coordinates may be calculated using a set of regression coordinates (e.g., four regression coordinates) and the corresponding anchors.

[0045] Each tertiary network may include a classification sub-network 232. The classification sub-network 232 may include one or more convolutional layers 234A-B, each followed by a ReLU activation, and the classification sub-network 232 may further include a final convolutional layer 238, followed by a sigmoid activation that outputs a prediction of the presence or absence of an object (i.e., the presence or absence of a malignant tumor and / or lesion). The classification sub-network 232 may be used to obtain one or more estimations of whether the patient has a malignant tumor and / or lesion at multiple spatial locations of the 2D slice 204. More specifically, each bounding box may be associated with an estimation score output by the classification sub-network. In some embodiments, the value of each estimation score may range from 0 to 1. One of the spatial locations may include one layer of the second sub-network 220, i.e., the entire first layer 222A. In this manner, the classification sub-network 232 can output an inference based on the 2D slices as to whether the patient has a malignant tumor and / or lesion. The final convolutional layer 238 can be followed by a softmax activation in a model that is trained to classify multiple types of malignant regions, such as multiple levels of malignancy (e.g., low risk regions, high risk regions, etc.).

[0046] The model 200 may include an output layer 250 that normalizes data across scales, calculates bounding box coordinates, and filters out low scoring bounding box predictions. The output layer 250 may receive the output of the tertiary sub-networks 224A-C and output one or more ROIs, each having an array of pixels scaled to the array size of the 2D slices 204 and an associated score. The pixel array may be a bounding box (e.g., a rectangular bounding box, etc.) calculated based on the regression coordinates and the anchors. The output layer 250 may filter out all scores below a predefined threshold, e.g., below 0.5. In some embodiments, the output layer 250 may receive the output from the tertiary sub-networks 224A-C and output a single malignancy likelihood score. In some embodiments, the score of the highest scoring bounding box may be selected as the single malignancy likelihood score.

[0047] 1 and 3, a process 300 for creating a composite image using a model trained to detect ROIs in 2D slices of 3D tomosynthesis data is shown. The process 300 may include one or more steps for selecting one or more of the most relevant ROIs output from the model and creating a composite image using the one or more ROIs. The process 300 may be implemented as instructions on one or more memories of a data processing system. The data processing system may further include one or more processors in communication with the one or more memories and configured to execute the instructions. The one or more processors may be configured to access a memory that stores 3D tomosynthesis data including multiple two-dimensional slices, and the memory may be included in the mass storage device 128.

[0048] At 304, the process 300 may receive 3D tomosynthesis data of breast tissue of a patient. The 3D tomosynthesis data may be generated by a 3D mammography imaging system, such as the x-ray imaging system 100. The 3D tomosynthesis data may include multiple 2D slices corresponding to a predetermined thickness, such as 1 mm, of breast tissue. Depending on the patient and / or imaging system, the 3D tomosynthesis data may include approximately 10-150 or more 2D slices. Each 2D slice may be a pixel array of a predetermined size, such as 2000×1500 pixels. The process 300 may then proceed to 308.

[0049] At 308, the process 300 may input each of the multiple 2D slices into a trained model that is capable of detecting an ROI based on the 2D slices. The trained model may be model 200 described above. In some embodiments, all of the 2D slices may be input into the trained model. In some embodiments, a subset of the 2D slices may be input into the trained model, for example, every fourth 2D slice. The process 300 may then proceed to 312. In some embodiments, the trained model may have been trained based on an image dataset that includes two-dimensional full-field digital mammography images that have been annotated by a physician.

[0050] At 312, the process 300 may receive multiple ROIs output by the model for each 2D slice input to the model. The model may output zero ROIs, one ROI, or multiple ROIs. Depending on the 2D slice and / or the model, most slices may not have any ROIs at all. As described above, an ROI may have a score and a sub-array of pixels of the 2D slice, with each pixel having an intensity value and a location in that 2D slice. Each ROI may be associated with a slice number that indicates which 2D slice of the 3D tomosynthesis data the ROI was generated from. The process 300 may then proceed to 316.

[0051] The process 300 may filter out from the plurality of ROIs at 316 all ROIs having a score below a predefined threshold. The threshold may be selected to include more ROIs in the final composite image (by selecting a higher) or fewer ROIs in the final composite image (by selecting a lower threshold), to reduce possible false negative results (by selecting a lower threshold), and / or to reduce possible false positive results (by selecting a higher threshold). For example, if the scores are in the range of 0 to 1, the user may select a threshold of 0.5. The process 300 may then remove from the plurality of ROIs all ROIs having a score below 0.5. The process 300 may then proceed to 320.

[0052] The process 300 may determine whether the plurality of ROIs includes at least one ROI at 320. The process 300 may then proceed to 324.

[0053] If the process 300 determines at 324 that the plurality of ROIs does not include at least one ROI ("No" at 324), then the process 300 may proceed to 328. Alternatively, if the process 300 determines that the plurality of ROIs does include at least one ROI, then the process 300 may proceed to 332.

[0054] The process 300 may generate the final composite image at 328 without using any ROI. In some embodiments, the process may select a default slice from the 2D slices in the 3D tomosynthesis data to be used as the final composite image. The default slice may be selected randomly, or may be the first slice from the 2D slices (e.g., the first slice from the 75 2D slices), or may be the last slice from the 2D slices (e.g., the 75th slice from the 75 2D slices), or may be the middle slice from the 2D slices (e.g., the 38th slice from the 75 2D slices). Each pixel in the default slice may then be included as a pixel in the final composite image.

[0055] In some embodiments, the process 300 can select a pixel having a maximum intensity value at a particular pixel location in the 2D slice (e.g., an (x,y) location in the array) as the pixel to be included in the final composite image. In other words, the process 300 can use a maximum intensity projection to generate a final composite shape based on the 2D slices. The final composite image can have dimensions equal to each of the 2D slices. The process 300 can determine which slice(s) has the maximum intensity value for each pixel location, and the maximum intensity value at that pixel location can be used in the final composite image. In this way, the pixel with the maximum intensity, which may correspond to a possible lesion, can be included in the final composite image.

[0056] In some embodiments, the process 300 can average two or more intensity values ​​of two or more 2D slices for each pixel location. The two or more 2D slices can be selected by selecting every xth 2D slice (specifically every 3rd or 5th slice), or by selecting the first and last 2D slices, or by selecting some other subset of the 2D slices, or by selecting all slices in the 2D slices. The intensity values ​​of each pixel location can then be averaged to provide an overall view of the breast tissue.

[0057] The process 300 may populate 332 a region of the preliminary composite image with one or more ROIs from the plurality of ROIs. The preliminary composite image may be an array of the same size as the final composite image with each pixel intensity value initialized to a null value. The process 300 may then add one or more ROIs to the preliminary composite image. The ROIs may be added based on one or more criteria. In some embodiments, the process may determine the ROI with the highest score for each pixel location. The intensity value of the preliminary composite image at that pixel location may then be set equal to the intensity value of the ROI at that pixel location. If multiple ROIs have the same score at a particular pixel location, the ROI with the largest number of pixels and / or the ROI with the largest number of pixels surrounding the pixel location and / or the ROI with the largest intensity value at the pixel location may be selected as the ROI at that pixel location. In some embodiments, the placement in the preliminary composite image may be based on an intersection-over-union (IoU) threshold. In the case where multiple ROIs overlap at a particular pixel, the process 300 may "suppress" all ROIs except the highest scoring ROI by not using the lower scoring ROIs for placement in the preliminary composite image. The ROI used for placement in the preliminary composite image may be removed from the plurality of ROIs, and any remaining ROIs in the plurality of ROIs may be used to create additional composite images, as described below. Alternatively, a high scoring ROI used for placement in the preliminary image may be placed in the additional composite image where it does not overlap with any low scoring ROIs that were not used in the preliminary image. For example, ROIs with scores of 0.85 and 0.75 may be used for placement in the preliminary image. The ROI with a score of 0.85 may overlap with a lower scoring ROI with a score of 0.65 (e.g., occupy at least one of the same pixel locations as the ROI with a score of 0.65), and the ROI with a score of 0.75 may not overlap with the ROI with a score of 0.65.An additional composite image may then be generated using the ROI with a score of 0.75 and the ROI with a score of 0.65, and each pixel of the preliminary composite image may be associated with the slice number of the ROI used to add the given intensity value of that particular pixel. Process 300 may then proceed to 336.

[0058] The process 300 may fill in the unplaced regions of the provisional composite image at 336. In some embodiments, the process 300 may use a watershed-by-flooding algorithm to fill in the unplaced regions. To perform the watershed-by-flooding algorithm, the process 300 may place the unplaced regions based on the 2D slice associated with each ROI. The process 300 may generate a coverage region for each 2D slice initialized with pixels of a particular ROI. For example, if an ROI has an array of pixels with corners at coordinates (10,11), (50,11), (10,40), and (50,40), the process may include this array of pixels in the coverage region of the 2D slice. The process 300 may then iteratively expand the coverage region until each pixel of the provisional composite image is associated with a slice number, and thus with a 2D slice by the expansion. After each pixel is associated with a 2D slice, the process 300 can set, for each pixel location, the intensity value of each pixel in the provisional composite image equal to the intensity value of the pixel at that particular pixel location in the associated 2D slice. The process 300 can iteratively expand each coverage region until all pixels in the provisional composite image are associated with a slice number, and thus associated with a 2D slice by the expansion. In some implementations, the process can expand each coverage region by one pixel up, down, left, or right. For example, if a coverage region has an array of pixels with corners at coordinates (10,11), (50,11), (10,40), and (50,40), the process can expand the array to have corners at coordinates (9,10), (51,10), (9,41), and (51,41). Process 300 may stop expanding the coverage area at a border with an adjacent coverage area where further expansion of the coverage area would extend into pixels already associated with another coverage area, or where the virtual composite image ends.In cases where process 300 extends multiple coverage regions to the same pixel, process 300 can extend the coverage region associated with the highest scoring ROI to that pixel. Process 300 can iteratively extend the coverage regions according to the protocol above until all pixels in the provisional composite image are associated with one coverage region, and thus one 2D slice, through the extension.

[0059] In some embodiments, the process 300 can use a "gradient-based" technique to fill in the unplaced regions. The process can iteratively expand the coverage area from a slice for a predetermined number of iterations, and then continue to place the unplaced regions using adjacent slices until the center slice is reached. As an example, the process 300 can use an ROI in slice 5 of a 40-slice DBT volume (i.e., slice 20 is the center slice) and expand the ROI to place the unplaced regions for a predetermined number of iterations, such as 10. After the 10th iteration, the process 300 can use the 6th slice to continue to place the unplaced regions. In this way, pixels that are placed within a distance of 10 pixels from the original ROI are included in the 5th slice, while the next 10 pixels are included in the 6th slice. The process 300 can continue to increment the slice number (or decrement the slice number if the original slice number is greater than the central slice, e.g., the 36th slice) used to place the unplaced regions until it reaches the central slice number (i.e., 20), at which point all other remaining extensions start at slice 20. This "tilt-based" approach can provide a smoother junction when two ROIs meet, as well as a more "classic" view of the breast. Additionally, the process 300 can adjust the spatial location of the original ROI to better match its spatial location relative to the central slice. For example, slices before and after the central slice can have breast borders that take up more image than the central slice, and the process can then adjust the location of the ROI in the final composite image.

[0060] In some embodiments, the process 300 can fill in the unplaced regions of the provisional composite image based on a default slice. In some embodiments, the process can select a default slice of a 2D slice included in the 3D tomosynthesis data to use to fill in the unplaced regions of the provisional composite slice. The default slice can be selected randomly, or the first slice of the 2D slices (e.g., the first slice of 75 2D slices), or the last slice of the 2D slices (e.g., the 75th slice of 75 2D slices), or the middle slice of the 2D slices (e.g., the 38th slice of 75 2D slices). The process 300 can associate each unplaced pixel with a slice number of the default slice and set the intensity value of each unplaced pixel of the provisional composite image to be equal to the intensity value of the pixel at the corresponding pixel location in the default slice. In some embodiments, the process 300 may set the intensity value of the unplaced pixels by selecting a pixel with a maximum intensity value at a particular pixel location in the 2D slice (e.g., an (x,y) location in the array, etc.) and setting the intensity value of the unplaced pixels equal to the maximum intensity value. In other words, the process 300 may use a projection of the maximum intensity to fill in the intensity values ​​of the unplaced pixels based on the 2D slices. For each pixel location, the process 300 may determine which slice(s) has the maximum intensity value, associate each pixel in the provisional composite image with the slice number of the 2D slice with the maximum intensity value, and set the intensity value of the unplaced pixels equal to the maximum intensity value at that pixel location across all the 2D slices. In this way, the provisional composite image may include pixels with the maximum intensity that may correspond to a possible lesion.

[0061] In some embodiments, after process 300 generates a preliminary composite image with intensity values ​​placed at every pixel, process 300 may proceed to post-processing the preliminary composite image at 340. In some embodiments, process 300 may proceed to 344 by setting the final composite image to be equal to the preliminary composite image.

[0062] The process 300 may perform post-processing on the provisional composite image at 340. In some embodiments, the process 300 may blend edges of coverage areas in the provisional image to reduce any boundary effects between the coverage areas. As described above, each pixel in the provisional image may be associated with a slice number and / or a 2D slice. For regions of pixels near where two coverage areas in the provisional image meet, the process 300 may set intensity values ​​for those pixels based on intensity values ​​at the pixel locations of the particular pixels in each coverage area. The process 300 may set intensity values ​​at pixel locations near where coverage areas in the provisional composite image meet as follows: iv p =w1iv j +w2iv k (1) In the above formula, iv p is the intensity value at a particular pixel location of the provisional composite image, w1 and w2 are the first and second weights, the sum of both weights is 1, and iv j is the intensity value at a particular pixel location in the first coverage region, and iv k is the intensity value at a particular pixel location in the second coverage region. In other words, iv j and iv. kis the intensity value at a particular pixel location in each corresponding 2D slice. For pixels located at the boundary between coverage regions, such as when the pixels are only one step apart in the x or y direction, the first weight w1 and the second weight w2 can each be set to 0.5, reflecting an equal emphasis on the "home" coverage region (i.e., the coverage region that contains the pixel) and the coverage region bordering the home coverage region. For pixels located further away from the boundary, such as a pixel located two pixels away from the boundary in the first coverage region, the first weight w1 can be set equal to 0.8 and the second weight w2 can be set equal to 0.2, reflecting a greater emphasis on the home coverage region of the pixel. In some embodiments, the process 300 can set the weights using a mathematical function, such as a linear increase / decrease function. For example, for pixels located on the boundary, pixels located one pixel away from the boundary, and pixels located two pixels away from the boundary, the process 300 may set the first weight of each pixel to 0.5, 0.75, and 1.0, respectively, and the second weight of each pixel to 0.5, 0.25, and 0.0, respectively. Other mathematical functions such as logarithmic or exponential increase / decrease of weights with distance from the boundary may be used to set the weights, as long as the sum of the weights is 1. The process 300 may use the above formula (1) to determine the intensity value of a pixel only if the pixel is located within a threshold distance of the boundary between adjacent coverage regions, e.g., less than three pixels away from the boundary. The process 300 may then use formula (1) and an applicable mathematical function, e.g., a linear increase / decrease function, to determine the intensity values ​​of all pixels within the threshold distance. Pixels not within the threshold distance may maintain their previously assigned intensity values. After the edges of the coverage areas have been blended, the process 300 may set the final composite image equal to the (processed) preliminary composite image. The process 300 may then proceed to 344.

[0063] At 344, the process 300 may output the final composite image to a memory for storage and / or use in other processes, and / or to a display, such as a computer monitor, for viewing by a human physician. For example, the process 300 may output the final composite image to a display of a medical device where a physician may view the final composite image and infer information about tumors and / or lesions, if any, present in the final composite image. As another example, additional processes (described below) may use the final composite image to predict the malignant potential of tumors and / or lesions in breast tissue. The final composite image may be used to replace the stored data for all slices included in the 3D tomosynthesis data, thereby reducing storage space by storing only an array of pixels of the final composite image and associated data about regions of interest, instead of storing 10-150 or more slices. This reduction in memory requirements may reduce the size of the servers required for information technology systems used by hospitals. The process 300 may then proceed to 348.

[0064] The process may determine at 348 whether an additional composite 2D image should be generated. The process 300 may determine whether the plurality of ROIs includes at least one ROI. If the plurality of ROIs does not include at least one ROI, the process 300 may determine that no more composite images should be generated. If the plurality of ROIs includes at least one ROI, the process 300 may determine that an additional composite image should be generated. The process may then proceed to 352.

[0065] If the process 300 determines that no more composite images should be generated at 352, the process 300 may proceed to end. If the process 300 determines that more composite images should be generated, the process 300 may proceed to 332.

[0066] 3 and 4, the ROIs associated with an example provisional composite image are shown. The ROIs are shown positioned on an array 404 of the provisional composite image. A first ROI 400A may be associated with a fifth 2D slice of the set of 2D slices. A second ROI 400B may be associated with a second 2D slice of the set of 2D slices. A third ROI 400C may be associated with an eighth 2D slice of the set of 2D slices. A fourth ROI 400D may be associated with a fourth 2D slice of the set of 2D slices. The array 404 may include one or more unpositioned pixels 408. FIG. 4 may represent the state of the provisional composite image after step 332 has been performed.

[0067] 3 and 4 and FIG. 5, the ROIs and coverage regions associated with the example provisional composite image of FIG. 4 are shown. The first ROI 400A may be associated with the first coverage region 500A. The second ROI 400B may be associated with the second coverage region 500B. The third ROI 400C may be associated with the third coverage region 500C. The fourth ROI 400D may be associated with the fourth coverage region 500D. Each pixel of the array 404 may be associated with one of the coverage regions 500A-D. FIG. 5 may represent the state of the provisional image after step 336 has been performed, and in particular after the watershed-by-flooding algorithm has been performed.

[0068] 3 and 6, an example quadratic model 600 for generating a malignancy likelihood score 654 is shown. The quadratic model 600 may receive an input composite image 604, which may be generated using process 300 as described above, and output a malignancy likelihood score 654. The malignancy likelihood score 654 may range from 0 to 1. In some embodiments, the malignancy likelihood score 654 may indicate a category of risk, such as a low, medium, or high risk category. In some embodiments, the malignancy likelihood score 654 may be selected from a numerical range, such as the integers 1 to 5, with 1 indicating the lowest risk level and 5 indicating the highest risk level.

[0069] In some embodiments, the secondary model 600 may include a neural network, such as a residual recurrent neural network. To train the secondary model 600, the model may be trained using a training dataset that includes synthetic images labeled as malignant or non-malignant. A human physician may label the synthetic images. For example, synthetic images corresponding to patients known to have cancer may be labeled with a "1" and synthetic images corresponding to patients known not to have cancer may be labeled with a "0". In some embodiments, the synthetic images may be 300. Once trained, the secondary model 600 may receive an input synthetic image and output a malignancy likelihood score indicating whether the breast tissue contains a malignant tumor and / or lesion.

[0070] The secondary model 600 may include multiple layers, such as convolutional layers. It is understood that some embodiments of the secondary model 600 may have a different number of layers, a different arrangement of layers, or other differences. However, in all embodiments, the secondary model 600 may receive an input 2D composite image and output a malignancy likelihood score. The secondary model 600 may be a single stage detection network that includes one or more sub-networks.

[0071] 2 and 6, the secondary model 600 may include a primary subnetwork 616 and a secondary subnetwork 620, which may be identical to the primary subnetwork 216 and the secondary subnetwork 220 of the model 200 described above. In some embodiments, the primary subnetwork 616 and the secondary subnetwork 620 may be identical to the primary subnetwork 216 and the secondary subnetwork 220 after the model 200 has been trained. In other words, the secondary model 600 may be initialized with the model weights prior to training.

[0072] The secondary model 600 is important because, although the model 200 described above can detect regions of interest in breast tissue, it may not be able to accurately determine whether the ROI is actually malignant. The secondary model 600 may be used to more accurately estimate malignancy of breast tissue using the composite images generated by the model 200 described above. In testing, the present model 600 was observed to have an increase of 0.03 in the area under the curve (AUC) of the receiver operating characteristic (ROC) when tested on 1000 first-time mammograms (100 cancers) using composite images generated using the process 300 as a control test against composite images generated using the imaging equipment manufacturer's default image generation technique.

[0073] The model 600 may include multiple tertiary sub-networks, such as a first tertiary network 624A, a second tertiary network 624B, and a third tertiary network 624C. Each tertiary network 624A-C may be connected to a layer of the second sub-network 620. The first tertiary network 624A may be connected to the first layer 622A, the second tertiary network 624B may be connected to the second layer 622B, and the third tertiary network 624C may be connected to the third layer 622C. Each tertiary network may receive features from a layer of the second sub-network 620 to estimate malignancy of breast tissue at different scale levels.

[0074] Each tertiary network may include a box regression sub-network 626. The box regression sub-network 626 may include one or more convolutional layers 628A-B, each followed by a rectified linear (ReLU) activation unit, and may also include a final convolutional layer 630 that outputs regression coordinates corresponding to anchors associated with a portion of one of the layers of the second sub-network 620 (the regression coordinates corresponding to an array of pixels of the input composite 2D slice 604). The anchors may be pre-determined sub-arrays of the layers of the second sub-network 620. The regression coordinates may represent predicted offsets between the anchors and the predicted bounding box. For each bounding box included in the ROI, the set of regression coordinates (e.g., four regression coordinates) and the corresponding anchors may be used to calculate the bounding box coordinates.

[0075] Each tertiary network may include a classification sub-network 632. The classification sub-network 632 may include one or more convolutional layers 634A-B, each followed by a ReLU activation, and may further include a final convolutional layer 638, followed by a sigmoid activation that outputs a prediction of the presence or absence of an object (i.e., the presence or absence of a malignant tumor and / or lesion). The classification sub-network 632 may be used to obtain one or more estimations of whether the patient has a malignant tumor and / or lesion at a plurality of spatial locations of the synthetic 2D slice 604. More specifically, each bounding box may be associated with an estimation score output by the classification sub-network 632. The bounding box may also be associated with a slice number as mentioned above. In some embodiments, the value of each estimation score may range from 0 to 1. One of the spatial locations may include all of one layer of the second sub-network 620, i.e., the first layer 622A. In this manner, the classification sub-network 632 can output an inference based on the 2D slices as to whether the patient has a malignant tumor and / or lesion. The final convolutional layer 638 can be followed by a softmax activation in a model that is trained to classify multiple types of malignant regions, such as multiple levels of malignancy (e.g., low risk regions, high risk regions, etc.).

[0076] The model 600 may include an output layer 650 that normalizes data across scales, calculates bounding box coordinates, and filters out bounding box predictions with low scores. The output layer 650 may receive the output of the tertiary sub-networks 624A-C and output one or more ROIs, each having an array of pixels scaled to the array size of the 2D slice 604 and an associated score. The pixel array may be a bounding box (e.g., a rectangular bounding box, etc.) calculated based on the regression coordinates and the anchors. The output layer 650 may filter out all scores below a predefined threshold, e.g., less than 0.5. After filtering, the output layer 650 may determine an array of pixels for each ROI based on one or more anchors associated with the other remaining scores. The output layer 650 may resize the anchors to match the scale of the 2D slice 604 before including them in the array of output ROIs. Such resizing may be necessary if an anchor is associated with a smaller layer of the second sub-network 620. The output layer 650 may receive outputs from the tertiary sub-networks 624A-C and output a malignancy likelihood score 654. In some embodiments, the highest bounding box score may be selected as the malignancy likelihood score 654. In some embodiments, the model 600 may output one or more ROIs 608, each having a score 608A and an array of pixels 608B. The pixel array 208B may be a rectangular bounding box. The one or more ROIs 608 may provide the physician with additional information about the likely malignant regions in the composite image 604.

[0077] 1, 3, and 6, and 7, a process 700 is shown for creating a composite image using a model trained to estimate malignancy of tumors and / or lesions in breast tissue and generating a malignancy likelihood score based on the composite image. The process 700 may be implemented as instructions on one or more memories of a data processing system. The data processing system may further include one or more processors in communication with the one or more memories and configured to execute the instructions. The one or more processors may be configured to access a memory that stores 3D tomosynthesis data including multiple two-dimensional slices, which may be included in mass storage device 128.

[0078] At 704, the process 700 may receive 3D tomosynthesis data of a patient's breast tissue. The 3D tomosynthesis data may be generated by a 3D mammography imaging system, such as the x-ray imaging system 100. The 3D tomosynthesis data may include multiple 2D slices corresponding to a predetermined thickness, such as 1 mm, of the breast tissue. Depending on the patient and / or the imaging system, the 3D tomosynthesis data may include about 10-150 or more 2D slices. Each 2D slice may be a pixel array of a predetermined size, such as 2000×1500 pixels. The process 700 may then proceed to 708.

[0079] At 708, the process 700 may generate one or more composite images (ie, mammograms) using at least one of the above steps 308-352. The process 700 may then proceed to 712.

[0080] At 712, the process 700 can input each of the composite images or images into a trained model to determine a malignancy likelihood score. The trained model can be model 600 described above. The process 700 can then proceed to 716.

[0081] The process 700 may receive a malignancy likelihood score from the trained model at 716. In some embodiments, the malignancy likelihood score may range from 0 to 1, with 1 indicating a high risk of malignancy and 0 indicating minimal or no risk of malignancy. In some embodiments, malignancy may be "yes" (i.e., 1) or "no" (i.e., 0) indicating whether the tumor is malignant or non-malignant. In some embodiments, the malignancy likelihood score may indicate a category of risk, for example, low, medium, or high risk. In some embodiments, the malignancy likelihood score may be selected from a numerical range, for example, from integers 1 to 5, with 1 indicating the lowest risk level and 5 indicating the highest risk level. In some embodiments, the process 700 may also receive one or more regions of interest generated by the trained model, such as region of interest 608 above. The process 700 may then proceed to 720.

[0082] The process 700 may output the malignancy likelihood score at 720 to a memory for storage and / or use in other processes and / or to a display such as a computer monitor for viewing by a human physician. For example, the process 300 may output the malignancy likelihood score to a display of a medical device where a physician may view the malignancy likelihood score and possibly make a patient diagnosis based on the malignancy likelihood score. The malignancy likelihood score may be stored in a medical record database for future analysis and / or research of breast cancer patients. In some embodiments, the process 700 may output one or more of the regions of interest received at 716 to a memory for storage and / or use in other processes and / or to a display such as a computer monitor for viewing by a human physician. The process 700 may output the malignancy likelihood score and / or the one or more regions of interest as a report. The process 700 may then end.

[0083] In testing, processes 300 and 700 were found to detect the presence of breast cancer in both dense and non-dense breasts. A first model according to model 200 of FIG. 2 was trained using the annotated FFDM dataset. A second model according to model 600 of FIG. 6 was trained using the annotated 3D breast image dataset. To train the second model, slices from the 3D images in the dataset were used to generate synthetic images using the process in process 300 of FIG. 3. The synthetic images were then annotated by a physician and input to the second model for training. Each synthetic image can be annotated as having or not having cancer at the full image level. In some embodiments, each synthetic image can be annotated based on annotations associated with the annotated 3D breast image dataset. In this way, the synthetic images can be weakly annotated. The first and second models were then tested using additional 3D breast image datasets using the process in process 700 of FIG. 7. The second model was able to identify cancer in non-dense breasts (BIRADS A, B) with a specificity of 88.9% and a sensitivity of 92%, with an area under the receiver operating characteristic (ROC) curve (AUC) of 0.96. Specificity was selected based on national averages. Additionally, the second model was able to identify cancer in dense breasts (BIRADS C, D) with a specificity of 88.9% and a sensitivity of 85%, with an AUC of 0.94. Overall, performance on additional 3D breast imaging datasets was 88.9% specific, with an AUC of 0.95 and a sensitivity of 91%.

[0084] 7 and 8, a visual data flow path of process 700 is shown. In general, process 700 uses a trained model to determine ROIs 802A-E for each slice 804A-E of the 3D DBT data. Then, a portion of the ROIs, including ROI 802A, ROI 802C, and ROI 802E, are used to synthesize a composite image 808. The composite image 808 is then input to a second trained model 812. The second trained model 812 then outputs a malignancy likelihood score 816.

[0085] 9, an example embodiment of a model 900 for generating a region of interest (ROI) for a second 2D slice 903 of 3D tomosynthesis data based on a first 2D slice 902 of the 3D tomosynthesis data, a second 2D slice 903 of the 3D tomosynthesis data, and a third 2D slice 904 of the 3D tomosynthesis data is shown. As described below, each of the first 2D slice 902, the second 2D slice 903, and the third 2D slice 904 can be input to the model 900, and the model 900 can determine an ROI corresponding to the second 2D slice 903 based on the first 2D slice 902, the second 2D slice 903, and the third 2D slice 904. The model 900 can have three input channels for receiving the three 2D slices. Each 2D slice and any ROIs associated with each 2D slice can be associated with an input channel of data analyzed by the model 900. For example, a first 2D slice 902 can be associated with a first input channel, a second 2D slice 903 can be associated with a second input channel, and a third 2D slice 904 can be associated with a third input channel. As described further below, a composite image associated with the second input channel can be generated using one or more ROIs associated with the second input channel.

[0086] The ROIs may be referred to as indicators. The model 900 may accept input 2D slices 902, 903, 904 and may output any number of ROIs associated with a second 2D slice 903. Each ROI may be associated with a second input 2D slice 903. For example, the model may output a first ROI 958 associated with the second 2D slice 903. Additionally, each ROI may be associated with a slice number indicating the 2D slice input to the second channel. For example, the second slice 903 may be the fourth slice in a set of 75 2D slices, and the first ROI 958 may be associated with slice number 4 (e.g., the fourth slice, etc.). The slice number may be used in selecting and / or combining ROIs to create a composite image, as described below. As described above, each ROI may have a region and a score. For example, the first ROI 958 may have a region 955 and a score 956.

[0087] Each slice number associated with the first 2D slice 902, the second 2D slice 903, and the third 2D slice 904 may be spaced apart by a predetermined value n. For example, if the second 2D slice 903 is associated with slice number x2, then the first 2D slice 902 may be associated with slice number x1=x2-n and the second 2D slice 904 may be associated with slice number x3=x2+n, where n may range from 1 to 5 or more. In some embodiments, the predetermined value n may be 3. In analyzing slice 903, slices 902 and 904 may provide volume context to model 900 that may aid in predicting whether malignant tumors and / or lesions are present in the breast tissue represented by slice 903.

[0088] Each of the first 2D slice 902, the second 2D slice 903 and the third 2D slice 904 may be in the form of an array of pixels of a predetermined size, e.g., 2000x1500. A pixel may have an intensity value (e.g., a white intensity value). Each pixel of the ROI may have an intensity value (e.g., a white intensity value) and a location in the 2D slice associated with it (e.g., a pixel at a particular (x,y) location in a 2000x1500 pixel slice).

[0089] In addition to the subarray of pixels, each ROI may further have a relevance score that indicates how relevant that subarray of pixels is in determining the malignancy likelihood score (as described below with reference to FIG. 11). The relevance score may be used to create a composite 2D image using one or more ROIs, as described in more detail below. The relevance score may be selected from a numerical range, such as 0-1. When identifying ROIs for the training dataset, a human physician may assign a relevance score to each ROI within the numerical range. Alternatively, the human physician may assign a relevance score using a different scale, such as 0-100, with a higher score indicating a higher likelihood of malignancy, which may then be normalized to the relevance score range used by the model 900. In some embodiments, the human physician may perform the task of identifying ROIs as benign in order to better train the model 900 to identify potentially malignant ROIs.

[0090] In some embodiments, the model 900 may include a neural network, such as a convolutional neural network. To train the model 900, a training data set consisting of slices from a set of 3D tomosynthesis images and pre-identified ROIs (e.g., by one or more physicians) may be used to train the model. When training the model 900 on 3D tomosynthesis images, 2D full-field digital mammography (FFDM) images, each with an ROI, may also be included, in which case the 2D images may be replicated across all three input channels. For either FFDM images or tomosynthesis slices, a human physician may examine a given 2D slice, identify ROIs by outlining any areas of potential interest using predefined shapes, such as rectangular boxes, and assign a relevance score to the predefined shapes based on his or her medical knowledge and / or experience in evaluating tumors and / or lesions. Alternatively, a relevance score may be assigned based on pathology results indicating whether the lesion is malignant or not. A large training database can be generated by having one or more physicians identify ROIs in 2D or 3D tomosynthesis image slices taken from multiple FFDM images (e.g., images from multiple patients). Model 900 can be trained by inputting three slices sequentially, offset by a predefined offset value (e.g., n=1) in the case of tomosynthesis slices, or by duplicating a particular FFDM image for the three input channels, including any associated ROIs in either case. Once trained, model 900 can receive three input 2D slices and output one or more ROIs, each containing an estimated relevance score and a subarray of pixels of the second 2D slice 903.

[0091] Model 900 may include multiple layers, such as convolutional layers. It is understood that there are embodiments in which model 900 has a different number of layers, a different arrangement of layers, or other differences. However, in all embodiments, model 900 may receive three 2D input slices and output any region of interest associated with a second 2D slice 903. Model 900 may be a single stage detection network that includes one or more sub-networks.

[0092] The model 900 may include a first sub-network 916. The first sub-network 916 may be a feed-forward residual neural network ("ResNet") having one or more layers 918A-C. Layer 918C may be an input layer configured to accept three input 2D slices 902, 903, and 904. In this manner, the model 900 may use data from all three slices 902, 903, and 904 to detect possible ROIs in a particular slice, such as the first 2D slice 902. A second sub-network 920 may be formed on top of the first sub-network 916 to effectively create a single neural network using the first sub-network 916 as the backbone of the single neural network. The second sub-network 920 may include multiple layers, including a first layer 922A, a second layer 922B, and a third layer 922C, although the number of layers used may vary (e.g., five layers), with three layers shown in the figure for simplicity. Each of the first layer 922A, the second layer 922B, and the third layer 922C may be a convolutional layer. Each layer may consist of a number of building blocks (not shown). Each building block may include a number of parameter layers, such as three parameter layers, each parameter layer having a number of filters (e.g., 256 filters) of a particular filter size (e.g., 3×3 filters). Each of the first layer 922A, the second layer 922B, and the third layer 922C may have an associated output size per input 2D slice, such as 144×144, 72×72, 36×36 filters, etc. The output size may vary from input slice to input slice based on pre-processing conditions and / or parameters. As the output size decreases between layers of the second sub-network 920, the number of filters in the parameter layers may be increased proportionally. That is, the number of filters doubles when the output size is halved. The second sub-network may also include a global average pooling layer connected to the final layer (i.e., the third layer 922C), a fully connected layer connected to the global average pooling layer, and a softmax layer with output size 1×1 (i.e., single-valued) connected to the fully connected layer.

[0093] The model 900 may include multiple tertiary sub-networks, such as a first tertiary network 924A, a second tertiary network 924B, and a third tertiary network 924C. Each tertiary network 924A-C may be connected to a layer of the second sub-network 920. The first tertiary network 924A may be connected to a first layer 922A, the second tertiary network 924B may be connected to a second layer 922B, and the third tertiary network 924C may be connected to a third layer 922C. Each tertiary network may receive features from a layer of the second sub-network 920 to detect tumors and / or lesions at different scale levels in each input 2D slice 902, 903, and 904.

[0094] Each tertiary network may include a box regression sub-network 926. The box regression sub-network 926 may include one or more convolutional layers 928A-B, each followed by a rectified linear (ReLU) activation unit, and the box regression sub-network 926 may also include a final convolutional layer 930 that outputs regression coordinates corresponding to anchors associated with a portion of one of the layers of the second sub-network 920 (the regression coordinates corresponding to an array of pixels of the second input 2D slice 903). The anchors may be pre-determined sub-arrays of the layers of the second sub-network 920. The regression coordinates may represent predicted offsets between the anchors and the predicted bounding box. For each bounding box included in the ROI, the set of regression coordinates (e.g., four regression coordinates) and the corresponding anchors may be used to calculate the bounding box coordinates.

[0095] Each tertiary network may include a classification sub-network 932. The classification sub-network 932 may include one or more convolutional layers 934A-B, each followed by a ReLU activation, and the classification sub-network 932 may further include a final convolutional layer 938, followed by a sigmoid activation that outputs a prediction of the presence or absence of an object (i.e., the presence or absence of a malignant tumor and / or lesion). The classification sub-network 932 may be used to obtain one or more estimations of whether the patient has a malignant tumor and / or lesion at multiple spatial locations of the second input 2D slice 903. More specifically, each bounding box may be associated with an estimation score output by the classification sub-network. In some embodiments, the value of each estimation score may range from 0 to 1. The final convolutional layer 938 may be followed by a softmax activation in a model trained to classify multiple types of malignant regions, such as multiple levels of malignancy (e.g., low risk regions, high risk regions, etc.).

[0096] The model 900 may include an output layer 950 that normalizes data across scales, calculates bounding box coordinates, and filters out low scoring bounding box predictions. The output layer 950 may receive the output of the tertiary sub-networks 924A-C and output one or more ROIs, each having an array of pixels scaled to the array size of the input 2D slices 903 and an associated score. The pixel array may be a bounding box (e.g., a rectangular bounding box, etc.) calculated based on the regression coordinates and the anchors. The output layer 950 may filter out all scores below a predefined threshold, e.g., below 0.5. In some embodiments, the output layer 950 may receive the output from the tertiary sub-networks 224A-C and output a single malignancy likelihood score. In some embodiments, the single malignancy likelihood score may be selected as the score of the highest scoring bounding box associated with any of the input 2D slices 903.

[0097] 1, 3, 6, 7, and 9, and FIG. 10, a process 1000 for generating a composite image using the model 900 is shown. In general, the process 1000 generates a composite image based on an ROI associated with a second input channel of the model 900. The ROI can be generated using the model 900 described above. After the process 1000 generates the composite image, the composite image can be used to determine a malignancy likelihood score using the process 700 in conjunction with the model 600 described above. The process 1000 can be implemented as instructions on one or more memories included in a data processing system. The data processing system can further include one or more processors configured to communicate with the one or more memories and execute the instructions. The one or more processors can be configured to access a memory that stores 3D tomosynthesis data including a number of two-dimensional slices, and the memory can be included in the mass storage device 128.

[0098] The process 1000 may receive 1004 3D tomosynthesis data of a patient's breast tissue. The 3D tomosynthesis data may include multiple 2D slices corresponding to a predetermined thickness of the breast tissue, such as 1 mm. Step 1004 may be identical to step 304 described above. The process 1000 may then proceed to 1008.

[0099] The process 1000 can input three 2D slices from the 3D tomosynthesis data to a trained model for simultaneous processing. The trained model can be model 900 described above. The process 1000 can provide three 2D slices from the 3D tomosynthesis data to a trained model for simultaneous processing. In some embodiments, the trained model can be trained on an image data set that includes two-dimensional full-field digital mammography images annotated by a physician. Each of the three 2D slices is input to a different input channel of the trained model. For example, the first 2D slice can be input to a first input channel, the second 2D slice can be input to a second input channel, and the third 2D slice can be input to a third input channel. The 2D slices can have associated slice numbers that differ by a predefined offset value. For example, for an offset value of two, the first 2D slice can be associated with slice number 1, the second 2D slice can be associated with slice number 3, and the third 2D slice can be associated with slice number 5. In some embodiments, the process 1000 may input three 2D slices to the process until all slices are input to the second input channel (by offsetting the slices with an offset value in the first and second channels). There may be 2D slices for which no other 2D slices have a slice number that is less than or greater than the slice number of the 2D slice by a given offset value. For example, if the 2D slice input to the second input channel is associated with slice number 1 and the offset value is 2, the slice number of the 2D slice input to the first input channel will be negative 1. In such a case, the process 1000 may input the next closest slice (e.g., the 2D slice associated with slice number 1) to the first channel. The process may then "move up" the 2D slices by incrementing by 1 each slice number associated with the input 2D slice.For example, the process 1000 may input 2D slices associated with slice numbers 1, 3, and 5, followed by 2D slices associated with slice numbers 2, 4, and 6. In some embodiments, the process 1000 may input a subset of the 2D slices to the second channel model 900, such as inputting every other slice, every third slice, etc. After all the 2D slices (or every other slice, every third slice, etc.) have been input to the trained model at least once, the process 1000 may proceed to 1012.

[0100] The process 1000 may receive a number of ROIs from the trained model at 1012. Each ROI may be associated with the second input channel and one of the 2D slices. Each ROI may be generated when its associated 2D slice is input to the trained model and the 2D slice is input to the first and third channels of the trained model. The process 1000 may then proceed to 1016.

[0101] The process 1000 may generate and output a composite image based on the multiple ROIs at 1016. Using the multiple ROIs generated at 1012, the process 1000 may perform at least a portion of steps 316-352 described above. The process 1000 may then proceed to 1024. The process 1000 may then end.

[0102] 10 and 11, an example secondary model 1100 for generating a malignancy likelihood score 1154 is shown. The secondary model 1100 can receive three input composite images including a first composite image 1102, a second composite image 1103, and a third composite image 1104. The second composite image 1103 can be generated using the process 1000 described above. The first and third composite images can be generated based on the second composite image and one or more 2D slices, as described below. The model 1100 can output a malignancy likelihood score 1154, which can range from 0 to 1. In some embodiments, the malignancy likelihood score 1154 can indicate a category of risk, such as a low, medium, or high risk category. In some embodiments, the malignancy likelihood score 1154 can be selected from a numerical range, for example, from integers 1 to 5, with 1 indicating the lowest risk level and 5 indicating the highest risk level. The second composite image 1103 can be generated using model 200 or model 1000 described above.

[0103] As discussed above, each pixel in the composite image can be associated with a slice number (or multiple slices, e.g., if blending techniques are used). For example, a first group of pixels in the second composite image 1103 can be associated with the seventh slice, and a second group of pixels in the second composite image can be associated with the fifth slice. The intensity value of each pixel in the first composite image 1102 and the third composite image 1104 can be determined by selecting the intensity value of the pixel in a slice that is plus or minus a predetermined offset number of slices relative to a slice associated with a pixel at a particular location in the second composite image 1103 (specifically, xn for the first composite image 1102 and x+n for the third composite image 1104, where x is the slice number associated with the pixel in the second composite image 1103 and n is the predetermined number of slices). For example, if the predetermined number of slices is 1, the intensity value of a pixel included in the first composite image 1102 and located at the same pixel position as the first pixel group (a pixel group included in the second composite image 1103 and associated with the seventh slice) can be set to be equal to the intensity value of a pixel included in the sixth slice and can be arranged at the same position as the first pixel group. In the case of the third composite image 1104, the intensity value of a pixel included in the third composite image 1104 and located at the same pixel position as the first pixel group (a pixel group included in the second composite image 1103 and associated with the seventh slice) can be set to be equal to the intensity value of a pixel included in the eighth slice and can be arranged at the same position as the first pixel group.

[0104] In an embodiment using blending to generate the second composite image 1103, the intensity value of each pixel in the first and third composite images 1102, 1104 can be set to be equal to the sum of the intensity values ​​of each pixel in slices that are a predetermined number of slices away from each slice used to generate the second composite image 1103, multiplied by the weights used to generate the second composite image 1103.

[0105] In some embodiments, the secondary model 1100 may include a neural network, such as a residual recurrent neural network. To train the secondary model 1100, the model may be trained using a training dataset that includes synthetic images labeled as malignant or non-malignant. A human physician may label the synthetic images. Once trained, the secondary model 1100 may receive three input synthetic images generated from 3D tomosynthesis data of breast tissue and output a malignancy likelihood score indicating whether the breast tissue contains a malignant tumor and / or lesion.

[0106] The secondary model 1100 may include multiple layers, such as convolutional layers. It is understood that some embodiments of the secondary model 1100 may have a different number of layers, a different arrangement of layers, or other differences. However, in all embodiments, the secondary model 1100 may receive three input 2D composite images and output a malignancy likelihood score. The secondary model 1100 may be a single stage detection network that includes one or more sub-networks.

[0107] 9 and 11, the secondary model 1100 may include a primary subnetwork 1116 and a secondary subnetwork 1120, which may be identical to the primary subnetwork 916 and the secondary subnetwork 920 of the model 900 described above. In some embodiments, the primary subnetwork 1116 and the secondary subnetwork 1120 may be identical to the primary subnetwork 916 and the secondary subnetwork 920 after the model 900 has been trained. In other words, the secondary model 1100 may be initialized with the weights of the model 900, the model 600, or the model 200 described above before training. The primary subnetwork 1116 may include an input layer 1118 configured to receive three input composite images.

[0108] The secondary model 1100 is important because, while the above-described model 900 can detect regions of interest in breast tissue, it may not be able to accurately determine whether the ROI is in fact malignant. The secondary model 1100 may be used to more accurately estimate the malignancy of breast tissue using a composite image generated by the above-described model 900.

[0109] The model 1100 may include multiple tertiary sub-networks, such as a first tertiary network 1124A, a second tertiary network 1124B, and a tertiary network 1124C. Each tertiary network 1124A-C may be connected to a layer of the second sub-network 1120. The first tertiary network 1124A may be connected to the first layer 1122A, the second tertiary network 1124B may be connected to the second layer 1122B, and the third tertiary network 1124C may be connected to the third layer 1122C. Each tertiary network may receive features from a layer of the second sub-network 1120 to estimate malignancy of breast tissue at different scale levels.

[0110] Each tertiary network may include a box regression sub-network 1126. The box regression sub-network 1126 may include one or more convolutional layers 1128A-B, each followed by a rectified linear (ReLU) activation unit, and may also include a final convolutional layer 1130 that outputs regression coordinates corresponding to anchors associated with a portion of one layer of the second sub-network 1120 (the regression coordinates corresponding to an array of pixels of the second input composite 2D slice 1103). The anchors may be pre-determined sub-arrays of the layers of the second sub-network 1120. The regression coordinates may represent a predicted offset between the anchors and a predicted bounding box. For each bounding box included in the ROI, the set of regression coordinates (e.g., four regression coordinates) and the corresponding anchors may be used to calculate the bounding box coordinates.

[0111] Each tertiary network may include a classification sub-network 1132. The classification sub-network 1132 may include one or more convolutional layers 1134A-B, each followed by a ReLU activation, and the classification sub-network 1132 may further include a final convolutional layer 1138, followed by a sigmoid activation that outputs a prediction of the presence or absence of an object (i.e., the presence or absence of a malignant tumor and / or lesion). The classification sub-network 1132 may be used to obtain one or more estimations of whether the patient has a malignant tumor and / or lesion at a plurality of spatial locations of the second input composite 2D slice 1103. More specifically, each bounding box may be associated with an estimation score output by the classification sub-network 1132. The bounding box may also be associated with a slice number as mentioned above. In some embodiments, the value of each estimation score may range from 0 to 1. One of the spatial locations may include the entirety of one layer, i.e., the first layer 1122A, of the second sub-network 1120. In this manner, the classification sub-network 1132 may output a prediction of whether the patient has a malignant tumor and / or lesion based on the composite 2D slice. The final convolutional layer 1138 may be followed by a softmax activation in a model that is trained to classify multiple types of malignant regions, such as multiple levels of malignancy (e.g., low risk regions, high risk regions, etc.).

[0112] The model 1100 may include an output layer 1150 that normalizes data across various scales, calculates bounding box coordinates, and filters out bounding box predictions with low scores. The output layer 1150 may receive output from the tertiary sub-networks 1124A-C and output one or more ROIs, each ROI having an array of pixels scaled to the array size of the second input composite 2D slice 1103 and an associated score. The pixel array may be a bounding box (e.g., a rectangular bounding box, etc.) calculated based on the regression coordinates and the anchors. The output layer 1150 may filter out all scores below a predefined threshold, e.g., less than 0.5. After filtering, the output layer 1150 may determine an array of pixels for each ROI based on one or more anchors associated with the other remaining scores. The output layer 1150 may resize the anchors to match the scale of the input composite 2D slices 1102, 1103, and 1104 before including them in the array of output ROIs. Such resizing may be necessary if an anchor is associated with a smaller layer of the second sub-network 1120. The output layer 1150 may receive outputs from the tertiary sub-networks 1124A-C and output a malignancy likelihood score 1154. In some embodiments, the highest bounding box score may be selected as the malignancy likelihood score 1154. In some embodiments, the model 1100 may output one or more ROIs 1108, each having a score 1108A and an array of pixels 1108B. The pixel array 1108B may be a rectangular bounding box. The one or more ROIs 1108 may provide the physician with additional information about the likely malignant regions in the second input composite 2D slice 1103.

[0113] 1, 9, and 11 and 12, a process 1200 is shown for creating three composite images using a trained model to estimate malignancy of tumors and / or lesions in breast tissue and generating a malignancy likelihood score based on the composite images. The process 1200 may be implemented as instructions on one or more memories of a data processing system. The data processing system may further include one or more processors in communication with the one or more memories and configured to execute the instructions. The one or more processors may be configured to access a memory that stores 3D tomosynthesis data including multiple two-dimensional slices, which may be included in mass storage device 128.

[0114] At 1204, the process 1200 may receive 3D tomosynthesis data of breast tissue of a patient. The 3D tomosynthesis data may be generated by a 3D mammography imaging system, such as the x-ray imaging system 100. The 3D tomosynthesis data may include multiple 2D slices corresponding to a predetermined thickness, such as 1 mm, of breast tissue. Depending on the patient and / or imaging system, the 3D tomosynthesis data may include about 10-150 or more 2D slices. Each 2D slice may be a pixel array of a predetermined size, such as 2000×1500 pixels. The process 1200 may then proceed to 1208.

[0115] The process 1200 may generate 1208 a composite image associated with the second input channel. The composite image may be referred to as a second composite image (i.e., a composite image associated with the second channel of the model 1100) and may be generated using at least one of steps 1008-1016 described above. The process 1200 may then proceed to 1212.

[0116] At 1212, the process 1200 may generate a composite image for the first input channel, referred to as a first composite image (i.e., a composite image for the first channel of the model 1100), based on the predetermined offset slice number and the second composite image. As described above, each pixel of the composite image may be associated with a slice number (or may be associated with multiple slices, e.g., if blending techniques are used). For example, a first group of pixels in the second composite image may be associated with the seventh slice, and a second group of pixels in the second composite image may be associated with the fifth slice. The process 1200 may determine an intensity value for each pixel in the first composite image by selecting an intensity value for the pixel in a slice that is associated with a pixel at a particular location in the second composite image minus the predetermined offset slice number (specifically, xn for the first composite image, where x is the slice number associated with the pixel in the second composite image and n is the predetermined number of slices). For example, if the predetermined number of slices is 1, the intensity value of a pixel in the first composite image at the same pixel location as the first group of pixels (the group of pixels in the second composite image associated with the seventh slice) may be set equal to the intensity value of a pixel in the sixth slice and may be located at the same location as the first group of pixels. In an embodiment using blending to generate the second composite image, process 1200 may set the intensity value of each pixel in the first composite image to be equal to the sum of the intensity values ​​of each pixel in slices that are a predetermined number of slices away from each slice used to generate the second composite image, multiplied by the weights used to generate the second composite image. The process may then proceed to 1216.

[0117] At 1216, the process 1200 may generate a composite image for the third input channel, referred to as a third composite image (i.e., a composite image for the third channel of the model 1100), based on the predetermined offset slice number and the second composite image. The process 1200 may determine an intensity value for each pixel in the third composite image by selecting an intensity value for the pixel in a slice associated with a pixel at a particular location in the second composite image plus the predetermined offset slice number (specifically, x+n for the third composite image, where x is the slice number associated with the pixel in the second composite image and n is the predetermined number of slices). For example, if the predetermined number of slices is 1, then the intensity value of a pixel in the third composite image that is at the same pixel location as the first group of pixels (the group of pixels in the second composite image associated with the seventh slice) may be set equal to the intensity value of a pixel in the eighth slice and placed at the same location as the first group of pixels. In an embodiment using blending to generate the second composite image, process 1200 may set the intensity value of each pixel in the first and third composite images to be equal to the sum of the intensity value of each pixel in a slice that is a predetermined number of slices away from each slice used to generate the second composite image, multiplied by the weight used to generate the second composite image. Process may then proceed to 1220.

[0118] The process 1200 can then provide 1220 the three composite images to a trained model for determining a malignancy likelihood score. The trained model can be model 1100 described above. The process 1200 can then proceed to 1216.

[0119] The process 1200 may receive a malignancy likelihood score from the trained model at 1224. In some embodiments, the malignancy likelihood score may range from 0 to 1, with 1 indicating a high risk of malignancy and 0 indicating minimal or no risk of malignancy. In some embodiments, malignancy may be "yes" (i.e., 1) or "no" (i.e., 0) indicating whether the tumor is malignant or non-malignant. In some embodiments, the malignancy likelihood score may indicate a category of risk, for example, low, medium, or high risk. In some embodiments, the malignancy likelihood score may be selected from a numerical range, for example, from integers 1 to 5, with 1 indicating the lowest risk level and 5 indicating the highest risk level. In some embodiments, the process 1200 may also receive one or more regions of interest generated by the trained model, such as region of interest 1108 above. The process 1200 may then proceed to 1220.

[0120] The process 1200 may output the likelihood of malignancy score at 1228 to a memory for storage and / or use in other processes and / or to a display such as a computer monitor for viewing by a human physician. For example, the process 1200 may output the likelihood of malignancy score to a display of a medical device where a physician may view the likelihood of malignancy score and possibly make a patient diagnosis based on the likelihood of malignancy score. The likelihood of malignancy score may be stored in a medical record database for future analysis and / or research of breast cancer patients. In some embodiments, the process 1200 may output one or more of the regions of interest received at 1216 to a memory for storage and / or use in other processes and / or to a display such as a computer monitor for viewing by a human physician. The process 1200 may output the likelihood of malignancy score and / or the one or more regions of interest as a report. The process 1200 may then end.

[0121] It can be appreciated that a composite image generated using an ROI generated by either model 200 or model 900 can be used to generate a malignancy likelihood score using either model 600 or model 1100 described above. Thus, the present disclosure provides a system and method for efficiently and uniformly creating composite 2D images from 3D tomography data of the breast and estimating the malignancy of breast tumors and / or lesions, if present.

[0122] Although the invention has been described with reference to one or more preferred embodiments, it should be understood that in addition to those explicitly stated equivalents, alternatives, variations, and improvements, many more equivalents, alternatives, variations, and improvements are possible and are within the scope of the invention.

Claims

1. 1. A method for determining a malignancy potential score for breast tissue of a patient, comprising: receiving a plurality of two-dimensional images of breast tissue derived from the three-dimensional image of the breast tissue; For each of the two-dimensional images, inputting the two-dimensional image into a first model including a trained first neural network; receiving a plurality of indicators from the first model, each indicator associated with a respective one of the two-dimensional images included in the plurality of two-dimensional images; generating a composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images; inputting the synthetic two-dimensional image into a second model comprising a second trained neural network; receiving a malignancy likelihood score from the second model; outputting a report including the malignancy likelihood score to at least one of a memory or a display; having each said indicator comprising an array of pixels each having an intensity value and a relevance score; The generating step includes: determining a first coverage area associated with a first indicator in the plurality of indicators, the first coverage area including a second array of pixels; determining a second coverage area associated with a second indicator in the plurality of indicators, the second coverage area including a third array of pixels; determining a first intensity value of a first pixel included in the composite two-dimensional image based on a second intensity value included in the first coverage area and a third intensity value included in the second coverage area; The method according to claim 1, further comprising:

2. the first coverage area includes at least one pixel that is not included in the first indicator, and the second coverage area includes at least one pixel that is not included in the second indicator; The method of claim 1.

3. The step of determining the first intensity value for the first pixel comprises: setting the first intensity value equal to the sum of the second intensity value multiplied by a first weight and the third intensity value multiplied by a second weight. having The method of claim 1.

4. the three-dimensional image is generated using digital breast tomosynthesis; The method of claim 1.

5. the trained first neural network includes a first sub-network and the trained second neural network includes a second sub-network; the first sub-network and the second sub-network have the same number of layers and filters; The method of claim 1.

6. the second trained neural network is trained using a set of weight values ​​of the first neural network as initial weight values; The method of claim 5.

7. extracting at least one indicator from the plurality of indicators; generating a second composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images; Further comprising The method of claim 1.

8. the first model is trained based on an image dataset including two-dimensional full-field digital mammography images with physician annotations; The method of claim 1.

9. each two-dimensional image of the breast tissue is a slice of a three-dimensional digital breast tomosynthesis volume; The method of claim 8.

10. 1. A system for determining a malignancy potential score for breast tissue of a patient, comprising: a memory configured to store a plurality of two-dimensional images of breast tissue derived from the three-dimensional images of the breast tissue; a processor configured to access the memory; Equipped with The processor, inputting each of the two-dimensional images into a first model including a trained first neural network; receiving, for each of the two-dimensional images, a plurality of indicators from the first model, each indicator associated with a respective one of the two-dimensional images included in the plurality of two-dimensional images; generating a composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images; inputting the synthetic two-dimensional image into a second model including a second trained neural network; The second model is used to determine a malignancy likelihood score. It is structured as follows: The system further comprises a display configured to display a report including the malignancy likelihood score; each said indicator comprising an array of pixels each having an intensity value and a relevance score; The processor, determining a first coverage area associated with a first indicator in the plurality of indicators, the first coverage area including a second array of pixels; determining a second coverage area associated with a second indicator in the plurality of indicators, the second coverage area including a third array of pixels; determining a first intensity value of a first pixel included in the composite two-dimensional image based on a second intensity value included in the first coverage area and a third intensity value included in the second coverage area; and generating the synthetic two-dimensional image by:

11. 1. A system for determining a malignancy potential score for breast tissue of a patient, comprising: a memory configured to store a plurality of two-dimensional images of breast tissue derived from the three-dimensional images of the breast tissue; a processor configured to access the memory; Equipped with The processor, inputting each of the two-dimensional images into a first model including a trained first neural network; receiving, for each of the two-dimensional images, a plurality of indicators from the first model, each indicator associated with a respective one of the two-dimensional images included in the plurality of two-dimensional images; generating a first composite two-dimensional image based on the plurality of indicators and at least one of the plurality of two-dimensional images; generating a second composite two-dimensional image based on the first composite two-dimensional image and at least one of the plurality of two-dimensional images; generating a third composite two-dimensional image based on the first composite two-dimensional image and at least one of the plurality of two-dimensional images; inputting the first synthetic two-dimensional image, the second synthetic two-dimensional image, and the third synthetic two-dimensional image into a second model including a trained second neural network; The second model is used to determine a malignancy likelihood score. It is structured as follows: The system further comprises a display configured to display a report including the malignancy likelihood score; each said indicator comprising an array of pixels each having an intensity value and a relevance score; The processor further comprises: determining a first coverage area associated with a first indicator in the plurality of indicators, the first coverage area including a second array of pixels; determining a second coverage area associated with a second indicator in the plurality of indicators, the second coverage area including a third array of pixels; determining a first intensity value of a first pixel included in each of the composite two-dimensional images based on a second intensity value included in the first coverage area and a third intensity value included in the second coverage area; A system characterized by:

Images