Mammary gland medical image processing system and method

Abstract

This application provides a breast medical image processing system and method. Digital breast tomography (DBT) can provide richer information than full-field digital mammography (FFDM). DBT data, such as DBT slices, can be processed based on deep learning techniques (such as using neural networks), and the DBT slices can be divided into groups, and a predetermined number of representative images can be derived based on these groups. The neural network can be configured to process the representative images to predict the presence or absence of breast diseases such as breast cancer.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and in particular to medical image processing systems and methods related to the breast. Background Technology

[0002] Breast cancer is a leading cause of death among women worldwide, accounting for a large proportion of new cancer cases each year and causing hundreds of thousands of deaths annually. Early screening and detection are crucial for improving breast cancer treatment outcomes and can be accomplished through mammography. However, conventional mammography is a two-dimensional (2D) imaging modality, which involves projecting information collected from the compressed breast into a 2D image. Because signals from objects above and below the pathology of interest can converge during projection, 2D images obtained using conventional mammography may include areas of tissue overlap and structural noise, making it difficult to detect underlying breast diseases (e.g., lesions) and frequently leading to false positive or false negative results. Summary of the Invention

[0003] This paper describes systems, methods, and apparatuses associated with processing digital breast tomography (DBT) data, such as DBT images. An apparatus capable of performing this task may include at least one processor configured to: acquire multiple slice images of the breast (e.g., DBT slices), group the multiple slice images into multiple groups, derive a predetermined number of representative images of the breast based on the groupings, and process the predetermined number of representative images via an artificial neural network (ANN) to detect abnormalities (e.g., breast cancer) in the breast. The multiple slice images may be reconstructed based on X-ray images of the breast captured from different angles, and the total number of representative images derived may be predetermined based on the settings of the artificial neural network (e.g., the number of input channels of the ANN (e.g., when the ANN includes a 2D convolutional neural network (CNN)), one dimension of the ANN (e.g., when the ANN includes a 3D CNN), etc.).

[0004] In some embodiments, deriving a predetermined number of representative images of the breast based on groups may include deriving one or more representative images of the breast for each group based on a subset of slice images included in each of the multiple groups. The number of subsets of slice images included in each of the multiple groups may be the same across the groups, or multiple groups (e.g., at least two of the multiple groups) may include different numbers of slice images. Additionally, slice images in one group may overlap with slice images in another group, and some slice images (e.g., slices associated with breast skin) that can be determined to be irrelevant to anomalies in predictions may be excluded from the group.

[0005] In some embodiments, one or more representative images derived for each of the multiple groups may include a slice image from the group (e.g., an intermediate slice in the group), while in other embodiments, one or more representative images for each group may be derived based on a corresponding statistical summary of a subset of the slice images in the group. The statistical summary may include, for example, one or more of the maximum, minimum, mean / average, standard deviation, principal component analysis (PCA), or singular value decomposition (SVD) of the slice images included in the group. In some embodiments, the multiple slice images obtained by the device described herein may be sequentially numbered based on a first sequence number set, while the multiple groups may be sequentially numbered based on a second sequence number set. However, slice images in two consecutively numbered groups (e.g., group 1 and group 2) may not be consecutive (e.g., they may be discontinuous).

[0006] This application provides a breast medical image processing apparatus and method. The apparatus includes: at least one processor configured to perform the following method: obtaining a plurality of slice images of the breast, wherein the slice images are reconstructed based on X-ray images of the breast captured from different angles; grouping the plurality of slice images into a plurality of groups, wherein each of the plurality of groups includes a subset of the plurality of slice images; deriving a predetermined number of representative images of the breast based on the groupings; and processing the predetermined number of representative images through an artificial neural network (ANN), wherein the ANN is trained to detect abnormalities in the breast, and wherein the result of the processing is a prediction of the abnormalities. Attached Figure Description

[0007] The examples disclosed herein can be understood in more detail from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0008] Figure 1A and Figure 1B This is a simplified diagram illustrating examples of mammography according to some embodiments described herein.

[0009] Figure 2 This is a simplified diagram illustrating an example of using machine learning (ML) techniques to process DBT data according to some embodiments described herein.

[0010] Figure 3A , Figure 3B and Figure 3C This is a simplified diagram illustrating example techniques for dividing DBT slices into groups and deriving representative images of each group according to some embodiments described herein.

[0011] Figure 4 This is a flowchart illustrating example methods for processing DBT data according to some embodiments described herein.

[0012] Figure 5 This is a flowchart illustrating an example method for training a neural network to perform one or more tasks as described with respect to some embodiments provided herein.

[0013] Figure 6 This is a simplified block diagram illustrating an example system or device for performing one or more tasks as described with respect to some embodiments provided herein. Detailed Implementation

[0014] The present disclosure is illustrated by way of example and not limitation in the accompanying drawings. A detailed description of illustrative embodiments will now be described with reference to the various drawings. Although detailed examples of possible implementations are provided in this specification, it should be noted that these details are intended to be exemplary and in no way intended to limit the scope of this application.

[0015] Mammography (mastogram) is used to capture images of the breast from different perspectives, such as cephalothorax (CC) and / or medial-lateral oblique (MLO) perspectives. Therefore, a standard mammogram may include four images, for example, left CC (LCC), left MLO (LMLO), right CC (RCC), and right MLO (RMLO). Figure 1A and Figure 1B An example of mammography (mastography) technique is illustrated, in which Figure 1A An example of full-field digital mammography (FFDM) is shown, while Figure 1B An example of digital breast tomography (DBT) is shown. Figure 1A As shown, FFDM can be considered a 2D imaging modality, which may involve passing a pulse train of X-rays 102 through a compressed breast 104 at an angle (e.g., perpendicular to the breast), capturing X-rays 102 on the opposite side (e.g., using a solid-state detector), and generating a 2D image 106 of the breast based on the captured signal (e.g., the captured X-rays 102 can be converted into electronic signals, which can then be used to generate the 2D image 106). Using FFDM, information from the entire breast can be incorporated into a single 2D image (e.g., 2D image 106), resulting in normal breast tissue (e.g., composed of...) Figure 1A (circles in the image represent) and potential lesions (e.g., caused by) Figure 1A The asterisk (indicated by an asterisk) indicates overlap in the resulting mammogram image. This overlap can obscure the presence of lesions and increase the chance of false positives or false negatives.

[0016] on the contrary, Figure 1BThe DBT technique shown can achieve or resemble the quality of a 3D imaging modality (e.g., DBT can be considered a pseudo-3D imaging modality). As shown, DBT technology can involve passing a pulse train of X-rays 102 through a compressed breast 104 at different angles (e.g., 0°, +15°, -15°, etc.) during scanning, acquiring one or more X-ray images of the breast at each angle, and reconstructing the individual X-ray images into a series of slices 108 (e.g., thin, high-resolution slice images), which can be displayed individually or as a film (e.g., in a dynamic movie mode). Thus, compared to... Figure 1A The example FFDM technique shown (e.g., which can project the breast 104 from only one angle) is different. Figure 1B The example DBT technique shown can project the breast from multiple angles and reconstruct multiple slice images 108 (e.g., multi-slice data) from data collected from these different angles, wherein normal breast tissue (e.g., composed of...) Figure 1B The circles in the diagram can clearly distinguish it from lesions (e.g., those caused by...). Figure 1B The asterisks in the text indicate that the images are separated. This technique can reduce or eliminate problems caused by 2D mammographic imaging (e.g., the FFDM technique described herein), resulting in improved diagnostic and screening accuracy.

[0017] It should be noted that, although Figure 1B Only three angles from which X-ray images of breast 104 were taken are shown; however, those skilled in the art will understand that more angles can be used and more images can be taken during an actual DBT procedure. For example, 15 images of the breast can be taken in an arc from the top and sides, and then reconstructed into multiple non-overlapping slices across the breast. Those skilled in the art will also understand that, although... Figure 1B As not shown in the image, DBT scans can include different perspectives of each breast, including, for example, LCC, LMLO, RCC, and RMLO.

[0018] The DBT technology described herein can provide richer information about breast diseases (e.g., breast cancer) than FFDM technology. However, the large amount of data generated during the DBT procedure (e.g., 40 to 80 slices per breast per view) may pose new challenges to clinicians, as reading DBT data can be more time-consuming than reading FFDM data. Therefore, embodiments of this disclosure employ artificial intelligence (AI) to profil, analyze, and / or summarize DBT data (e.g., DBT slice images). For example, as Figure 2As illustrated in the example, machine learning (ML) techniques such as artificial neural networks (ANNs) can be used to acquire knowledge about breast diseases (e.g., abnormalities such as lesions) through training, and subsequently automatically detect breast diseases on a given patient's DBT data (e.g., slice images). Various techniques can also be employed to prepare (e.g., preprocess) DBT data so that a co-trained ML model (e.g., an artificial neural network) can be used on different types of DBT data (e.g., which may include different numbers of slices) to reduce the data load on the ML model, etc.

[0019] like Figure 2 As shown, an AI-based system or device configured to process DBT data can be configured to obtain multiple DBT slices 202 (e.g., slice images) of the breast, wherein the slice images can be reconstructed based on X-ray images of the breast captured from different angles (e.g., as described in Figure 1). The AI-based system or device can also be configured to group or divide the multiple slice images of the breast into multiple groups 204 (e.g., each group includes a subset of the multiple slice images 202) based on the groupings, and derive a predetermined (e.g., fixed) number of representative images 206 of the breast based on the groupings. For example, one or more representative images can be derived for a group based on a subset of the slice images included in each of the multiple groups, and the total number of representative images derived for the multiple groups can be fixed (e.g., based on the criteria described herein). The AI-based system or device can then process the representative images 206 via an artificial neural network (ANN) 208 to predict, for example, abnormalities (e.g., lesions) that may be present in the breast at the output of the ANN 208. The examples provided in this article may refer to ANN 208 in the singular, but those skilled in the art will understand that one or more ANNs can be used to accomplish the tasks described herein.

[0020] ANN 208 can include various types of neural networks, including, for example, 2D convolutional neural networks (CNNs), 3D convolutional neural networks, Transformer neural networks, etc. In the example where ANN 208 includes a 2D CNN with multiple input channels, the number of representative images (e.g., the total number) derived for multiple groups 204 can be equal to the number of input channels of the ANN (e.g., individual representative images 206 can be provided to the corresponding input channels of the 2D CNN). For example, if the 2D CNN has four input channels, four representative images 206 can be derived (e.g., based on four or fewer groups), and if the 2D CNN has six input channels, six representative images 206 can be derived (e.g., based on six or fewer groups). In the example where ANN 208 includes a 3D CNN or another type of neural network, the techniques described herein can still be utilized, for example, to reduce the number of DBT slices or the amount of data that the neural network can be assigned to process. For example, if the 3D CNN has a W-dimensional dimension (e.g., corresponding to the width of the input image), an H-dimensional dimension (e.g., corresponding to the height of the input image), and a Z-dimensional dimension (e.g., corresponding to the temporal or depth values ​​in space), the number of representative images 206 generated can be predetermined (e.g., fixed) based on the Z-dimensional dimension of the 3D CNN. As will be described in more detail below, the grouping or partitioning of DBT slice images 202 and / or the derivation of representative images 206 can be performed in various ways. Furthermore, in at least some examples, not all DBT slice images 202 can be included in group 204 (e.g., some slice images can be excluded if they are determined to be irrelevant to the anomaly prediction), and DBT slice images 202 may be assigned to or not assigned to group 204 based on the order of the DBT slice images.

[0021] ANN 208 may include multiple layers, such as one or more convolutional layers, one or more pooling layers, and / or one or more fully connected layers. Each convolutional layer may include multiple convolutional kernels or filters configured to extract features from an input image (e.g., representative image 206) received at the input channel. Following the convolution operation may be batch normalization and / or linear (or non-linear) activation, and the features extracted by the convolutional layers may be downsampled by pooling layers and / or fully connected layers to reduce feature redundancy and / or dimensionality, thereby obtaining a representation of the downsampled features (e.g., in the form of feature vectors or feature maps). In some examples (e.g., in the detection of breast abnormalities, which involves segmenting abnormal regions in corresponding medical images), ANN 208 may also include one or more non-pooling layers and one or more transposed convolutional layers, which may be configured to upsample and deconvolve the features extracted by the above operations. As a result of upsampling and deconvolution, a dense feature representation (e.g., a dense feature map) of the input image can be derived, and an ANN 208 can be trained (e.g., the parameters of the ANN can be tuned) to predict the presence or absence of abnormalities (e.g., lesions) in the input image based on the feature representation. As will be described in more detail below, the training of the ANN 208 can be based on publicly available DBT slice images including breast lumps and / or architectural distortions, and the parameters of the ANN 208 can be tuned (e.g., learned) based on various loss functions.

[0022] Figure 3A , Figure 3B and Figure 3C An example method is illustrated for grouping multiple DBT slices (e.g., slice images) and deriving representative images for each group. As shown in the example, multiple DBT slices 302 can be divided into multiple groups or blocks 304, the number of which can be fixed or variable. Grouping can be performed in various ways, for example, to adapt to the structure of an ANN and / or optimize the performance of the ANN. For example, DBT slices 302 can be grouped in an approximately uniform manner based on the total number of groups (e.g., as shown in the example). Figure 3A As shown) the division, or DBT slice 302 can be divided unevenly (e.g., the number of slices in different groups can be different, such as...). Figure 3B and Figure 3C (As shown). At least one group 304 may include multiple DBT slices, and subsets of DBT slices between two different groups may overlap (e.g., slices 1-5 and slices 3-7 may be placed in group 1 and group 2, respectively). Some slices may be excluded during grouping or partitioning. For example, as... Figure 3BAs shown, if a slice (e.g., a slice associated with breast skin) is determined to be irrelevant to the detection of breast abnormalities (e.g., contributing little or no contribution), then that slice may not be included in any group 304. Furthermore, while DBT slices 302 and / or group 304 may be associated with corresponding sequence numbers (e.g., first sequence number and second sequence number), slices assigned to consecutive groups may themselves be consecutive or may not be consecutive. For example, slices 1 through 5 in DBT slices 302 may be assigned to group 1, while slices 7 through 11 in DBT slices 302 may be assigned to group 2, and so on.

[0023] For each group 304 described herein, one or more representative images 306 can be derived based on a subset of slices included in that group, and can be (e.g., based on artificial neural networks (e.g., Figure 2 The structure and / or settings of the ANN 208 are fixed (e.g., for all groups 304), resulting in a total number of representative images 306. The export of these representative images can be performed in various ways. For example, one or more representative images for each group 304 may include a slice from the group (e.g., an intermediate slice in the group may be used as a representative image of the group). As another example, representative images for each group 304 may include images exported based on a statistical summary of the slices in the group, including, for example, maximum, mean / average, minimum, standard deviation, or combinations thereof associated with the slices of the group. As yet another example, representative images for each group 304 may include images exported based on principal component analysis (PSA) or singular value decomposition (SVD) of the slices in the group. For example, representative images for group 304 can be exported by selecting the maximum value at each corresponding pixel location across all slices in the group. The number of representative images 306 exported for each group 304 and / or the manner in which these images are exported can be the same or different. For example, Figures 3A to 3C The different shaded areas in the diagram illustrate that different derivation methods can be applied to different groups (e.g., some images can be derived based on the maximum image value, while other images can be derived based on the average or mean), and different numbers of representative images can be derived for different groups (e.g., some groups can have one representative image, while other groups can have two or more representative images).

[0024] Figure 4Example method 400 for processing DBT data (e.g., DBT slices) according to some embodiments of the present disclosure is illustrated. Method 400 may be implemented by an AI-based system or device as described herein. The method may include: obtaining multiple reconstructed slice images (e.g., DBT slices) of the breast at 402, wherein the slice images may be reconstructed based on X-ray images of the breast captured at different angles (e.g., as described in Figure 1). Method 400 may further include: dividing the multiple slice images of the breast into a predetermined number of groups at 404 (e.g., in a preprocessing step), wherein each group comprises a subset of the multiple slice images, and deriving one or more representative images of the breast for each group based on the subset of slice images included in each of the predetermined number of groups at 406. Method 400 may additionally include: processing one or more representative images associated with each of the predetermined number of groups via an artificial neural network (ANN) at 408 to predict abnormalities (e.g., lesions) detected in the representative images. As described herein, the total number of groups into which the slice images are divided may be fixed, for example, based on the number of input channels of the ANN. Similarly, as described in this article, an ANN can be pre-trained based on training data including normal breasts and breasts with abnormal conditions to acquire knowledge about abnormalities, so that when given a patient's DBT data, the ANN can use the knowledge acquired through training to identify abnormal conditions.

[0025] Figure 5 A flowchart is shown illustrating an example process 500 for training a neural network (e.g., an ML model implemented by a neural network) to perform one or more tasks described herein. As shown, the training process 500 may include: at 502 initializing the execution parameters of the neural network (e.g., weights associated with the individual layers of the neural network), for example by sampling from a probability distribution or by replicating the parameters of another neural network with a similar structure. The training process 500 may also include: at 504 processing the input (e.g., a training image such as the representative image 306 shown in Figure 3) using the currently assigned parameters of the neural network, and at 506 making a prediction about the presence of a breast disease (e.g., a lesion). At 508, the prediction may be compared to a gold standard indicating the true state of the breast disease (e.g., whether the disease actually exists) to determine a loss associated with the prediction based on a loss function. The loss function used for training may be selected based on the specific task the neural network is trained to perform. For example, if the task involves classifying or segmenting the input image, a loss function based on the mean squared error between the predicted result and the gold standard, such as the L1 norm, L2 norm, etc., can be used. And if the task involves detecting the location of anomalies around anomalies (e.g., by drawing bounding boxes), a loss function based on the generalized intersection-union (GIOU) can be used.

[0026] At step 510, the loss calculated using one or more of the techniques described above can be used to determine whether one or more training termination criteria are met. For example, if the loss is below a threshold or if the change in loss between two training iterations is below a threshold, it can be determined that the training termination criteria are met. If the termination criteria are determined to be met at step 510, training can end; otherwise, at step 512, for example, before training returns to step 506, the currently assigned network parameters can be adjusted by backpropagating the gradient descent of the loss function through the network.

[0027] For the sake of simplicity, the training steps are depicted and described in a specific order herein. However, it should be understood that training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that may be included in the training method are depicted and described herein, and not all exemplified operations need to be performed.

[0028] The systems, methods, and / or apparatuses described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 6 An example device 600 is illustrated that can be configured to perform the tasks described herein. As shown, device 600 may include a processor (e.g., one or more processors) 602, which may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller, reduced instruction set computer (RISC) processor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), or any other circuitry or processor capable of performing the functions described herein. Device 600 may also include communication circuitry 604, memory 606, mass storage device 608, input device 610, and / or communication link 612 (e.g., communication bus) through which one or more components shown in the figures exchange information.

[0029] Communication circuitry 604 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including local area networks (LANs), wide area networks (WANs), the Internet, and wireless data networks (e.g., Wi-Fi, 3G, 4G / LTE, or 5G networks). Memory 606 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause processor 602 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Mass storage device 608 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROMs or DVD-ROMs, etc., on which instructions and / or data may be stored for operation of processor 602. Input device 610 may include a keyboard, mouse, voice-controlled input device, touch-sensitive input device (e.g., touch screen), etc., for receiving user input from device 600.

[0030] It should be noted that device 600 can operate as a standalone device or can be connected to other computing devices (e.g., networked or clustered) to perform the tasks described herein. And even in Figure 6 Only one example of each component is shown in the figure, and those skilled in the art will understand that device 600 may include multiple instances of one or more components shown in the figure.

[0031] Although this disclosure has been described according to certain embodiments and generally associated methods, changes and variations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure. Furthermore, unless specifically stated otherwise, discussions using terms such as “analyze,” “determine,” “enable,” “identify,” and “modify” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data representing physical (e.g., electronic) quantities within the registers and memories of the computer system into other data representing physical quantities within the computer system's memory or other such information storage, transmission, or display devices.

[0032] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art after reading and understanding the above description.

Claims

1. A breast medical image processing device, comprising at least one processor configured to: Obtain multiple slice images of the breast, among which, The slice images are reconstructed based on X-ray images of the breast captured from different angles; The plurality of slice images are grouped into a plurality of groups, wherein each of the plurality of groups comprises a subset of the plurality of slice images; While satisfying ANN performance, the multiple slice images of the breast are sequentially numbered based on a first sequence number set, and the multiple groups of slice images are sequentially numbered based on a second sequence number set. Furthermore, in consecutive groups, the slice images in two adjacent groups are not consecutively numbered, and the slice images in two adjacent groups are consecutively numbered. Based on the grouping, a predetermined number of representative images of the breast are derived, wherein one or more representative images of the breast derived for each of the plurality of groups are derived based on a corresponding statistical summary of a subset of slice images included in the group; and The predetermined number of representative images are processed by an artificial neural network (ANN), wherein the ANN is trained to detect abnormalities in the breast, and the result of the processing is a prediction of the abnormality.

2. The device according to claim 1, wherein, The ANN includes a two-dimensional convolutional neural network (2D CNN) with multiple input channels, and The predetermined number of representative images is equal to the number of input channels of the 2D CNN.

3. The device according to claim 1, wherein, The ANN includes a three-dimensional convolutional neural network (3D CNN), and The predetermined number of representative images is determined based on one dimension of the 3D CNN.

4. The device according to claim 1, wherein, The at least one processor is configured to export the predetermined number of representative images of the breast, including: The at least one processor is configured to derive one or more representative images of the breast for each of the plurality of groups based on a subset of slice images included in each of the plurality of groups.

5. The device according to claim 4, wherein, The statistical summary of the subset of slice images in each of the plurality of groups includes one or more of the following: maximum, minimum, mean, standard deviation, principal component analysis, or singular value decomposition of the subset of slice images in the group.

6. The device according to claim 1, wherein, At least two of the multiple groups include different numbers of slice images, or, At least two of the plurality of groups include overlapping slice images.

7. The device according to claim 1, wherein, The at least one processor is configured to group the plurality of sliced ​​images into the plurality of groups, including: The at least one processor is configured to determine that one or more of the plurality of slice images are unrelated to the prediction of the anomaly and exclude the one or more slice images from the plurality of groups.

8. A method for processing a breast medical image, the method comprising: Multiple slice images of the breast are obtained, wherein the slice images are reconstructed based on X-ray images of the breast captured from different angles; The plurality of slice images are grouped into a plurality of groups, wherein each of the plurality of groups comprises a subset of the plurality of slice images; Based on the grouping, a predetermined number of representative images of the breast are derived, wherein, for each of the plurality of groups, one or more representative images of the breast are derived based on a corresponding statistical summary of a subset of slice images included in the group; and The predetermined number of representative images are processed using an artificial neural network (ANN), wherein the ANN is trained to detect abnormalities in the breast, and the result of the processing is a prediction of the abnormality. Specifically, while satisfying ANN performance requirements, the multiple slice images of the breast are sequentially numbered based on a first sequence number set, and the multiple groups of slice images are sequentially numbered based on a second sequence number set. Furthermore, the numbering of the slice images in the first group and the slice images in the second group are not consecutive, while the numbering of the second group and the first group are consecutive.

Images