A method for auxiliary screening of breast diseases based on mammography

Abstract

The present invention relates to a method for assisting in the screening of breast diseases based on mammography images, which includes: performing data preparation, including: collecting mammography images, where each piece of data contains subject information as metadata; performing data preprocessing on the collected images; performing data training on the preprocessed data, and the training includes image feature extraction, metadata feature extraction, and feature classification. The image feature extraction network outputs the first high-dimensional feature through a feature extraction network based on a transformer; the metadata feature extraction combines factors related to breast diseases, and the metadata constructs shallow features through linear regression and constructs deep features through linear regression, and outputs the second high-dimensional feature; the first high-dimensional feature and the second high-dimensional feature are spliced together, and after feature classification, a breast disease classification model is obtained; the subject is taken a mammography image, and then it is input into the breast disease classification model to obtain the health status of the subject's breast and the confidence level of the corresponding result.

Description

Technical Field

[0001] This invention patent relates to the field of medical auxiliary screening, and particularly to the field of auxiliary screening for breast diseases. Background Art

[0002] In recent years, with the popularization of medical examinations in China, more and more people have attached importance to the early screening of diseases; breast screening has always been a routine examination method, which can effectively screen out breast-related diseases and achieve early treatment and prevention. Breast diseases are one of the common diseases among women. From breast hyperplasia, benign breast masses, breast inflammation to atypical breast hyperplasia, intraductal papilloma, breast cancer, various types of breast diseases bring economic and physiological losses to women at different stages. Especially breast cancer, according to the "Status and Trends of Cancers in China 2017" released by the National Cancer Center: Currently, the incidence rate of breast cancer has become the top of malignant tumors among Chinese women. In the United States, about 1 in every 8 women is unfortunately diagnosed with breast cancer, which is the second leading cause of cancer death among American women, second only to lung cancer. And the incidence rate of breast cancer is also increasing year by year in our country and showing a trend of getting younger. Therefore, the earlier breast diseases can be screened out, the earlier treatment can be carried out, achieving the effect of early detection and early treatment.

[0003] Mammography, full name mammography X-ray imaging examination, is currently the preferred, simplest, and most reliable non-invasive detection method for diagnosing breast diseases. It is simple and easy to perform, with high resolution, good repeatability, and the images taken can be used for comparison before and after. It is not restricted by age and body shape and has now become a routine examination. Its feature is that it can detect breast masses that doctors cannot feel. Especially for large breasts and fatty breasts, its diagnostic accuracy can be as high as 95%. For T0-stage breast cancer with only a little microcalcification as the only manifestation, it can only be detected and diagnosed early by soft X-ray examination. Figure 1 shows three cases of mammography: among them Figure 1a is the normal breast tissue structure, Figure 1b is a benign lesion, Figure 1c is malignant breast cancer; each picture respectively contains: the craniocaudal view (CC) on the left and right, and the mediolateral oblique view (MLO) on the left and right.

[0004] Deep learning is a type of machine learning that forms more abstract high-level representations of attribute categories or features by combining low-level features to discover the distributed feature representations of data. Currently, algorithms based on deep learning have reached or exceeded the human level in many fields. A large number of manual film readings are required in the screening task of mammography. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for auxiliary screening of breast diseases based on mammography images, which is used to solve the problems of a large number of film readings and insufficient human experience during breast physical examinations.

[0006] An auxiliary screening method for breast diseases based on mammography images according to the present invention, which includes: data preparation, data preprocessing, data training, and model verification; data preparation is carried out, including: collecting mammography images, including benign, malignant, and normal data as image features, and each piece of data contains subject information as metadata; performing data preprocessing on the collected images; performing data training on the preprocessed data, and the training includes image feature extraction, metadata feature extraction, and feature classification. The image feature extraction network outputs the first high-dimensional feature through a feature extraction network based on transformer; the metadata feature extraction combines factors related to breast diseases, and the metadata constructs shallow features through linear regression and constructs deep features through linear regression, and outputs the second high-dimensional feature; the first high-dimensional feature and the second high-dimensional feature are spliced together, and after feature classification, a breast disease classification model is obtained; taking a mammography image of the subject and then inputting it into the breast disease classification model to obtain the health status of the subject's breast and the confidence level of the corresponding result.

[0007] According to an embodiment of the auxiliary screening method for breast diseases based on mammography images of the present invention, the subject information includes: age, gender, weight, height, whether often drink alcohol, whether often smoke, age of menarche, age of menopause, whether give birth, whether have a family history of breast diseases, whether the breast has received radiotherapy, and age of the first childbirth.

[0008] According to an embodiment of the auxiliary screening method for breast diseases based on mammography images of the present invention, each piece of data of the subject information is used as metadata to extract metadata features.

[0009] According to an embodiment of the auxiliary screening method for breast diseases based on mammography images of the present invention, each piece of data includes the cranio-caudal views on the left and right, the mediolateral oblique views on the left and right, and subject information;

[0010] Performing data preprocessing on the collected images includes: splicing the four images of the cranio-caudal views on the left and right and the mediolateral oblique views on the left and right, then scaling the image, and then using data augmentation to increase the robustness of the data. The data augmentation includes rotation, translation, scaling, color space adjustment, self-contrast training augmentation, multi-image combination, and MixUp augmentation, and digitalizing the subject information.

[0011] In an embodiment of the mammary gland disease assisted screening method based on molybdenum target images according to the present invention, the image feature extraction network is a feature extraction network based on a transformer, and the output is a first high-dimensional feature of 1024 dimensions; the metadata extraction part takes into account factors related to mammary gland diseases and outputs a second high-dimensional feature of 32 dimensions through a two-layer network; the first high-dimensional feature and the second high-dimensional feature are concatenated, and a multi-layer perceptron is used as a classifier. After feature classification, a mammary gland disease classification model is obtained.

[0012] In an embodiment of the mammary gland disease assisted screening method based on molybdenum target images according to the present invention, metadata feature extraction includes: the metadata constructs a shallow feature of 512 dimensions through linear regression, and then through normalization, activation function, and random sampling, constructs a deep feature of 32 dimensions through linear regression, and then through normalization and activation function, outputs a metadata feature of 32 dimensions.

[0013] In an embodiment of the mammary gland disease assisted screening method based on molybdenum target images according to the present invention, image feature extraction includes;

[0014] The image obtained after preprocessing is a 256*256*3 image, which is evenly cropped into 8*8 = 64 parts to construct an image group of 64*32*32*3. The position information is made into a learnable vector E pos , E pos is a 65*1024 vector representing the position information of the sequence. Expand X p ∈ the image group of 64*32*32*3 into 64 one-dimensional sequences 64*(32*32*3) = 64*3072, multiply by the learnable vector E, the dimension of E is (32*32*3)*1024, and the output of the multiplication result is 64*1024 dimensions. Formula 1 is as follows:

[0015]

[0016] The dimension of 1024 is a learnable embedding vector; the 0th position in the position encoding plays an indexing role, and then add the position vector E pos ; to form the input of the Transformer encoding. The network depth of the repeated architecture of the Transformer is selected to be 24 layers, and a 1*1024 high-dimensional feature vector is obtained through the encoding of the Transformer.

[0017] In one embodiment of the method for assisting in breast disease screening based on mammography according to the present invention, the feature classification includes: using a perceptron network to splice 32-dimensional metadata features and 1024-dimensional image features into a 1056-dimensional feature. The 1056-dimensional feature contains image and metadata features, and then the 1056-dimensional feature is input into a multi-layer perceptron network for classification. The multi-layer perceptron network is in turn a fully connected layer -> activation function layer -> dropout layer -> fully connected layer -> dropout layer -> output result. The output result is compared with the true value, and then optimized training is performed using a loss function to obtain a breast disease classification model.

[0018] The present invention can solve the problems of a large number of film readings and insufficient human experience during breast physical examinations, thereby improving the film reading efficiency of doctors. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. 1 is a breast example: Figure 1a is normal breast tissue, Figure 1b is benign, Figure 1c is malignant breast cancer;

[0020] Figure 2 is a flowchart of the breast disease screening algorithm;

[0021] Figure 3 is a data preprocessing diagram;

[0022] Figure 4 is a schematic diagram of the data training process;

[0023] Figure 5 is a schematic diagram of image feature extraction based on a transformer. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To make the objectives, content, and advantages of the present invention clearer, the following further describes in detail the specific embodiments of the present invention with reference to the accompanying drawings and embodiments.

[0025] The present invention comprises three parts: an image feature extraction part, a metadata feature extraction part, and a feature classification part. Breast cancer, as a highly prevalent cancer, has always been a key focus of general screening in the country. However, the large number of collected images, strong repeatability, and heavy workload lead to a certain degree of missed detection by doctors. To solve this problem, the present invention mimics the process of doctors viewing mammogram images. First, it examines various information of the target object, including age, gender, weight, height, whether they often drink alcohol, whether they often smoke, age of menarche, age of menopause, whether they have given birth, whether they have a family history of breast diseases, whether the breast has received radiotherapy, and age of first childbirth. These factors are referred to as metadata in the present invention. The present invention uses a neural network method for feature extraction in the metadata feature extraction part. To obtain more mutual relationships between information, the present invention uses a two-layer neural network to extract metadata features.

[0026] Then, doctors will view the mammogram based on experience to find possible lesion features in the image. In this step, the present invention uses Transformer in the self-attention-based architecture. This structure is a feature extraction network structure proposed by Google. The present invention uses Transformer for mammogram image feature extraction. To effectively extract the features of the image, the network depth used in the present invention is 24 layers, and its parameter size is 307 * 10^6. On most machines with GPUs, it can complete the extraction of data features within 1 minute. And in terms of feature extraction, Transformer has been proven to be more effective than convolutional neural networks. Through the image feature extraction part, the present invention can obtain a 1024-dimensional feature vector, which mimics the process of human doctors viewing images, but the machine will be more efficient and focused than human doctors.

[0027] Finally, it combines the feature information of the image and the feature information of the metadata to determine whether the target object has breast cancer. In this process, the present invention uses a multi-layer perceptron (MLP) for feature classification, and the classification results are normal, benign, and malignant. The multi-layer perceptron uses fully connected layers. The present invention uses 2 hidden layers. On the one hand, it reduces the computational amount. On the other hand, since feature extraction has been performed before, all the features here are high-dimensional and effective features, so the classification network should not have too many layers. From the classification effect, the multi-layer perceptron is also very suitable as a classifier for a small amount of feature data.

[0028] As Figure 2 shown, an embodiment of the present invention uses a method comprising three parts: data preparation, data preprocessing, data training, and model verification.

[0029] Step 1: First, it is necessary to collect mammography images, including 500 cases of benign, 500 cases of malignant, and 500 cases of normal data. Each case of data should include the cranio-caudal (CC) views of the left and right breasts, the mediolateral oblique (MLO) views of the left and right breasts, and the subject information: age, gender, height, weight, whether they often drink alcohol, whether they often smoke, age of menarche, age of menopause, whether they have given birth, whether they have a family history of breast diseases, whether the breast has received radiotherapy, and the age of the first childbirth;

[0030] Step 2: Data preprocessing. First, process the image data. Stitch the four images of the cranio-caudal views of the left and right breasts and the mediolateral oblique views of the left and right breasts. After stitching, the size is 4600*2300*3, and then scale it to an image of size 256*256*3 as Figure 3 shown. Then use data augmentation to increase the robustness of the data, including rotation, translation, scaling, color space adjustment, self-adversarial training augmentation, multi-image combination, and MixUp. Then process and organize the collected table. All data is digitized, with 1 for male, 2 for female, and 0 for other cases such as not filled; 1 for drinking alcohol, 2 for not drinking alcohol, and 0 for other cases such as not filled; whether they have given birth, whether they have a family history of breast diseases, whether the breast has received radiotherapy, and the age of the first childbirth are processed similarly. After processing, a metadata will be obtained, which is convenient for subsequent processing.

[0031] Step 3: Data training. The training includes three parts: image feature extraction, metadata feature extraction, and feature classification. As Figure 4 shown, the image feature extraction network is a feature extraction network based on transformer, and the output is a high-dimensional feature of 1024 dimensions; the metadata extraction part takes into account the factors related to breast diseases and outputs a high-dimensional feature of 32 dimensions through a two-layer network, as Figure 4 shown. In the metadata feature extraction part, it is divided into two layers. The first layer outputs a high-dimensional feature of 512 dimensions. The more dimensions, the more weights between the metadata can be learned. Therefore, 512 layers are selected here (too many will cause overfitting). The second layer is regressed to 32 layers as a trade-off with the number of image features to avoid the excessive influence of metadata features; then the two high-dimensional features are concatenated (concat), and then a multi-layer perceptron is used as the classifier (MLP) to obtain the final result. The parameters of the network are batchsize (the number of samples used in one iteration) = 16, learning rate = 1e-5, and epochs (the number of times all samples in the training set are trained) = 100.

[0032] Step 3.1: The metadata feature processing part of the present invention, as Figure 4As shown, the metadata first constructs a 512-dimensional shallow feature through linear regression; then goes through normalization, activation function, and random sampling (dropout), where the sampling coefficient is 0.3; then constructs a 32-dimensional deep feature through linear regression again; also goes through normalization and activation function; finally outputs a 32-dimensional metadata feature.

[0033] Step 3.2: The image feature extraction structure of the present invention is as Figure 5 shown. The Transformer algorithm is used. Transformer is a classic model for natural language processing proposed by the Google team in 2017. The Transformer model uses the self-attention mechanism and does not adopt the sequential structure of the recurrent neural network, enabling the model to be trained in parallel and having global information. The image obtained after preprocessing is a 256*256*3 image, which is then evenly cropped into 8*8 = 64 parts to construct a 64*32*32*3 image group. At the same time, the position information is made into a 1024-dimensional learnable vector E pos , E pos is a 65*1024 vector representing the position information of the sequence. Then, the X p ∈64*32*32*3 image group is expanded into 64 one-dimensional sequences 64*(32*32*3) = 64*3072, and then multiplied by the learnable vector E. The dimension of E is (32*32*3)*1024, and the output of the multiplication result is 64*1024 dimensions. The formula 1 is as follows:

[0034]

[0035] For the subsequent classification task; a vector is added here; the dimension 1024 is a learnable embedding vector; the 0th position in the position encoding plays an indexing role, and then the position vector E pos is added; to form the input of the Transformer encoding. Among them, the repeated architecture of the Transformer, the network depth selected in the present invention is 24 layers. Through the encoding of the Transformer, a 1*1024 high-dimensional feature vector can be obtained.

[0036] Step 3.3: The last part is feature classification. Using a multi-layer perceptron network (MLP), the 32-dimensional metadata features and the 1024-dimensional image features are concatenated into a 1056-dimensional feature, which contains both image and metadata features. Then, the 1056-dimensional feature is input into the multi-layer perceptron network for classification. The MLP consists of a fully connected layer -> activation function layer -> dropout layer -> fully connected layer -> dropout layer -> output result in sequence. The output result is compared with the ground truth, and then optimized and trained using a loss function. After 100 epochs of training, a breast disease classification model is obtained.

[0037] Step 4: Model verification. When conducting breast cancer mass screening, first, let the subjects fill in the corresponding electronic questionnaire, then take mammogram images of the subjects, and then input them into the network to obtain the breast health status of the subject: normal, benign, or malignant tumor, as well as the confidence level of the corresponding result. Most subjects can be screened out automatically. For images with a malignant confidence level greater than 20% or a benign confidence level greater than 30%, they are rejudged by a doctor. Through the above method, the doctor's time for viewing images can be effectively reduced, achieving the purpose of assisting in screening.

[0038] In summary, the present invention can train a classification model for breast diseases. The model can effectively classify whether the breast is normal, benign, or malignant from metadata and mammogram images. These three parts are optimized as a whole during the training process to achieve the best effect. After training, the model is verified, that is, the model assists doctors in screening during the breast screening process to reduce the workload and improve the efficiency of film reading.

[0039] The present invention is an algorithm that combines deep learning and breast mammogram images to reduce doctors' film reading and errors and serves as an auxiliary screening method to automatically screen out problems such as normal, benign, and malignant in breast diseases in the medical system.

[0040] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principles of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. An auxiliary screening method for breast diseases based on mammography, characterized in that, Including: Data preparation, data preprocessing, data training, and model verification; Perform data preparation, including: collecting mammogram images, including benign, malignant, and normal data as image features, and each piece of data contains subject information as metadata; Perform data preprocessing on the collected images; each piece of data contains the craniocaudal views of the left and right breasts, the mediolateral oblique views of the left and right breasts, and subject information; Perform data preprocessing on the collected images, including: stitching the four images of the craniocaudal views of the left and right breasts and the mediolateral oblique views of the left and right breasts, then scaling the images, and then using data augmentation to increase the robustness of the data. The data augmentation includes rotation, translation, scaling, color space adjustment, self-contrastive training augmentation, multi-image combination, and MixUp augmentation. Digitalize the subject information data; Perform data training on the preprocessed data. The training includes image feature extraction, metadata feature extraction, and feature classification. The image feature extraction network outputs the first high-dimensional feature through a feature extraction network based on a transformer; by constructing a 64×32×32×3 image group, and at the same time making the position information into a learnable vector Epos of 1024 dimensions, then expand the image group Xp∈64×32×32×3 into 64 one-dimensional sequences 64×(32×32×3)=64×3072, and then multiply by the learnable vector E to form the input of the Transformer encoding. Through the Transformer encoding, a 1×1024 high-dimensional feature vector can be obtained. The metadata feature extraction combines factors related to breast diseases. The metadata constructs shallow features through linear regression and constructs deep features through linear regression, and outputs the second high-dimensional feature; concatenate the first high-dimensional feature and the second high-dimensional feature, and after feature classification, use a perceptron network to concatenate the 32-dimensional metadata feature and the 1024-dimensional image feature into a 1056-dimensional feature. The 1056-dimensional feature contains image and metadata features, and then input the 1056-dimensional feature into a multi-layer perceptron network for classification. The multi-layer perceptron network is in turn a fully connected layer -> activation function layer -> dropout layer -> fully connected layer -> dropout layer -> output result. Compare the output result with the true value, and then use the loss function for optimization training to obtain a breast disease classification model; Take a mammogram of the subject, and then input it into the breast disease classification model to obtain the classification result of the mammogram image and the confidence of the corresponding result.

2. The method for auxiliary screening of breast diseases based on mammography images according to claim 1, wherein Subject information includes: age, gender, weight, height, whether often drink alcohol, whether often smoke, age of menarche, age of menopause, whether give birth, whether have a family history of breast diseases, whether the breast has received radiotherapy, and age of the first childbirth.

3. The method for auxiliary screening of breast diseases based on mammography images according to claim 2, wherein Each piece of data of the subject information is used as metadata to extract metadata features.

4. The method for auxiliary screening of breast diseases based on mammography images according to claim 1, wherein, The image feature extraction network is a feature extraction network based on a transformer, and the output is a 1024-dimensional first high-dimensional feature; The metadata extraction part takes into account factors related to breast diseases and outputs a 32-dimensional second high-dimensional feature through a two-layer network; the first high-dimensional feature and the second high-dimensional feature are concatenated, and a multi-layer perceptron is used as a classifier. After feature classification, a breast disease classification model is obtained.

5. The method for auxiliary screening of breast diseases based on mammography images according to claim 1, characterized in that, Metadata feature extraction includes: The metadata constructs a 512-dimensional shallow feature through linear regression, and then through normalization, activation function, and random sampling, constructs a 32-dimensional deep feature through linear regression, and then through normalization and activation function, outputs a 32-dimensional metadata feature.

6. The method for auxiliary screening of breast diseases based on mammography images according to claim 1, characterized in that Image feature extraction includes; The image is preprocessed to obtain an image of 256*256*3, which is evenly cropped into 8*8 = 64 parts to construct an image group of 64*32*32*3. The position information is made into a learnable vector E of 1024 dimensions pos , E pos is a vector of 65*1024 representing the position information of the sequence. Let X p ∈ the image group of 64*32*32*3 is expanded into 64 one-dimensional sequences 64*(32*32*3) = 64*3072, multiplied by the learnable vector E. The dimension of E is (32*32*3)*1024, and the output of the multiplication result is 64*1024 dimensions. Formula 1 is as follows: E ∈ R (32*32*3)*1024 , E pos ∈ R (64+1)*1024 (1) Dimension 1024 is a learnable embedding vector; the 0th position in the positional encoding serves as a kind of index, and then the positional vector E is added pos ; which constitutes the input of the Transformer encoding. Among them, the network depth selected by the repeated architecture of the Transformer is 24 layers, and a 1*1024 high-dimensional feature vector is obtained through the encoding of the Transformer.

Images