Construction method of early screening model for breast cancer and screening auxiliary system
By employing cross-modal feature fusion and multi-task collaboration strategies, an early breast cancer screening model was constructed. This model addresses the interpretability and data insufficiency issues of deep neural networks in breast cancer screening, enabling rapid and accurate breast cancer screening and improving the reliability and transparency of the model.
Patent Information
- Application Number
- CN202410372922.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-03-29
AI Technical Summary
Existing technologies make it difficult to build a rapid and accurate early breast cancer screening system using limited breast ultrasound data. Deep neural networks lack interpretability and require a large amount of data for training, leading to misdiagnosis and overfitting problems.
Employing semantically driven interpretability units and a multi-task collaborative strategy, an early breast cancer screening model is constructed by fusing cross-modal features from breast ultrasound images and radiology reports. Combining a clinical report-driven interpretability mechanism and a multi-task collaborative diagnostic strategy, morphological and interpretable features are extracted to achieve rapid and accurate breast cancer screening.
It improves the accuracy and interpretability of early breast cancer screening, reduces the risk of misdiagnosis, enhances the transparency and reliability of the model, simulates the clinical diagnostic process, and improves the model's generalization ability.
Smart Images

Figure CN118299032B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a construction method of an early screening model for breast cancer and a screening auxiliary system, and belongs to the field of medical image processing and small sample learning. Background Art
[0002] Breast cancer is the most common malignant tumor in the world and one of the most common cancers in women. Due to the large population base in my country, the number of female breast cancer patients in China has increased year by year, and the age of patients has also become younger and younger. Therefore, how to improve the early screening capabilities of breast cancer has become increasingly important for the prevention, control and treatment of breast cancer in my country.
[0003] In recent years, with the rapid development of medical imaging technology, there are more flexible and accurate ways to screen breast cancer early. At present, commonly used breast imaging technologies include breast ultrasound and breast dynamic contrast-enhanced magnetic resonance imaging. Considering the medical level and cost of my country and developing countries, breast ultrasound examination is the most effective early screening method for breast cancer that suits the national conditions of my country and developing countries, because breast ultrasound imaging diagnosis has the advantages of low cost, non-invasiveness and no radiation compared with other breast imaging technologies. However, the current breast ultrasound diagnosis has extremely high requirements for clinicians, requiring clinicians to have very rich diagnostic experience and technology. In addition, clinicians need to interpret a large number of breast ultrasound images, which can easily lead to fatigue and misdiagnosis of clinicians. Recently, deep neural networks have achieved great success in breast ultrasound image diagnosis. However, due to the black box problem in deep neural networks, it is impossible to show the training process of the network through some effective formulas, and the results shown only by accuracy and other methods will not be actually applied. These networks still lack sufficient interpretability. In addition, deep neural network algorithms require a large amount of data to train so that the network can learn sufficient experience. As we all know, it is not easy to collect labeled medical image data due to the privacy of patients. When the training data is insufficient, the deep neural network algorithm cannot learn enough experience, which often leads to overfitting. Therefore, it is a challenging problem to build a fast and accurate early breast cancer screening auxiliary system using a small amount of breast ultrasound data. Summary of the invention
[0004] In order to solve the above problems, the present invention provides a method for constructing an early screening model for breast cancer and a screening auxiliary system. The technical solution is as follows:
[0005] The first object of the present invention is to provide a method for constructing an early screening model for breast cancer, the method comprising:
[0006] Step 1: Obtain breast ultrasound images and breast ultrasound radiology reports;
[0007] Step 2: preprocessing the breast ultrasound image and breast ultrasound radiology report;
[0008] Step 3: Based on the preprocessed ultrasound images and radiology reports, a paired ultrasound image and radiology report dataset is constructed and divided into a support set and a query set;
[0009] Step 4: construct two encoders to extract morphological features and interpretability features from paired ultrasound images and radiology reports, respectively, and implement cross-modal combination of the morphological features and interpretability features through a semantically driven interpretability unit to obtain fused features;
[0010] Step 5: construct a multi-task collaboration strategy, train a neural network model based on the fusion features, and obtain a trained breast cancer early screening model, wherein the breast cancer early screening model outputs a breast cancer classification result based on the input fusion features.
[0011] Optionally, step 4 includes:
[0012] Step 41: construct a breast ultrasound image encoder and a breast ultrasound radiology report encoder;
[0013] Step 42: Encode the breast ultrasound image x using the breast ultrasound image encoder to extract morphological features Where C is the number of channels of the output feature, H and W are the height and width respectively;
[0014] Step 43: Encode the breast ultrasound radiology report r using the breast ultrasound radiology report encoder to extract semantic features
[0015] Step 44: Generate new morphological features:
[0016]
[0017] in is the weight of the fully connected layer, b c1 , b c2 is the bias term of the corresponding fully connected layer, l1 and l2 are the number of output features;
[0018] Step 45: Generate interpretable features:
[0019]
[0020] σ is the sigmoid function, is the weight of the fully connected layer, b e1 , b e2 is the bias term of the corresponding fully connected layer, l3 and l4 are the number of output features;
[0021] Step 46: The cross-modal combination of the two features is achieved through a semantically driven interpretability unit, as follows:
[0022]
[0023] in, is the fusion feature, F mul Represents the new morphological feature f morphology and interpretability feature f explanation Point multiplication between .
[0024] Optionally, step 5 includes:
[0025] Step 51: Construct training set D train and the test set D test , the training set D train Includes: Training Task Task train and test tasks test ;
[0026] Step 52: Training Task train In the paper, the clinical report-driven interpretability model is defined as a neural network F with parameters θ θ , randomly initialize θ=θ0;
[0027] For training tasks train The support set task in Perform G gradient update processes to obtain θ G , this process is called the inner loop normal training process, where j is the index of a batch of tasks, G is the number of gradient descents in the inner loop training, and in the inner loop normal training, the support set The adaptive parameters after calculating the gradient descent are:
[0028]
[0029] in, is the semantically driven interpretability model weights for task j after running G gradient updates, is the loss of the support set task during normal inner loop training, and α is the learning rate of normal inner loop training;
[0030] Step 53: Given the total size of each batch of tasks as J, according to all query set tasks The performance of θ0 is optimized by averaging the total loss. This set of optimizations is named the outer loop to assist the training process. The update result of θ0 is expressed as:
[0031]
[0032] Among them, β is the learning rate of the outer loop to assist training, The loss when using the query set task in the outer loop to assist training;
[0033] Step 54: Using the task loss, the clinical report driven interpretability model accepts a single input and generates a single output. The task loss is expressed as the error between the output S and the target Q, and its formula is:
[0034]
[0035] in, is the task loss, S i is the task output, Q i It is the mission objective.
[0036] Optionally, the preprocessing process in step 2 includes:
[0037] Step 21: rescaling all breast ultrasound images to a uniform pixel value size to obtain a uniform input image dimension;
[0038] Step 22: normalize all ultrasound image pixel values to obtain a standard normal distribution;
[0039] Step 23: After removing non-English sentences and erroneous sentences, construct a standard radiology report;
[0040] Step 24: Use Tokenizer technology to obtain feature representations of words in the radiology report.
[0041] Optionally, step 3 includes:
[0042] Step 31: Construct N-way C-shot breast cancer early screening task;
[0043] Step 32: Given a training set, divide the training set into support and query sets.
[0044] The support set consists of N breast cancer types, each of which includes C pairs of breast ultrasound images and breast ultrasound radiology reports;
[0045] The query set consists of N breast cancer types, but M pairs of breast ultrasound images and breast ultrasound radiology reports are randomly selected from each breast cancer type;
[0046] The purpose of the N-way C-shot breast cancer early screening task is to predict the query set samples based on N types of breast cancer and their N×C samples in the support set.
[0047] A second object of the present invention is to provide an auxiliary system for early screening of breast cancer, the system comprising:
[0048] An image acquisition module, used for acquiring a breast ultrasound image to be screened;
[0049] A clinical report acquisition module, used for acquiring a breast ultrasound radiology report corresponding to the breast ultrasound image to be screened;
[0050] A data preprocessing module, used for preprocessing the breast ultrasound image and breast ultrasound radiology report;
[0051] A feature extraction module, including a breast ultrasound image encoder and a breast ultrasound radiology report encoder, which are used to extract morphological features and interpretability features respectively, and realize a cross-modal combination of the morphological features and the interpretability features through a semantically driven interpretability unit to obtain a fusion feature;
[0052] A classification module, wherein the classification module is an early screening model for breast cancer constructed by any of the methods described above, and the model outputs a breast cancer classification result based on the input fusion features.
[0053] Optionally, the breast ultrasound image encoder adopts a multi-feature fusion capsule network model.
[0054] Optionally, the breast ultrasound radiology report encoder adopts a long short-term memory network.
[0055] The third object of the present invention is to provide an auxiliary device for early screening of breast cancer, comprising a memory and a processor;
[0056] The memory is used to store computer programs;
[0057] The processor is used to obtain a breast cancer classification result by using any of the above-mentioned breast cancer early screening auxiliary systems when executing the computer program.
[0058] A fourth object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, a breast cancer classification result is obtained by using an early breast cancer screening auxiliary system as described in any one of the above items.
[0059] The beneficial effects of the present invention are:
[0060] (1) The breast cancer early screening auxiliary system of the present invention combines the clinical report-driven interpretability mechanism with the multi-task collaborative diagnosis strategy to achieve rapid and accurate early screening of breast cancer. This is a cross-modal mutual learning method that can use the clinical information in the breast ultrasound clinical radiology report to interpret and enhance the information of the breast ultrasound image, thereby improving the reliability and transparency of the model, making the results generated by the model easier for clinicians to understand and accept.
[0061] (2) The present invention simulates the process of clinicians diagnosing patients and invents a clinical report-driven interpretability mechanism, which is similar to the process of clinicians diagnosing patients and interprets the information of breast ultrasound images by extracting clinical information from breast ultrasound clinical radiology reports.
[0062] (3) The present invention imitates the inspiration of mutual help in the human learning process and invents a new small sample learning strategy, namely, the multi-task collaborative strategy. This strategy enables the models to learn to learn by enabling them to collaborate with each other for training, thereby improving the generalization ability of the models.
[0063] (4) The present invention uses a small number of paired breast ultrasound images and breast ultrasound clinical radiology reports. By combining the clinical report-driven interpretability mechanism with a multi-task collaborative diagnosis strategy, it not only avoids overfitting and achieves competitive diagnostic results, but also improves the reliability and transparency of the diagnostic results. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0065] Figure 1 It is a flow chart of the method for constructing an early screening model for breast cancer of the present invention.
[0066] Figure 2 It is a network structure diagram of the feature extraction part of the second embodiment of the present invention.
[0067] Figure 3 It is a schematic diagram of dividing the support set and the query set in the second embodiment of the present invention.
[0068] Figure 4 This is a diagram of the clinical report-driven interpretability mechanism structure (feature fusion process) of Example 2 of the present invention.
[0069] Figure 5 It is a structural diagram of a breast ultrasound encoder according to the second embodiment of the present invention.
[0070] Figure 6 It is a structural diagram of a breast ultrasound radioactivity reporting encoder according to the second embodiment of the present invention.
[0071] Figure 7 This is a diagram of the LSTM network structure of the second embodiment of the present invention.
[0072] Figure 8 This is a diagram of the multi-task collaborative training network structure of the second embodiment of the present invention.
[0073] Fig. 9 This is a diagram of breast cancer classification results according to the second embodiment of the present invention. DETAILED DESCRIPTION
[0074] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0075] Embodiment 1:
[0076] This embodiment provides a method for constructing an early screening model for breast cancer. Figure 1 As shown, the method includes:
[0077] Step 1: Obtain breast ultrasound images and breast ultrasound radiology reports;
[0078] Step 2: Preprocessing breast ultrasound images and breast ultrasound radiology reports;
[0079] Step 3: Based on the preprocessed ultrasound images and radiology reports, a paired ultrasound image and radiology report dataset is constructed and divided into a support set and a query set;
[0080] Step 4: Construct two encoders to extract morphological features and interpretability features from paired ultrasound images and radiology reports, respectively, and use semantically driven interpretability units to realize cross-modal combination of morphological features and interpretability features to obtain fused features;
[0081] Step 5: Construct a multi-task collaborative strategy, train the neural network model based on the fusion features, and obtain a trained early breast cancer screening model.
[0082] Embodiment 2:
[0083] This example provides a method for constructing an interpretable small sample learning model for early breast cancer screening driven by clinical reports. Figure 1 , the method comprising:
[0084] (1) Obtain breast ultrasound images and ultrasound clinical reports.
[0085] (2) Process the acquired images and reports.
[0086] (3) Construct a paired image and report dataset and divide it into a support set and a query set.
[0087] (4) Design a clinical report-driven interpretability mechanism that simultaneously learns morphological (image) and clinical (radiology report) information from paired breast ultrasound images and breast ultrasound radiology reports, and then uses the clinical information obtained from the radiology reports as interpretable information of the images, thereby promoting the reliability, transparency, and interpretability of the black-box AI model.
[0088] (5) Construct a multi-task collaboration strategy, which enables artificial intelligence models to collaborate in training, so that the models can learn to learn, thereby improving the generalization ability of the models.
[0089] (6) By combining the clinical report-driven interpretability mechanism and multi-task collaboration strategy, the small sample learning method can effectively and accurately achieve early screening of breast cancer based on a small number of paired ultrasound images and breast ultrasound radiology reports.
[0090] Furthermore, step (1) obtaining a breast ultrasound image and an ultrasound clinical report specifically includes the following steps:
[0091] (11) A total of 1128 examinations were performed on 995 patients using a fully automatic breast ultrasound scanner from GE Healthcare. The size of the generated 3D ultrasound data was 900×1280×720, and the slice thickness was 0.3 mm. One examination of a patient included multiple scanning directions, including five scanning directions of the left breast: LAP, LLAT, LMED, LUOQ, and LSUP, and five scanning directions of the right breast: RAP, RLAT, RMED, RUOQ, and RSUP.
[0092] (12) Through the examination, a PDF-formatted examination report is obtained for each patient. The report describes the patient's basic information, scanning direction, and whether there is a tumor. If there is a tumor, the approximate location of the tumor is marked, and the location, size, and grade (benign or malignant) of the tumor are described.
[0093] Furthermore, step (2) processes the acquired images and reports, specifically including the following steps:
[0094] (21) According to the physicians’ annotations, all breast ultrasound images were cropped and their resolutions were rescaled to 256×256×256 pixel values to obtain a uniform input image dimension.
[0095] (22) All ultrasound image pixel values are switched from [0, 255] to [0, 1] to obtain a standard normal distribution.
[0096] (23) After removing non-English sentences and erroneous sentences, a standard radiology report was constructed.
[0097] (24) Tokenizer technology is used to obtain the feature representation of words in radiology reports. The Tokenizer technology first removes English punctuation marks and converts uppercase letters to lowercase letters; then, word frequency counting is performed on the radiology report and the words are sorted according to word frequency; then, each word is converted into a numerical sequence based on word frequency; finally, a zero-padding method is used to obtain the required length of the numerical sequence.
[0098] Furthermore, step (3) constructs a paired image and report dataset and divides it into a support set and a query set; Figure 3 As shown, the specific steps include:
[0099] (31) Construct an N-way C-shot breast cancer early screening task, which includes 2-way 1-shot, 2-way 2-shot, 2-way 3-shot, 2-way 4-shot, 2-way 5-shot, and 2-way 6-shot early screening tasks.
[0100] (32) Given a training set, divide the training set into a support set and a query set.
[0101] (33) The support set consists of two breast cancer types, but each breast cancer type only contains C = {1, 2, 3, 4, 5, 6} pairs of breast ultrasound images and breast ultrasound radiology reports.
[0102] (34) The query set also consists of two breast cancer types, but M = 5 pairs of breast ultrasound images and breast ultrasound radiology reports are randomly selected from each breast cancer type.
[0103] (35) The purpose of the N-way C-shot task is to predict the query set samples based on the two types of breast cancer and their 2×{1, 2, 3, 4, 5, 6} samples in the support set.
[0104] Furthermore, step (4) designs a clinical report-driven interpretability mechanism, which simultaneously learns morphological (image) and clinical (radiology report) information from paired breast ultrasound images and breast ultrasound radiology reports, and then uses the clinical information obtained from the radiology reports as interpretable information of the images, thereby promoting the reliability, transparency, and interpretability of the black-box AI model, such as Figure 4 As shown, the specific steps include:
[0105] (41) Two encoders are constructed, namely, Breast Ultrasound Image Encoder (BUIE) and Breast Ultrasound Radiology Report Encoder (BURPE). The encoders can be constructed using a neural network to realize the function of feature extraction.
[0106] (42) The breast ultrasound image encoder uses a multi-feature fusion capsule network model, such as Figure 5 As shown, the first step of the model is the input vector u of the capsule i Using the affine transformation operation, use the formula Get the transformed prediction vector And set the number of iterations r of dynamic routing; the second step is to initialize the prior probability b between the low-level capsule i and the high-level capsule j ij , b ij =0; the third step coupling coefficient c ij With the prior probability b ij The formula is used to change with the change of The coupling coefficient c is obtained ij ; Step 4 All predicted capsules Using the coupling coefficient c ij To weight and sum, use the formula Get the total input vector S j ; The fifth step uses the nonlinear activation function Squash to process the input vector S j Perform the extrusion transformation operation, that is, use the formula Get the capsule output vector v j ; Step 6 predicts the vector for each low-level capsule and the output vector v j Perform a dot product operation, that is, use the formula Update the prior probability; in the seventh step, when the set number of dynamic routing iterations is reached, the update of the capsule network ends, otherwise jump to the third step to continue; finally, add a capsule layer after each convolutional layer of the model, and save the features extracted by the capsule layer, then add a pooling layer for downsampling to reduce the noise of the image, and make the receptive field of the capsule layer larger, and then merge the capsule features saved at different levels.
[0107] (43) The breast ultrasound radiology report encoder uses a long short-term memory network (LSTM), such as Figure 6As shown, the network uses three gates, input gate, forget gate, and output gate, to control the retention of information. The formula for each gate is:
[0108] i t =sigmoid(W ri r t +W hi h t-1 +b i )
[0109] f t =sigmoid(W rf r t +W hf h t-1 +b f )
[0110] o t =sigmoid(W ro r t +W ho h t-1 +b o )
[0111] Where W ri , W rf and W ro is to update the current input vector r t The weight matrix W hi , W hf and W ho Indicates updating the output h at time t-1 t-1 Parameters of b i , b f and b o Represents the bias term of the corresponding update operation. The input gate (it) is used to manage the valuable information stored in the memory unit; the forget gate (f t ) is used to decide which previous information to keep or forget, and the output gate (o t ) is used to determine the amount of information to be displayed. The input gate consists of two parts, the sigmoid layer and the tanh layer, which work together to retain the information. The formula is as follows:
[0112] g t =tanh(W rg r t +W hg h t-1 +b g )
[0113] c t =i t ⊙gt +f t ⊙c t-1
[0114] h t =o t ⊙tanh(c t )
[0115] Where ⊙ is the matrix dot product, c t is the cell state value, h t is a hidden output.
[0116] (44) Use BUIE to encode breast ultrasound image x and extract morphological (image) features Where C is the number of channels of the output feature, H and W are the height and width respectively.
[0117] (45) Use BURPE to encode breast ultrasound radiology reports and output clinical (interpretability) features
[0118] (46) Generate new morphological features using the following formula:
[0119]
[0120] is the weight of the fully connected layer, b c1 , b c2 is the bias term of the corresponding fully connected layer, l1=128, l2=64 is the number of output features.
[0121] (47) The following formula is used to generate interpretable features:
[0122]
[0123] σ is the sigmoid function, is the weight of the fully connected layer, b e1 , b e2 is the bias term of the corresponding fully connected layer, l3=128, l4=64 is the number of output features.
[0124] (48) After obtaining the morphological features and semantic features, the cross-modal combination of these two features is realized through the semantically driven interpretability unit. The formula is as follows:
[0125]
[0126] Among them, f morphology , Represents new morphological features and interpretability features, F mul represents fmorphology and f explanation Point multiplication between ; For the original morphological characteristics, is a semantic feature, in particular, Global average pooling is used so that its shape is consistent with match.
[0127] The clinical report-driven interpretability mechanism is the process of clinicians diagnosing patients in real life. The encoding process of breast ultrasound images can be seen as the process of clinicians interpreting images, and the encoding process of breast ultrasound radiology reports can be seen as the process of clinicians interpreting patients' pathology reports.
[0128] Furthermore, step (5) constructs a multi-task collaboration strategy, which enables the artificial intelligence models to collaborate with each other in training, so that the models can learn to learn, thereby improving the generalization ability of the models, such as Figure 8 As shown, the specific steps include:
[0129] (51) Given a dataset D consisting of breast ultrasound images and related breast radiology reports.
[0130] (52) Divide the dataset D into two parts: training set (D train ) and the test set (D test ), training set D train By training task train ) and test tasks test )composition.
[0131] (53) From D train Randomly select some pairs of images and radiology reports to construct Task train and Task test .Task train and Task test Contains some support sets and query sets.
[0132] (54) In the training task train In the paper, the clinical report-driven interpretability model is defined as a neural network F with parameters θ θ , randomly initialize θ = θ0; next, for the training task Task train The support set task in Perform G gradient update processes to obtain θ G , this process is called the inner loop normal training process, where j is the index of a batch of tasks, G is the number of gradient descents in the inner loop training, and in the inner loop normal training, the support set The adaptive parameters after calculating the gradient descent are:
[0133]
[0134] in, is the semantically driven interpretability model weights for task j after running G gradient updates, is the loss of the support set task during normal inner loop training, and α is the learning rate of normal inner loop training;
[0135] (55) Given the total size of each batch of tasks as J, we then calculate the total size of each batch of tasks based on all query set tasks. The performance of θ0 is optimized by averaging the total loss. This set of optimizations is named the outer loop to assist the training process. The update result of θ0 can be expressed as:
[0136]
[0137] β is the learning rate of the outer loop to assist training, The loss when using the query set task in the outer loop to assist training;
[0138] (56) Using task loss, the clinical report-driven interpretability model accepts a single input and generates a single output. The task loss is expressed as the error between the output S and the target Q, and its formula is:
[0139]
[0140] is the task loss, S i is the task output, Q i It is the mission objective.
[0141] (57) In the inner loop, the multi-task collaborative diagnosis strategy uses the support set tasks to normally train the clinical report-driven interpretability model, and then optimizes the model through the query set tasks in the outer loop. Specifically, multi-collaborative training uses the experience gained from training the support set tasks in the inner loop to guide the classification of the query set tasks in the outer loop; then, the experience gained from training the query set tasks in the outer loop is further used to optimize the experience gained from training the support set tasks in the inner loop. Through this continuous cycle of optimization training, different support set tasks and query set tasks can collaborate with each other for training and learning. Therefore, after a series of training, the proposed clinical report-driven interpretability model can learn collaborative learning, learn fast learning, and learn to learn by itself, that is, the clinical report-driven interpretability model learns by itself how to learn quickly from a small amount of data.
[0142] After the model is trained, it can be used for early screening of breast cancer. The screening process is as follows:
[0143] Doctors obtain breast ultrasound images and breast ultrasound radiology reports of patients through ultrasound examinations, and then input the breast ultrasound images and breast ultrasound radiology reports into the breast cancer early screening system. The breast cancer early screening system uses the breast ultrasound image encoder BUIE and the breast ultrasound radiology report encoder BURPE to extract the morphological features of breast ultrasound images and the interpretable features of breast ultrasound radiology reports, respectively, and further obtain fusion features, which are then input into the trained classification neural network F. θ In the neural network F θ The final classification result is output and the doctor informs the patient of the result.
[0144] Embodiment three:
[0145] This embodiment provides a breast cancer early screening auxiliary system, including:
[0146] An image acquisition module, used for acquiring a breast ultrasound image to be screened;
[0147] A clinical report acquisition module, used for acquiring a breast ultrasound radiology report corresponding to the breast ultrasound image to be screened;
[0148] A data preprocessing module, used for preprocessing breast ultrasound images and breast ultrasound radiology reports;
[0149] A feature extraction module, including a breast ultrasound image encoder and a breast ultrasound radiology report encoder, which are used to extract morphological features and interpretability features respectively, and realize the cross-modal combination of morphological features and interpretability features through a semantically driven interpretability unit to obtain fusion features;
[0150] The classification module is a breast cancer early screening model constructed by the method of Embodiment 1 or Embodiment 2. The breast cancer early screening model is a trained neural network that can output breast cancer classification results based on input fusion features.
[0151] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.
[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for constructing an early screening model for breast cancer, characterized in that: The method comprises: Step 1: Obtain breast ultrasound images and breast ultrasound radiology reports; Step 2: preprocessing the breast ultrasound image and breast ultrasound radiology report; Step 3: Based on the preprocessed ultrasound images and radiology reports, a paired ultrasound image and radiology report dataset is constructed and divided into a support set and a query set; Step 4: construct two encoders to extract morphological features and interpretability features from paired ultrasound images and radiology reports, respectively, and implement cross-modal combination of the morphological features and interpretability features through a semantically driven interpretability unit to obtain fused features; Step 5: construct a multi-task collaboration strategy, train a neural network model based on the fusion features, and obtain a trained breast cancer early screening model, wherein the breast cancer early screening model outputs a breast cancer classification result based on the input fusion features; The step 4 comprises: Step 41: construct a breast ultrasound image encoder and a breast ultrasound radiology report encoder; Step 42: Utilize the breast ultrasound image encoder to encode the breast ultrasound image Encode and extract morphological features ,in is the number of channels of the output feature, and are height and width respectively; Step 43: Use the breast ultrasound radiology report encoder to encode the breast ultrasound radiology report. Encode and extract semantic features ; Step 44: Generate new morphological features: in , is the weight of the fully connected layer, , is the bias term of the corresponding fully connected layer, , is the number of output features; Step 45: Generate interpretable features: is the sigmoid function, , is the weight of the fully connected layer, , is the bias term of the corresponding fully connected layer, , is the number of output features; Step 46: The cross-modal combination of the two features is achieved through a semantically driven interpretability unit, as follows: in, To fusion features, Represents the new morphological features and interpretability features Point multiplication between ; The step 5 comprises: Step 51: Construct training set and test set , the training set Includes: Training missions and test tasks ; Step 52: Training Tasks In this paper, the clinical report-driven interpretability model is defined as a model with parameters Neural Network , randomly initialized ; For training tasks The support set task in implement A gradient update process is used to obtain , this process is called the inner loop normal training process, where is the index of a batch of tasks, is the number of gradient descents in the inner loop training. In normal inner loop training, the support set The adaptive parameters after calculating the gradient descent are: in, For the task In operation The semantically driven interpretability model weights after gradient updates, is the loss of normal training in the inner loop using the support set task, is the learning rate of normal training of the inner loop; Step 53: Give the total size of each batch of tasks as , according to all query set tasks The average of the total loss is used to optimize This set of optimizations is named as the outer loop to assist the training process. The update result is expressed as: in, is the learning rate of the outer loop assisting training, The loss when using the query set task in the outer loop to assist training; Step 54: Using the task loss, the clinical report driven interpretability model accepts a single input and generates a single output, and the task loss is calculated using the output and goals The error between is expressed as follows: in, It's a mission loss. is the task output, It is the mission objective.
2. The method for constructing an early screening model for breast cancer according to claim 1, characterized in that: The pretreatment process in step 2 includes: Step 21: rescaling all breast ultrasound images to a uniform pixel value size to obtain a uniform input image dimension; Step 22: normalize all ultrasound image pixel values to obtain a standard normal distribution; Step 23: After removing non-English sentences and erroneous sentences, construct a standard radiology report; Step 24: Use Tokenizer technology to obtain feature representations of words in the radiology report.
3. The method for constructing an early screening model for breast cancer according to claim 1, characterized in that: The step 3 comprises: Step 31: Construction -way -shot breast cancer early screening task; Step 32: Given a training set, divide the training set into support and query sets. Query Set ; The support set is given by Each breast cancer type contains Paired breast ultrasound images and breast ultrasound radiology reports; The query set is composed of breast cancer types, but randomly selected from each breast cancer type Paired breast ultrasound images and breast ultrasound radiology reports; Said -way -shot breast cancer early screening mission aims to support the concentration Breast cancer and its The samples are used to predict the query set samples.
4. A breast cancer early screening auxiliary system, characterized in that: The system comprises: An image acquisition module, used for acquiring a breast ultrasound image to be screened; A clinical report acquisition module, used for acquiring a breast ultrasound radiology report corresponding to the breast ultrasound image to be screened; A data preprocessing module, used for preprocessing the breast ultrasound image and breast ultrasound radiology report; A feature extraction module, including a breast ultrasound image encoder and a breast ultrasound radiology report encoder, which are used to extract morphological features and interpretability features respectively, and realize a cross-modal combination of the morphological features and the interpretability features through a semantically driven interpretability unit to obtain a fusion feature; A classification module, wherein the classification module is an early screening model for breast cancer constructed by the method according to any one of claims 1 to 3, and the model outputs a breast cancer classification result according to the input fusion feature; The process of extracting morphological features and interpretability features by the breast ultrasound image encoder and the breast ultrasound radiology report encoder respectively comprises: Encode and extract morphological features ,in is the number of channels of the output feature, and height and width respectively; for breast ultrasound radiology report Encode and extract semantic features ; Then new morphological features and interpretability features are generated, and the new morphological features are: in , is the weight of the fully connected layer, , is the bias term of the corresponding fully connected layer, , is the number of output features; The interpretability features are: is the sigmoid function, , is the weight of the fully connected layer, , is the bias term of the corresponding fully connected layer, , is the number of output features; The semantically driven interpretability unit realizes the cross-modal combination of the morphological features and the interpretability features, and the process of obtaining the fusion features is expressed as follows: in, To fusion features, Represents the new morphological features and interpretability features Point multiplication between .
5. The breast cancer early screening auxiliary system according to claim 4, characterized in that: The breast ultrasound image encoder adopts a multi-feature fusion capsule network model.
6. The breast cancer early screening auxiliary system according to claim 4, characterized in that: The breast ultrasound radiology report encoder adopts a long short-term memory network.
7. An auxiliary device for early screening of breast cancer, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to obtain a breast cancer classification result by using the breast cancer early screening auxiliary system according to any one of claims 4 to 6 when executing the computer program.
8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the breast cancer classification result is obtained by using the breast cancer early screening auxiliary system according to any one of claims 4 to 6.
