Multi-mode-based breast cancer risk hierarchical model construction method and device and multi-mode-based breast cancer risk hierarchical model prediction method
By constructing a multimodal breast cancer risk stratification model, combining cytology diagnosis and ultrasound imaging data, the noise and contrast problems of image recognition in breast cancer diagnosis are solved, and the accuracy and diagnostic efficiency of breast cancer risk stratification prediction are improved.
Patent Information
- Application Number
- CN202411152350.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-06
AI Technical Summary
In the diagnosis of breast cancer, the ultrasound image recognition has high noise and low contrast, which leads to the impact of the diagnosis results due to doctor experience, misjudgment and misjudgment, and the image data processing workload is overloaded, and the multimodal data feature fusion is complex and cannot be obtained. Accurate results.
Using deep learning methods based on attention mechanism, combining cytology diagnostic data and ultrasound imaging data, a multimodal breast cancer risk hierarchical model was constructed, and a convolutional neural network and pre-trained VGG16 network were used to adjust the Softmax layer to perform images and Characteristic fusion of text modality improves the accuracy of breast cancer risk stratification prediction.
It improves the accuracy of breast cancer risk stratification prediction, simplifies the multimodal feature fusion process, reduces the rate of diagnosis errors, reduces the workload of doctors, and improves diagnostic efficiency and accuracy.
Smart Images

Figure CN119092097A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for constructing a multimodal breast cancer risk stratification model. Background Art
[0002] Breast cancer is a malignant tumor that seriously endangers human health and is one of the three major malignant tumors in women. Usually, fine needle aspiration is the gold standard for preoperative diagnosis of breast cancer, but in order to avoid excessive puncture, ultrasound examination is the preferred imaging examination and preoperative evaluation method for breast nodules. In the past, ultrasound examination and diagnosis mainly relied on manual discrimination. There was overlap in the image performance of benign / malignant nodules in ultrasound images. Ultrasound images themselves also had the disadvantages of high noise and low contrast. The diagnostic results were limited by the clinical experience of doctors, which could easily lead to misjudgment and missed judgments. On the other hand, the growth in the number of physicians is far less than the growth in imaging data. In the future, the workload of imaging physicians in processing imaging data will be far overloaded, which will further affect the correctness of the diagnosis.
[0003] In the prior art, there is a method for generating an ultrasonic multimodal automatic recognition model based on the molecular classification of breast cancer. The method performs noise reduction preprocessing on the grayscale, color ultrasound and elastic ultrasound images of the molecular classification of breast cancer; performs data enhancement on the three ultrasound images after preprocessing; performs modal model training on the three enhanced ultrasound images; and finally integrates the grayscale, color ultrasound and elastic models formed by training. However, the multimodal data of this method is only image data, and it is still impossible to obtain accurate results. In addition, there is a breast cancer ultrasound image classification method based on multimodal deep learning imaging omics, which extracts a large number of features from multimodal data such as pictures, videos and texts through deep learning and imaging omics technology, and combines these multimodal information to construct a nomogram, realizing the function of identifying and classifying ultrasound images of benign and malignant breast tumors, but it needs to use the obtained multimodal features to perform factor fusion to construct a nomogram to reflect the nature of the image, which is relatively complicated to implement. Summary of the invention
[0004] In view of the problems existing in the prior art, the present invention adopts a deep learning method based on the use of an attention mechanism to learn multimodal data of the breast and establish a model that can predict the risk stratification of breast cancer (a breast cancer risk stratification model based on multimodality, thereby improving the accuracy of breast cancer risk stratification prediction and further assisting medical diagnosis.
[0005] The technical solution adopted by the present invention specifically includes:
[0006] The first aspect discloses a method for constructing a multimodal breast cancer risk stratification model, comprising:
[0007] Acquiring multimodal data, wherein the multimodal data includes cytological diagnosis data and ultrasound imaging data;
[0008] The cytological diagnosis data and ultrasound image data were preprocessed and divided into training set and validation set respectively;
[0009] Build a deep learning model for cytological diagnosis data. The deep learning model for cytological diagnosis data is a convolutional neural network composed of multiple hidden layers.
[0010] Build a deep learning model for ultrasound image data. The deep learning model for ultrasound image data uses the weight initialization model of the pre-trained VGG16 network and adjusts the last Softmax layer of the VGG16 network according to the classification task;
[0011] Based on the deep learning model of cytological diagnosis data and the deep learning model of ultrasound imaging data, a multimodal breast cancer risk stratification model was constructed, and the multimodal breast cancer risk stratification model was trained using the training sets of cytological diagnosis data and ultrasound imaging data to obtain a trained model.
[0012] In one embodiment, the cytological diagnosis data and the ultrasound image data are preprocessed separately, including:
[0013] Perform case conversion, non-ASCII character deletion, punctuation replacement, and word segmentation on cytological diagnosis data;
[0014] Scale, normalize, and annotate images in ultrasound imaging data.
[0015] In one embodiment, the deep learning model of cytological diagnosis data includes multiple convolutional layers, multiple pooling layers, fully connected layers and output layers, wherein the input cytological diagnosis data is subjected to a series of convolutions and pooling to obtain a feature vector of a fixed size, and then the feature vector of the text data is obtained through one or more fully connected layers and output layers.
[0016] In one embodiment, the last Softmax layer of the VGG16 network is adjusted according to the classification task, including:
[0017] The output size of the last Softmax layer of the VGG16 network is adjusted to the number of categories for breast cancer risk stratification.
[0018] In one embodiment, a multimodal breast cancer risk stratification model is constructed based on a deep learning model of cytological diagnosis data and a deep learning model of ultrasound imaging data, including:
[0019] For the image modality, an adjusted VGG16 network is used, and for the text modality, a convolutional neural network is used;
[0020] Before the image modality and the text modality are shared, one or more hidden layers are set after the adjusted VGG16 network and the convolutional neural network;
[0021] After connecting the two modes, a hidden layer is added, and finally a Softmax layer is set.
[0022] In one embodiment, before training the multimodal breast cancer risk stratification model using the cytological diagnosis data training set and the ultrasound image data training set, the method further includes:
[0023] Zero padding was performed on the texts in the cytology diagnosis data training set to obtain texts of equal length;
[0024] Convert texts of equal length into a token matrix, where each row in the matrix represents a token in the cytology diagnosis data extracted using the pre-trained word2vec model.
[0025] Based on the same inventive concept, the second aspect of the present invention discloses a device for constructing a multimodal breast cancer risk stratification model, comprising:
[0026] A multimodal data acquisition module, used to acquire multimodal data, wherein the multimodal data includes cytological diagnosis data and ultrasound image data;
[0027] A data preprocessing module is used to preprocess the cytological diagnosis data and ultrasound image data respectively, and divide them into a training set and a validation set;
[0028] Convolutional neural network building module, used to build a deep learning model for cytological diagnosis data. The deep learning model for cytological diagnosis data is a convolutional neural network composed of multiple hidden layers;
[0029] Ultrasound image data deep learning model building module, used to build a deep learning model for ultrasound image data. The ultrasound image data deep learning model uses the weight initialization model of the pre-trained VGG16 network and adjusts the last Softmax layer of the VGG16 network according to the classification task;
[0030] The multimodal breast cancer risk stratification model construction module is used to build a multimodal breast cancer risk stratification model based on the deep learning model of cytological diagnosis data and the deep learning model of ultrasound imaging data, and use the training set of cytological diagnosis data and the training set of ultrasound imaging data to train the multimodal breast cancer risk stratification model to obtain a trained model.
[0031] Based on the same inventive concept, the third aspect of the present invention discloses a multimodal breast cancer risk stratification prediction method, comprising:
[0032] The cytological diagnosis data and ultrasound image data to be processed are pre-processed and then input into the trained model described in the first aspect to obtain the breast cancer risk stratification prediction results.
[0033] Based on the same inventive concept, the fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for constructing a multimodal breast cancer risk stratification model as described in the first aspect.
[0034] Based on the same inventive concept, the fifth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for constructing a multimodal breast cancer risk stratification model as described in the first aspect is implemented.
[0035] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:
[0036] The present invention proposes a method for constructing a breast cancer risk stratification model based on multimodality. First, multimodal data is acquired, then preprocessed, and divided into a training set and a validation set; then a deep learning model for cytological diagnosis data is constructed, which is a convolutional neural network composed of multiple hidden layers; a deep learning model for ultrasound image data is constructed, which uses a weight initialization model of a pre-trained VGG16 network, and adjusts the last Softmax layer of the VGG16 network according to a classification task; finally, a breast cancer risk stratification model based on multimodality is constructed based on the deep learning model for cytological diagnosis data and the deep learning model for ultrasound image data, and the breast cancer risk stratification model based on multimodality is trained using the training set of cytological diagnosis data and the training set of ultrasound image data to obtain a trained model. The method of the present invention uses the adjusted VGG16 network for the image modality and uses a convolutional neural network composed of multiple hidden layers for the text modality. Not only text features but also image features can be learned. Two feature vectors are fused to form a feature pool, and finally a predicted classification result can be obtained, which can greatly improve the accuracy of the prediction. Moreover, only a deep learning neural network model module is needed for the learning of the text modality, and no other modules are required, and no nomogram needs to be constructed. The classification result can be directly obtained, thereby improving the efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0038] Figure 1 A flowchart of a method for constructing a multimodal breast cancer risk stratification model provided in an embodiment of the present invention;
[0039] Figure 2 is a structural diagram of a multimodal breast cancer risk stratification model constructed in an embodiment of the present invention;
[0040] Figure 3 This is a flowchart of breast cancer risk stratification prediction based on multimodality in an embodiment of the present invention. DETAILED DESCRIPTION
[0041] Deep learning is a branch of machine learning, and its model is a multi-layer network structure based on neural networks. In recent years, with the improvement of computer computing power and the rapid development of storage space, deep learning has been greatly developed and quickly occupied the top of the rankings of many computer competitions. Multimodal deep learning is a type of deep learning that combines information from multiple modalities such as text, images, audio and video to make more accurate and comprehensive predictions. It involves training a deep neural network on data containing multiple types of information and using the network to make predictions based on this combined data. Multimodal deep learning has many applications, including image captioning, speech recognition, natural language processing, and self-driving cars. By combining information from multiple modalities, multimodal deep learning can improve the accuracy and robustness of the model.
[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0043] Embodiment 1
[0044] The present invention discloses a method for constructing a multimodal breast cancer risk stratification model, see Figure 1 , the method comprising:
[0045] S1: acquiring multimodal data, wherein the multimodal data includes cytological diagnosis data and ultrasound imaging data;
[0046] S2: Preprocess the cytological diagnosis data and ultrasound image data respectively, and divide them into training set and validation set;
[0047] S3: Build a deep learning model for cytological diagnosis data. The deep learning model for cytological diagnosis data is a convolutional neural network composed of multiple hidden layers.
[0048] S4: Build a deep learning model for ultrasound image data. The deep learning model for ultrasound image data uses the weights of the pre-trained VGG16 network to initialize the model, and adjust the last Softmax layer of the VGG16 network according to the classification task;
[0049] S5: Based on the deep learning model of cytological diagnosis data and the deep learning model of ultrasound imaging data, a multimodal breast cancer risk stratification model is constructed, and the multimodal breast cancer risk stratification model is trained using the training set of cytological diagnosis data and the training set of ultrasound imaging data to obtain a trained model.
[0050] Specifically, S1 is the acquisition of multimodal data. When it is implemented, it can be done by collecting breast pathology reports and establishing a text database, and by collecting breast ultrasound image data and establishing an image database.
[0051] S2 is the preprocessing of data, which is mainly to facilitate feature extraction by the network.
[0052] S3 built a deep learning model for cytological diagnosis data to process text modal data, S4 built a deep learning model for ultrasound imaging data to process image modal data, and S5 combined the two models to build a multi-modal breast cancer risk stratification model.
[0053] In one embodiment, the cytological diagnosis data and the ultrasound image data are preprocessed separately, including:
[0054] Perform case conversion, non-ASCII character deletion, punctuation replacement, and word segmentation on cytological diagnosis data;
[0055] Scale, normalize, and annotate images in ultrasound imaging data.
[0056] In the specific implementation process, the preprocessing of cytological diagnosis data includes:
[0057] a. Convert uppercase numbers to lowercase numbers, uppercase letters to lowercase letters, and delete non-ASCII characters;
[0058] b. Replace all punctuation marks with spaces;
[0059] c. Use jieba for word segmentation;
[0060] d. Randomize the dataset and set 60% of the dataset as the training set; the remaining 40% as the validation set.
[0061] The preprocessing of ultrasound image data includes:
[0062] a. Scale the pixels of the image to between 0 and 1, and then normalize each channel according to the dataset;
[0063] b. Data annotation: use annotation tools to annotate breast nodules;
[0064] c. Randomize the dataset and set 60% of the dataset as the training set; set the remaining 40% as the validation set.
[0065] In one embodiment, the deep learning model of cytological diagnosis data includes multiple convolutional layers, multiple pooling layers, fully connected layers and output layers, wherein the input cytological diagnosis data is subjected to a series of convolutions and pooling to obtain a feature vector of a fixed size, and then the feature vector of the text data is obtained through one or more fully connected layers and output layers.
[0066] In the specific implementation process, the number of each layer can be set according to the actual situation. In this embodiment, a convolutional neural network consisting of 6 hidden layers is created. After the text is input, a fixed-size feature vector is obtained through a series of convolution and pooling operations. Here we apply the average pooling layer. Then, these fixed-size feature vectors pass through one or more fully connected hidden layers (fully connected layers), and finally pass through the output layer to obtain the feature vector of the text data. Among them, ReLU or Softmax is used as the activation function in the convolution layer, the fully connected layer and the output layer.
[0067] In addition, the training of deep learning models for cytological diagnosis data is also included. In specific implementation, the deep learning model for cytological diagnosis data is trained using the Ada optimizer, using the same pooling length as the filter window size, and using batch normalization. When optimizing the loss, the learning rate is set to 0.005, the dropout ratio is set to 0.5, and the maximum number of epochs is set to 100. 100 and 200 filters are used, respectively, with corresponding window sizes of 2 and 4. The same pooling length as the filter window size is used, and batch normalization is used.
[0068] In one embodiment, the last Softmax layer of the VGG16 network is adjusted according to the classification task, including:
[0069] The output size of the last Softmax layer of the VGG16 network is adjusted to the number of categories for breast cancer risk stratification.
[0070] Specifically, for the image modality, transfer learning is used. The existing weights of the pre-trained model are used. Specifically, the weights of the pre-trained VGG16 network are used to initialize the model. Finally, the output size of the last Softmax layer of the VGG16 network is adjusted to the number of categories of breast cancer risk stratification.
[0071] In addition, it also includes the training of deep learning models for ultrasound imaging data. The Adam optimizer is used to train the image model. The learning rate adopts a dynamic adjustment strategy to balance the convergence speed and avoid the parameters from oscillating back and forth on both sides of the optimal solution. In specific implementation, the learning rate adopts a dynamic adjustment strategy. The initial learning rate is 0.0001 to improve the convergence speed; when the accuracy stops improving within 100 epochs, the learning rate is reduced by 10 times to avoid the parameters from oscillating back and forth on both sides of the optimal solution. The maximum number of epochs is set to 1000.
[0072] In one embodiment, a multimodal breast cancer risk stratification model is constructed based on a deep learning model of cytological diagnosis data and a deep learning model of ultrasound imaging data, including:
[0073] For the image modality, an adjusted VGG16 network is used, and for the text modality, a convolutional neural network is used;
[0074] Before the image modality and the text modality are shared, one or more hidden layers are set after the adjusted VGG16 network and the convolutional neural network;
[0075] After connecting the two modes, a hidden layer is added, and finally a Softmax layer is set.
[0076] See also Figure 2 , is a structural diagram of a multimodal breast cancer risk stratification model constructed in an embodiment of the present invention.
[0077] Specifically, there are multiple hidden layers on each side of the VGG16 network and the convolutional neural network before forming the shared features from the two modalities.
[0078] After the two modes are connected, a feature pool is formed, followed by a hidden layer and a softmax layer. Feature fusion is performed in the feature pool as the input of the hidden layer; the hidden layer is used for nonlinear processing of input data and feature extraction; the role of softmax is to perform normalized multi-classification, specifically, it converts the original scores of the previous hidden layer into a probability distribution with a non-zero sum of 1, so as to perform classification.
[0079] In one embodiment, before training the multimodal breast cancer risk stratification model using the cytological diagnosis data training set and the ultrasound image data training set, the method further includes:
[0080] Zero padding was performed on the texts in the cytology diagnosis data training set to obtain texts of equal length;
[0081] Convert texts of equal length into a token matrix, where each row in the matrix represents a token in the cytology diagnosis data extracted using the pre-trained word2vec model.
[0082] Among them, the word2vec model (a group of related models used to generate word vectors) is trained using the CBOW (continuous bag of words model) method.
[0083] In the specific implementation process, when training the multimodal breast cancer risk stratification model, the Adam optimizer with a mini-batch size of 64 is used to train the model. ReLU or Softmax is used as the activation function, and an early stopping condition is used.
[0084] During the training process, the processed cytological diagnosis data is input into the convolutional neural network, which outputs a set of feature vectors of the cytological diagnosis data, and the ultrasound image data is input into the adjusted VGG16 network, which outputs a set of feature vectors of medical imaging data. The two sets of feature vectors are used as input parameters of the multimodal deep learning model, and are trained according to the value of the loss function to obtain the final classification results.
[0085] Different from the traditional training method based on breast medical imaging data, the present invention uses multimodal data based on breast medical imaging and cytological diagnosis data for training, thereby increasing the comprehensiveness of the data and further improving the prediction accuracy of the model. In actual application, it can reduce the necessity of patients undergoing puncture examinations and surgical operations due to unclear imaging examinations, reduce the economic and psychological burden of patients, and improve patient satisfaction.
[0086] For ultrasound image data, the VGG16 network architecture is used; for cytological diagnosis data, the word2vec model is used for preprocessing, and the convolutional neural network architecture is used; finally, the two models are shared, and the weights of the two models are adjusted according to actual needs. Specifically, the weights of the two model features are adjusted according to the loss function in the training process.
[0087] Embodiment 2
[0088] Based on the same inventive concept, this embodiment discloses a device for constructing a multimodal breast cancer risk stratification model, comprising:
[0089] A multimodal data acquisition module, used to acquire multimodal data, wherein the multimodal data includes cytological diagnosis data and ultrasound image data;
[0090] A data preprocessing module is used to preprocess the cytological diagnosis data and ultrasound image data respectively, and divide them into a training set and a validation set;
[0091] Convolutional neural network building module, used to build a deep learning model for cytological diagnosis data. The deep learning model for cytological diagnosis data is a convolutional neural network composed of multiple hidden layers;
[0092] Ultrasound image data deep learning model building module, used to build a deep learning model for ultrasound image data. The ultrasound image data deep learning model uses the weight initialization model of the pre-trained VGG16 network and adjusts the last Softmax layer of the VGG16 network according to the classification task;
[0093] The multimodal breast cancer risk stratification model construction module is used to build a multimodal breast cancer risk stratification model based on the deep learning model of cytological diagnosis data and the deep learning model of ultrasound imaging data, and use the training set of cytological diagnosis data and the training set of ultrasound imaging data to train the multimodal breast cancer risk stratification model to obtain a trained model.
[0094] Since the device introduced in the second embodiment of the present invention is a device used to implement the method for constructing a breast cancer risk stratification model based on multimodality in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the device, so it is not repeated here. All devices used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.
[0095] Embodiment 3
[0096] This embodiment provides a multi-modality-based breast cancer risk stratification prediction method, including:
[0097] The cytological diagnosis data and ultrasound image data to be processed are pre-processed and then input into the trained model of Example 1 to obtain the breast cancer risk stratification prediction result.
[0098] For details, please see Figure 3 , is a flowchart of breast cancer risk stratification prediction based on multimodality in an embodiment of the present invention.
[0099] Embodiment 4
[0100] Based on the same inventive concept, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in the first embodiment is implemented.
[0101] Since the computer-readable storage medium introduced in the fourth embodiment of the present invention is a computer-readable storage medium used to implement the method for constructing a multimodal breast cancer risk stratification model in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, so it is not repeated here. All computer-readable storage media used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.
[0102] Embodiment 5
[0103] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in Embodiment 1 when executing the program.
[0104] Since the computer device introduced in the fifth embodiment of the present invention is a computer device used to implement the method for constructing a breast cancer risk stratification model based on multimodality in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer device, so it is not repeated here. All computer devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.
[0105] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0106] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0107] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic creative concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for constructing a multimodal breast cancer risk stratification model. It is characterized in that include: Acquiring multimodal data, wherein the multimodal data includes cytological diagnosis data and ultrasound imaging data; The cytological diagnosis data and ultrasound image data were preprocessed and divided into training set and validation set respectively; Build a deep learning model for cytological diagnosis data. The deep learning model for cytological diagnosis data is a convolutional neural network composed of multiple hidden layers. Build a deep learning model for ultrasound image data. The deep learning model for ultrasound image data uses the weight initialization model of the pre-trained VGG16 network and adjusts the last Softmax layer of the VGG16 network according to the classification task; Based on the deep learning model of cytological diagnosis data and the deep learning model of ultrasound imaging data, a multimodal breast cancer risk stratification model was constructed, and the multimodal breast cancer risk stratification model was trained using the training sets of cytological diagnosis data and ultrasound imaging data to obtain a trained model.
2. The method for constructing a multimodal breast cancer risk stratification model according to claim 1, It is characterized in that The cytological diagnosis data and ultrasound image data are preprocessed separately, including: Perform case conversion, non-ASCII character deletion, punctuation replacement, and word segmentation on cytological diagnosis data; Scale, normalize, and annotate images in ultrasound imaging data.
3. The method for constructing a multimodal breast cancer risk stratification model according to claim 1, It is characterized in that The deep learning model of cytological diagnosis data includes multiple convolutional layers, multiple pooling layers, fully connected layers and output layers. The input cytological diagnosis data undergoes a series of convolutions and pooling to obtain a feature vector of a fixed size, and then passes through one or more fully connected layers and output layers to obtain the feature vector of the text data.
4. The method for constructing a multimodal breast cancer risk stratification model according to claim 1, It is characterized in that Adjust the final Softmax layer of the VGG16 network according to the classification task, including: The output size of the last Softmax layer of the VGG16 network is adjusted to the number of categories for breast cancer risk stratification.
5. The method for constructing a multimodal breast cancer risk stratification model according to claim 1, It is characterized in that Based on the deep learning model of cytological diagnosis data and ultrasound imaging data, a multimodal breast cancer risk stratification model is constructed, including: For the image modality, an adjusted VGG16 network is used, and for the text modality, a convolutional neural network is used; Before the image modality and the text modality are shared, one or more hidden layers are set after the adjusted VGG16 network and the convolutional neural network; After connecting the two modes, a hidden layer is added, and finally a Softmax layer is set.
6. The method for constructing a multimodal breast cancer risk stratification model according to claim 1, It is characterized in that Before training the multimodal breast cancer risk stratification model using the cytological diagnosis data training set and the ultrasound image data training set, the method further includes: Zero padding was performed on the texts in the cytology diagnosis data training set to obtain texts of equal length; Convert texts of equal length into a token matrix, where each row in the matrix represents a token in the cytology diagnosis data extracted using the pre-trained word2vec model.
7. A device for constructing a multimodal breast cancer risk stratification model. It is characterized in that include: A multimodal data acquisition module, used to acquire multimodal data, wherein the multimodal data includes cytological diagnosis data and ultrasound imaging data; A data preprocessing module is used to preprocess the cytological diagnosis data and ultrasound image data respectively, and divide them into a training set and a validation set; Convolutional neural network building module, used to build a deep learning model for cytological diagnosis data. The deep learning model for cytological diagnosis data is a convolutional neural network composed of multiple hidden layers; Ultrasound image data deep learning model building module, used to build a deep learning model for ultrasound image data. The ultrasound image data deep learning model uses the weight initialization model of the pre-trained VGG16 network and adjusts the last Softmax layer of the VGG16 network according to the classification task; The multimodal breast cancer risk stratification model construction module is used to build a multimodal breast cancer risk stratification model based on the deep learning model of cytological diagnosis data and the deep learning model of ultrasound imaging data, and use the training set of cytological diagnosis data and the training set of ultrasound imaging data to train the multimodal breast cancer risk stratification model to obtain a trained model.
8. A multimodal breast cancer risk stratification prediction method. It is characterized in that include: The cytological diagnosis data and ultrasound image data to be processed are pre-processed and then input into the trained model as described in any one of claims 1 to 6 to obtain the breast cancer risk stratification prediction result.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the method for constructing a multimodal breast cancer risk stratification model as described in any one of claims 1 to 6 is implemented.
10. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the method for constructing a multimodal breast cancer risk stratification model as described in any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Method and system for identifying plasmocyte mastitis in inflammatory mass stage
CN119851010A
