Breast cancer detection method and system based on multi-modal input

Abstract

The invention relates to the technical field of breast cancer detection based on multi-modal input, in particular to a breast cancer detection method and system based on multi-modal input. The method comprises the following steps: acquiring breast tumor original clinical data, including breast X-ray image data, medical history text data and clinical index data; preprocessing the obtained original clinical data of the breast tumor; performing feature extraction on the preprocessed breast tumor original clinical data; performing feature fusion on the extracted features to form a fusion feature vector; and performing breast cancer category classification by using the fusion feature vector. According to the application, the related information of the breast cancer is more comprehensively captured by fusing three types of multi-modal input data of the past medical history, the mammary gland X-ray and the clinical data of the patient. According to the multi-modal fusion method, the advantages of different types of data can be fully utilized, richer information is provided, and therefore the accuracy and reliability of detection are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal input breast cancer detection, and in particular to a multimodal input-based breast cancer detection method and system. Background Art

[0002] Breast cancer is the most common malignant tumor in women worldwide, with a mortality rate second only to lung cancer, colorectal cancer, and liver cancer. It poses a significant threat to women's health. Early and accurate diagnosis of breast cancer is crucial for improving patient outcomes. Early diagnosis, targeted treatment, and ongoing monitoring significantly increase cure and survival rates.

[0003] Current AI-based early breast cancer screening methods rely on single-source data, such as imaging, often ignoring key relevant information such as medical history and clinical data. Traditional deep learning algorithms are often used to train and evaluate the fused features of multimodal data, primarily as follows.

[0004] Disease detection based on imaging data primarily uses mammograms, breast ultrasound images, or breast MRI images as input. For two-dimensional images, traditional image processing techniques are employed, such as using convolutional layers to extract deep semantic features, followed by classification using fully connected layers to learn the morphological and texture characteristics of masses, calcifications, and other features. Some methods also employ image segmentation to extract features of tumor regions. Based on the extracted image features, models such as ResNet and DenseNet are constructed to detect breast cancer. For breast MRI images, network structures such as 3D U-Net are used to process the three-dimensional information of breast MRI images. Some research has also introduced attention mechanisms into CNNs, enabling the model to focus more on important areas in the image, thereby improving the detection and diagnosis capabilities of breast cancer lesion features.

[0005] Patients' clinical data and medical history are also crucial for breast cancer diagnosis. Traditional statistical models and machine learning methods, such as logistic regression, support vector machines, and random forests, are often used to analyze clinical data and predict breast cancer risk. These models typically use features such as a patient's age, family history, past medical history, and breast exam results as input, and build predictive models by learning the relationship between these features and breast cancer risk. However, these traditional models often rely on manually selected and extracted features and may not fully exploit the complex patterns and nonlinear relationships in the data.

[0006] (1) Disease detection based on imaging data is highly dependent on the results of image examinations, but the interpretation of the results often depends on the experience of the radiologist. There may be differences in diagnostic results between different doctors. Inexperienced doctors may miss some early signs of lesions, resulting in false negative results; while overly sensitive interpretations may increase false positive results.

[0007] (2) Traditional statistical models and machine learning methods usually require manual feature selection and extraction when processing breast cancer clinical data. However, these models have difficulty fully utilizing the complex patterns and nonlinear relationships in the data and may not be able to fully tap the potential information in the data, thus limiting the accuracy and reliability of predictions.

[0008] (3) Fusion of multimodal imaging data typically requires the construction of complex deep learning model architectures with large parameter counts to process the data characteristics of different modalities. These models are difficult to train and optimize, and are prone to overfitting. Furthermore, complex model architectures require extensive computing resources for training and inference, increasing hardware costs and potentially limiting their practical application. Summary of the Invention

[0009] In order to solve the above-mentioned problems, the present invention provides a breast cancer detection method and system based on multimodal input.

[0010] In a first aspect, the present invention provides a breast cancer detection method based on multimodal input, which adopts the following technical solutions: A breast cancer detection method based on multimodal input, comprising: Obtain original clinical data on breast tumors, including mammary X-ray image data, medical history text data, and clinical indicator data; Preprocess the acquired original clinical data of breast tumors; Feature extraction is performed on the pre-processed raw clinical data of breast tumors; Perform feature fusion on the extracted features to form a fused feature vector; Breast cancer classification using fused feature vectors.

[0011] Furthermore, the obtained original clinical data of breast tumors are preprocessed, including grayscale normalization, and the pixel values ​​of the image are normalized according to the formula Normalized to the range [0, 1], where is the pixel value of the original image, and The maximum and minimum pixel values ​​of the image are respectively used, and then the median filter is applied to remove salt and pepper noise and retain the image details. The adaptive histogram equalization algorithm is used to enhance the image contrast, and the clip limit parameter of CLAHE is set to limit over-enhancement and highlight the tiny lesions in the breast tissue.

[0012] Furthermore, the preprocessing of the obtained original clinical data of breast tumors also includes preprocessing the obtained original clinical data of breast tumors, converting the clinical numerical data of patients admitted to the hospital into binary features based on dummy coding One-Hot Encoding, aligning the unit information of each data, and finally using a histogram to detect whether there are outliers, eliminating samples with outliers, and filling missing values ​​with normal values ​​of indicators.

[0013] Furthermore, the feature extraction of the pre-processed original clinical data of breast tumors includes feature extraction of breast X-ray images using a DenseNet-121 network, wherein, in the first dense block, the image is operated by multiple composite functions, each composite function extracts and outputs a new feature map, and the features extracted by the current layer and the feature information of all previous layers are integrated to form a rich feature set; the feature map is used as the input of the transition block, normalized and activated, and then the number of channels is compressed through a 1×1 convolution layer to reduce the amount of data; the feature map is then downsampled using a 2×2 average pooling layer; after processing by four dense blocks and three transition blocks, the original image is converted into an abstract feature of 7*7*512, and then passed through a BN module, a ReLu activation function, a pooling layer and a 7×7 global pooling layer with a step size of 1 and ceil_mode=True to obtain breast X-ray dimension information with a dimension of 1024.

[0014] Furthermore, the feature extraction of the pre-processed original clinical data of breast tumors also includes mapping the medical history text data to a vector space using an improved BERT-base-chinese model, wherein data enhancement includes semantic expansion and synonym replacement. The medical dictionary is used as the gold standard, and 15% of non-key medical terms are randomly selected in the medical history text and replaced with synonyms; then, for the medical terms in 10% of the text, relevant hypernyms are introduced for expansion; the enhanced data and the original data are processed in a unified format, and the model is segmented based on the WordPiece algorithm, and the patient's medical history is divided into vocabulary units in the vocabulary, while processing the situation of unregistered disease name words that may appear in the medical text; the processed medical history text is input into the BERT-base-chinese model to extract semantic features.

[0015] Furthermore, the feature fusion of the extracted features to form a fused feature vector includes sequentially aligning and fusing the extracted image features, text features and processed numerical data to form a fused feature vector; wherein, for the image features, the fully connected layer Adjust its dimension to n=768, where is the weight matrix, is the image feature dimension, is the bias vector; the numerical features are transformed linearly Its dimension is also expanded to n dimensions, where is the weight matrix, is the patient's numerical feature dimension, is the bias vector; the image feature vector , text feature vector and numerical eigenvectors based on The stitching matrix is ​​obtained by stitching .

[0016] Furthermore, the feature fusion of the extracted features to form a fused feature vector also includes completing feature fusion based on the self-attention mechanism, dynamically determining the correlation between features by calculating the dot product between the query, key and value, and assigning different weights to different features, thereby achieving weighted fusion of features, wherein the input feature matrix is ​​linearly transformed to generate the query matrix I, the key-value matrix B and the value matrix P, and the dot product is performed using the query matrix and the value matrix to obtain the attention score matrix ; Apply the Softmax function to the SA matrix to convert the attention score into a probability distribution and obtain the attention weight matrix , based on Generate weighted feature matrix ;right Splicing get , based on the formula , and the dimensions are Fusion feature vector ,in is the weight matrix, is the bias vector.

[0017] Furthermore, the method of using fusion feature vectors to classify breast cancer types includes taking fusion features as model input and using a fully connected multilayer perceptron as the basic architecture for breast cancer discrimination tasks; the first layer is the input layer, and the number of neurons in the layer is the same as the fusion feature vector. Dimension Same, used to receive the input feature vector, where two hidden layers are set; the first hidden layer is set neurons, using the ReLU activation function, which is used to alleviate the gradient disappearance problem. The second hidden layer is set neurons, using the ReLU activation function; adding a Dropout layer after each hidden layer, setting the dropout probability to 0.5; the final output layer contains only one neuron, using the Sigmoid activation function The output value is mapped to the interval (0, 1). When the output value is greater than 0.5, the detection result is breast cancer, otherwise it is not breast cancer.

[0018] Furthermore, the method of using the fused feature vector to classify breast cancer types also includes introducing a regularized cross entropy loss function. Compile, where is the true label, is the model prediction value, is the regularization coefficient, It is the square term of the model weight. If a certain weight is too large, then its square term will lead to an increase in the overall loss. The optimization process tends to reduce the value of the weight. Adam is used for optimization, the initial learning rate is set to 0.001, and an attenuation mechanism is introduced.

[0019] In a second aspect, a breast cancer detection system based on multimodal input includes: The data acquisition module is configured to acquire original clinical data of breast tumors, including breast X-ray image data, medical history text data, and clinical indicator data; The preprocessing module is configured to preprocess the acquired raw clinical data of breast tumors; The feature extraction module is configured to extract features from the preprocessed raw clinical data of breast tumors; The feature fusion module is configured to perform feature fusion on the extracted features to form a fused feature vector; The classification module is configured to classify breast cancer types using the fused feature vector.

[0020] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device for the method for detecting breast cancer based on multimodal input.

[0021] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is configured to implement various instructions; and the computer-readable storage medium is configured to store a plurality of instructions, wherein the instructions are suitable for being loaded by the processor and executed by the method for detecting breast cancer based on multimodal input.

[0022] In summary, the present invention has the following beneficial technical effects: 1. This application fuses three types of multimodal input data: a patient's medical history, mammograms, and clinical data. This method captures more comprehensive information about breast cancer. This multimodal fusion approach leverages the strengths of different data types, providing richer information and improving detection accuracy and reliability. Compared to existing technologies that rely on a single data modality, this application can effectively reduce false positive and false negative rates.

[0023] 2. Combining the characteristics of medical history text and mammograms, and leveraging the strengths of DenseNet, we learned more representative and discriminative image features from mammograms. Furthermore, we designed a text enrichment mechanism suitable for breast cancer, leveraging the Chinese semantic understanding capabilities of the BERT-base-Chinese model to deeply mine key information from medical history text. Compared to traditional manual feature extraction methods or simple deep learning models, this application can better capture complex patterns and nonlinear relationships in the data, improving the expressive power of features.

[0024] 3. Based on the attention mechanism and MLP network, a feature fusion method is designed to dynamically learn the correlation and importance between different modal data. Compared with existing multimodal fusion methods, the feature fusion method of this application can better utilize the complementarity of each modal data and improve the model's comprehensive analysis ability of multimodal data. At the same time, the feature fusion and classification model based on the attention mechanism and MLP network has high computational efficiency and can complete the prediction of new samples in a shorter time. Compared with some models in the prior art that have high computational complexity and long time consumption, this model has moderate parameters and is more suitable for medical scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 4 is a schematic diagram of a breast cancer detection method based on multimodal input according to Example 1 of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the accompanying drawings.

[0027] Example 1 Reference Figure 1 , a breast cancer detection method based on multimodal input in this embodiment includes: Obtain original clinical data on breast tumors, including mammary X-ray image data, medical history text data, and clinical indicator data; Preprocess the acquired original clinical data of breast tumors; Feature extraction is performed on the pre-processed raw clinical data of breast tumors; Perform feature fusion on the extracted features to form a fused feature vector; Breast cancer classification using fused feature vectors.

[0028] Specifically: 1. Data processing, Different types of raw data (mammographic images, patient medical history text data, and patient clinical indicator numerical data) for benign and malignant cases are standardized and preprocessed. Benign patients are labeled as 0 and malignant patients are labeled as 1. The following are the detailed steps of the data processing stage: (1) Mammographic images, Mammographic X-ray images were collected in the hospital's picture archiving and communication system (PACS) and converted from DICOM format images to PNG format. Image normalization, image denoising, contrast enhancement, and data augmentation were performed sequentially to improve image quality and enhance feature expression.

[0029] First, grayscale normalization is performed to normalize the pixel values ​​of the image according to the formula Normalized to the range [0, 1], where is the pixel value of the original image, and The maximum and minimum pixel values ​​of the image are used, respectively. A median filter is then applied to remove salt-and-pepper noise, preserving image detail. The adaptive histogram equalization (CLAHE) algorithm is used to enhance image contrast. The clip limit parameter of CLAHE is set to limit over-enhancement and highlight subtle lesions in breast tissue. Based on the actual image size, the image is resized to a fixed size of 224*224 pixels to adapt to the input size while preserving key image information. Finally, operations such as rotation, flipping, scaling, and translation are used to increase the diversity of the training data and improve the model's generalization ability.

[0030] (2) Patient's medical history, The text data includes three types of medical history data: past medical history, family medical history, and treatment history. The text data is cleaned to remove special characters and punctuation marks, retaining only Chinese characters, letters, time, and numbers. If the text contains traditional Chinese characters, it is converted to simplified Chinese.

[0031] (3) Patient clinical data, Numerical clinical data from this patient admission included over ten clinical data items, including age, weight, menopausal status, age at menarche, childbearing age, and breast density; vital signs such as heart rate, respiratory rate, and blood pressure; and laboratory test results such as white blood cell count and platelet count from routine blood tests. Categorical variables such as menopausal status were converted to binary features using one-hot encoding, and the unit information of each data point was aligned. Finally, a histogram was used to detect outliers, and samples with outliers were removed. Missing values ​​were filled with the normal value of the indicator.

[0032] 2. Feature extraction, (1) Mammographic image feature extraction, Taking into account the size and complexity of mammary X-ray images and the balance of computing resources, this solution selects DenseNet-121 as the basic architecture for feature extraction. DenseNet strengthens feature propagation. During the training process, this gradient flow helps to extract deep features from mammary X-rays. At the same time, the feature maps of each layer can be shared by all subsequent layers, which can increase its generalization ability and is more suitable for relatively small datasets such as mammary X-ray images, improving the stability and reliability of the model. DenseNet consists of multiple dense blocks and transition blocks. In the feature extraction of mammary X-ray images, this dense connection can more efficiently utilize the edges, textures and other features extracted by the previous layers, which helps to retain the spatial structure information of the image, allowing the network to learn detailed information more comprehensively and adapt to lesions of different scales.

[0033] Each dense block consists of multiple composite functions, including a batch normalization module, a ReLU activation function, and a 3x3 convolutional layer. Within a dense block, the output of each composite function is used as the input for all subsequent composite functions, achieving dense feature connections. Each transition block structure consists of a batch normalization module, a ReLU activation function, a 1x1 convolutional layer, and a 2x2 average pooling layer, compressing and reducing the size of the feature map to obtain higher-level semantic information. The detailed feature extraction scheme is described below: In the first dense block, the image is processed through multiple composite functions, each of which extracts and outputs a new feature map. This composite function combines the features extracted by the current layer with the features from all previous layers, forming a rich feature set. The feature map serves as input to the transition block, undergoing normalization and activation. It then passes through a 1×1 convolutional layer to reduce the number of channels and data size. A 2×2 average pooling layer then downsamples the feature map. After processing by the dense and transition blocks, the feature map contains higher-level feature representations, better capturing the overall information in the mammographic image. The feature map output by each layer is then fed into subsequent dense and transition blocks for further processing, continuously extracting and fusing deeper features. After processing through four dense blocks and three transition blocks, the original image is converted into a 7×7×512 abstract feature map. This feature map then passes through a batch normalization module, a Reinforced Lu (ReLU) activation function, a pooling layer, and a 7×7 global pooling layer with a stride of 1 and ceil_mode=True to obtain a mammographic dimension of 1024.

[0034] Among them, the preprocessed image is converted into a suitable input for the Densenet-121 network Tensor form. Indicates that the number of samples used in each iteration during training is 1, and c=1 indicates that the input image is a grayscale image with 1 channel. Represents height, Represents width. The tensor passes through the initial convolutional layer, undergoing a 7×7 convolution operation with a stride of 2, outputting a 112×112×64 feature map. The output feature map is resized to 56×56×64 using a 3×3 maximum pooling operation with a stride of 2. It then passes through the first Dense Block, which contains 6 Bottleneck layers. Each layer first performs a 1×1 convolution for dimensionality reduction, followed by a 3×3 convolution for feature extraction. The output feature maps are stacked in sequence, ultimately outputting a 56×56×256 feature map. It then passes through the first Transition Layer, using a 1×1 convolution to adjust the number of channels to 128, followed by a 2×2 average pooling operation with a stride of 2 to reduce the feature map size to 28×28×128. It then passes through the second Dense Block, which contains 12 Bottleneck layers, similar to the first Dense Block, outputting a 28×28×512 feature map. Next, the second Transition Layer uses a 1×1 convolution to adjust the number of channels to 256. Then, 2×2 average pooling with a stride of 2 is used to reduce the feature map size to 14×14×256. The third Dense Block, consisting of 24 Bottleneck layers, outputs a 14×14×1024 feature map. This output feature map is then passed to the third Transition Layer, where a 1×1 convolution is used to adjust the number of channels to 512. Then, 2×2 average pooling with a stride of 2 is used to reduce the feature map size to 7×7×512. The fourth Dense Block, consisting of 16 Bottleneck layers, outputs a 7×7×1024 feature map. Finally, global average pooling is performed on the 7×7×1024 feature map, resulting in a 1×1×1024 feature vector, or a 1024-dimensional feature vector.

[0035] (2) Extraction of past medical history features, To adapt to the patient medical history task and capture its complex medical semantic information, this solution uses a data enhancement strategy to fine-tune the pre-trained BERT-base-chinese model and map the patient's medical history into a vector space. The details are as follows: The main methods of data augmentation include semantic expansion and synonym replacement. Using medical dictionaries as the gold standard, we randomly selected 15% of non-critical medical terms in the medical history text and replaced them with synonyms. This is shown in Table 1 below: Table 1 Data augmentation For some medical terms in 10% of the text, relevant hypernym expansions are introduced. For example, "hypertension" is expanded to "hypertension, cardiovascular diseases", and "chemotherapy" is expanded to "chemotherapy, anti-tumor treatment", etc.

[0036] Process the enhanced data and the original data in a unified format. Perform word segmentation on the model based on the WordPiece algorithm, segment the patient's medical history into lexical units in the vocabulary, and simultaneously handle the situation of out-of-vocabulary disease name words that may appear in the medical text. For example, for the word "nodule", if it exists in the vocabulary, it will be segmented into one token; if it does not exist, it will be split into two sub-word units, "knot" and "section". Add special start symbol [CLS] and end symbol [SEP], pad or truncate the input sequence to make its length consistent, and perform encoding.

[0037] Initialize the BERT-base-chinese model, load the pre-trained parameters, select the cross-entropy loss function as the loss function, set Adam as the optimizer, set the initial learning rate to 0.001, and adopt a learning rate decay strategy. Input the processed medical history text into the BERT-base-chinese model to extract semantic features. Finally, perform average pooling on the outputs of all tokens to generate a text feature vector with a dimension of 768.

[0038] Among them, first, map the segmented lexical units to the corresponding word embedding vectors. After word segmentation, a series of tokens are obtained. For each token, obtain the corresponding word embedding matrix , where n is the size of the vocabulary and d = 768 is the embedding dimension. When the index corresponding to a word in the vocabulary is i, the corresponding word embedding vector is .

[0039] At the same time, generate paragraph embedding vectors and position embedding vectors. The former is used to distinguish different sentences. For the token in the j-th sentence, its paragraph embedding vector is . The latter represents position information, and the position embedding matrix , where, is the maximum sequence length supported by the model. When the length of the input sequence is x, the corresponding position indices are 0, 1,..., x - 1, x, and the corresponding position embedding vectors are . Add the three to obtain a tensor suitable for subsequent input .

[0040] The input tensor is passed through the 12 Transformer encoder layers of the BERT-base-chinese model. Each encoder layer consists of two main sub-layers: a multi-head self-attention mechanism and a feed-forward neural network. After each sub-layer, a residual connection and layer normalization operation are performed. The multi-head self-attention mechanism in the first encoder layer The query matrix I-tran, the key matrix B-tran and the value matrix P-tran are obtained through linear transformation.

[0041] , , , in 、 、 , where dB=64 represents the dimension of each attention head. Then calculate the attention score matrix of each attention head , where h=1, 2, ..., 12.

[0042] , in, and is the query matrix and key matrix corresponding to the current attention head, by and Then, we can get the output of each attention head by dividing it into columns: , Finally, the outputs of all attention heads are concatenated column by column and linearly transformed to obtain the final output of the multi-head self-attention mechanism:

[0043] in is the weight matrix of the linear transformation. The input and output of the multi-head attention mechanism are added together to obtain the residual connection result. After the result is layer normalized, the normalized result is Enter the feedforward neural network.

[0044] , in, and are the weights of the feedforward neural network, and is the bias term. The output of the feedforward neural network is added to the layer-normalized matrix, and then after layer normalization, the output of the first Transformer encoder layer is obtained.

[0045] The calculation process of the second to twelfth Transformer encoder layers is similar to the first encoder, except that the weight matrix used is different. After processing 12 Transformer encoder layers, the output matrix is ​​obtained , where each row vector corresponds to the semantic representation of a token in the input sequence. The vector corresponding to the first [CLS] tag in the output matrix is ​​taken as the feature vector of the entire text, with a dimension of 768.

[0046] 3. Feature fusion, The extracted image features, text features and processed numerical data are aligned, fused and spliced ​​in sequence to form a fused feature vector.

[0047] For image features, through the fully connected layer Adjust its dimension to n=768, where is the weight matrix, is the image feature dimension, is the bias vector. Numerical features are transformed linearly Its dimension is also expanded to n dimensions, where is the weight matrix, is the patient's numerical feature dimension, Is the bias vector. The image feature vector , text feature vector and numerical eigenvectors based on The stitching matrix is ​​obtained by stitching .

[0048] Feature fusion is accomplished based on the self-attention mechanism. The correlation between features is dynamically determined by calculating the dot product between the query, key, and value. Different weights are assigned to different features, thereby achieving weighted feature fusion. The input feature matrix is ​​linearly transformed to generate the query matrix I, the key-value matrix B, and the value matrix P. The generation formula is as follows: , , , in, is the weight matrix, and are the dimensions of the query matrix and the value matrix respectively. The query matrix and the value matrix are dot-producted as shown in the formula to obtain the attention score matrix .

[0049] , in, is a scaling factor to prevent the attention score from being too large. Applying the Softmax function to the SA matrix converts the attention score into a probability distribution and obtains the attention weight matrix , based on Generate weighted feature matrix .

[0050] , , right Splicing get , based on the formula , and the dimensions are Fusion feature vector ,in is the weight matrix, is the bias vector.

[0051] 4. Classification tasks, In the classification task, a multi-layer perceptron model is used. The input layer of the multi-layer perceptron model contains neurons, which receive input .

[0052] In the first hidden layer, the number of neurons is set to , the weight matrix of this layer is , the bias vector is , using the ReLU activation function, its formula is: , During the forward propagation process, the output of the first hidden layer is: , in, Represents the activation value vector of the first hidden layer. A Dropout layer with a dropout probability of 0.5 is added after the first hidden layer, and the output of some neurons is randomly set to 0 in each iteration.

[0053] The second hidden layer is set neurons, also using the ReLU activation function with a dropout rate of 0.5. The forward propagation calculation is:

[0054] in, represents the weight matrix of the second hidden layer, Represents the bias vector.

[0055] Finally, we enter the output layer, which contains 1 neuron and uses the Sigmoid activation function to map the output value to the (0, 1) interval. The Sigmoid activation function formula is as follows: , The output of the forward propagation is: , in, It indicates the probability of predicting breast cancer. If it is greater than 0.5, it is judged as breast cancer, otherwise it is judged as non-breast cancer.

[0056] The fusion feature is used as the model input, and a fully connected multi-layer perceptron is used as the basic architecture for breast cancer discrimination. The first layer is the input layer, and the number of neurons in it is the same as the fusion feature vector. Dimension The same is used to receive the input feature vector. In order to balance the need to learn complex descriptions and the situation where the network is too deep, which makes it difficult to train and overfit, the present invention sets two hidden layers. The first hidden layer is set The second hidden layer is set up with ReLU activation function, which can effectively alleviate the gradient vanishing problem and speed up the model training. neurons, also using the ReLU activation function. A Dropout layer is added after each hidden layer, with a dropout probability of 0.5. Finally, the output layer contains only one neuron, as this is a binary classification task.

[0057] 5. Model training and validation, This patent chooses to introduce a regularized cross entropy loss function Compile, where is the true label, is the model prediction value, is the regularization coefficient, The square term of the model weights. If a weight is too large, its square term will lead to an increase in overall loss, so the optimization process tends to reduce the weight value. Adam is used for optimization, with an initial learning rate of 0.001 and an attenuation mechanism.

[0058] An existing annotated dataset is divided into training, validation, and test sets in a 7:2:1 ratio. The training set is used to update and learn model parameters, while the validation set is used to monitor model performance during training. Training is terminated early when the loss on the validation set stops decreasing, and the model's learning rate, regularization, and hyperparameters are adjusted based on the performance on the validation set. The test set is used to ultimately evaluate the model's generalization and diagnostic performance. During each training cycle, the training set data is fed into the model in batches. The model output is calculated through forward propagation, and the loss function is used to calculate the loss value. The gradient is then calculated through backpropagation, and the model parameters are updated using an optimization algorithm. After sufficient training and optimization, accuracy, precision, recall, and F1 score are used as evaluation metrics to represent the model's performance on the test set.

[0059] Example 2 This embodiment provides a breast cancer detection system based on multimodal input, including: The data acquisition module is configured as follows: A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device for a method for detecting breast cancer based on multimodal input.

[0060] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executed by a breast cancer detection method based on multimodal input.

[0061] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A breast cancer detection method based on multimodal input, characterized in that: include: Obtain original clinical data on breast tumors, including mammary X-ray image data, medical history text data, and clinical indicator data; Preprocess the acquired original clinical data of breast tumors; Feature extraction is performed on the pre-processed raw clinical data of breast tumors; Perform feature fusion on the extracted features to form a fused feature vector; Breast cancer classification using fused feature vectors.

2. The breast cancer detection method based on multimodal input according to claim 1, characterized in that: The preprocessing of the original clinical data of breast tumors obtained includes grayscale normalization, and the pixel values ​​of the image are normalized according to the formula Normalized to the range [0, 1], where is the pixel value of the original image, and The maximum and minimum pixel values ​​of the image are respectively used, and then the median filter is applied to remove salt and pepper noise and retain the image details. The adaptive histogram equalization algorithm is used to enhance the image contrast, and the clip limit parameter of CLAHE is set to limit over-enhancement and highlight the tiny lesions in the breast tissue.

3. The breast cancer detection method based on multimodal input according to claim 2, characterized in that: The preprocessing of the acquired original clinical data of breast tumors also includes converting the clinical numerical data of patients admitted to the hospital into binary features based on dummy coding One-Hot Encoding, aligning the unit information of each data, and finally using a histogram to detect whether there are outliers, eliminating samples with outliers, and filling missing values ​​with normal values ​​of indicators.

4. The breast cancer detection method based on multimodal input according to claim 3, characterized in that: The method extracts features from the pre-processed original clinical data of breast tumors, including extracting features from breast X-ray images using a DenseNet-121 network, wherein in a first dense block, the image is operated by multiple composite functions, each composite function extracts and outputs a new feature map, and the features extracted by the current layer and the feature information of all previous layers are integrated to form a rich feature set; the feature map is used as input of a transition block, normalized and activated, and then the number of channels is compressed through a 1×1 convolution layer to reduce the amount of data; the feature map is downsampled using a 2×2 average pooling layer; after processing by four dense blocks and three transition blocks, the original image is converted into an abstract feature of 7*7*512, and then passed through a BN module, a ReLu activation function, a pooling layer, and a 7×7 global pooling layer with a step size of 1 and ceil_mode=True to obtain breast X-ray dimension information with a dimension of 1024.

5. The breast cancer detection method based on multimodal input according to claim 4, characterized in that: The feature extraction of the pre-processed original clinical data of breast tumors also includes mapping the medical history text data to a vector space using an improved BERT-base-chinese model, wherein data enhancement includes semantic expansion and synonym replacement, using a medical dictionary as a gold standard, randomly selecting 15% of non-key medical terms in the medical history text and replacing them with synonyms; then, for the medical terms in 10% of the text, introducing related hypernyms for expansion; processing the enhanced data and the original data in a unified format, performing word segmentation processing on the model based on the WordPiece algorithm, dividing the patient's medical history into vocabulary units in the vocabulary, and processing the situation where unregistered disease name words may appear in the medical text; the processed medical history text is input into the BERT-base-chinese model to extract semantic features.

6. The method for detecting breast cancer based on multimodal input according to claim 5, wherein: The feature fusion of the extracted features to form a fusion feature vector includes sequentially aligning and fusing the extracted image features, text features and processed numerical data to form a fusion feature vector; wherein, for the image features, the fully connected layer Adjust its dimension to n=768, where is the weight matrix, is the image feature dimension, is the bias vector; the numerical features are transformed linearly Its dimension is also expanded to n dimensions, where is the weight matrix, is the patient's numerical feature dimension, is the bias vector; the image feature vector , text feature vector and numerical eigenvectors based on The stitching matrix is ​​obtained by stitching .

7. The method for detecting breast cancer based on multimodal input according to claim 6, wherein: The feature fusion of the extracted features to form a fused feature vector also includes completing feature fusion based on the self-attention mechanism, dynamically determining the correlation between features by calculating the dot product between the query, key and value, and assigning different weights to different features, thereby achieving weighted fusion of features, wherein the input feature matrix is ​​linearly transformed to generate the query matrix I, the key value matrix B and the value matrix P, and the dot product is performed using the query matrix and the value matrix to obtain the attention score matrix ; Apply the Softmax function to the SA matrix to convert the attention score into a probability distribution and obtain the attention weight matrix , based on Generate weighted feature matrix ;right Splicing get , based on the formula , and the dimensions are Fusion feature vector ,in is the weight matrix, is the bias vector.

8. The method for detecting breast cancer based on multimodal input according to claim 7, wherein: The method of using fusion feature vector to classify breast cancer includes taking fusion feature as model input and using fully connected multi-layer perceptron as the basic architecture of breast cancer discrimination task; the first layer is input layer, and the number of neurons in it is the same as that of fusion feature vector. Dimension Same, used to receive the input feature vector, where two hidden layers are set; the first hidden layer is set neurons, using the ReLU activation function, which is used to alleviate the gradient disappearance problem. The second hidden layer is set neurons, using the ReLU activation function; adding a Dropout layer after each hidden layer, setting the dropout probability to 0.5; the final output layer contains only one neuron, using the Sigmoid activation function The output value is mapped to the interval (0, 1). When the output value is greater than 0.5, the detection result is breast cancer, otherwise it is not breast cancer.

9. The method for detecting breast cancer based on multimodal input according to claim 8, wherein: The method of using the fusion feature vector to classify breast cancer types also includes introducing a regularized cross entropy loss function. Compile, where is the true label, is the model prediction value, is the regularization coefficient, It is the square term of the model weight. If a certain weight is too large, then its square term will lead to an increase in the overall loss. The optimization process tends to reduce the value of the weight. Adam is used for optimization, the initial learning rate is set to 0.001, and an attenuation mechanism is introduced.

10. A breast cancer detection system based on multimodal input, characterized in that: include: The data acquisition module is configured to acquire original clinical data of breast tumors, including breast X-ray image data, medical history text data, and clinical indicator data; The preprocessing module is configured to preprocess the acquired raw clinical data of breast tumors; The feature extraction module is configured to extract features from the preprocessed raw clinical data of breast tumors; The feature fusion module is configured to perform feature fusion on the extracted features to form a fused feature vector; The classification module is configured to classify breast cancer types using the fused feature vector.

Images