A method for classifying benign and malignant breast lesions

Abstract

This invention provides a method for classifying benign and malignant breast lesions, comprising the following steps: Step 1, obtaining N sample pairs; Step 2, constructing a benign and malignant breast lesion classification model; Step 3, training the benign and malignant breast lesion classification model; Step 4, obtaining the benign and malignant breast lesion classification results. This invention can effectively improve the segmentation accuracy of breast lesions in radiological images, providing more accurate and reliable benign and malignant classification results, and has potential clinical application value.

Description

Technical Field

[0001] This invention relates to a method for classifying benign and malignant breast lesions, belonging to the field of image processing technology. Background Technology

[0002] Breast lesions can be classified as benign or malignant. Benign lesions mainly include fibroadenomas, cysts, and intraductal papillomas. They have clear and regular boundaries, minimal changes in breast structure, and generally do not develop into cancer, usually requiring only regular follow-up. Malignant lesions mainly include invasive ductal carcinoma and invasive lobular carcinoma. They have irregular edges such as indistinct spiculated or lobulated margins. Cancer cells in these lesions can easily spread from the breast to other parts of the body, requiring a combination of surgery, chemotherapy, radiotherapy, and targeted therapy.

[0003] In recent years, radiographic imaging techniques such as mammography, CT, and MRI have provided powerful tools for observing the morphology of breast lesions. Therefore, accurate classification of benign and malignant breast lesions based on radiographic imaging can provide scientific guidance for the selection of clinical intervention strategies, reduce the waste of medical resources, and thus improve the effectiveness of breast cancer diagnosis and treatment. However, automatic classification of benign and malignant breast lesions based on radiographic imaging still faces some challenges. On the one hand, due to the differences in the imaging principles of different imaging modalities, the same lesion exhibits significant feature differences on different images, making it difficult to fuse and unify the analysis of multimodal image features. On the other hand, because radiographic imaging can only present the macroscopic anatomical structure of lesions and lacks the microscopic structural features provided by histopathological examination, benign and malignant lesions still exhibit highly similar features on images, resulting in low classification sensitivity. Summary of the Invention

[0004] This invention provides a method for classifying benign and malignant breast lesions. Guided by text features, it can effectively improve the segmentation accuracy of breast lesions in radiological images and provide more accurate and reliable benign and malignant classification results.

[0005] To address the aforementioned technical problems, this invention provides a method for classifying benign and malignant breast lesions, comprising the following steps: Step 1, acquiring N sample pairs: each sample pair includes mammography X-ray images, CT images, MRI images, and pathological text; Step 2, constructing a classification model for benign and malignant breast lesions: the classification model includes a multimodal image feature fusion model, a text feature extraction model, a common semantic space, and a mask prediction model based on a stacked multi-layer bidirectional Transformer decoder; the multimodal image feature fusion model includes a first image feature extraction module, a second image feature extraction module, a third image feature extraction module, and an image feature fusion module; the text feature extraction model is a FastText word vector model, and the hidden layer of the FastText word vector model is a KAN neural network; the outputs of the first image feature extraction module, the second image feature extraction module, and the third image feature extraction module are all connected to the image feature fusion module, and the output of the image feature fusion module is connected to the mask. The output of the prediction model and the text feature extraction model are connected to the mask prediction model. A contrastive learning loss function based on the cosine similarity matrix of image features and text features is constructed in the common semantic space. The contrastive learning loss function is used to jointly train the multimodal image feature fusion model and the text feature extraction model. During the joint training of the multimodal image feature fusion model and the text feature extraction model, the image feature fusion module outputs image features to the common semantic space, and the FastText word vector model outputs text features to the common semantic space. Step 3: Train the benign and malignant breast lesion classification model: The contrastive learning method is used to jointly train the multimodal image feature fusion model and the text feature extraction model, and the supervised training method is used to train the mask prediction model. Step 4: Obtain the benign and malignant breast lesion classification results: The sample pairs of breast lesion benign and malignant classification, including mammography X-ray images, CT images, MRI images and pathological texts, are input into the trained benign and malignant breast lesion classification model to obtain the benign and malignant breast lesion classification results.

[0006] In one specific implementation, the bidirectional Transformer decoder includes a self-attention layer, a first cross-attention layer, a second cross-attention layer, and a multilayer perceptron B. The output of the self-attention layer is connected to the first cross-attention layer, the output of the first cross-attention layer is connected to the multilayer perceptron B, and the output of the multilayer perceptron B is connected to the second cross-attention layer. The multilayer perceptron B outputs updated text features, and the second cross-attention layer outputs updated image features.

[0007] In one specific implementation, the mask prediction model of the stacked multilayer bidirectional Transformer decoder further includes a transposed convolution module, a multilayer perceptron A, and a dynamic linear classification layer; the output of the text feature extraction model is connected to the self-attention layer of the first-layer bidirectional Transformer decoder, the output of the image feature fusion module is connected to the first cross-attention layer and the second cross-attention layer of the first-layer bidirectional Transformer decoder, the updated image features output from the second cross-attention layer of the last-layer bidirectional Transformer decoder are output to the transposed convolution module, the transposed convolution module downsamples them and outputs them to the multilayer perceptron A, the multilayer perceptron A processes them and outputs them to the dynamic linear classifier to obtain the classification result of benign or malignant lesions, the outputs of the second cross-attention layers of the first-layer and intermediate-layer bidirectional Transformer decoders are all connected to the first cross-attention layer and the second cross-attention layer of the corresponding next layer, and the outputs of the multilayer perceptron B of the first-layer and intermediate-layer bidirectional Transformer decoders are all connected to the self-attention layer of the corresponding next layer.

[0008] In one specific implementation, the KAN neural network of the FastText word vector model performs a non-linear mapping on the word vector x output from the input layer of the FastText word vector model to obtain... , The calculation formula is as follows: The KAN neural network of the FastText word vector model will generate The output is fed to the output layer of the FastText word vector model, where text features are extracted using Softmax; where b(x) is the SiLU activation function. spline(x) is a linear combination function of B-splines. w b w are the weighting coefficients of b(x). s B represents the weighting coefficients of spline(x). l (x) is the l-th B-spline function, c l The coefficients are those of the l-th B-spline function.

[0009] In one specific implementation, the first image feature extraction module, the second image feature extraction module, and the third image feature extraction module are all Hiera models.

[0010] In one specific implementation, the image feature fusion module is a graph convolutional neural network.

[0011] In one specific implementation, step 3, which employs a contrastive learning method to jointly train the multimodal image feature fusion model and the text feature extraction model, specifically involves: Step 3.1: Inputting N pairs of mammogram X-ray images into the first image feature extraction module, N pairs of CT images into the second image feature extraction module, and N pairs of MRI images into the third image feature extraction module. The first, second, and third image feature extraction modules, along with the image feature fusion module, work together to obtain N image features; Step 3.2: Inputting N pairs of pathological texts into the FastText word vector model to obtain N text features; Step 3.3: Inputting the N image features and N text features into a common semantic space, aligning the input N image features and N text features within the common semantic space; Step 3.4: Defining matching image features and text features as positive sample pairs, and non-matching image features and text features as negative sample pairs, calculating the cosine similarity matrix of the N image features and N text features, where the i-th image feature I... i and the j-th text feature J j Cosine similarity s between i,j It is expressed as follows: ,in, J is the temperature scaling parameter. k Let i be the k-th text feature, where i is a positive integer and k is a positive integer; Step 3.5: Use the contrastive learning loss function to jointly train the parameters of the multimodal image feature fusion model and the text feature extraction model through the error backpropagation algorithm, and set the loss value L of the contrastive learning loss function. cl The calculation formula is as follows: , where J i For the i-th text feature, I k Let k be the kth image feature.

[0012] In one specific implementation, step 4 specifically comprises: Step 4.1, inputting the mammography X-ray images, CT images, and MRI images of the sample pairs for which benign and malignant breast lesions need to be classified into benign and malignant forms into the trained multimodal image feature fusion model; the first image feature extraction module extracts multi-scale features from the mammography X-ray images and outputs them to the image feature fusion module; the second image feature extraction module extracts multi-scale features from the CT images and outputs them to the image feature fusion module; and the third image feature extraction module extracts multi-scale features from the MRI images and outputs them to the image feature fusion module; Step 4.2, image feature fusion. The module stitches and fuses multi-scale features from mammography X-ray images, CT images, and MRI images to obtain image features. The image feature fusion module outputs the image features to the trained mask prediction model. Step 4.3 inputs the pathological text of the sample pairs for which benign and malignant breast lesions need to be classified into benign and malignant forms into the trained text feature extraction model to obtain text features, which are then input into the trained mask prediction model. Step 4.4, guided by the text features, the trained mask prediction model decodes and reconstructs the image features to obtain the benign and malignant classification results of breast lesions.

[0013] In one specific implementation, the training data for the mask prediction model uses image-text sample pairs with benign and malignant lesion segmentation labels, and the mask prediction model parameters are trained using a multivariate cross-entropy loss function and an error backpropagation algorithm.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows.

[0015] 1. This invention constructs a classification model for benign and malignant breast lesions, comprising a multimodal image feature fusion model, a text feature extraction model, a common semantic space, and a mask prediction model based on a stacked multi-layer bidirectional Transformer decoder. The multimodal image feature fusion model includes a first image feature extraction module, a second image feature extraction module, a third image feature extraction module, and an image feature fusion module. The multimodal image feature fusion model extracts multi-scale features from mammography X-ray images, CT images, and MRI images, and performs alignment and fusion to effectively obtain image features. The text feature extraction model acquires text features. Guided by the text features, the mask prediction model decodes and reconstructs the image features to obtain the classification results for benign and malignant breast lesions. The constructed classification model for benign and malignant breast lesions has potential clinical application value.

[0016] 2. The method of classifying benign and malignant breast lesions using the constructed benign and malignant classification model of this invention can effectively improve the segmentation accuracy of breast lesions in radiological images and provide more accurate and reliable benign and malignant classification results, which has potential clinical application value.

[0017] 3. This invention uses the Hiera model based on the hierarchical visual Transformer structure to extract multi-scale features of mammogram X-ray images, CT images, and MRI images. It then uses graph convolutional neural networks to align and fuse these multi-scale features to effectively obtain image features.

[0018] 4. The text feature extraction model of this invention adopts the FastText word vector model with KAN neural network as the hidden layer, which effectively avoids the curse of dimensionality problem when processing high-dimensional text features. This allows the model to maintain interpretability while taking into account the dynamic nonlinear semantic expression ability of word vectors, and effectively improves efficiency.

[0019] 5. This invention employs a contrastive learning approach to jointly train a multimodal image feature fusion model and a text feature extraction model, effectively enhancing the visual-textual joint representation of breast lesions.

[0020] 6. This invention uses text features as guidance in the mask prediction model, thereby improving the accuracy of classifying benign and malignant breast lesions. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the method for classifying benign and malignant breast lesions according to the present invention.

[0022] Figure 2 This is a schematic diagram of the structure of the benign and malignant breast lesion classification model of the present invention.

[0023] Figure 3 This is a schematic diagram of the bidirectional Transformer decoder of the present invention.

[0024] Figure 4 This is a schematic diagram of the classification results of benign and malignant breast lesions in radiographic images according to an embodiment of the present invention.

[0025] Figure labels: Benign lesion 1, Malignant lesion 2. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the embodiments and accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0027] refer to Figure 1 A method for classifying benign and malignant breast lesions includes the following steps: Step 1: Obtain N sample pairs: Each sample pair includes mammograms, CT images, MRI images, and pathological text.

[0028] Step 2: Construct a classification model for benign and malignant breast lesions: The classification model for benign and malignant breast lesions includes a multimodal image feature fusion model, a text feature extraction model, a common semantic space, and a mask prediction model based on a stacked multi-layer bidirectional Transformer decoder. The structure of the classification model for benign and malignant breast lesions is referenced below. Figure 2 .

[0029] The multimodal image feature fusion model includes a first image feature extraction module, a second image feature extraction module, a third image feature extraction module, and an image feature fusion module. Furthermore, the first image feature extraction module, the second image feature extraction module, and the third image feature extraction module are all Hiera models, and the image feature fusion module is a graph convolutional neural network.

[0030] The Hiera model employs a hierarchical visual Transformer structure, consisting of four downsampling stages each containing multiple Transformer modules. Each Transformer module supports window self-attention and uses absolute window position encoding, combined with interpolated global positional information to enhance spatial awareness. Furthermore, the Hiera model fuses feature maps from different downsampling stages through a feature pyramid network to further enhance its multi-scale representation capabilities.

[0031] The text feature extraction model is the FastText word vector model, and the hidden layer of the FastText word vector model is the KAN neural network.

[0032] Preferably, the KAN neural network of the FastText word vector model performs a non-linear mapping on the word vector x output from the input layer of the FastText word vector model to obtain... , The calculation formula is as follows: The KAN neural network of the FastText word vector model will generate The output is fed to the output layer of the FastText word vector model, where text features are extracted using Softmax; where b(x) is the SiLU activation function. spline(x) is a linear combination function of B-splines. w b w are the weighting coefficients of b(x). s B represents the weighting coefficients of spline(x). l (x) is the l-th B-spline function, c l The coefficients are those of the l-th B-spline function.

[0033] Unlike traditional fully connected layers, the KAN neural network simplifies the modeling of complex nonlinear mappings by using a finite number of learnable B-spline functions and a dynamic grid mechanism. This avoids the curse of dimensionality problem when dealing with high-dimensional text features, allowing the model to maintain interpretability while taking into account the dynamic nonlinear semantic expression capabilities of word vectors, and effectively improving efficiency.

[0034] The outputs of the first image feature extraction module, the second image feature extraction module, and the third image feature extraction module are all connected to the image feature fusion module. The output of the image feature fusion module is connected to the mask prediction model, and the output of the text feature extraction model is connected to the mask prediction model.

[0035] refer to Figure 3 Preferably, the bidirectional Transformer decoder includes a self-attention layer, a first cross-attention layer, a second cross-attention layer, and a multilayer perceptron B. The output of the self-attention layer is connected to the first cross-attention layer, the output of the first cross-attention layer is connected to the multilayer perceptron B, and the output of the multilayer perceptron B is connected to the second cross-attention layer. The multilayer perceptron B outputs updated text features, and the second cross-attention layer outputs updated image features.

[0036] Furthermore, the mask prediction model of the stacked multilayer bidirectional Transformer decoder also includes a transposed convolution module, a multilayer perceptron A, and a dynamic linear classification layer; the output of the text feature extraction model is connected to the self-attention layer of the first-layer bidirectional Transformer decoder, the output of the image feature fusion module is connected to the first cross-attention layer and the second cross-attention layer of the first-layer bidirectional Transformer decoder, the updated image features output from the second cross-attention layer of the last-layer bidirectional Transformer decoder are output to the transposed convolution module, the transposed convolution module downsamples them and outputs them to the multilayer perceptron A, the multilayer perceptron A processes them and outputs them to the dynamic linear classifier to obtain the classification result of benign or malignant lesions, the output of the second cross-attention layer of the first-layer and intermediate-layer bidirectional Transformer decoders are all connected to the first cross-attention layer and the second cross-attention layer of the corresponding next layer, and the output of the multilayer perceptron B of the first-layer and intermediate-layer bidirectional Transformer decoders are all connected to the self-attention layer of the corresponding next layer.

[0037] A contrastive learning loss function based on the cosine similarity matrix of image features and text features is constructed in the public semantic space. The contrastive learning loss function is used to jointly train the multimodal image feature fusion model and the text feature extraction model.

[0038] During the joint training of the multimodal image feature fusion model and the text feature extraction model, the image feature fusion module outputs image features to the common semantic space, and the FastText word vector model outputs text features to the common semantic space.

[0039] Step 3, train the benign and malignant classification model of breast lesions: use contrastive learning to jointly train the multimodal image feature fusion model and the text feature extraction model to explore the potential semantic association between image features and text features; use supervised training to train the mask prediction model.

[0040] Specifically, step 3, which uses a contrastive learning method to jointly train the multimodal image feature fusion model and the text feature extraction model, is as follows: Step 3.1, input the N sample pairs of mammogram X-ray images into the first image feature extraction module, input the N sample pairs of CT images into the second image feature extraction module, and input the N sample pairs of MRI images into the third image feature extraction module. The first image feature extraction module, the second image feature extraction module, the third image feature extraction module, and the image feature fusion module work together to obtain N image features.

[0041] Step 3.2: Input the pathological texts of N sample pairs into the FastText word vector model to obtain N text features.

[0042] Step 3.3: Input the N image features and N text features into the common semantic space, and align the input N image features and N text features within the common semantic space.

[0043] Step 3.4: Define matching image features and text features as positive sample pairs, and non-matching image features and text features as negative sample pairs. Calculate the cosine similarity matrix of N image features and N text features. The i-th image feature I... i and the j-th text feature J j Cosine similarity s between i,j It is expressed as follows: ,in, J is the temperature scaling parameter. k Let i be the k-th text feature, where i is a positive integer and k is a positive integer.

[0044] Step 3.5: Using a contrastive learning loss function, the parameters of the multimodal image feature fusion model and the text feature extraction model are jointly trained through the error backpropagation algorithm. The loss value L of the contrastive learning loss function is... cl The calculation formula is as follows: , where J i For the i-th text feature, I k Let k be the kth image feature.

[0045] Specifically, the training data for the mask prediction model uses image-text sample pairs with benign and malignant lesion segmentation labels, and the mask prediction model parameters are trained using a multivariate cross-entropy loss function and an error backpropagation algorithm.

[0046] The loss value L of the multivariate cross-entropy loss function ce The calculation formula is as follows: , where p z,v Let y be the probability that the z-th pixel in the corresponding image in the training data belongs to category v, where v = 0 represents background, v = 1 represents benign, and v = 2 represents malignant. z,v To determine whether the z-th pixel in the corresponding image in the training data belongs to category v, y z,v A value of 1 indicates that the z-th pixel in the corresponding image in the training data belongs to category v, y z,v A value of 0 indicates that the z-th pixel in the corresponding image in the training data does not belong to category v, and M is the total number of pixels in the image.

[0047] Step 4: Input the sample pairs of mammograms, CT images, MRI images, and pathological texts that need to be classified as benign or malignant breast lesions into the trained benign or malignant breast lesion classification model to obtain the classification results.

[0048] Furthermore, step 4 specifically includes: Step 4.1, inputting the mammogram, CT, and MRI images of the sample pairs for which benign and malignant breast lesions need to be classified into benign and malignant forms into the trained multimodal image feature fusion model; the first image feature extraction module extracts multi-scale features of the mammogram images and outputs them to the image feature fusion module; the second image feature extraction module extracts multi-scale features of the CT images and outputs them to the image feature fusion module; and the third image feature extraction module extracts multi-scale features of the MRI images and outputs them to the image feature fusion module.

[0049] Step 4.2: The image feature fusion module stitches and fuses the multi-scale features of mammogram X-ray images, CT images, and MRI images to obtain image features. The image feature fusion module outputs the image features to the trained mask prediction model.

[0050] The multi-scale features of mammogram X-ray images, CT images, and MRI images are spliced ​​together along the channel dimension. The spliced ​​features are then input into the corresponding nodes of a graph convolutional neural network. The output of the last layer of the graph convolutional neural network is used to represent the image features after the final multimodal alignment and fusion.

[0051] Step 4.3 Input the pathological text of the sample pairs for which benign and malignant breast lesions need to be classified into benign and malignant features into the text feature extraction model to obtain text features, and then input them into the mask prediction model that has been trained.

[0052] The text feature extraction model is the FastText word vector model with KAN neural network as the hidden layer. KAN neural network simplifies the modeling of complex nonlinear mapping by using a finite number of learnable B-spline functions and dynamic grid mechanism, thereby avoiding the curse of dimensionality problem when dealing with high-dimensional text features. This allows the model to maintain interpretability while taking into account the dynamic nonlinear semantic expression ability of word vectors, and effectively improves efficiency.

[0053] Step 4.4: Guided by text features, the trained mask prediction model decodes and reconstructs image features to obtain the classification results of benign and malignant breast lesions.

[0054] The mask prediction model's backbone consists of a stack of multiple bidirectional Transformer decoders. First, text features are input into an attention layer for optimization. The optimized text and image features are then fed into a first cross-attention layer to obtain a fused cross-feature. Next, this fused cross-feature is input into a multilayer perceptron (B) to obtain updated text features. These updated text and image features are then fed into a second cross-attention layer to obtain updated image features. The updated image and text features serve as input to the next bidirectional Transformer decoder. Finally, the image features updated by the last bidirectional Transformer decoder are sequentially passed through a transposed convolutional module, a multilayer perceptron (A), and a dynamic linear classifier to obtain the benign / malignant probability at each location in the image, thus yielding the benign / malignant classification result for breast lesions.

[0055] from Figure 4 It can be seen that the constructed benign and malignant classification model for breast lesions can effectively obtain the classification results of benign and malignant breast lesions.

[0056] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions and substitutions can be made without departing from the inventive concept, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for classifying benign and malignant breast lesions, characterized in that, Includes the following steps: Step 1: Obtain N sample pairs: Each sample pair includes mammograms, CT images, MRI images, and pathological text; Step 2: Construct a classification model for benign and malignant breast lesions: The classification model for benign and malignant breast lesions includes a multimodal image feature fusion model, a text feature extraction model, a common semantic space, and a mask prediction model based on a stacked multi-layer bidirectional Transformer decoder; The multimodal image feature fusion model includes a first image feature extraction module, a second image feature extraction module, a third image feature extraction module, and an image feature fusion module; The text feature extraction model is the FastText word vector model, and the hidden layer of the FastText word vector model is the KAN neural network; The outputs of the first image feature extraction module, the second image feature extraction module, and the third image feature extraction module are all connected to the image feature fusion module. The output of the image feature fusion module is connected to the mask prediction model, and the output of the text feature extraction model is connected to the mask prediction model. A contrastive learning loss function based on the cosine similarity matrix of image features and text features is constructed in the public semantic space. The contrastive learning loss function is used to jointly train the multimodal image feature fusion model and the text feature extraction model. During the joint training of the multimodal image feature fusion model and the text feature extraction model, the image feature fusion module outputs image features to the common semantic space, and the FastText word vector model outputs text features to the common semantic space. Step 3: Training the benign and malignant classification model of breast lesions: The contrastive learning method is used to jointly train the multimodal image feature fusion model and the text feature extraction model, and the supervised training method is used to train the mask prediction model; Step 4: Obtain the classification results of benign and malignant breast lesions: Input the sample pairs of breast lesion benign and malignant classification, including mammograms, CT images, MRI images and pathological texts, into the trained breast lesion benign and malignant classification model to obtain the classification results.

2. The method for classifying benign and malignant breast lesions as described in claim 1, characterized in that, The bidirectional Transformer decoder includes a self-attention layer, a first cross-attention layer, a second cross-attention layer, and a multilayer perceptron B. The output of the self-attention layer is connected to the first cross-attention layer, the output of the first cross-attention layer is connected to the multilayer perceptron B, and the output of the multilayer perceptron B is connected to the second cross-attention layer. The multilayer perceptron B outputs updated text features, and the second cross-attention layer outputs updated image features.

3. The method for classifying benign and malignant breast lesions as described in claim 2, characterized in that, The mask prediction model of the stacked multilayer bidirectional Transformer decoder also includes a transposed convolution module, a multilayer perceptron A, and a dynamic linear classification layer. The output of the text feature extraction model is connected to the self-attention layer of the first-layer bidirectional Transformer decoder. The output of the image feature fusion module is connected to the first cross-attention layer and the second cross-attention layer of the first-layer bidirectional Transformer decoder. The updated image features output from the second cross-attention layer of the last-layer bidirectional Transformer decoder are output to the transposed convolution module. The transposed convolution module downsamples the features and outputs them to the multilayer perceptron A. After processing by the multilayer perceptron A, the features are output to the dynamic linear classifier to obtain the classification result of benign or malignant lesions. The outputs of the second cross-attention layers of the first-layer and intermediate-layer bidirectional Transformer decoders are connected to the first cross-attention layer and the second cross-attention layer of the corresponding next layer. The outputs of the multilayer perceptron B of the first-layer and intermediate-layer bidirectional Transformer decoders are connected to the self-attention layer of the corresponding next layer.

4. The method for classifying benign and malignant breast lesions as described in claim 3, characterized in that, The KAN neural network of the FastText word vector model performs a non-linear mapping on the word vector x output from the input layer of the FastText word vector model to obtain... , The calculation formula is as follows: The KAN neural network of the FastText word vector model will generate The output is fed to the output layer of the FastText word vector model, where text features are extracted using Softmax. Where b(x) is the SiLU activation function, spline(x) is a linear combination function of B-splines. w b w are the weighting coefficients of b(x). s B represents the weighting coefficients of spline(x). l (x) is the l-th B-spline function, c l The coefficients are those of the l-th B-spline function.

5. The method for classifying benign and malignant breast lesions as described in claim 4, characterized in that, The first image feature extraction module, the second image feature extraction module, and the third image feature extraction module are all Hiera models.

6. The method for classifying benign and malignant breast lesions as described in claim 5, characterized in that, The image feature fusion module is a graph convolutional neural network.

7. The method for classifying benign and malignant breast lesions as described in claim 6, characterized in that, Step 3, which involves jointly training the multimodal image feature fusion model and the text feature extraction model using a contrastive learning method, specifically involves: Step 3.1: Input the mammograms of N sample pairs into the first image feature extraction module, input the CT images of N sample pairs into the second image feature extraction module, and input the MRI images of N sample pairs into the third image feature extraction module. The first image feature extraction module, the second image feature extraction module, the third image feature extraction module, and the image feature fusion module work together to obtain N image features. Step 3.2: Input the pathological texts of N sample pairs into the FastText word vector model to obtain N text features; Step 3.3: Input the N image features and N text features into the common semantic space, and align the input N image features and N text features within the common semantic space; Step 3.4: Define matching image features and text features as positive sample pairs, and non-matching image features and text features as negative sample pairs. Calculate the cosine similarity matrix of N image features and N text features. The i-th image feature I... i and the j-th text feature J j Cosine similarity s between i,j It is expressed as follows: ,in, J is the temperature scaling parameter. k Let i be the k-th text feature, where i is a positive integer and k is a positive integer. Step 3.5: Using a contrastive learning loss function, the parameters of the multimodal image feature fusion model and the text feature extraction model are jointly trained through the error backpropagation algorithm. The loss value L of the contrastive learning loss function is... cl The calculation formula is as follows: , where J i For the i-th text feature, I k Let k be the kth image feature.

8. The method for classifying benign and malignant breast lesions as described in claim 7, characterized in that, Step 4 specifically involves: Step 4.1: Input the mammography X-ray images, CT images, and MRI images of the sample pairs for which benign and malignant breast lesions need to be classified into benign and malignant forms into the trained multimodal image feature fusion model. The first image feature extraction module extracts multi-scale features of the mammography X-ray images and outputs them to the image feature fusion module. The second image feature extraction module extracts multi-scale features of the CT images and outputs them to the image feature fusion module. The third image feature extraction module extracts multi-scale features of the MRI images and outputs them to the image feature fusion module. Step 4.2: The image feature fusion module stitches and fuses the multi-scale features of mammogram X-ray images, CT images, and MRI images to obtain image features. The image feature fusion module outputs the image features to the trained mask prediction model. Step 4.3 Input the pathological text of the sample pairs for which benign and malignant breast lesions need to be classified into benign and malignant features into the trained text feature extraction model to obtain text features and input them into the trained mask prediction model. Step 4.4: Guided by text features, the trained mask prediction model decodes and reconstructs image features to obtain the classification results of benign and malignant breast lesions.

9. The method for classifying benign and malignant breast lesions as described in claim 8, characterized in that, The training data for the mask prediction model uses image-text sample pairs with benign and malignant lesion segmentation labels, and the parameters of the mask prediction model are trained using a multivariate cross-entropy loss function and an error backpropagation algorithm.

Images