Breast image classification and evaluation method and system based on deep learning

Abstract

The present invention relates to the technical field of image classification, and discloses a method and system for breast image classification evaluation based on deep learning, wherein the method comprises: preprocessing sample breast image data to obtain comprehensive feature data; performing feature enhancement based on the comprehensive feature data to obtain enhanced feature data; performing multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fusion feature data; constructing and training a semi-supervised local feature-preserving densely connected context-aware graph convolutional network based on the multimodal fusion feature data to obtain a graph convolution classification model; constructing and training a multi-branch residual attention reconstruction network according to the multimodal fusion feature data to obtain a multiscale feature fusion model; and classifying and evaluating real-time breast image data and integrating prediction results through the graph convolution classification model and the multiscale feature fusion model to obtain a target breast image classification result, wherein the method improves the accuracy and reliability of breast image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a breast image classification and evaluation method and system based on deep learning. Background Art

[0002] Breast cancer is one of the most common malignant tumors in women worldwide, and it poses a serious threat to women's health. Early diagnosis and accurate classification are crucial for the treatment and prognosis of breast cancer. With the development of medical imaging technology, breast images have become an important tool for breast cancer diagnosis. However, traditional manual diagnostic methods have problems such as strong subjectivity and low efficiency, and are unable to meet the growing diagnostic needs.

[0003] In recent years, deep learning technology has made significant progress in the field of medical image analysis, providing new possibilities for the automated classification of breast images. However, existing deep learning methods still face some challenges when processing complex breast images. The features of breast images are often multimodal and multi-scale, and a single feature extraction method is difficult to fully capture the key information of the image. Secondly, the contradiction between limited labeled samples and a large amount of unlabeled data restricts the performance improvement of the model. In addition, existing methods often ignore the spatial structural information of breast tissue and find it difficult to fully utilize the correlation between local features. At the same time, how to effectively fuse features of different modalities and scales, and how to improve the model's attention to key areas, are issues that need to be addressed. Summary of the invention

[0004] The present invention provides a deep learning-based breast image classification evaluation method and system for improving the accuracy and reliability of breast image classification.

[0005] In a first aspect, the present invention provides a breast image classification and evaluation method based on deep learning, the breast image classification and evaluation method based on deep learning comprising:

[0006] Preprocessing the sample breast image data to obtain comprehensive feature data;

[0007] Based on the comprehensive feature data, feature enhancement is performed by coordinating the attention module and the multimodal attention module to obtain enhanced feature data;

[0008] Performing multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fused feature data;

[0009] Based on the multimodal fusion feature data, a semi-supervised local feature preserving densely connected context-aware graph convolutional network is constructed and trained to obtain a graph convolution classification model;

[0010] According to the multimodal fusion feature data, a multi-branch residual attention reconstruction network is constructed and trained to obtain a multi-scale feature fusion model;

[0011] The real-time breast image data is classified, evaluated and predicted by the graph convolution classification model and the multi-scale feature fusion model to obtain a target breast image classification result.

[0012] In a second aspect, the present invention provides a breast image classification and evaluation system based on deep learning, wherein the breast image classification and evaluation system based on deep learning comprises:

[0013] A preprocessing module, used for preprocessing sample breast image data to obtain comprehensive feature data;

[0014] A feature enhancement module, configured to perform feature enhancement based on the comprehensive feature data by coordinating the attention module and the multimodal attention module to obtain enhanced feature data;

[0015] A feature fusion module, used for performing multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fused feature data;

[0016] A first construction module is used to construct and train a semi-supervised local feature preserving densely connected context-aware graph convolutional network based on the multimodal fusion feature data to obtain a graph convolution classification model;

[0017] A second construction module is used to construct and train a multi-branch residual attention reconstruction network according to the multimodal fusion feature data to obtain a multi-scale feature fusion model;

[0018] The integration module is used to classify and evaluate the real-time breast image data and integrate the prediction results through the graph convolution classification model and the multi-scale feature fusion model to obtain the target breast image classification result.

[0019] The third aspect of the present invention provides a breast image classification and evaluation device based on deep learning, comprising: a memory and at least one processor, wherein instructions are stored in the memory; the at least one processor calls the instructions in the memory so that the breast image classification and evaluation device based on deep learning performs the above-mentioned breast image classification and evaluation method based on deep learning.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the above-mentioned deep learning-based breast image classification and evaluation method.

[0021] In the technical solution provided by the present invention, by preprocessing and pyramid decomposition of sample breast image data, combined with a coordinated attention module and a multimodal attention module, effective extraction and enhancement of multimodal and multiscale features are achieved. Various features of breast images can be fully captured to improve the accuracy and robustness of classification. By constructing a semi-supervised local feature-retaining densely connected context-aware graph convolutional network, limited labeled samples and a large amount of unlabeled data are effectively utilized. This strategy not only improves the generalization ability of the model, but also reduces the dependence on a large amount of labeled data and reduces the cost of manual annotation. By using superpixel segmentation and graph convolutional networks, the method can effectively capture the spatial structural information of breast tissue and make full use of the correlation between local features. This feature extraction method based on graph structure can better characterize the complex structure of breast tissue and improve the accuracy of classification. By introducing a multi-level attention mechanism in a graph convolutional network and a multi-branch residual network, it is possible to adaptively focus on key areas and important features in the image and improve the discrimination ability of the model. By constructing a multi-branch residual attention reconstruction network and designing an adaptive feature fusion strategy, effective integration of features of different scales is achieved. This fusion method can make full use of multi-scale information and improve the model's ability to identify lesions of different sizes. By combining the prediction results of the graph convolution classification model and the multi-scale feature fusion model, model integration is achieved, further improving the accuracy and stability of classification. By adopting an end-to-end training method, by jointly optimizing feature extraction, attention mechanism, classification tasks and reconstruction tasks, the coordinated optimization of each module is achieved, improving the performance of the overall model.

[0022] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of an embodiment of a breast image classification and evaluation method based on deep learning in an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of an embodiment of a breast image classification and evaluation system based on deep learning in an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of an embodiment of a breast image classification and evaluation device based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.

[0029] To facilitate understanding of this embodiment, a breast image classification and evaluation method based on deep learning disclosed in an embodiment of the present invention is first introduced in detail. Figure 1 As shown, the method comprises the following steps:

[0030] 101. Preprocessing the sample breast image data to obtain comprehensive feature data;

[0031] It is understandable that the execution subject of the present invention may be a breast image classification and evaluation system based on deep learning, or may be a terminal or a server, which is not specifically limited here. The embodiment of the present invention is described by taking a server as an execution subject as an example.

[0032] Specifically, the sample breast image data is standardized, and the pixel values ​​of the image data are adjusted to a uniform range to reduce the differences in image brightness and contrast, thereby obtaining a standardized image. The standardized image is pyramid decomposed, and multi-scale images of different resolutions are generated by constructing a multi-scale representation of the image to capture the detailed features of different levels in the breast image. The multi-scale images are convolved respectively, and initial features such as edges and textures in the image are extracted through filters to obtain an initial feature map. The initial feature map is batch normalized to accelerate the convergence speed of network training and avoid gradient vanishing or gradient explosion to obtain a normalized feature map. The normalized feature map is nonlinearly processed by applying the ReLU activation function to enhance the model's ability to express complex features and generate an activated feature map. The activated feature map is subjected to a maximum pooling operation to obtain a downsampled feature map, and the spatial dimension of the feature map is reduced by maximum pooling while retaining the most significant feature information to obtain a downsampled feature map. In order to prevent information loss, the downsampled feature map is processed by a residual connection technique, and the input feature map is directly added to the output feature map to retain more original feature information to obtain a residual feature map. The attention mechanism is applied to the residual feature map to perform attention feature weighting processing. By introducing the attention mechanism, the model can automatically focus on more important areas in the breast image and obtain attention weighted feature maps of different modalities and scales. The attention weighted feature maps of different modalities and scales are spliced ​​in the channel dimension to integrate the feature information of different modalities and scales, so that the model can use multiple features to classify images at the same time and finally generate comprehensive feature data.

[0033] 102. Based on the comprehensive feature data, feature enhancement is performed by coordinating an attention module and a multimodal attention module to obtain enhanced feature data;

[0034] Specifically, a global average pooling operation is performed on the comprehensive feature data to compress the spatial dimension of each feature map into a feature vector, retaining the global spatial information. Through global average pooling, the generated feature vector contains the average information of all pixels in the breast image. A 1x1 convolution operation is performed on the feature vector. The 1x1 convolution can effectively compress and recombine features, so that the feature vector is more suitable for subsequent attention calculations, and the initial weight of spatial attention is generated. The sigmoid activation function is applied to the initial weight of spatial attention. The output value of the sigmoid function is between 0 and 1, which is used to indicate the importance of each feature in the spatial dimension. The activated spatial attention weight map indicates the attention of each spatial position. The maximum pooling operation is performed on the comprehensive feature data. The channel dimension is compressed by maximum pooling, while retaining the most significant features, to obtain the feature matrix after the channel dimension is compressed. For the feature matrix after the channel dimension is compressed, a fully connected layer is used for processing, and the weights of different channels are redistributed to generate the initial weight of channel attention. The sigmoid activation function is applied to the initial weight of channel attention. The channel attention weight vector after sigmoid activation is used to reflect the importance of each channel. The spatial attention weight map is multiplied by the channel attention weight vector to generate the coordinated attention weight. The comprehensive feature data is modally grouped, and different types of features are divided into multiple modal feature subsets, each representing a different data modality for separate processing. For each modal feature subset, the query matrix, key matrix, and value matrix are calculated. Through the operation of these matrices, the key attention information is captured in each modal subset. Through matrix multiplication, the query matrix and the key matrix are multiplied to generate attention scores. These attention scores are normalized by the softmax function to represent the correlation between each feature. The multimodal attention score is multiplied by the value matrix to obtain the weighted multimodal feature. In order to enhance the multimodal features, the multimodal attention score is multiplied by the value matrix, combining the feature information of different modalities and the comprehensive weights of space and channels to finally generate enhanced feature data.

[0035] 103. Perform multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fused feature data;

[0036] Specifically, the enhanced feature data is grouped according to different modalities, and feature information from different sources or different types is separated into multiple modal feature groups, so that the data features of each modality can be processed and enhanced separately in subsequent operations to maximize the expressiveness of each modality. Each modal feature group is subjected to 1x1 convolution and global average pooling operations in turn. The purpose of the 1x1 convolution operation is to compress and reorganize the channels of the feature map while retaining the spatial structure information of the feature map. Global average pooling further compresses the spatial dimension into a single global descriptor, so that each modality can extract global feature information. Based on the global descriptor, the feature map of the current modality is element-wise multiplied to generate forward flow features. The global descriptor of the previous modality is multiplied element by element with the features of the current modality to ensure that the information of the previous modality can be passed to the current modality to form a forward information flow. Similarly, based on the global descriptor of the next modality, the feature map of the current modality is element-wise multiplied to generate reverse flow features. Through the bidirectional flow mechanism, the current mode is not only affected by the previous mode, but also interacts with the next mode, so that the features of different modes can be interconnected. The forward flow features and the reverse flow features are weighted and summed to integrate the bidirectional feature information and obtain the bidirectional modulation features. The bidirectional modulation features are subjected to deep separable convolution to reduce the amount of calculation while retaining the independence of channel and spatial information to obtain channel space features. The target weights are calculated based on the channel space features to determine the importance of each feature in the classification task. The target weights are element-wise multiplied with the bidirectional modulation features to obtain weighted fusion features. The weighted fusion features are residually connected with the enhanced feature data to retain the original enhanced feature data and superimpose the weighted fusion features to ensure that the final multimodal fusion feature data not only contains the key information in the enhanced feature data, but also integrates the results of multimodal bidirectional fusion to obtain a richer and more accurate feature expression.

[0037] 104. Based on the multimodal fusion feature data, construct and train a semi-supervised local feature preserving densely connected context-aware graph convolutional network to obtain a graph convolution classification model;

[0038] Specifically, the multimodal fusion feature data is segmented by superpixels, and the image is divided into several regions, each of which contains pixels with similar features. These superpixel regions are used as nodes in the graph to form a graph structure containing multiple nodes. Based on the k-nearest neighbor relationship of the nodes and the Gaussian similarity function, the adjacency matrix and degree matrix of the graph are constructed. By calculating the similarity between nodes and combining their spatial proximity, the connection relationship between nodes in the graph is defined to obtain the adjacency matrix of the graph. The adjacency matrix is ​​normalized by matrix operations to ensure the stability and consistency of feature propagation and form the initial graph structure. A semi-supervised local feature preserving densely connected context-aware graph convolutional network is constructed. The network consists of 5 layers of graph convolutional layers, each of which contains graph convolution operations, batch normalization and LeakyReLU activation functions. The graph convolution operation generates the output features of each layer by performing matrix operations on the input features, adjacency matrix and learnable weight matrix. Batch normalization ensures numerical stability during training, and the LeakyReLU activation function introduces nonlinearity to enhance the expressiveness of the model. In order to improve the feature representation ability of the model, dense connections are introduced between every two adjacent graph convolutional layers. Dense connections concatenate the output features of the current layer with the input features of all subsequent layers, so that subsequent layers can directly access the feature representation of the previous layer, achieving efficient propagation and reuse of features. At the same time, a layer attention module is inserted after the 2nd and 4th graph convolutional layers. This layer attention module calculates the attention weights through two layers of fully connected layers and softmax activation function. These weights are multiplied with the enhanced features to obtain the attention-enhanced graph convolutional features. A global average pooling operation is performed on the output features of the 5th graph convolutional layer to aggregate all node features into a fixed-dimensional graph-level feature representation. The graph-level feature representation is input into two layers of fully connected layers, where the first layer contains 256 neurons and the number of neurons in the second layer is the same as the number of classification categories. The first classification prediction result of the labeled sample is output through the softmax function. For unlabeled samples, data augmentation operations are performed to generate enhanced samples from two perspectives. By performing different perturbations or changes on the same unlabeled sample, two enhanced samples are obtained, and they are input into the graph convolutional network to obtain feature representations from two perspectives. In order to ensure the robustness of the model to unlabeled data, training is performed by minimizing the cross entropy loss of labeled samples and the consistency regularization loss of unlabeled samples. The consistency regularization loss is measured by calculating the Euclidean distance between the two perspective feature representations, that is, the degree of difference between the two enhanced samples in the feature space. The weighted sum of the cross entropy loss of labeled samples and the consistency regularization loss of unlabeled samples is used as the total loss function to guide the end-to-end training of the model. In this way, the entire semi-supervised local feature preserving densely connected context-aware graph convolutional network is gradually optimized to obtain the final graph convolution classification model.This model can not only effectively process labeled data, but also learn the potential features of unlabeled data through consistency regularization, thereby improving the accuracy and generalization ability of breast image classification.

[0039] In the graph convolution network, the output features of the i-th graph convolution layer are transformed in dimension to ensure that these output features can match the input dimensions of the subsequent layers. The feature map is concatenated with the input features from the i+1th layer to the 5th layer along the channel dimension to form an initial enhanced feature containing multiple layers of features. In this way, the dense connection fuses the output of the current layer with the input features of the subsequent layer, allowing the network to reuse the previous features and enhance the feature representation ability of the model. The initial enhanced features are subjected to a 1x1 convolution operation, and the channel dimension is compressed through convolution to reduce the complexity of the features and obtain the compressed enhanced features. The compressed enhanced features are input into the first fully connected layer. The number of neurons in this fully connected layer is set to half of the dimension of the compressed features. Through this dimensionality reduction process, the features are extracted and simplified to obtain an intermediate feature representation. The ReLU activation function is operated on the intermediate feature representation. The nonlinear characteristics of ReLU can enhance the model's ability to express complex features, effectively introduce nonlinear transformations, and generate intermediate features after nonlinear transformations. The intermediate features after nonlinear transformation are input into the second fully connected layer. The number of neurons in this fully connected layer is the same as the number of channels of the compressed enhanced features. The purpose is to generate a corresponding attention weight for each channel to form the original attention weight. The softmax function is applied to the original attention weight. The softmax function normalizes the original attention weight so that the weight value of each channel is between 0 and 1, and the sum of all weights is 1. Through the normalization operation, the network can determine the relative importance of each channel and generate normalized attention weights. The normalized attention weights are multiplied channel by channel with the compressed enhanced features. Through the weighted multiplication operation channel by channel, the network adaptively enhances the attention to important channels during the forward propagation process, and weakens those channels that contribute less to the task, and obtains the attention-enhanced graph convolution features.

[0040] 105. According to the multimodal fusion feature data, a multi-branch residual attention reconstruction network is constructed and trained to obtain a multi-scale feature fusion model;

[0041] Specifically, the multimodal fusion feature data is decomposed into pyramid scales, and the input feature data is divided into three different scales. The feature maps of each scale are used as inputs of three parallel branches. In each branch, the input feature map is processed by five residual blocks. Each residual block consists of two 3x3 convolutional layers, a batch normalization layer, and a ReLU activation function. It also contains a short-circuit connection, which enables the input feature map to be transmitted in a jump manner, thereby effectively solving the gradient vanishing problem in deep networks. Through the processing of residual blocks, initial residual features are generated. These features have strong hierarchical information transmission capabilities and can retain the original information of the input image. The spatial attention module is inserted into the second and fourth residual blocks of each branch to weight the feature importance of the initial residual features in the spatial dimension. The spatial attention module can make the model focus more on the key areas in the breast image by learning the importance of different positions in the image, and generate spatial attention enhanced features. By effectively guiding the model to pay attention to important local features in the image, the accuracy of feature extraction is improved. The channel attention module is applied to the spatial attention enhanced features. The channel attention module compresses the feature map in the spatial dimension through global average pooling, and then calculates the weight of each channel through the fully connected layer and the sigmoid activation function. In this way, the network can identify the important features on different channels and weight them to obtain the dual attention enhanced features. A reconstruction module is added to the last residual block of each branch. The reconstruction module consists of 3 transposed convolutional layers, each of which is followed by batch normalization and ReLU activation function. Its main function is to gradually upsample the feature map to the size of the original input image. Through the reconstruction module, a reconstructed feature map is generated to ensure that the network can not only classify, but also learn the detailed information of the input image through the reconstruction process, and improve the generalization ability of the model more comprehensively. In order to optimize the network, during the training process, the mean square error (MSE) between the reconstructed feature map and the original input image is calculated. This error is the reconstruction loss. The reconstruction loss is used to measure the accuracy of the network in the image reconstruction process. At the same time, the reconstruction loss is added to the cross entropy loss in the classification task as the overall optimization goal of the network. Based on the dual attention enhanced features of the three branches, adaptive feature fusion is performed. By calculating the Euclidean distance between each branch feature and the ideal feature, the feature quality of each branch is obtained, and the corresponding fusion weight is calculated according to the proportional relationship of these distances. The features of each branch are weighted and summed according to the weights, and finally the trained multi-scale fusion features are obtained. This fusion feature contains key information from different scales and different modes, which can effectively improve the classification accuracy of the model. The trained multi-scale fusion features are input into the fully connected layer and the softmax layer to generate the second classification prediction results of the labeled samples.Combining the reconstruction loss and classification cross entropy loss, the entire network is trained end-to-end through the back-propagation algorithm, and finally a multi-scale feature fusion model is obtained.

[0042] Specifically, a global average pooling operation is performed on the dual attention enhanced features of each branch to compress the high-dimensional feature map into a feature vector of fixed dimension to obtain the global feature representation of each branch. An ideal feature vector is constructed as a reference for feature fusion. The element value of the ideal feature vector is 1 and the dimension is the same as the global feature representation of each branch. By comparing the actual features of the branch with this ideal feature vector, the degree of difference between the features of each branch and the ideal situation is evaluated. The Euclidean distance between the global feature representation of each branch and the ideal feature vector is calculated. The Euclidean distance can measure the difference between two vectors in the feature space. By calculating the Euclidean distance between the three branches and the ideal features, three distance values ​​are obtained, which correspond to the degree of difference between the features of the three branches. The larger the distance, the greater the difference between the branch feature and the ideal feature; the smaller the distance, the closer the branch feature is to the ideal state. In order to make these distance values ​​more comparable, the three distance values ​​are normalized. Each distance value is divided by the sum of all distance values ​​to obtain the normalized distance ratio. Based on the normalized distance ratio, the fusion weight of each branch is calculated. Subtract the normalized distance ratio of each branch from 1, so that branches with features closer to the ideal features get larger weights, while branches with larger differences from the ideal features get smaller weights. Normalize these weights so that the weights of each branch are reasonably distributed. Multiply the fusion weights by the dual attention enhancement features of the corresponding branches. The features of each branch will adjust their contribution to the final feature fusion according to the degree of difference relative to the ideal features. Add the three weighted features element by element to obtain the final trained multi-scale fusion features.

[0043] 106. Classify and evaluate the real-time breast image data and integrate the prediction results through the graph convolution classification model and the multi-scale feature fusion model to obtain a target breast image classification result.

[0044] Specifically, the real-time breast image data is preprocessed. The preprocessing operation includes standardizing the input breast image data and pyramid scale decomposing to generate multi-scale image data. The preprocessed multi-scale image data is input into the graph structure generation layer of the graph convolution classification model. In this layer, the breast image is divided into regions using superpixel segmentation technology, and a graph structure representation is constructed based on the similarity of each superpixel region. Through the graph structure representation, the graph convolution network can better process the local region and global relationship of the image, and map the pixel points of the image to the connection relationship between the graph nodes. The graph structure representation is processed in turn through the 5 layers of graph convolution layers in the graph convolution classification model. Each graph convolution layer contains graph convolution operations, batch normalization, and LeakyReLU activation functions. These operations work together to extract the graph convolution features of the breast image layer by layer. At the same time, the layer attention module is applied after the second and fourth layers of the graph convolution layer. By adaptively adjusting the weights of the features of each layer, the model can more accurately focus on the key parts of the breast image and obtain real-time graph convolution features. The real-time graph convolution features are input into the global average pooling layer to compress them into a global feature representation of a fixed dimension. The global features are input into two fully connected layers, and the first classification probability distribution of the graph convolution classification model is generated using the softmax function. This reflects the preliminary classification results of the graph convolution network for breast images. At the same time, the preprocessed multi-scale image data is input into the three parallel branches of the multi-scale feature fusion model. Each branch contains 5 residual blocks, each of which has two convolutional layers, batch normalization layers, ReLU activation functions, and a short-circuit connection. Each branch applies the spatial attention module and the channel attention module after the second and fourth residual blocks, respectively. Through the dual attention mechanism, the model can enhance the representation ability of features in the spatial and channel dimensions, and obtain real-time dual attention enhanced features. The dual attention enhanced features of these three branches are adaptively fused. The fusion weights are obtained by calculating the Euclidean distance between each branch feature and the ideal feature vector and assigning weights based on these distances. The features of each branch are weighted and multiplied by these weights, and then the weighted features are added element by element to obtain real-time multi-scale fusion features. The real-time multi-scale fusion features are input into the fully connected layer and the softmax layer to obtain the second classification probability distribution of the multi-scale feature fusion model. This probability distribution is based on the dual attention mechanism of the multi-scale feature fusion model, which further enhances the accuracy of the classification results. The first classification probability distribution of the graph convolution classification model and the second classification probability distribution of the multi-scale feature fusion model are weighted averaged, and the final classification result of the target breast image is obtained by effectively integrating the two classification results.

[0045] In an embodiment of the present invention, by preprocessing and pyramid decomposing the sample breast image data, combined with the coordinated attention module and the multimodal attention module, the effective extraction and enhancement of multimodal and multiscale features are achieved. Various features of breast images can be fully captured to improve the accuracy and robustness of classification. By constructing a semi-supervised local feature-preserving densely connected context-aware graph convolutional network, limited labeled samples and a large amount of unlabeled data are effectively utilized. This strategy not only improves the generalization ability of the model, but also reduces the dependence on a large amount of labeled data and reduces the cost of manual labeling. By using superpixel segmentation and graph convolutional networks, the method can effectively capture the spatial structural information of breast tissue and make full use of the correlation between local features. This feature extraction method based on graph structure can better characterize the complex structure of breast tissue and improve the accuracy of classification. By introducing a multi-level attention mechanism in the graph convolutional network and the multi-branch residual network, it is possible to adaptively focus on the key areas and important features in the image and improve the discrimination ability of the model. By constructing a multi-branch residual attention reconstruction network and designing an adaptive feature fusion strategy, the effective integration of features of different scales is achieved. This fusion method can make full use of multi-scale information and improve the model's recognition ability for lesions of different sizes. By combining the prediction results of the graph convolution classification model and the multi-scale feature fusion model, model integration is achieved, further improving the accuracy and stability of classification. By adopting an end-to-end training method, by jointly optimizing feature extraction, attention mechanism, classification tasks and reconstruction tasks, the coordinated optimization of each module is achieved, improving the performance of the overall model.

[0046] In a specific embodiment, the process of executing step 101 may specifically include the following steps:

[0047] Performing standardization processing on sample breast image data to obtain a standardized image, and performing pyramid decomposition on the standardized image to obtain a multi-scale image;

[0048] Performing convolution operations on the multi-scale images respectively to obtain initial feature maps, and performing batch normalization processing on the initial feature maps to obtain normalized feature maps;

[0049] Performing a ReLU activation function operation on the normalized feature map to obtain an activated feature map, and performing a maximum pooling operation on the activated feature map to obtain a downsampled feature map;

[0050] Performing residual connection on the downsampled feature map to obtain a residual feature map, and performing attention feature weighting processing on the residual feature map to obtain attention weighted feature maps of different modalities and different scales;

[0051] The attention-weighted feature maps of different modalities and different scales are concatenated in the channel dimension to obtain comprehensive feature data.

[0052] Specifically, the input sample breast image data is standardized to adjust the pixel values ​​of the image data to a consistent range, reducing the impact of contrast differences between images on model training. Through standardization, the interference of illumination changes and noise is effectively reduced, and the robustness of the model to image data is enhanced. Assume that the input breast image data is , the standardized processing can be expressed as

[0053] ;

[0054] in, represents the normalized image, is the pixel mean of the image, is the pixel standard deviation. Through the standardization operation, all pixel values ​​of the image are adjusted to a range of mean 0 and standard deviation 1. The standardized image is subjected to pyramid decomposition to decompose the image data into multiple scale images of different resolutions. Pyramid decomposition continuously downsamples the image to generate image layers of different resolutions, thereby capturing the global and local features of the image at different scales. Assume that the standardized image Decomposed into three different scales , and , these images correspond to high-resolution, medium-resolution and low-resolution feature representations respectively. Perform convolution operations on images of different scales to extract the initial feature maps. Scan the image through the convolution kernel to extract low-level features such as edges and textures. Assume that the size of the convolution kernel is , by performing convolution operations on the images of each scale, the corresponding initial feature map is obtained. The convolution process can be expressed as:

[0055] ;

[0056] in, represents the feature map after convolution, is the weight of the convolution kernel, is the local area of ​​the image. Through the convolution operation, the local features in the initial feature map are extracted, such as edges, corners, etc. In order to ensure the training stability of the model and the uniform distribution of features, the obtained initial feature map is batch normalized. The output features of each layer are renormalized to eliminate internal covariate shift, thereby accelerating the training of the model. The normalization process adjusts the output of each feature map to a range of mean 0 and variance 1 to maintain the distribution stability of the features during training. The ReLU activation function is performed on the normalized feature map. ReLU is a commonly used activation function that introduces nonlinear features by setting negative values ​​to zero and retaining positive values. The mathematical expression of ReLU is:

[0057] ;

[0058] After activation by ReLU, negative information in the feature map is suppressed, and only positive information is retained, which helps the model better learn important features in the image. On the activated feature map, the maximum pooling operation is performed to reduce the spatial dimension of the feature map while retaining the most important features. Maximum pooling reduces the resolution of the feature map by taking the maximum value in the local area, which can reduce the amount of calculation and enhance the robustness of the model to the features. After maximum pooling, the downsampled feature map is obtained. In order to retain some of the information lost during the downsampling process, the residual connection technology is used to process the downsampled feature map. The residual connection directly adds the input to the output, so that the input information can be directly passed to the subsequent layers while passing through the convolution layer, avoiding the gradient vanishing problem that may occur in the deep network. Through the residual connection, the residual feature map obtained can effectively retain the initial information. The residual feature map is weighted by attention features. The spatial attention mechanism is used to weight the spatial dimension of the residual feature map to highlight the area with important features in the image. Through the channel attention mechanism, the feature maps of different channels are weighted to enhance the key features expressed in different channels. After attention weighting processing, attention weighted feature maps of different modalities and scales are obtained. All the attention weighted feature maps of different modalities and scales are spliced ​​in the channel dimension, combining multiple features together to form a comprehensive feature representation, ensuring that the model can use multiple features at the same time to classify breast images.

[0059] In a specific embodiment, the process of executing step 102 may specifically include the following steps:

[0060] Performing a global average pooling operation on the comprehensive feature data to obtain a feature vector after spatial dimension compression;

[0061] Performing a 1x1 convolution operation on the feature vector to obtain an initial spatial attention weight, and performing a sigmoid activation function on the initial spatial attention weight to obtain a spatial attention weight map;

[0062] Performing a maximum pooling operation on the comprehensive feature data to obtain a feature matrix after channel dimension compression;

[0063] Performing a fully connected layer operation on the feature matrix after the channel dimension is compressed to obtain the initial channel attention weight, and performing a sigmoid activation function on the initial channel attention weight to obtain a channel attention weight vector;

[0064] Multiplying the spatial attention weight map by the channel attention weight vector to obtain a coordinated attention weight;

[0065] Performing modality grouping on the comprehensive feature data to obtain a plurality of modality feature subsets;

[0066] Calculating a query matrix, a key matrix, and a value matrix for each of the modal feature subsets, and calculating a multimodal attention score through matrix multiplication and a softmax function;

[0067] The multimodal attention score is multiplied by the value matrix to obtain a multimodal weighted feature, and the multimodal weighted feature is multiplied by the coordinated attention weight to obtain enhanced feature data.

[0068] Specifically, global average pooling is performed on the comprehensive feature data. By averaging all feature points in each channel in the spatial dimension, the high-dimensional spatial features are compressed into a feature vector of fixed length. Assume that the comprehensive feature data is a three-dimensional tensor ,in is the height, is the width, is the number of channels, and the global average pooling can be expressed as:

[0069] ;

[0070] in, Indicates channel The average value on Indicates the spatial position of the feature map After global average pooling, the feature map containing spatial information is compressed to a length of The feature vector of . Perform a 1x1 convolution operation on the feature vector and perform a linear transformation on the feature vector to obtain the initial weight of the spatial attention. Assume that the weight matrix of the convolution kernel is , then the 1x1 convolution operation can be expressed as:

[0071] ;

[0072] in, is the initial weight of spatial attention obtained by 1x1 convolution. Apply the sigmoid activation function to this initial weight and map its value to the range of 0 to 1 to form the final spatial attention weight map. The expression of the sigmoid function is:

[0073] ;

[0074] Spatial attention weight map after sigmoid activation , indicating the importance of each spatial position. At the same time, the comprehensive feature data is subjected to the maximum pooling operation to obtain the compressed feature matrix on the channel dimension. Maximum pooling reduces the data dimension by taking the maximum value of the local area of ​​each channel to obtain more representative channel features. After maximum pooling, the spatial dimension of the feature matrix is ​​further compressed, reducing the complexity of calculation. The fully connected layer operation is performed on the compressed feature matrix. The features are transformed by learning parameters to generate the initial weights of the channel attention. Assume that the weight matrix of the fully connected layer is , then the initial weight of the channel attention can be expressed as:

[0075] ;

[0076] in, is the feature matrix after maximum pooling, is the initial weight of the channel attention. Similar to the spatial attention mechanism, the sigmoid activation function is used to activate the initial weight of the channel attention to generate the channel attention weight vector This vector indicates the relative importance of each channel in the feature representation. Multiply it with the channel attention weight vector to calculate the coordinated attention weight. The coordinated attention weight can be calculated by element-wise multiplication:

[0077] ;

[0078] in, Indicates spatial location and Channel Coordinated attention weights on . This step combines the two attention mechanisms of space and channel, so that the model can focus on important spatial regions and channel features at the same time, enhancing the expressiveness of features. The comprehensive feature data is grouped by modality to obtain multiple modality feature subsets. These subsets can be divided according to different types of features or different resolutions in breast images, and each modality feature subset represents a specific feature type in the image. For each modality feature subset, the query matrix, key matrix, and value matrix are calculated to learn the association between different modalities through the attention mechanism. Query matrix , key matrix Sum Matrix It can be obtained by linearly transforming each modal feature subset. Assume that the input modal feature subset is , the transformation matrices are , and , then the query, key, and value matrices are calculated as:

[0079] ;

[0080] The query matrix is ​​multiplied with the key matrix through matrix multiplication, and the multimodal attention score is calculated using the softmax function:

[0081] ;

[0082] in, is the dimension of the key matrix. Through this attention mechanism, the model can learn the correlation between different modal features and use the value matrix The multimodal weighted features are obtained by weighted calculation. The multimodal weighted features are multiplied by the coordinated attention weights to generate the final enhanced feature data. Combining multiple attention information from space, channels, and modalities ensures that the model can effectively utilize features from different levels and modalities, thereby performing more accurate classification and evaluation of breast images.

[0083] In a specific embodiment, the process of executing step 103 may specifically include the following steps:

[0084] Grouping the enhanced feature data according to different modalities to obtain multiple modal feature groups;

[0085] Performing a 1x1 convolution operation and a global average pooling operation on each of the modal feature groups to obtain a global descriptor of the modality;

[0086] Performing elemento-wise multiplication on the feature map of the current modality based on the global descriptor of the previous modality to obtain forward flow features, and performing elemento-wise multiplication on the feature map of the current modality based on the global descriptor of the next modality to obtain reverse flow features;

[0087] Performing a weighted sum operation on the forward flow feature and the reverse flow feature to obtain a bidirectional modulation feature, and performing a depth-separable convolution operation on the bidirectional modulation feature to obtain a channel space feature;

[0088] Calculating a target weight based on the channel space feature, and multiplying the target weight with the bidirectional modulation feature to obtain a weighted fusion feature;

[0089] The weighted fusion feature is residually connected with the enhanced feature data to obtain multimodal fusion feature data.

[0090] Specifically, the enhanced feature data is grouped according to different modalities to obtain multiple modal feature groups. The grouping is based on different modal attributes, and each modal feature group represents a specific type of feature information. For example, the enhanced feature data of a breast image may contain multiple different modal features, which may correspond to the density, texture information or edge features of breast tissue respectively. A 1x1 convolution operation is performed on each modal feature group, and the feature map of each modality is compressed and linearly transformed to reduce the feature dimension and integrate the feature information. Assume that a modal feature group is ,in and Represent the height and width of the feature map respectively, Indicates the number of channels of the modal feature group. Through 1x1 convolution, the output feature map They have the same spatial dimensions, but the number of channels can be controlled by the number of convolution kernels. The 1x1 convolution operation can be expressed as:

[0091] ;

[0092] in, is the weight of the 1x1 convolution kernel, is the feature map after convolution, Represents the spatial position in the feature map. Through this operation, the features of each modality can be compressed and key information can be extracted. Global average pooling is performed on the feature map after 1x1 convolution to obtain the global descriptor of the modality. Global average pooling generates a feature vector with a length of the number of channels by averaging the feature map over the entire spatial dimension. Global descriptor It reflects the global feature distribution of the current mode. The operation of global average pooling can be expressed as:

[0093] ;

[0094] in, It is a channel The global average value on represents the overall information of the channel in the modal feature group. The feature map of the current modality is multiplied element by element based on the global descriptor of the previous modality to obtain the forward flow feature. With the current modal feature map Perform element-wise multiplication to apply the global information of the previous mode to each spatial position of the current mode feature map to obtain the forward flow feature

[0095] ;

[0096] This operation ensures the interaction of features between different modalities, so that the features of the current modality can be modulated by the global information of the previous modality. Similarly, the feature map of the current modality is multiplied element by element based on the global descriptor of the next modality to obtain the reverse flow feature. . The global descriptor of the latter mode Perform elementwise multiplication with the current modality feature map:

[0097] ;

[0098] Through the forward and reverse flow features, the features of the current modality not only interact with the previous modality, but also combine with the next modality. The bidirectional flow mechanism can ensure that information is fully propagated and integrated between different modalities. The forward flow features and the reverse flow features are weighted and summed to obtain the bidirectional modulation features. Assume that the weights are assigned to the forward flow features. , assign weights to the reverse flow features , then the bidirectional modulation characteristic The calculation formula is:

[0099] ;

[0100] in, and It can be dynamically adjusted through the learning mechanism to ensure that the fusion of features can adapt to different task requirements. The bidirectional modulation features are subjected to a deep separable convolution operation. The deep separable convolution consists of two steps: performing spatial convolution independently on each channel, and then performing 1x1 convolution on the channel dimension for feature fusion. This operation can reduce the amount of convolution calculation while retaining the independence of channel and spatial information. Through the deep separable convolution operation, the information of the feature map obtained in both the spatial dimension and the channel dimension is further integrated. The target weight is calculated based on the channel spatial feature. By analyzing the importance of channel and spatial information, a weight map is obtained for weighting the bidirectional modulation features. The target weight can be calculated through global pooling and fully connected layers, and normalized using the sigmoid function to ensure that the weight range is between 0 and 1. The target weight is multiplied element-by-element with the bidirectional modulation feature to generate a weighted fusion feature, which highlights the most important part of the feature map for the classification task, while suppressing those areas that contribute less to the final decision. The weighted fusion feature is residually connected with the original enhanced feature data to obtain multimodal fusion feature data. The input features are added to the weighted output features, thereby preserving the original information and enhancing it through the weighted fusion features.

[0101] In a specific embodiment, the process of executing step 104 may specifically include the following steps:

[0102] Performing superpixel segmentation on the multimodal fusion feature data, taking each superpixel region as a node in a graph, and obtaining a graph structure node set;

[0103] Based on the k-nearest neighbor relationship and Gaussian similarity function of the nodes of the graph structure, an adjacency matrix and a degree matrix of the graph are constructed, and a normalized adjacency matrix is ​​obtained through matrix operation to form an initial graph structure;

[0104] A semi-supervised local feature preserving densely connected context-aware graph convolutional network is constructed, wherein the graph convolutional network includes 5 graph convolutional layers, each of which includes a graph convolution operation, batch normalization, and a LeakyReLU activation function, wherein the output of each layer is obtained by performing a series of matrix operations on the input features, the adjacency matrix, and the learnable weight matrix;

[0105] Add dense connections between every two adjacent graph convolutional layers, concatenate the output features of the current layer with the input features of all subsequent layers to obtain enhanced features; insert a layer attention module after the 2nd and 4th graph convolutional layers, the layer attention module includes two fully connected layers and a softmax activation function, and obtain attention-enhanced graph convolutional features by calculating the attention weights and multiplying them with the enhanced features;

[0106] Perform global average pooling on the graph convolution features output by the 5th layer to obtain a graph-level feature representation with a fixed dimension;

[0107] The graph-level feature representation is input into two fully connected layers, the first layer contains 256 neurons, and the number of neurons in the second layer is the same as the number of categories. Finally, the first classification prediction result of the labeled sample is obtained through the softmax function;

[0108] Performing data augmentation operations on the unlabeled samples to obtain augmented samples of two perspectives, inputting the augmented samples of the two perspectives into a graph convolutional network to obtain feature representations of the two perspectives;

[0109] By minimizing the cross entropy loss of labeled samples and the consistency regularization loss of unlabeled samples, where the consistency regularization loss is obtained by calculating the Euclidean distance between the two perspective feature representations, the weighted sum of the two losses is used as the total loss function, and the semi-supervised local feature preserving densely connected context-aware graph convolutional network is end-to-end trained to obtain a graph convolution classification model.

[0110] Specifically, the multimodal fusion feature data is segmented into superpixels, and the features in the breast image are divided into multiple local regions, each of which is a superpixel. These superpixels can aggregate similar pixels in the image to form several small segments with similar features. Each superpixel region is regarded as a node in the graph, and the center position and feature information of the superpixel are used as the attributes of the node to obtain a set of graph structure nodes. Assume that the breast image is segmented into superpixel regions, the node set can be expressed as , where each node Represents a superpixel region in an image. The neighbor relationship and Gaussian similarity function are used to construct the adjacency matrix and degree matrix of the graph. ,pass The nearest neighbor algorithm finds the closest neighbor nodes to build the connectivity between nodes. On this basis, the Gaussian similarity function is used to measure the similarity between adjacent nodes. The expression of the Gaussian similarity function is:

[0111] ;

[0112] in, Representation Node and nodes The similarity between For Node The characteristic vector of is the parameter that controls the similarity attenuation. Similarity matrix Each element of Represents the similarity strength between nodes. Based on the similarity matrix, the degree matrix is ​​calculated , whose diagonal elements are the degrees of each node, that is, By the adjacency matrix Perform normalization to obtain the normalized adjacency matrix , and its calculation formula is:

[0113] ;

[0114] The normalized adjacency matrix is ​​used in the graph convolution operation to ensure that the feature contribution between different nodes can be balanced during information propagation to form the initial graph structure. A semi-supervised local feature preserving densely connected context-aware graph convolution network is constructed. The network consists of 5 layers of graph convolution layers, each of which contains graph convolution operations, batch normalization, and LeakyReLU activation functions. The graph convolution operation is performed through the adjacency matrix and a learnable weight matrix The basic formula for graph convolution is to transform the input node features:

[0115] ;

[0116] in, Indicates The input feature matrix of the layer, For the The weight matrix of the layer, is a nonlinear activation function (such as LeakyReLU). Through this matrix operation, the features output by each graph convolution layer are updated and propagated according to the features and weights of the adjacent nodes, and the local and global information in the graph structure are gradually learned. Dense connections are added between every two adjacent graph convolution layers. The function of dense connections is to splice the output features of the current layer with the input features of the subsequent layer, so that each layer can directly access the feature representation of the previous layer to avoid information loss in the deep network. Dense connections can significantly improve the propagation efficiency of features, reduce the gradient vanishing problem, and obtain enhanced features. In order to improve the feature expression ability of the graph convolution network, a layer attention module is inserted after the 2nd and 4th graph convolution layers. The layer attention module learns the weights of the output features of each layer through two fully connected layers and softmax activation functions. By calculating the attention weights and multiplying them with the enhanced features, the model can adaptively focus on the features at different levels and improve the model's ability to recognize key features. On the features output by the 5th graph convolution layer, a global average pooling operation is performed to aggregate the features of all nodes into a fixed-dimensional graph-level feature representation. The purpose of global average pooling is to compress the spatial information of nodes and obtain the global features of the entire graph structure. The graph-level features are input into two fully connected layers. The first fully connected layer contains 256 neurons, and the number of neurons in the second fully connected layer is equal to the number of categories. The classification probability of labeled samples is output through the softmax function. The formula of the softmax function is:

[0117] ;

[0118] in, Indicates category The probability of is the output value of the fully connected layer, is the number of categories. For unlabeled samples, data augmentation operations are performed to generate enhanced samples from two perspectives. Data augmentation can include transformations such as rotation, scaling, or color adjustment of the image. Through the augmentation operation, enhanced samples from two different perspectives are obtained. The enhanced samples from these two perspectives are input into the graph convolutional network to obtain feature representations from the two perspectives. In order to ensure that the model can maintain consistency on unlabeled data, the difference between the two perspective feature representations is measured by calculating the Euclidean distance:

[0119] ;

[0120] in, and They are the feature representations from two perspectives respectively. is their Euclidean distance. In order to encourage the model to maintain consistency in feature representation under different viewpoints, this distance is used as part of the consistency regularization loss. The model is optimized by minimizing the cross entropy loss of labeled samples and the consistency regularization loss of unlabeled samples. The cross entropy loss is used to supervise the classification results of labeled samples, while the consistency regularization loss ensures that the feature representation of unlabeled samples under different viewpoints is consistent. The total loss function can be expressed as:

[0121] ;

[0122] in, is the cross entropy loss, is the consistency regularization loss, To balance the hyperparameters of the two. Through end-to-end training, the semi-supervised local feature preserving densely connected context-aware graph convolutional network can be jointly optimized on labeled and unlabeled samples, and finally a graph convolution classification model is obtained. This model can not only process the multimodal features of breast images, but also effectively capture the local and global information of the image through graph convolution operations and attention mechanisms, thereby improving classification accuracy and robustness.

[0123] In a specific embodiment, the execution step adds a dense connection between every two adjacent graph convolution layers, concatenates the output features of the current layer with the input features of all subsequent layers to obtain enhanced features; inserts a layer attention module after the second and fourth graph convolution layers, the layer attention module includes two fully connected layers and a softmax activation function, and obtains the attention-enhanced graph convolution features by calculating the attention weights and multiplying them with the enhanced features. The process can specifically include the following steps:

[0124] Transform the output features of the i-th graph convolutional layer to obtain a feature map that matches the input dimensions of the subsequent layers;

[0125] The feature map is concatenated with the input features of the i+1th layer to the 5th layer in the channel dimension to obtain an initial enhanced feature;

[0126] Performing a 1x1 convolution operation on the initial enhanced features to obtain compressed enhanced features;

[0127] Inputting the compressed enhanced features into a first fully connected layer to obtain an intermediate feature representation, wherein the number of neurons in the first fully connected layer is half of the dimension of the compressed enhanced features;

[0128] Performing a ReLU activation function operation on the intermediate feature representation to obtain the intermediate feature after nonlinear transformation;

[0129] Inputting the intermediate features after the nonlinear transformation into a second fully connected layer to obtain original attention weights, wherein the number of neurons in the second fully connected layer is the same as the number of channels of the compressed enhanced features;

[0130] Performing a softmax function operation on the original attention weight to obtain a normalized attention weight;

[0131] The normalized attention weight is multiplied channel by channel with the compressed enhanced feature to obtain an attention-enhanced graph convolution feature.

[0132] Specifically, the output features of the i-th graph convolutional layer are transformed to obtain a feature map that matches the input dimensions of the subsequent layers. The output features of the layer are ,in and are the height and width of the feature map, respectively. is the number of channels. In order to match the subsequent layers, a 1x1 convolution operation is used to adjust the number of channels of the feature map to match the input dimension of the subsequent layers. Keep it consistent. The role of 1x1 convolution is to perform a linear transformation on each spatial position without changing the spatial dimension, only adjusting the number of channels. Its calculation formula is:

[0133] ;

[0134] in, represents the transformed feature map, is the weight matrix of the 1x1 convolution kernel, Indicates The layer output feature map is at the spatial position The value on the feature map after dimension transformation With The input features from layer 1 to layer 5 are concatenated in the channel dimension. Assume that the input features of the subsequent layers are , these features have the number of channels By concatenating these features in the channel dimension, we get the initial enhanced features This concatenation operation can be achieved by stacking the features of each layer, ensuring that the subsequent layers can access the feature information of the current layer and deeper layers at the same time. In order to reduce the amount of calculation and compress the channel dimension of the feature, the initial enhanced feature Perform a 1x1 convolution operation. This operation compresses the channel dimension of the feature to a smaller value while retaining important information. The calculation of this convolution operation is similar to the aforementioned dimensional transformation. A 1x1 convolution kernel is used to perform a linear transformation on each spatial position to obtain the compressed enhanced feature. ,in is the number of channels after compression, which is usually set to a smaller value of the original number of channels to reduce computational complexity. The compressed enhanced features are input to the first fully connected layer to obtain an intermediate feature representation. The function of the first fully connected layer is to flatten the two-dimensional feature map into a one-dimensional feature vector and perform a linear transformation on it to obtain a new feature representation. Assume that the number of neurons in the first fully connected layer is the number of channels after compression. The calculation formula of the fully connected layer is:

[0135] ;

[0136] in, is the intermediate feature representation, is the weight matrix of the first fully connected layer, is the bias term. For the intermediate feature representation Perform ReLU activation function operation. ReLU is a commonly used nonlinear activation function, and its formula is:

[0137] ;

[0138] Through ReLU activation, the negative values ​​in the intermediate feature representation are set to zero, retaining the positive information, thereby enhancing the model's ability to express nonlinear features. After ReLU activation, the intermediate features after nonlinear transformation are obtained. The intermediate features after nonlinear transformation are input to the second fully connected layer. The number of neurons in the second fully connected layer is equal to the number of channels of the enhanced features after compression. , which is used to generate the original attention weight The calculation formula of the fully connected layer is:

[0139] ;

[0140] in, represents the original attention weight, is the weight matrix of the second fully connected layer, is the bias term. For the original attention weight Perform softmax function operation to obtain normalized attention weights The function of the softmax function is to normalize the original weights into a probability distribution, and its calculation formula is:

[0141] ;

[0142] in, For Channel The normalized attention weights on , ensure that the sum of the weights of all channels is 1. Through softmax activation, the model can assign an attention weight to each channel, indicating the importance of the channel in feature expression. Enhanced features after compression Perform channel-by-channel multiplication to obtain the attention-enhanced graph convolution features. This operation weights the features channel by channel to ensure that the model can pay more attention to those channels with high weights, thereby improving the expressiveness of the features. The calculation formula for channel-by-channel multiplication is:

[0143] ;

[0144] in, Indicates the spatial position of the feature map after attention weighting and Channel The value on .

[0145] In a specific embodiment, the process of executing step 105 may specifically include the following steps:

[0146] Performing pyramid scale decomposition on the multimodal fusion feature data to obtain feature maps of three different scales, which are respectively used as inputs of three parallel branches;

[0147] Apply 5 residual blocks to the input feature map of each branch. Each residual block contains two 3x3 convolutional layers, a batch normalization layer, a ReLU activation function, and a short-circuit connection to obtain the initial residual features.

[0148] Inserting a spatial attention module after the second and fourth residual blocks of each branch, weighting the feature importance of the spatial dimension of the initial residual features, and obtaining spatial attention enhanced features;

[0149] Applying a channel attention module to the spatial attention enhancement feature, calculating the channel weight through global average pooling, a fully connected layer and a sigmoid activation function, and multiplying it with the feature map to obtain a dual attention enhancement feature;

[0150] A reconstruction module is added after the last residual block of each branch. The reconstruction module includes three transposed convolutional layers, each of which is followed by batch normalization and ReLU activation function, which is used to upsample the feature map to the original input size to obtain a reconstructed feature map.

[0151] Calculate the mean square error between the reconstructed feature map and the original input image to obtain the reconstruction loss, and add the reconstruction loss to the classification loss as the overall optimization target of the network;

[0152] Adaptive feature fusion is performed on the dual attention enhancement features of the three branches, specifically including: calculating the Euclidean distance between each branch feature and the ideal feature, obtaining the fusion weight based on the distance ratio, performing weighted summation, and obtaining the training multi-scale fusion feature;

[0153] The trained multi-scale fusion features are input into the fully connected layer and the softmax layer to obtain the second classification prediction results of the labeled samples, and the reconstruction loss and the classification cross entropy loss are combined to perform end-to-end training on the entire network through the back propagation algorithm to obtain a multi-scale feature fusion model.

[0154] Specifically, the multimodal fusion feature data is decomposed into pyramid scales. Pyramid scale decomposition downsamples the input feature map to obtain feature maps of different resolutions, capturing the feature information of the image at different scales. Assume that the input multimodal fusion feature data is , through pyramid decomposition, three feature maps of different scales can be obtained: , and . These feature maps serve as inputs to three parallel branches respectively, providing a basis for subsequent multi-scale feature processing. For the input feature map of each branch, 5 residual blocks are applied for processing. Each residual block contains two 3x3 convolutional layers, a batch normalization layer, and a ReLU activation function. The design of the residual block enables the network to maintain gradient flow while increasing in depth, avoiding the problem of gradient vanishing. Each convolution layer in the residual block convolves the input feature map with a 3x3 convolution kernel. The convolved features are normalized through a batch normalization layer to ensure the stability of the feature distribution during training, and then nonlinearity is introduced through the ReLU activation function. In order to avoid information loss, the residual block introduces a short-circuit connection, which directly adds the input features to the output features to form the initial residual features. The output of the residual block can be expressed as:

[0155] ;

[0156] in, is the output feature of the residual block, is the input feature, Conv represents the convolution operation, BN is batch normalization, and ReLU is the activation function. Through the processing of 5 residual blocks, rich initial residual features are obtained. The spatial attention module is inserted into the 2nd and 4th residual blocks of each branch to enhance the expressiveness of the features in the spatial dimension. The spatial attention module enables the network to pay more attention to important areas in the image by performing weighted operations on the spatial dimensions of the feature map. Global average pooling and global maximum pooling are performed on the residual features to generate two different spatial attention maps. These attention maps are combined together through convolution operations and mapped to the range of 0 to 1 through the sigmoid function to obtain the final spatial attention weight. This weight is multiplied element-by-element with the original feature map to obtain the spatial attention enhanced feature:

[0157] ;

[0158] in, Indicates spatial location The spatial attention weights on is the spatial attention enhancement feature. After obtaining the spatial attention enhancement feature, the channel attention module is applied. The channel attention module weights the channel dimension according to the importance of each channel in the feature representation. Global average pooling is performed on each channel to obtain the channel-level feature vector, and then the weight of each channel is calculated through the fully connected layer and normalized by the sigmoid function. The final channel weight is multiplied by the original feature map channel by channel to obtain the dual attention enhancement feature:

[0159] ;

[0160] in, For Channel The weight of Enhance features for dual attention. A reconstruction module is added to the last residual block of each branch to upsample the downsampled feature map back to the original input size for use in subsequent feature fusion and loss calculation. The reconstruction module contains 3 transposed convolution layers, each of which is followed by batch normalization and ReLU activation function. Transposed convolution (also known as deconvolution) is used to gradually restore the spatial resolution of the feature map. After processing by the reconstruction module, a reconstructed feature map with the same size as the original input image is generated. In order to evaluate the reconstruction effect, the mean square error (MSE) between the reconstructed feature map and the original input image is calculated. The calculation formula for the mean square error is:

[0161] ;

[0162] in, is the pixel value of the original image, To reconstruct the pixel value of the feature map, is the total number of pixels. Reconstruction loss It is used to measure the performance of the model in the reconstruction task. The final loss function is the weighted sum of the reconstruction loss and the classification loss. The classification loss usually adopts the cross entropy loss, which is expressed as:

[0163] ;

[0164] in, is the cross entropy loss, is a weight hyperparameter used to balance the impact of reconstruction and classification losses. Adaptive feature fusion is performed on the dual attention enhancement features of the three branches. The Euclidean distance between each branch feature and the ideal feature is calculated. The ideal feature can be a predefined feature benchmark. The formula for the Euclidean distance is:

[0165] ;

[0166] in, Indicates The dual attention enhancement feature of each branch, is the ideal feature. Based on the distance between each branch and the ideal feature, the fusion weight of each branch is calculated and normalized according to the ratio of the distance. The features of the three branches are weighted and summed according to these weights to obtain the final multi-scale fusion feature:

[0167] ;

[0168] in, For the The trained multi-scale fusion features are input into the fully connected layer and the softmax layer to generate the second classification prediction results of the labeled samples. The final loss function of the model combines the reconstruction loss and the classification loss. The whole network is trained end-to-end through the back-propagation algorithm, and the weights are gradually optimized to finally obtain the multi-scale feature fusion model.

[0169] In a specific embodiment, the execution step performs adaptive feature fusion on the dual attention enhancement features of the three branches, specifically including: calculating the Euclidean distance between each branch feature and the ideal feature, obtaining the fusion weight based on the distance ratio relationship, performing weighted summation, and obtaining the process of training multi-scale fusion features can specifically include the following steps:

[0170] Perform a global average pooling operation on the dual attention enhanced features of each branch to obtain a feature vector of fixed dimension as the global feature representation of the branch;

[0171] Construct an ideal feature vector whose element value is 1 and whose dimension is the same as the global feature representation, as a reference for feature fusion;

[0172] Calculate the Euclidean distance between the global feature representation of each branch and the ideal feature vector, and obtain three distance values, which correspond to the degree of feature difference of the three branches respectively;

[0173] The three distance values ​​obtained are normalized, and each distance value is divided by the sum of the three distance values ​​to obtain a normalized distance ratio;

[0174] Based on the normalized distance ratio, calculate the fusion weight of each branch, subtract the normalized distance ratio of the branch from 1, and then normalize again to ensure that the sum of the weights of the three branches is 1;

[0175] The calculated fusion weights are weighted and multiplied with the dual attention enhancement features of the corresponding branches, and the three weighted features are added element by element to obtain the training multi-scale fusion features.

[0176] Specifically, a global average pooling operation is performed on the dual attention enhanced features of each branch. By averaging all feature points of each channel in the spatial dimension, the high-dimensional spatial features are compressed into a feature vector of fixed length. Assume that The dual attention enhancement feature of each branch is ,in and Represent the height and width of the feature map respectively, Represents the number of channels. Through the global average pooling operation, the output feature vector Representing the global information of each channel, it can be expressed as:

[0177] ;

[0178] in, Indicates channel The global average value on reflects the average response of the channel in the entire feature map. The global feature vector As the The global feature representation of each branch is used for the subsequent feature fusion step. Construct an ideal feature vector , each element value is 1, and the dimension is the same as the global feature representation The ideal feature vector is used as a reference for feature fusion to measure the degree of feature difference of each branch. The ideal feature vector represents the ideal state of the feature, where the average response of each channel is the maximum value. In order to quantify the difference between the global feature representation of each branch and the ideal feature vector, the Euclidean distance is used for calculation. The Euclidean distance is used to measure the distance between two vectors in the feature space. The larger the distance, the greater the difference between the feature of the branch and the ideal feature. branches, whose global features represent With the ideal eigenvector The Euclidean distance between It can be calculated as:

[0179] ;

[0180] in, For the The Euclidean distance of the branches, For branch No. The global eigenvalue of the channels, is the value of the corresponding channel in the ideal feature vector (all 1 in this example). By calculating the Euclidean distance, three distance values ​​are obtained, corresponding to the degree of feature difference of the three branches. Normalize. Convert these distance values ​​to ratios so that they can be weighted. Divide each distance value by the sum of the three distance values ​​to ensure that their sum is 1. Normalized distance ratio The calculation formula is:

[0181] ;

[0182] in, For the The normalized distance ratio of the branches. Three distance ratios The sum of is 1. Based on the normalized distance ratio , calculate the fusion weight of each branch. In order to ensure that the weight can reflect the quality of the feature, the fusion weight is calculated by subtracting the normalized distance ratio from 1, that is:

[0183] ;

[0184] The branches whose features are closer to the ideal features will get larger weights, while the branches with larger differences will get smaller weights. These weights are normalized to get the final fusion weights

[0185] ;

[0186] in, After normalization, The final fusion weight of the branches is calculated to ensure that the sum of the weights is 1. They are weighted and multiplied with the dual attention enhancement features of the corresponding branches. Assume that The dual attention enhancement feature of each branch is , then the weighted features can be expressed as:

[0187] ;

[0188] in, Represents the weighted The three weighted features are added element by element to obtain the final training multi-scale fusion feature. The features of different scales are fused together to form a comprehensive feature representation that captures the key information of the image at different scales and modalities. The calculation formula for the fusion feature is:

[0189] ;

[0190] Through this weighted summation operation, the final fusion feature It contains multi-scale information from each branch, and by adjusting the fusion weight, it effectively balances the influence of different branch features. In a specific embodiment, the process of executing step 106 may specifically include the following steps:

[0191] Performing preprocessing operations on real-time breast image data to obtain preprocessed multi-scale image data;

[0192] Inputting the preprocessed multi-scale image data into the graph structure generation layer of the graph convolution classification model, and obtaining a graph structure representation through superpixel segmentation and graph construction operations;

[0193] The graph structure representation is sequentially passed through 5 layers of graph convolution layers of the graph convolution classification model, each layer includes graph convolution operation, batch normalization and LeakyReLU activation, and a layer attention module is applied after the 2nd and 4th layers to obtain real-time graph convolution features;

[0194] The real-time graph convolution features are input into the global average pooling layer for global average pooling operation, and the features of all nodes are aggregated into a graph-level feature representation of fixed dimensions. The graph-level feature representation is input into two fully connected layers. The first fully connected layer contains 256 neurons, and the number of neurons in the second fully connected layer is the same as the number of categories of breast image classification. The probability distribution of each category is calculated by the softmax function to obtain the first classification probability distribution of the graph convolution classification model;

[0195] Inputting the preprocessed multi-scale image data into three parallel branches of a multi-scale feature fusion model, each branch comprising 5 residual blocks, a spatial attention module and a channel attention module, to obtain real-time dual attention enhanced features;

[0196] Adaptively fusing the real-time dual attention enhancement features, and obtaining real-time multi-scale fusion features by calculating Euclidean distance and weight distribution;

[0197] Inputting the real-time multi-scale fusion features into the fully connected layer and the softmax layer to obtain a second classification probability distribution of the multi-scale feature fusion model;

[0198] A weighted average is performed on the first classification probability distribution and the second classification probability distribution to obtain a classification result of the target breast image.

[0199] Specifically, the image data is preprocessed to decompose the input breast image into multiple scales to generate image data of different resolutions and capture the detailed information of the image at different scales. , through pyramid scale decomposition, generate image data of three different scales , ,and The preprocessed multi-scale image data is input into the graph structure generation layer of the graph convolution classification model. The graph structure generation layer structures the breast image data into graph nodes and generates a graph structure through superpixel segmentation and graph construction operations. The purpose of superpixel segmentation is to divide the image into superpixel regions with similar features, each region represents a node. Assume that the image is segmented into superpixel regions, each superpixel region becomes a node in the graph structure , the attributes of each node include its location information and feature vector. Through the graph construction operation, the adjacency matrix of the graph is generated based on the spatial and feature similarities between nodes. , each element of the adjacency matrix Representation Node and nodes In order to ensure the stability of the graph structure, the adjacency matrix is ​​usually normalized to generate a normalized adjacency matrix. , and its calculation formula is:

[0200]

[0201] in, is the degree matrix, whose diagonal elements Representation Node The graph structure is represented by five graph convolution layers of the graph convolution classification model. Each graph convolution layer contains graph convolution operation, batch normalization and LeakyReLU activation function. The graph convolution operation is to pass the features of the graph nodes through the graph adjacency matrix Propagate, the output features of each layer Calculated by the following formula:

[0202] ;

[0203] in, For the The input features of the layer, is the learnable weight matrix, is the LeakyReLU activation function. The layer attention module is introduced in the 2nd and 4th layers. The layer attention module enhances the focus on key features by learning the importance of the output features of each layer and assigning weights to these features. This module usually calculates the attention weights through two fully connected layers and the softmax function , and then weighted multiply it with the feature to obtain the attention-enhanced graph convolution feature. Perform a global average pooling operation on the obtained graph convolution feature to aggregate the features of all nodes into a graph-level feature representation of a fixed dimension. The graph-level feature representation is input into two fully connected layers. The first fully connected layer usually contains 256 neurons, and the number of neurons in the second fully connected layer is the same as the number of categories for breast image classification. The probability distribution of each category is calculated by the softmax function to obtain the first classification probability distribution of the graph convolution classification model:

[0204] ;

[0205] in, For the The score of the class, is the number of categories. At the same time, the preprocessed multi-scale image data is input into the multi-scale feature fusion model. The three parallel branches of the multi-scale feature fusion model process image data of different scales respectively. Each branch contains 5 residual blocks, each of which consists of two 3x3 convolutional layers, a batch normalization layer and a ReLU activation function, and also contains short-circuit connections to ensure that information can flow smoothly in the deep network. The residual block of each branch is used to extract local features of images of different scales, and further enhance the feature expression through the spatial attention module and the channel attention module. The spatial attention module focuses on important spatial areas in the feature map, while the channel attention module strengthens the importance of different channels by calculating the weights of each channel. The dual attention enhanced features of the three branches are adaptively fused. The Euclidean distance between the features of each branch and the ideal feature vector is calculated to measure the feature difference. Assume that The global feature of each branch is expressed as , the ideal eigenvector is , then the Euclidean distance The calculation formula is:

[0206] ;

[0207] The three distance values ​​obtained are normalized to ensure that the fusion weights can be reasonably distributed. For each branch, the normalized distance ratio is used. Calculate fusion weights

[0208] ;

[0209] These fusion weights are used to weight the features of different branches, and the weighted features are added element by element to obtain the final multi-scale fusion features.

[0210] ;

[0211] The multi-scale fusion features are input into the fully connected layer and the softmax layer to generate the second classification probability distribution of the multi-scale feature fusion model. Second classification probability distribution with multi-scale feature fusion model Perform weighted average. Assume is the weight of the first classification probability, is the weight of the second classification probability, then the final classification result It can be expressed as:

[0212] ;

[0213] Through this weighted average method, the classification results of the two models are combined to obtain the final classification result of the target breast image.

[0214] The above describes the breast image classification and evaluation method based on deep learning in the embodiment of the present invention. The following describes the breast image classification and evaluation system based on deep learning in the embodiment of the present invention. Figure 2 In one embodiment of the present invention, a breast image classification and evaluation system based on deep learning includes:

[0215] A preprocessing module 201 is used to preprocess the sample breast image data to obtain comprehensive feature data;

[0216] A feature enhancement module 202, configured to perform feature enhancement based on the comprehensive feature data by using a coordinated attention module and a multimodal attention module to obtain enhanced feature data;

[0217] A feature fusion module 203 is used to perform multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fused feature data;

[0218] A first construction module 204 is used to construct and train a semi-supervised local feature preserving densely connected context-aware graph convolutional network based on the multimodal fusion feature data to obtain a graph convolution classification model;

[0219] The second construction module 205 is used to construct and train a multi-branch residual attention reconstruction network according to the multimodal fusion feature data to obtain a multi-scale feature fusion model;

[0220] The integration module 206 is used to perform classification evaluation and prediction result integration on the real-time breast image data through the graph convolution classification model and the multi-scale feature fusion model to obtain the target breast image classification result.

[0221] Through the synergy of the above components, by preprocessing and pyramid decomposition of sample breast image data, combined with coordinated attention module and multimodal attention module, the effective extraction and enhancement of multimodal and multiscale features are achieved. It can comprehensively capture various features of breast images and improve the accuracy and robustness of classification. By constructing a semi-supervised local feature-preserving densely connected context-aware graph convolutional network, limited labeled samples and a large amount of unlabeled data are effectively utilized. This strategy not only improves the generalization ability of the model, but also reduces the dependence on a large amount of labeled data and reduces the cost of manual annotation. Using superpixel segmentation and graph convolutional network, this method can effectively capture the spatial structural information of breast tissue and make full use of the correlation between local features. This feature extraction method based on graph structure can better characterize the complex structure of breast tissue and improve the accuracy of classification. By introducing a multi-level attention mechanism in the graph convolutional network and the multi-branch residual network, it can adaptively focus on the key areas and important features in the image and improve the discrimination ability of the model. By constructing a multi-branch residual attention reconstruction network and designing an adaptive feature fusion strategy, the effective integration of features of different scales is achieved. This fusion method can make full use of multi-scale information and improve the model's ability to identify lesions of different sizes. By combining the prediction results of the graph convolution classification model and the multi-scale feature fusion model, model integration is achieved, further improving the accuracy and stability of classification. By adopting an end-to-end training method, by jointly optimizing feature extraction, attention mechanism, classification tasks and reconstruction tasks, the coordinated optimization of each module is achieved, improving the performance of the overall model.

[0222] above Figure 2 The breast image classification and evaluation system based on deep learning in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The breast image classification and evaluation device based on deep learning in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0223] Figure 3It is a structural schematic diagram of a deep learning-based breast image classification and evaluation device provided by an embodiment of the present invention. The deep learning-based breast image classification and evaluation device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more massive storage device terminals) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the deep learning-based breast image classification and evaluation device 300. Furthermore, the processor 310 may be configured to communicate with the storage medium 330, and execute a series of instruction operations in the storage medium 330 on the deep learning-based breast image classification and evaluation device 300 to implement the steps of the deep learning-based breast image classification and evaluation method.

[0224] The deep learning-based breast image classification and evaluation device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 3 The structure of the deep learning-based breast image classification and evaluation device shown does not constitute a limitation on the deep learning-based breast image classification and evaluation device provided by the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0225] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the deep learning-based breast image classification and evaluation method.

[0226] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0227] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0228] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A breast image classification and evaluation method based on deep learning, characterized in that: The method comprises: Preprocessing the sample breast image data to obtain comprehensive feature data; Based on the comprehensive feature data, feature enhancement is performed by coordinating the attention module and the multimodal attention module to obtain enhanced feature data; Performing multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fused feature data; Based on the multimodal fusion feature data, a semi-supervised local feature preserving densely connected context-aware graph convolutional network is constructed and trained to obtain a graph convolution classification model; According to the multimodal fusion feature data, a multi-branch residual attention reconstruction network is constructed and trained to obtain a multi-scale feature fusion model; The real-time breast image data is classified, evaluated and predicted by the graph convolution classification model and the multi-scale feature fusion model to obtain a target breast image classification result.

2. The breast image classification and evaluation method based on deep learning according to claim 1, characterized in that: The preprocessing of the sample breast image data to obtain comprehensive feature data includes: Performing standardization processing on sample breast image data to obtain a standardized image, and performing pyramid decomposition on the standardized image to obtain a multi-scale image; Performing convolution operations on the multi-scale images respectively to obtain initial feature maps, and performing batch normalization processing on the initial feature maps to obtain normalized feature maps; Performing a ReLU activation function operation on the normalized feature map to obtain an activated feature map, and performing a maximum pooling operation on the activated feature map to obtain a downsampled feature map; Performing residual connection on the downsampled feature map to obtain a residual feature map, and performing attention feature weighting processing on the residual feature map to obtain attention weighted feature maps of different modalities and different scales; The attention-weighted feature maps of different modalities and different scales are concatenated in the channel dimension to obtain comprehensive feature data.

3. The breast image classification and evaluation method based on deep learning according to claim 1, characterized in that: Based on the comprehensive feature data, feature enhancement is performed by coordinating the attention module and the multimodal attention module to obtain enhanced feature data, including: Performing a global average pooling operation on the comprehensive feature data to obtain a feature vector after spatial dimension compression; Performing a 1x1 convolution operation on the feature vector to obtain an initial spatial attention weight, and performing a sigmoid activation function on the initial spatial attention weight to obtain a spatial attention weight map; Performing a maximum pooling operation on the comprehensive feature data to obtain a feature matrix after channel dimension compression; Performing a fully connected layer operation on the feature matrix after the channel dimension is compressed to obtain the initial channel attention weight, and performing a sigmoid activation function on the initial channel attention weight to obtain a channel attention weight vector; Multiplying the spatial attention weight map by the channel attention weight vector to obtain a coordinated attention weight; Performing modality grouping on the comprehensive feature data to obtain a plurality of modality feature subsets; Calculating a query matrix, a key matrix, and a value matrix for each of the modal feature subsets, and calculating a multimodal attention score through matrix multiplication and a softmax function; The multimodal attention score is multiplied by the value matrix to obtain a multimodal weighted feature, and the multimodal weighted feature is multiplied by the coordinated attention weight to obtain enhanced feature data.

4. The breast image classification and evaluation method based on deep learning according to claim 1, characterized in that: The step of performing multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fused feature data includes: Grouping the enhanced feature data according to different modalities to obtain a plurality of modal feature groups; Performing a 1x1 convolution operation and a global average pooling operation on each of the modal feature groups to obtain a global descriptor of the modality; Performing elemento-wise multiplication on the feature map of the current modality based on the global descriptor of the previous modality to obtain forward flow features, and performing elemento-wise multiplication on the feature map of the current modality based on the global descriptor of the next modality to obtain reverse flow features; Performing a weighted sum operation on the forward flow feature and the reverse flow feature to obtain a bidirectional modulation feature, and performing a depth-separable convolution operation on the bidirectional modulation feature to obtain a channel space feature; Calculating a target weight based on the channel space feature, and multiplying the target weight with the bidirectional modulation feature to obtain a weighted fusion feature; The weighted fusion feature is residually connected with the enhanced feature data to obtain multimodal fusion feature data.

5. The breast image classification and evaluation method based on deep learning according to claim 1, characterized in that: Based on the multimodal fusion feature data, a semi-supervised local feature-preserving densely connected context-aware graph convolutional network is constructed and trained to obtain a graph convolution classification model, including: Performing superpixel segmentation on the multimodal fusion feature data, taking each superpixel region as a node in a graph, and obtaining a graph structure node set; Based on the k-nearest neighbor relationship and Gaussian similarity function of the graph structure node set, an adjacency matrix and a degree matrix of the graph are constructed, and a normalized adjacency matrix is ​​obtained through matrix operation to form an initial graph structure; A semi-supervised local feature preserving densely connected context-aware graph convolutional network is constructed, wherein the graph convolutional network includes 5 graph convolutional layers, each of which includes a graph convolution operation, batch normalization, and a LeakyReLU activation function, wherein the output of each layer is obtained by performing a series of matrix operations on input features, an adjacency matrix, and a learnable weight matrix; Add dense connections between every two adjacent graph convolutional layers, concatenate the output features of the current layer with the input features of all subsequent layers to obtain enhanced features; insert a layer attention module after the 2nd and 4th graph convolutional layers, the layer attention module includes two fully connected layers and a softmax activation function, and obtain attention-enhanced graph convolutional features by calculating the attention weights and multiplying them with the enhanced features; Perform global average pooling on the graph convolution features output by the 5th layer to obtain a graph-level feature representation with a fixed dimension; The graph-level feature representation is input into two fully connected layers, the first layer contains 256 neurons, and the number of neurons in the second layer is the same as the number of categories. Finally, the first classification prediction result of the labeled sample is obtained through the softmax function; Performing data augmentation operations on the unlabeled samples to obtain augmented samples of two perspectives, inputting the augmented samples of the two perspectives into a graph convolutional network to obtain feature representations of the two perspectives; By minimizing the cross entropy loss of labeled samples and the consistency regularization loss of unlabeled samples, where the consistency regularization loss is obtained by calculating the Euclidean distance between the two perspective feature representations, the weighted sum of the two losses is used as the total loss function, and the semi-supervised local feature preserving densely connected context-aware graph convolutional network is end-to-end trained to obtain a graph convolution classification model.

6. The breast image classification and evaluation method based on deep learning according to claim 5 is characterized in that: The method adds a dense connection between every two adjacent graph convolutional layers, concatenates the output features of the current layer with the input features of all subsequent layers, and obtains enhanced features; A layer attention module is inserted after the 2nd and 4th graph convolution layers. The layer attention module includes two fully connected layers and a softmax activation function. By calculating the attention weight and multiplying it with the enhanced feature, the attention-enhanced graph convolution feature is obtained, including: Transform the output features of the i-th graph convolutional layer to obtain a feature map that matches the input dimensions of the subsequent layers; The feature map is concatenated with the input features of the i+1th layer to the 5th layer in the channel dimension to obtain an initial enhanced feature; Performing a 1x1 convolution operation on the initial enhanced features to obtain compressed enhanced features; Inputting the compressed enhanced features into a first fully connected layer to obtain an intermediate feature representation, wherein the number of neurons in the first fully connected layer is half of the dimension of the compressed enhanced features; Performing a ReLU activation function operation on the intermediate feature representation to obtain the intermediate feature after nonlinear transformation; Inputting the intermediate features after the nonlinear transformation into a second fully connected layer to obtain original attention weights, wherein the number of neurons in the second fully connected layer is the same as the number of channels of the compressed enhanced features; Performing a softmax function operation on the original attention weight to obtain a normalized attention weight; The normalized attention weight is multiplied channel by channel with the compressed enhanced feature to obtain an attention-enhanced graph convolution feature.

7. The breast image classification and evaluation method based on deep learning according to claim 1, characterized in that: According to the multimodal fusion feature data, a multi-branch residual attention reconstruction network is constructed and trained to obtain a multi-scale feature fusion model, including: Performing pyramid scale decomposition on the multimodal fusion feature data to obtain feature maps of three different scales, which are respectively used as inputs of three parallel branches; Apply 5 residual blocks to the input feature map of each branch. Each residual block contains two 3x3 convolutional layers, a batch normalization layer, a ReLU activation function, and a short-circuit connection to obtain the initial residual features. Inserting a spatial attention module after the second and fourth residual blocks of each branch, weighting the feature importance of the spatial dimension of the initial residual features, and obtaining spatial attention enhanced features; Applying a channel attention module to the spatial attention enhancement feature, calculating the channel weight through global average pooling, a fully connected layer and a sigmoid activation function, and multiplying it with the feature map to obtain a dual attention enhancement feature; A reconstruction module is added after the last residual block of each branch. The reconstruction module includes three transposed convolutional layers, each of which is followed by batch normalization and ReLU activation function, which is used to upsample the feature map to the original input size to obtain a reconstructed feature map. Calculate the mean square error between the reconstructed feature map and the original input image to obtain the reconstruction loss, and add the reconstruction loss to the classification loss as the overall optimization target of the network; Adaptive feature fusion is performed on the dual attention enhancement features of the three branches, specifically including: calculating the Euclidean distance between each branch feature and the ideal feature, obtaining the fusion weight based on the distance ratio, performing weighted summation, and obtaining the training multi-scale fusion feature; The trained multi-scale fusion features are input into the fully connected layer and the softmax layer to obtain the second classification prediction results of the labeled samples, and the reconstruction loss and the classification cross entropy loss are combined to perform end-to-end training on the entire network through the back propagation algorithm to obtain a multi-scale feature fusion model.

8. The deep learning-based breast image classification and evaluation method according to claim 7, characterized in that: The adaptive feature fusion of the dual attention enhancement features of the three branches specifically includes: calculating the Euclidean distance between each branch feature and the ideal feature, obtaining the fusion weight based on the distance ratio relationship, performing weighted summation, and obtaining the training multi-scale fusion feature, including: Perform a global average pooling operation on the dual attention enhanced features of each branch to obtain a feature vector of fixed dimension as the global feature representation of the branch; Construct an ideal feature vector whose element value is 1 and whose dimension is the same as the global feature representation as a reference for feature fusion; Calculate the Euclidean distance between the global feature representation of each branch and the ideal feature vector, and obtain three distance values, which correspond to the degree of feature difference of the three branches respectively; The three distance values ​​obtained are normalized, and each distance value is divided by the sum of the three distance values ​​to obtain a normalized distance ratio; Based on the normalized distance ratio, calculate the fusion weight of each branch, subtract the normalized distance ratio of the branch from 1, and then normalize again to ensure that the sum of the weights of the three branches is 1; The calculated fusion weights are weighted and multiplied with the dual attention enhancement features of the corresponding branches, and the three weighted features are added element by element to obtain the training multi-scale fusion features.

9. The breast image classification and evaluation method based on deep learning according to claim 1, characterized in that: The method of performing classification evaluation and prediction result integration on real-time breast image data through the graph convolution classification model and the multi-scale feature fusion model to obtain a target breast image classification result includes: Performing preprocessing operations on real-time breast image data to obtain preprocessed multi-scale image data; Inputting the preprocessed multi-scale image data into the graph structure generation layer of the graph convolution classification model, and obtaining a graph structure representation through superpixel segmentation and graph construction operations; The graph structure representation is sequentially passed through 5 layers of graph convolution layers of the graph convolution classification model, each layer includes graph convolution operation, batch normalization and LeakyReLU activation, and a layer attention module is applied after the 2nd and 4th layers to obtain real-time graph convolution features; Input the real-time graph convolutional features into the global average pooling layer for global average pooling operation, aggregate the features of all nodes into a graph-level feature representation of a fixed dimension, input the graph-level feature representation into two fully connected layers, the first fully connected layer contains 256 neurons, the number of neurons in the second fully connected layer is the same as the number of categories of breast image classification, calculate the probability distribution of each category by the softmax function, and obtain the first classification probability distribution of the graph convolution classification model; Inputting the preprocessed multi-scale image data into three parallel branches of a multi-scale feature fusion model, each branch comprising 5 residual blocks, a spatial attention module and a channel attention module, to obtain real-time dual attention enhanced features; Adaptively fusing the real-time dual attention enhancement features, and obtaining real-time multi-scale fusion features by calculating Euclidean distance and weight distribution; Inputting the real-time multi-scale fusion features into the fully connected layer and the softmax layer to obtain a second classification probability distribution of the multi-scale feature fusion model; A weighted average is performed on the first classification probability distribution and the second classification probability distribution to obtain a classification result of the target breast image.

10. A breast image classification and evaluation system based on deep learning, characterized in that: Used to perform the deep learning-based breast image classification and evaluation method according to any one of claims 1 to 9, the system comprising: A preprocessing module, used for preprocessing sample breast image data to obtain comprehensive feature data; A feature enhancement module, configured to perform feature enhancement based on the comprehensive feature data by coordinating the attention module and the multimodal attention module to obtain enhanced feature data; A feature fusion module, used for performing multimodal bidirectional feature fusion on the enhanced feature data to obtain multimodal fused feature data; A first construction module is used to construct and train a semi-supervised local feature preserving densely connected context-aware graph convolutional network based on the multimodal fusion feature data to obtain a graph convolution classification model; A second construction module is used to construct and train a multi-branch residual attention reconstruction network according to the multimodal fusion feature data to obtain a multi-scale feature fusion model; The integration module is used to classify and evaluate the real-time breast image data and integrate the prediction results through the graph convolution classification model and the multi-scale feature fusion model to obtain the target breast image classification result.

Images