Cervical cancer white light and fluorescence picture fusion detection method based on attention mechanism

Abstract

According to the cervical cancer white light and fluorescence picture fusion detection method based on the attention mechanism, a multi-source medical image fusion network is built, and the multi-source medical image fusion network can extract the characteristics of a white light picture and a fluorescence picture and is used for diagnosing and recognizing early-stage cervical cancer symptoms. The multi-source medical image fusion network comprises a feature extraction module, a multi-scale feature fusion module, a multi-modal feature fusion module and a multi-head attention mechanism module. Firstly, a feature extraction module is used for extracting multi-source depth features of a white light picture and a fluorescent picture, then a multi-modal feature fusion module is used for fusing the multi-source depth features, and a multi-head attention mechanism module is combined for target accurate positioning. Experimental results show that compared with a single-source image method, the accuracy of the method on an experimental data set is improved by 7%, the recall rate is improved by 6%, and experiments verify that the method improves the detection accuracy of the early-stage symptoms of the cervical cancer.

Description

Technical Field

[0001] The present invention relates to the field of medical detection technology, and in particular to a cervical cancer white light and fluorescence image fusion detection method based on an attention mechanism. Background Art

[0002] With the continuous advancement of medical technology, the detection and identification technology of cervical cancer is also developing. The difficulty of cervical cancer detection and identification is related to many factors, including the type of cancer, the stage of development, and the application of detection technology. Cervical cancer is mainly divided into two categories: squamous cell carcinoma and adenocarcinoma. Early cervical cancer is usually carcinoma in situ, with a low degree of malignancy, and the peak age is 30 to 35 years old. Late cervical cancer is usually invasive cancer, and cancer cells may break through the basement membrane to infiltrate or metastasize to the stroma or distant tissues and organs. The age of onset is mainly 45 to 55 years old. Early cervical cancer often has no obvious symptoms and signs. As the lesion develops, vaginal bleeding and vaginal discharge will occur, but the cervix is ​​smooth or difficult to distinguish from cervical columnar epithelium ectopic, which is easy to miss or misdiagnose. In the late stage, different secondary symptoms appear according to the scope of cancer involvement, such as frequent urination, urgency, constipation, lower limb swelling and pain, and anemia. With the advancement of science and technology and medicine, especially the development of colposcopy technology and HPV vaccine, strong technical support has been provided for early screening and prevention of cervical cancer. The detection technology of cervical cancer has developed from traditional cytology (such as Pap smear) to HPV DNA-based detection technology. As the preferred screening method, HPV detection marks the official transition of cervical cancer screening from the cytology era to the HPV DNA era. The development of HPV detection technology has brought a milestone change in the field of cervical cancer screening, sounding the alarm for potential cases during the infection period when there is no cell lesions.

[0003] In the field of intelligent detection of cervical cancer, research based on deep learning has made significant progress in recent years. Wang et al. (2023) developed a deep learning detection system using the ResNet architecture, and achieved a detection accuracy of 91.5% on a large-scale data set through transfer learning methods, providing a new technical solution for early screening. Schmidt et al. (2023) first introduced Vision Transformer (ViT) into cervical cancer detection. The self-attention framework they proposed achieved an accuracy of 93.2% when processing more than 2,000 medical images, especially in the recognition of complex lesion areas. The attention-enhanced CNN model developed by Davies et al. (2023) has a detection sensitivity of 89.7% for subtle lesions, which reduces the missed detection rate by 15% compared with traditional methods. In terms of real-time detection, the detection system developed by Tanaka et al. (2023) based on the YOLOv5 architecture achieved a processing speed of 30 frames per second, significantly improving clinical detection efficiency while maintaining an accuracy of 87.3%. Chen et al. (2023) used a lightweight model built with EfficientNet to compress the model size to 15MB, and still maintained an accuracy of 90.1% on the validation set, greatly improving the applicability of the model in primary medical institutions. In terms of domestic research, the multi-scale feature extraction network based on DenseNet proposed by Li Ming et al. (2023) improved the recognition ability of small lesions by optimizing the network structure, and the accuracy rate reached 92.3% in the verification of 3000 clinical data. The Transformer-CNN hybrid architecture designed by Xia Shaojun et al. (2023) made a breakthrough in feature extraction, and the accuracy of the model was improved to 94.1% when processing multi-scale lesions. Du Dexi et al. (2023) based on the improved model of U-Net++ achieved a Dice coefficient of 0.89 in the lesion area segmentation task, providing a reliable tool for accurate identification of lesions. However, existing research still faces challenges. As Wang et al. (2023) pointed out, single-modality images are easily affected by acquisition conditions, resulting in incomplete feature extraction. Schmidt et al. (2023) showed that even advanced ViT models still have a 5-8 percentage point drop in accuracy when processing white light images with uneven illumination. Davies et al. (2023) found that in early lesion detection, due to the lack of obvious features, the detection sensitivity is generally 10-15 percentage points lower than that of late lesions.

[0004] Although white light image processing technology has made some progress in the field of cervical cancer detection, white light image detection is a conventional means and method for cervical cancer detection, but there are still some limitations and shortcomings. White light images can usually only provide limited information. For example, white light images mainly provide morphological information of tissues. White light images are difficult to fully capture the complex characteristics of cervical cancer, such as metabolic changes and microenvironment differences. Due to the lack of multi-angle information, white light images have limitations in the early diagnosis and grading of cervical cancer, resulting in false negative or false positive results. White light images mainly reflect the morphological structure of tissues, but cannot provide information about tissue function and metabolic status, which is particularly important for the early detection of cervical cancer. The feature vectors of different modalities are located in different subspaces. This heterogeneity gap hinders the comprehensive use of data in deep learning models. Single-modal data enhancement methods are difficult to simulate the complexity of multi-source data, which limits the completeness of the representation learned by the model. Summary of the invention

[0005] To solve the above technical problems, the present invention proposes a fusion detection method for cervical cancer white light and fluorescence images based on an attention mechanism, which includes a feature extraction module, a multimodal feature fusion module, and a multi-head attention mechanism module. By fusing fluorescence images and white light images, the morphological information (white light image) and functional information (fluorescence image) of the tissue can be obtained simultaneously, thereby enhancing the feature extraction capability of the model. The multimodal feature fusion module can provide richer contextual information to help doctors diagnose cervical cancer and its precursor lesions more accurately, and improve the accuracy of diagnosis and early detection capabilities. The fused image can be used for treatment planning. By combining the biomarker information of the fluorescence image and the anatomical information of the white light image, the tumor boundary can be more accurately located and the treatment plan can be optimized. By combining the fluorescence image and the white light image, the radiation dose to the patient can be reduced to a certain extent, especially when multiple imaging is required.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] The method for detecting cervical cancer by fusion of white light and fluorescent images based on attention mechanism is characterized by comprising the following steps:

[0008] S1. Experimental data and preprocessing:

[0009] S11. Data collection:

[0010] Collect medical image data related to cervical cancer in hospitals, clinics or public databases. The medical image data include cervical cytology images, histopathology images, and colposcopy images.

[0011] S12. Data preprocessing:

[0012] Preprocess the collected medical image data, including digitization, denoising, enhancement, and segmentation of the images, in preparation for subsequent analysis. Professional doctors or technicians will annotate the key features in the medical data images, and these annotation information will serve as labels for network training;

[0013] S13. Data classification:

[0014] Classification is performed based on cell features and morphological knowledge. In each cell image, the cytoplasm and nucleus areas are manually marked. The medical image dataset is classified into training set, validation set and test set. The training set is used for network training, the validation set is used for network tuning, and the test set is used to evaluate network performance.

[0015] S14. Data enhancement:

[0016] Medical image data is preprocessed by Min-Max Scaling, i.e., minimum and maximum standardization. Min-Max Scaling is a preprocessing technique that scales data features to a specified range, usually the interval [0,1]. The minimum value of the original data is mapped to 0 and the maximum value is mapped to 1 through linear transformation, while the intermediate values ​​are scaled proportionally.

[0017] The calculation formula for Min-Max Scaling is as follows:

[0018]

[0019] Where x is the original data, min(x) is the minimum value in the medical image dataset, max(x) is the maximum value in the medical image dataset, and x′ is the scaled data;

[0020] Use image enhancement technology to enhance the medical image dataset. The formula is as follows

[0021] Enhanced image = F (original image, enhancement technique, enhancement parameters)

[0022] f is a function representing the process of image enhancement technology, the original image is a picture in the medical image dataset, and the enhancement technology is an autonomously selected enhancement method;

[0023] S2. Multi-source medical image fusion network:

[0024] The multi-source medical image fusion network includes a feature extraction module, a multi-modal feature fusion module, and a multi-head attention mechanism module. The feature extraction module is used to extract features of the fluorescent image and the white light image in the medical image dataset. After extracting the features of the two, the multi-modal feature fusion module is used to fuse the features. After the feature fusion, the multi-head attention mechanism module is used to segment the input fusion features and perform linear transformation in each segmentation space. Then, the output in each segmentation space is calculated separately. Finally, all the outputs are spliced ​​together and a final linear transformation and layer normalization are performed to obtain the output of the multi-head attention mechanism module. Finally, the output of the multi-head attention mechanism module is classified and predicted.

[0025] S21, feature extraction module:

[0026] The feature extraction module uses the idea of ​​the FsterNet module. FasterNet is a new neural network architecture that reduces redundant calculations and memory accesses by using partial convolution. Partial convolution is a special convolution operation that only considers part of the input feature map when performing convolution. Its calculation formula is as follows

[0027]

[0028] Point-by-point convolution is a 1x1 convolution operation that performs weighted combination of feature maps in the depth direction to generate a new feature map. The calculation formula of point-by-point convolution can be expressed as

[0029] y=W T x+b

[0030] Among them, y is the output feature map, W is the 1x1 convolution kernel, and b is the bias term.

[0031] The purpose of point-by-point convolution is to merge multiple feature maps after depth convolution into the required number of output feature maps.

[0032] Use four layers of PConv+PWConv and pooling layers to extract features;

[0033] S22, multimodal feature fusion module:

[0034] The multimodal feature fusion module can perform channel attention and spatial attention operations on the features FA and FB of the same scale to obtain F1 and F2, and then add these two features to obtain the fused feature F′, and then perform channel attention and spatial attention operations on F′ to obtain F″. Finally, F′ and F″ are added to obtain the final fused feature F cat

[0035] The whole process is expressed as:

[0036]

[0037] Where Cat represents the cascade operation, F SA represents spatial attention, F CA represents channel attention, F' and Fcat are the features generated after two fusions;

[0038] S23, multi-head attention mechanism module:

[0039] The multi-head attention mechanism module allows the network to learn information in different representation subspaces in parallel. It is achieved by splitting the input vector into multiple subspaces and performing self-attention operations in each subspace. Each subspace is linearly transformed using an independent weight matrix to generate query, key, and value matrices. Then, each head calculates the attention weight and weights the sum value matrix accordingly to obtain the output of each head. Finally, the outputs of all heads are concatenated and passed through a final linear transformation and layer normalization to obtain the output of the multi-head attention. The calculation of the multi-head attention mechanism can be expressed as the following formula:

[0040] MultiHead(Q,K,V)=Concat(head1,head2...)w o

[0041] Where Q, K, V represent query, key and value matrices respectively, h represents the number of heads,

[0042] head i =Attention(QW i Q , KW i k , VW i v ) represents the output of the i-th head, W o is the output transformation matrix, and the self-attention calculation of each can be expressed as: Where Q, K are query and key matrices respectively, V is the value matrix, d k is the dimension of the key vector, and the softmax function is used to normalize the attention weights;

[0043] S3. Experiments and Results:

[0044] S31. Network training:

[0045] First, data enhancement techniques are used to reduce overfitting.

[0046] Then the training settings are made: the training cycle is 400 rounds, the batch size is set to 2, SGD is used as the optimizer, and the training will stop when the network accuracy no longer improves;

[0047] Then set the loss function, which is the mean square error loss function. By calculating the square of the difference between the predicted value and the true value, and then taking the average value, the mathematical formula of the mean square error loss function is as follows:

[0048]

[0049] Where n is the number of samples, yi is the true value of the i-th sample, and Y' is the predicted value of the i-th sample.

[0050] All network training and testing were implemented in Python 3.8 using the PyTorch environment on a Linux server with NVIDIA RTX 4060ti;

[0051] S32, evaluation indicators:

[0052] Evaluation indicators are set. Four key indicators are used: accuracy, recall, precision and F1 score. These indicators provide a comprehensive evaluation of the network prediction results. Accuracy is the basic indicator to measure the overall performance of the network. It reflects the proportion of samples correctly predicted by the network to the total number of samples.

[0053] The calculation formula is:

[0054]

[0055] Among them, true positive TP is the number of samples correctly predicted as positive examples, false positive FP is the number of samples incorrectly predicted as positive examples, true negative TN is the number of samples correctly predicted as negative examples, and false negative FN is the number of samples incorrectly predicted as negative examples.

[0056] Recall, also known as true positive rate or sensitivity, measures the proportion of samples that the network correctly predicts as positive among all samples that are actually positive.

[0057]

[0058] Precision measures the proportion of samples that the network predicts to be positive that are actually positive.

[0059]

[0060] The F1 score is the harmonic mean of precision and recall, which strikes a balance between the two. Precision and recall are positively proportional to the F1 score.

[0061]

[0062] S33, comparative experiment:

[0063] Comparison of the multi-source fusion model without the multi-source medical fusion network: After training with the same data set, the evaluation indicators were obtained and compared. Compared with the image detection network without multi-source fusion, the accuracy of the network with multi-source fusion increased by 9 percentage points and the recall rate increased by 8 percentage points. Therefore, it is proved that the multi-source fusion network is better than the network without multi-source fusion.

[0064] Comparison with existing models: After training the Resnet50 model and the mobilenet model with the same data set, the evaluation indicators were obtained and compared. The accuracy was 12 percentage points higher than that of the Mobilenet model and 7 percentage points higher than that of the Resnet50 model. The recall was 11 percentage points higher than that of the Resnet50 model and 8 percentage points higher than that of the Mobilenet model.

[0065] S34, ablation experiment:

[0066] In order to verify that spatial attention and channel attention mechanisms can improve detection accuracy, a multi-source fusion network A without attention mechanism was constructed and trained with the same parameters on the same dataset. From the training results, it was found that the performance of the model including attention mechanism was higher than that of the model without attention mechanism.

[0067] As a preferred technical solution of the present invention: in step S12, the labeling includes marking the cell nucleus, cell type, and lesion area in the medical image data.

[0068] As a preferred technical solution of the present invention: in step S14, the image enhancement technology includes geometric transformation, color transformation, adding noise, random erasing and mixed enhancement, the geometric transformation includes rotation, flipping, scaling and shearing, the color transformation includes adjusting brightness, contrast, saturation and hue, and the adding noise includes adding Gaussian noise and adding salt and pepper noise.

[0069] As a preferred technical solution of the present invention: in step S31, the data enhancement technology includes image rotation, scaling, inversion and cropping.

[0070] The present invention proposes a fusion detection method for cervical cancer white light and fluorescence images based on the attention mechanism. By building a multi-source medical image fusion network, the features of white light images and fluorescence images can be extracted through the multi-source medical image fusion network for diagnosis, positioning and grading of early cervical cancer symptoms. The multi-source medical image fusion network includes a feature extraction module, a multi-modal feature fusion module and a multi-head attention mechanism module. First, the feature extraction module is used to extract the multi-source deep features of the white light image and the fluorescence image, and then the multi-modal feature fusion module is used to fuse the multi-source deep features, and the multi-head attention mechanism module is combined to accurately locate the target. The experimental results show that compared with the single-source image method, the accuracy of this method on the experimental data set is improved by 7 percentage points, and the recall rate is improved by 6 percentage points. The experiment verifies that this method improves the detection accuracy of early symptoms of cervical cancer.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] The multi-source medical image feature fusion network proposed in the present invention includes a feature extraction module, a multi-scale feature fusion module, a multi-source image fusion module and an attention decision module. By fusing fluorescence images and white light images, the morphological information (white light image) and functional information (fluorescence image) of the tissue can be obtained simultaneously, thereby enhancing the feature extraction capability of the network. Multi-source image fusion can provide richer contextual information, help doctors diagnose cervical cancer and its precursor lesions more accurately, and improve the accuracy of diagnosis and early detection capabilities. Fused images can be used for treatment planning. By combining the biomarker information of fluorescence images and the anatomical information of white light images, the tumor boundary can be more accurately located and the treatment plan can be optimized. By combining fluorescence images and white light images, the radiation dose to the patient can be reduced to a certain extent, especially when multiple imaging is required.

[0073] The multi-source medical image feature fusion network proposed in the present invention effectively realizes the recognition of cervical cancer by combining a multi-scale feature fusion module and a multi-source medical image feature fusion module. The design of the network makes full use of the characteristics of images of different modalities and enhances the network's recognition ability of cervical cancer by weighted fusion of features of different modalities.

[0074] In the field of multi-source fusion, various fusion strategies establish connections between different layers of the network so that the information of deep fusion can be used by shallow layers, avoiding information loss while maintaining the advantages of post-fusion. In addition, the fusion of multi-source data can provide more information for network decision-making, thereby improving the accuracy of the overall decision-making results. For example, by utilizing multi-source fusion features, performance can be improved in cross-media analysis tasks such as video classification, event detection, sentiment analysis, and cross-modal translation.

[0075] The application of multi-source medical image feature fusion in the field of cervical cancer detection proposed in this paper not only improves the recognition rate of lesions, but also provides a more accurate basis for clinical decision-making. With the continuous advancement of deep learning technology, future research is expected to be applied in a wider range of cancer treatment fields, providing patients with more accurate treatment options and improved survival expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a principle framework diagram of a multi-source medical image fusion network;

[0077] Figure 2 It is a multi-source medical image fusion network connection diagram;

[0078] Figure 3 This is a comparison chart with and without multi-source fusion;

[0079] Figure 4 It is the accuracy-round comparison trend chart with and without multi-source fusion;

[0080] Figure 5 This is the loss-round comparison trend chart with and without multi-source fusion;

[0081] Figure 6 This is a comparison chart with the Resnet50 model and the mobilenet model;

[0082] Figure 7 This is the accuracy-round comparison chart with the Resnet50 model and the mobilenet model

[0083] Figure 8 This is a comparison chart of recall rate-rounds with the Resnet50 model and the mobilenet model.

[0084] Fig. 9 This is a comparison chart of ablation experiments;

[0085] Fig.10 This is the ablation experiment accuracy-round trend comparison chart

[0086] Fig.11 This is the comparison chart of recall rate-round trend in ablation experiment

[0087] Fig.12 This is the network structure diagram of the four-layer PConv+PWConv and pooling layer;

[0088] Fig.13 This is the network structure diagram of the four-layer PConv+PWConv and pooling layer;

[0089] Fig.14 This is the network structure diagram of the four-layer PConv+PWConv and pooling layer;

[0090] Fig.15 It is medical image data related to cervical cancer 1;

[0091] Fig.16 It is medical image data related to cervical cancer 2. DETAILED DESCRIPTION

[0092] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0093] like Figure 1-16 As shown, the present invention proposes a cervical cancer white light and fluorescence image fusion detection method based on an attention mechanism, comprising the following steps:

[0094] S1. Experimental data and preprocessing:

[0095] S11. Data collection:

[0096] Collect medical image data related to cervical cancer in hospitals, clinics or public databases. The medical image data include cervical cytology images, histopathology images, and colposcopy images.

[0097] The dataset for this application study comes from clinical case data from three tertiary hospitals, Nanjing Drum Tower Hospital, Nanjing Maternal and Child Health Hospital, and Jiangsu Provincial People's Hospital, from 2020 to 2023. It was collected with the approval of the hospital ethics committee and the informed consent of the patients. Fig.15 and Fig.16 shown.

[0098] S12. Data preprocessing:

[0099] Preprocess the collected medical image data, including digitization, denoising, enhancement, and segmentation of the images, in preparation for subsequent analysis. Professional doctors or technicians will annotate the key features in the medical data images, and these annotation information will serve as labels for network training;

[0100] S13. Data classification:

[0101] Classification is performed based on cell features and morphological knowledge. In each cell image, the cytoplasm and nucleus areas are manually marked. The medical image dataset is classified into training set, validation set and test set. The training set is used for network training, the validation set is used for network tuning, and the test set is used to evaluate network performance.

[0102] S14. Data enhancement:

[0103] Medical image data is preprocessed by Min-Max Scaling, i.e., minimum and maximum standardization. Min-Max Scaling is a preprocessing technique that scales data features to a specified range, usually the interval [0,1]. The minimum value of the original data is mapped to 0 and the maximum value is mapped to 1 through linear transformation, while the intermediate values ​​are scaled proportionally. The calculation formula of Min-Max Scaling is as follows:

[0104]

[0105] Where x is the original data, min(x) is the minimum value in the medical image dataset, max(x) is the maximum value in the medical image dataset, and x′ is the scaled data;

[0106] Use image enhancement technology to enhance the medical image dataset. The formula is as follows

[0107] Enhanced image = F (original image, enhancement technique, enhancement parameters)

[0108] f is a function representing the process of image enhancement technology, the original image is a picture in the medical image dataset, and the enhancement technology is an autonomously selected enhancement method;

[0109] The dataset for this study comes from clinical case data from three tertiary hospitals, Nanjing Drum Tower Hospital, Nanjing Maternal and Child Health Hospital, and Jiangsu Provincial People's Hospital, from 2020 to 2023. It was collected after approval by the hospital ethics committee and informed consent from the patients. The dataset contains white light and fluorescence image pairs of 13,500 cervical cancer screening cases, and each pair of images is synchronously acquired by professional medical equipment at the same viewing angle. Based on cell characteristics and morphological characteristics, the cytopathology expert team of the three hospitals annotated the dataset into 21 categories, including normal cervix, mild erosion, simple erosion and other pathological types. The dataset was randomly divided into a training set (9,450 cases), a validation set (1,350 cases), and a test set (2,700 cases) in a ratio of 7:1:2. All image data undergo a standardized preprocessing process, including digitization, denoising, contrast enhancement, and lesion area segmentation, to ensure data quality. The entire data collection and use process strictly abides by patient privacy protection regulations, uses de-identification processing to protect patient privacy, and ensures data security through encrypted storage.

[0110] Data augmentation is a key technology to improve network performance in machine learning, especially in the field of deep learning. It expands the training set by generating data variants, effectively improving the generalization ability of the network and reducing overfitting. The importance of data augmentation is reflected in improving the network's learning of the inherent laws of the data, reducing overfitting of the training set, enhancing the robustness of the network in the face of noise and disturbance, and reducing the cost of obtaining a large amount of high-quality annotated data. Common data augmentation techniques include geometric transformations such as rotation, flipping, scaling and shearing, color transformations such as adjusting brightness, contrast, saturation and hue, adding noise such as Gaussian noise and salt and pepper noise, random erasing and hybrid enhancement such as Mixup technology, and even using generative adversarial networks (GANs) to generate new data samples. These technologies are widely used in many fields such as image recognition, natural language processing, speech recognition, medical image analysis and autonomous driving. By simulating different conditions and environments, the adaptability and robustness of the network are enhanced. With the continuous deepening of research, data augmentation will continue to play an important role in the development of machine learning technology and promote the development and application of more innovative methods.

[0111] S2. Multi-source medical image fusion network:

[0112] The multi-source medical image fusion network includes a feature extraction module, a multi-modal feature fusion module, and a multi-head attention mechanism module. The feature extraction module is used to extract features of the fluorescent image and the white light image in the medical image dataset. After extracting the features of the two, the multi-modal feature fusion module is used to fuse the features. After the feature fusion, the multi-head attention mechanism module is used to segment the input fusion features and perform linear transformation in each segmentation space. Then, the output in each segmentation space is calculated separately. Finally, all the outputs are spliced ​​together and a final linear transformation and layer normalization are performed to obtain the output of the multi-head attention mechanism module. Finally, the output of the multi-head attention mechanism module is classified and predicted.

[0113] S21, feature extraction module:

[0114] The feature extraction module uses the idea of ​​the FsterNet module. FasterNet is a new neural network architecture that reduces redundant calculations and memory accesses by using partial convolution. Partial convolution is a special convolution operation that only considers part of the input feature map when performing convolution. Its calculation formula is as follows

[0115]

[0116] Point-by-point convolution is a 1x1 convolution operation that performs weighted combination of feature maps in the depth direction to generate a new feature map. The calculation formula of point-by-point convolution can be expressed as

[0117] y=W T x+b

[0118] Among them, y is the output feature map, W is the 1x1 convolution kernel, and b is the bias term.

[0119] The purpose of point-by-point convolution is to merge multiple feature maps after depth convolution into the required number of output feature maps.

[0120] Use four layers of PConv+PWConv and pooling layers to extract features;

[0121] Feature Extraction Module With the help of the idea of ​​FsterNet module, FasterNet is a new neural network architecture that reduces redundant calculations and memory accesses by using partial convolution (PConv), thereby increasing the network's running speed. The core of FasterNet is partial convolution (PConv), which performs convolution operations only on part of the input feature map, rather than the full application of traditional convolution operations. In this way, PConv can reduce unnecessary calculations and memory accesses because it ignores the parts of the input that are considered redundant. This approach is particularly suitable for running deep learning networks on devices with limited resources because it can significantly reduce computing requirements without sacrificing too much performance.

[0122] Use four layers of PConv+PWConv and pooling layers to extract features. The network structure is as follows Figure 12-14 shown.

[0123] S22, multimodal feature fusion module:

[0124] The multimodal feature fusion module can perform channel attention and spatial attention operations on the features FA and FB of the same scale to obtain F1 and F2, and then add these two features to obtain the fused feature F′, and then perform channel attention and spatial attention operations on F′ to obtain F″. Finally, F′ and F″ are added to obtain the final fused feature F cat

[0125] The whole process is expressed as:

[0126]

[0127] Where Cat represents the cascade operation, F SA represents spatial attention, F CA represents channel attention, F' and Fcat are the features generated after two fusions;

[0128] S23, multi-head attention mechanism module:

[0129] The multi-head attention mechanism module allows the network to learn information in different representation subspaces in parallel. It is achieved by splitting the input vector into multiple subspaces and performing self-attention operations in each subspace. Each subspace is linearly transformed using an independent weight matrix to generate query, key, and value matrices. Then, each head calculates the attention weight and weights the sum value matrix accordingly to obtain the output of each head. Finally, the outputs of all heads are concatenated and passed through a final linear transformation and layer normalization to obtain the output of the multi-head attention. The calculation of the multi-head attention mechanism can be expressed as the following formula:

[0130] MultiHead(Q,K,V)=Concat(head1,head2......)w o

[0131] Where Q, K, V represent query, key and value matrices respectively, h represents the number of heads,

[0132] head i =Attention(QW i Q , KW i k , VW i v ) represents the output of the i-th head, W o is the output transformation matrix, and the self-attention calculation of each can be expressed as: Where Q, K are query and key matrices respectively, V is the value matrix, d k is the dimension of the key vector, and the softmax function is used to normalize the attention weights;

[0133] S3. Experiments and Results:

[0134] S31. Network training:

[0135] First, reduce overfitting through data augmentation technology. Data augmentation technology is a commonly used method in machine learning and deep learning. It aims to expand the training set by creating data variants, thereby improving the generalization ability of the network and reducing overfitting, including rotation, scaling, inversion, etc.

[0136] Then the training settings are made: the training cycle is 400 rounds, the batch size is set to 2, SGD is used as the optimizer, and the training will stop when the network accuracy no longer improves;

[0137] Then set the loss function, which is the mean square error loss function. By calculating the square of the difference between the predicted value and the true value, and then taking the average value, the mathematical formula of the mean square error loss function is as follows:

[0138]

[0139] Where n is the number of samples, yi is the true value of the i-th sample, and Y' is the predicted value of the i-th sample.

[0140] All network training and testing were implemented in Python 3.8 using the PyTorch environment on a Linux server with NVIDIA RTX 4060ti;

[0141] Mean Squared Error (MSE) is one of the commonly used loss functions in regression problems, which is used to measure the difference between the network prediction value and the actual value.

[0142] S32, evaluation indicators:

[0143] Evaluation indicators are set. Four key indicators are used: accuracy, recall, precision and F1 score. These indicators provide a comprehensive evaluation of the network prediction results. Accuracy is the basic indicator to measure the overall performance of the network. It reflects the proportion of samples correctly predicted by the network to the total number of samples.

[0144] The calculation formula is:

[0145]

[0146] Among them, true positive TP is the number of samples correctly predicted as positive examples, false positive FP is the number of samples incorrectly predicted as positive examples, true negative TN is the number of samples correctly predicted as negative examples, and false negative FN is the number of samples incorrectly predicted as negative examples.

[0147] Recall, also known as true positive rate or sensitivity, measures the proportion of samples that the network correctly predicts as positive among all samples that are actually positive.

[0148]

[0149] Precision measures the proportion of samples that the network predicts to be positive that are actually positive.

[0150]

[0151] The F1 score is the harmonic mean of precision and recall, which strikes a balance between the two. Precision and recall are positively proportional to the F1 score.

[0152]

[0153] These indicators together provide a performance evaluation of the network in different aspects, helping us understand the performance of the network in practical applications. When dealing with unbalanced datasets, relying solely on accuracy is not enough, and recall and precision are particularly important. The F1 score provides an indicator that comprehensively considers precision and recall, and is particularly suitable for evaluating network performance on unbalanced datasets.

[0154] S33, comparative experiment:

[0155] Comparison of the multi-source fusion model without the multi-source medical fusion network: After training with the same data set, the evaluation indicators were obtained and compared. Compared with the image detection network without multi-source fusion, the accuracy of the network with multi-source fusion increased by 9 percentage points and the recall rate increased by 8 percentage points. Therefore, it is proved that the multi-source fusion network is better than the network without multi-source fusion.

[0156] Comparison with existing models: After training with the same data set in the resnet50 model and the mobilenet model, the evaluation indicators were obtained and compared. The accuracy was 3 percentage points higher than that of the Mobilenet model.

[0157] The resnet50 model is 6 percentage points higher, the recall rate is 7 percentage points higher than the resnet50 model, and 5 percentage points higher than the Mobilenet model;

[0158] In order to verify the effectiveness and accuracy of the multi-source medical image fusion network proposed in the present invention, the multi-source fusion model without the multi-source medical image fusion network was compared, and the evaluation indicators were obtained after training with the same data set for comparison. Compared with the image detection network without multi-source fusion, the accuracy of the network with multi-source fusion increased by 9 percentage points, and the recall rate increased by 8 percentage points. It is proved that the multi-source fusion network is better than the network without multi-source fusion.

[0159] In addition, the present invention reproduces the detection methods in multiple documents, such as the resnet50 model and the mobilenet model. By comparing the present invention with the above models, the accuracy of the network is 3 percentage points higher than that of the Mobilenet architecture and 6 percentage points higher than that of the resnet50 architecture. The recall rate is 7 percentage points higher than that of the resnet50 architecture and 5 percentage points higher than that of the Mobilenet architecture.

[0160] S34, ablation experiment:

[0161] In order to evaluate the contribution of each module to the performance of early cervical cancer symptom detection, an ablation experiment was designed to remove key modules from the model one by one to observe their impact on model performance. The ablation experiment mainly includes the following configurations:

[0162] 1. Complete PFM Model: includes all core modules and serves as the benchmark model.

[0163] 2. Remove feature extraction module (No Feature Extraction): Remove the image feature extraction module and only retain the image fusion and decision modules.

[0164] 3. Remove the multi-scale feature fusion module (No Multiscale Fusion): Remove the multi-scale feature fusion module and only use single-scale features for training.

[0165] 4. Remove the multimodal feature fusion module (No Multimodal Fusion): Remove the multimodal feature fusion module and use only a single modality (such as white light or fluorescence image).

[0166] 5. Remove the attention decision module (No Attention): Remove the attention mechanism module, and the model makes decisions only based on the fused features.

[0167] The above description is only a preferred embodiment of the present invention and does not constitute any other form of limitation to the present invention. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A fusion detection method for cervical cancer white light and fluorescence images based on attention mechanism, characterized by: The following steps are included: S1. Experimental data and preprocessing: S11. Data collection: Collect medical image data related to cervical cancer in hospitals, clinics or public databases. The medical image data include cervical cytology images, histopathology images, and colposcopy images. S12. Data preprocessing: Preprocess the collected medical image data, including digitization, denoising, enhancement, and segmentation of the images, in preparation for subsequent analysis. Professional doctors or technicians will annotate the key features in the medical data images, and these annotation information will serve as labels for network training; S13. Data classification: Classification is performed based on cell features and morphological knowledge. In each cell image, the cytoplasm and nucleus areas are manually marked. The medical image dataset is classified into training set, validation set and test set. The training set is used for network training, the validation set is used for network tuning, and the test set is used to evaluate network performance. S14. Data enhancement: Medical image data is preprocessed by Min-Max Scaling, i.e., minimum and maximum standardization. Min-Max Scaling is a preprocessing technique that scales data features to a specified range, usually the interval [0,1]. The minimum value of the original data is mapped to 0 and the maximum value is mapped to 1 through linear transformation, while the intermediate values ​​are scaled proportionally. The calculation formula for Min-Max Scaling is as follows: Where x is the original data, min(x) is the minimum value in the medical image dataset, max(x) is the maximum value in the medical image dataset, and x′ is the scaled data; Use image enhancement technology to enhance the medical image dataset. The formula is as follows Enhanced image = F (original image, enhancement technique, enhancement parameters) f is a function representing the process of image enhancement technology, the original image is a picture in the medical image dataset, and the enhancement technology is an autonomously selected enhancement method; S2. Multi-source medical image fusion network: The multi-source medical image fusion network includes a feature extraction module, a multi-modal feature fusion module, and a multi-head attention mechanism module. The feature extraction module is used to extract features of the fluorescent image and the white light image in the medical image dataset. After extracting the features of the two, the multi-modal feature fusion module is used to fuse the features. After the feature fusion, the multi-head attention mechanism module is used to segment the input fusion features and perform linear transformation in each segmentation space. Then, the output in each segmentation space is calculated separately. Finally, all the outputs are spliced ​​together and a final linear transformation and layer normalization are performed to obtain the output of the multi-head attention mechanism module. Finally, the output of the multi-head attention mechanism module is classified and predicted. S21, feature extraction module: The feature extraction module uses the idea of ​​the FsterNet module. FasterNet is a new type of neural network architecture that reduces redundant calculations and memory accesses by using partial convolution. Partial convolution is a special convolution operation that only considers part of the input feature map when performing convolution. Its calculation formula is as follows Point-by-point convolution is a 1x1 convolution operation that performs weighted combination of feature maps in the depth direction to generate a new feature map. The calculation formula of point-by-point convolution can be expressed as y=W T x+b Among them, y is the output feature map, W is the 1x1 convolution kernel, and b is the bias term. The purpose of point-by-point convolution is to merge multiple feature maps after depth convolution into the required number of output feature maps. Use four layers of PConv+PWConv and pooling layers to extract features; S22, multimodal feature fusion module: The multimodal feature fusion module can perform channel attention and spatial attention operations on the features FA and FB of the same scale to obtain F1 and F2, and then add these two features to obtain the fused feature F′, and then perform channel attention and spatial attention operations on F′ to obtain F″. Finally, F′ and F″ are added to obtain the final fused feature F cat The whole process is expressed as: Where Cat represents the cascade operation, F SA represents spatial attention, F CA represents channel attention, F' and Fcat are the features generated after two fusions; S23, multi-head attention mechanism module: The multi-head attention mechanism module allows the network to learn information in different representation subspaces in parallel. It is achieved by splitting the input vector into multiple subspaces and performing self-attention operations in each subspace. Each subspace is linearly transformed using an independent weight matrix to generate query, key, and value matrices. Then, each head calculates the attention weight and weights the sum value matrix accordingly to obtain the output of each head. Finally, the outputs of all heads are concatenated and passed through a final linear transformation and layer normalization to obtain the output of the multi-head attention. The calculation of the multi-head attention mechanism can be expressed as the following formula: MultiHead(Q,K,V)=Concat(head1,head2......)w o Where Q, K, V represent query, key and value matrices respectively, h represents the number of heads, head i =Attention(QW i Q , KW i k , VW i v ) represents the output of the i-th head, w o is the output transformation matrix, and the self-attention calculation of each can be expressed as: Where Q, K are query and key matrices respectively, V is the value matrix, d k is the dimension of the key vector, and the softmax function is used to normalize the attention weights; S3. Experiments and Results: S31. Network training: First, data enhancement techniques are used to reduce overfitting. Then the training settings are made: the training cycle is 400 rounds, the batch size is set to 2, SGD is used as the optimizer, and the training will stop when the network accuracy no longer improves; Then set the loss function, which is the mean square error loss function. By calculating the square of the difference between the predicted value and the true value, and then taking the average value, the mathematical formula of the mean square error loss function is as follows: Where n is the number of samples, yi is the true value of the i-th sample, and Y' is the predicted value of the i-th sample. All network training and testing were implemented in Python 3.8 using the PyTorch environment on a Linux server with NVIDIA RTX 4060ti; S32, evaluation indicators: Evaluation indicators are set. Four key indicators are used: accuracy, recall, precision and F1 score. These indicators provide a comprehensive evaluation of the network prediction results. Accuracy is a basic indicator to measure the overall performance of the network. It reflects the proportion of samples correctly predicted by the network to the total number of samples. The calculation formula is: Among them, true positive TP is the number of samples correctly predicted as positive examples, false positive FP is the number of samples incorrectly predicted as positive examples, true negative TN is the number of samples correctly predicted as negative examples, and false negative FN is the number of samples incorrectly predicted as negative examples. Recall, also known as true positive rate or sensitivity, measures the proportion of samples that the network correctly predicts as positive among all samples that are actually positive. Precision measures the proportion of samples that the network predicts to be positive that are actually positive. The F1 score is the harmonic mean of precision and recall, which strikes a balance between the two. Precision and recall are positively proportional to the F1 score. S33, comparative experiment: Comparison of the multi-source fusion model without the multi-source medical fusion network: After training with the same data set, the evaluation indicators were obtained and compared. Compared with the image detection network without multi-source fusion, the accuracy of the network with multi-source fusion increased by 9 percentage points and the recall rate increased by 8 percentage points. Therefore, it is proved that the multi-source fusion network is better than the network without multi-source fusion. Comparison with existing models: After training the Resnet50 model and the mobilenet model with the same data set, the evaluation indicators were obtained and compared. The accuracy was 12 percentage points higher than that of the Mobilenet model and 7 percentage points higher than that of the Resnet50 model. The recall was 11 percentage points higher than that of the Resnet50 model and 8 percentage points higher than that of the Mobilenet model. S34, ablation experiment: In order to verify that spatial attention and channel attention mechanisms can improve detection accuracy, a multi-source fusion network A without attention mechanism was constructed and trained with the same parameters on the same dataset. From the training results, it was found that the performance of the model including attention mechanism was higher than that of the model without attention mechanism.

2. The method for detecting cervical cancer by fusion of white light and fluorescence images based on attention mechanism according to claim 1, characterized in that: In step S12, the labeling includes marking the cell nucleus, cell type, and lesion area in the medical image data.

3. The method for detecting cervical cancer by fusion of white light and fluorescence images based on attention mechanism according to claim 1, characterized in that: In step S14, the image enhancement techniques include geometric transformation, color transformation, noise addition, random erasing and hybrid enhancement. The geometric transformation includes rotation, flipping, scaling and shearing. The color transformation includes adjusting brightness, contrast, saturation and hue. The noise addition includes adding Gaussian noise and adding salt and pepper noise.

4. The method for detecting cervical cancer by fusion of white light and fluorescence images based on attention mechanism according to claim 1, characterized in that: In step S31, the data enhancement technology includes image rotation, scaling, inversion and cropping.

Images