Target segmentation method based on multi-source cross attention fusion

Abstract

The invention discloses a target segmentation method based on multi-source cross attention fusion, and particularly relates to the technical field of image processing and target segmentation, and the method comprises the steps: obtaining multi-source image data, and carrying out the corresponding preprocessing operation of a data set; a segmentation network based on multi-source image data fusion is constructed, and a segmentation network based on multi-source cross attention fusion is constructed by introducing a double-branch encoder, an autonomously constructed CAFFM (cross attention feature fusion module) and a feature decoder CU2C module, and is used for training and prediction of multi-source data target segmentation in different fields; and sending the data in the training set and the corresponding labels into the segmentation network for training the segmentation network of the multi-source data target segmentation task, calculating a loss function of the segmentation network, iteratively updating parameters, and stopping training if the accuracy of the network test set is not improved after training for a certain round.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation and target segmentation, and in particular to a target segmentation method based on multi-source cross attention fusion. Background Art

[0002] Image segmentation is one of the important fields of artificial intelligence. Analyzing a large amount of image data can build a more appropriate segmentation network model, so as to automatically identify and classify specific targets in different fields and improve the accuracy and efficiency of target segmentation. In image segmentation tasks in different fields, a single information source often cannot provide enough details, especially in complex backgrounds or blurred boundaries. For example, in the retinal arteriovenous segmentation task, this task involves the accurate distinction and segmentation of arteriovenous vessels in retinal images, which is challenging because arteries and veins often show similar structural features in images, and due to the complex background, interlaced blood vessels, and image resolution limitations, single-modality image data is usually difficult to obtain accurate segmentation results; in the remote sensing image object classification task, this task involves the accurate distinction and segmentation of different objects in remote sensing images, which is also challenging. Due to the complexity and diversity of a single SAR image, traditional object classification methods based on artificial rules and mathematical models are often limited by the limitations of feature selection and classification algorithms, and it is difficult to handle complex object coverage and extract detailed features. Although SAR images have the ability to overcome complex environments, they contain limited information and it is difficult to obtain accurate object classification results. Therefore, designing a general and good segmentation method is crucial for multiple fields.

[0003] Due to the complexity and diversity of image segmentation tasks in different fields, the segmentation method based on a single source lacks consideration of the multi-faceted features of the image, thus affecting the accuracy of segmentation. For example, in the field of medical image segmentation, in 2022, Hu et al. proposed a multi-scale interactive network with an A / V discriminator for retinal artery and vein recognition. The model has high accuracy in three publicly available fundus image datasets (AV-DRIVE, HRF, LES-AV). However, some small blood vessels are still difficult to segment; in the field of remote sensing images, in 2021, Huang et al. proposed a new unsupervised learning method for learning the discriminative physical perception features of synthetic aperture radar images using deep neural networks, and designed a convolutional neural network for extracting high-level features from detected SAR images. However, SAR images have the ability to overcome complex environments, but the information they contain is limited, while optical images have more details but are not feasible for complex environments. Therefore, with the development of deep learning and multimodal data fusion technology, the method of multi-source cross-attention fusion has gradually become a research hotspot in the field of retinal vascular artery and vein segmentation. By combining multi-source data from different imaging modalities, these methods can capture richer vascular feature information, thereby significantly improving segmentation accuracy. For example, color fundus images and optical coherence tomography angiography (OCTA) in retinal images each have unique advantages. Color fundus images can provide color and morphological information of blood vessels, while OCTA can display the three-dimensional structure and depth information of blood vessels. However, using the information of these modalities alone may not be enough to deal with the accurate distinction and segmentation of arteries and veins. In the field of remote sensing image segmentation, the fusion of multi-source data also shows great potential. Traditional remote sensing image segmentation methods are usually based on a single optical image or SAR data, which makes it difficult to take into account both the details of the image and the ability to adapt to the environment. In order to overcome these challenges, a segmentation method based on multi-source fusion of optical and SAR images can be adopted. Therefore, designing a segmentation network based on multi-source cross-attention fusion, combining information sources of different modalities in different fields, can obtain richer contextual information and improve the accuracy of target segmentation.

[0004] In addition, the target segmentation method based on multi-source cross-attention fusion faces three major challenges: first, the accuracy problem. The method needs to accurately identify different types of targets in different fields and overcome the differences in data sources and fusion difficulties; second, the real-time problem. The method must quickly process and analyze a large amount of image data to provide timely segmentation results; finally, the versatility problem. The system needs to have good generalization capabilities, adapt to image sources in different fields, and handle various data formats and resolutions. Therefore, in order to overcome these problems, it is of great practical significance to design a target segmentation method based on multi-source cross-attention fusion. Summary of the invention

[0005] The purpose of the present invention is to provide a target segmentation method based on multi-source cross-attention fusion for target segmentation combining multiple source data sources in different fields to improve segmentation accuracy and solve the problems raised in the background technology, in view of the low target segmentation accuracy and insufficient generalization ability caused by factors such as the information limitation of a single data source in the prior art.

[0006] In different fields, the target segmentation method of multi-source cross-attention fusion integrates multi-source data in the field, fully utilizes the advantages of different data sources, and extracts richer feature information, thereby improving the effect of target segmentation. It has broad application value in various fields such as smart medical care, smart remote sensing, and land planning.

[0007] In order to achieve the above object, the present invention provides the following technical solution: a target segmentation method based on multi-source cross attention fusion, comprising:

[0008] Acquire multi-source image data and perform corresponding preprocessing operations on the data set;

[0009] Construct a segmentation network based on multi-source image data fusion. By introducing a dual-branch encoder, a self-built CAFFM (Cross Attention Feature Fusion Module) and a feature decoder CU2C module, a segmentation network based on multi-source cross attention fusion is constructed for training and prediction of multi-source data target segmentation in different fields.

[0010] The data and corresponding labels in the training set are sent to the segmentation network to train the segmentation network for the multi-source data target segmentation task. The loss function of the segmentation network is calculated and the parameters are updated iteratively. After a certain number of training rounds, if the accuracy of the network test set no longer improves, the training is stopped.

[0011] Select the optimal model weights and test the training effect of the model in the multi-source test set;

[0012] The test model is effective in the retinal artery and vein segmentation task in the medical field and the land object classification task in the remote sensing field.

[0013] Preferably, Figure 1As shown in the figure, the preprocessing operation is as follows: before training the segmentation model for the general segmentation task, the multi-source data is first preprocessed. For different multi-source data sets in different fields, the image preprocessing is different. For example, in the multi-source ground object classification task in the remote sensing field, the image preprocessing is to first read the single-channel SAR image data. Considering that the single-channel SAR image only has phase and amplitude features, but in practice the phase of the single-channel SAR system does not provide useful information, and the amplitude (or intensity) is the only useful information. The absolute value function is used to extract the amplitude feature of the SAR image, and the optical image data is read and converted into a grayscale image. The edge detection factor is used to extract the contour of the ground object in the optical image from the grayscale image, and the edge information is enhanced to the original optical image, thereby improving the distinction between different ground objects and facilitating feature learning; after image preprocessing, the data set is divided into training set, validation set, and test set in a ratio of 8:1:1, and finally random horizontal rotation and random vertical flipping are used for data enhancement.

[0014] Preferably, Figure 1 As shown in the figure, in the retinal arteriovenous segmentation task in the medical field, image preprocessing takes into account the high contrast of blood vessels in optical fundus images, which helps to identify the rough morphology of blood vessels, while the detailed information in OCTA images helps to accurately distinguish arteries and veins. Therefore, the optical fundus images are first grayed and the CLAHE contrast is enhanced. The OCTA images are denoised and data enhanced. The optical and corresponding OCTA fundus images are randomly cropped to increase the number of data sets and improve the generalization ability of the model. After image preprocessing, the data set is divided into training set, validation set, and test set in a ratio of 8:1:1. Finally, random horizontal rotation and random vertical flipping are used for data enhancement.

[0015] Preferably, the segmentation network based on multi-source image data fusion includes a dual-branch encoder module (Encoder), a self-built CAFFM module (cross attention feature fusion module) and a feature decoder CU2C module (Decoder); firstly, considering that the target segmentation method based on a single source lacks consideration of many detailed features of the image, thereby affecting the classification accuracy, a dual-branch encoder is used to extract features of multi-source image data of the same target, such as Figure 2 As shown in Encoder;

[0016] The dual-branch encoder module uses ResNet-50 as the backbone network, and selects shallower and deeper feature layers from it respectively; for low-level features, the output of the shallow Stage2 of the corresponding network is selected, and the low-level features of data source D1 and data source D2 are marked as and For high-level features, the output of the deepest layer Stage4 of the corresponding network is selected and marked as the high-level features of data source D1 and data source D2 respectively. and

[0017] In order to deal with the complex background problem of optical images, the pre-trained Resnet50 weights pre-trained on the ImageNet benchmark dataset are loaded into the optical feature extraction branch in the Encoder. The previously learned feature representation can be used to accelerate the model learning of feature information and feature convergence in optical images. Since non-optical images do not have optical features, the pre-trained weights are not processed. In order to make full use of the prominent features and complementary features of D1 and D2 images to make the classification of objects more accurate, it is decided to use the self-built CAFFM module, which is as follows: Figure 2 As shown in CAFFM, the CAFFM module converts the low-level feature maps of data source D1 and data source D2 output by stage2 and High-level feature graphs of data sources D1 and D2 output by stage4 and Perform CAFFM feature fusion respectively, the CAFFM module is improved on the basis of the attention fusion model;

[0018] The specific improvements are as follows:

[0019] First, perform a similarity operation on the Q feature of data source D1 and the K feature of data source D2, that is, compare the similarity between each element in the D1 feature and the corresponding element in the D2 feature, and determine the output of the V feature of D2 accordingly;

[0020] At the same time, the similarity operation is performed on the Q feature of D2 and the K feature of D1, that is, the similarity between each element in the D2 feature and the corresponding element in the D1 feature is compared, and the output of the V feature of D1 is determined accordingly; the Hadamard product of the two features is performed to fuse the features, so as to obtain and In short, the cross-modal attention fusion module quantifies the similarities between features of different modalities and then adjusts the weights of features based on these similarities to achieve interaction and fusion of cross-modal information.

[0021] at last, By comparing with D1 and D2 low-level feature maps and Perform concatenation and convolution to obtain low-level fusion feature maps of D1 and D2 By comparing with D1 and D2 high-level feature maps and Perform splicing, ASPP, and upsampling to obtain D1 and D2 high-level fusion feature maps Finally, in order to make the final output segmentation result map better display the learned features, it is decided to use the CU2C decoder module, which consists of a splicing, an upsampling layer, and two convolutional layers. First, the D1 and D2 low-level fusion feature maps obtained in the Encoder are And D1, D2 advanced fusion feature map Channel splicing is performed, and then the feature map size is restored to the original image size through the upsampling layer. Finally, the feature map is convolved twice to obtain the output result map of the target segmentation.

[0022] Preferably, the segmentation network is optimized through the comprehensive loss function of CrossEntropyLoss and Di ceLoss, and its comprehensive loss function is as follows:

[0023] Total Loss=α×CrossEntropyLoss+β×DiceLoss;

[0024] Where CrossEntropyLoss is the cross entropy loss, DiceLoss is the Dice loss, α and β are weight parameters, which are used to balance the relative importance of CrossEntropyLoss and DiceLoss in the total loss, that is, to balance the classification accuracy and the coherence of the overall segmentation; and before model training, the initial weight of each target category is set according to the proportion of the target category in the data, and the category with a lower proportion is given a higher degree of attention.

[0025] The present invention has the following advantages:

[0026] The present invention solves the problem of information limitation of a single data source. It utilizes multi-source data information fusion to automatically and accurately classify the target and accurately segment the area. It is used for target segmentation combining multi-source data sources in different fields to improve segmentation accuracy. It can provide reference and service for various fields such as remote sensing and medicine. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A diagram of the data preprocessing process provided by the present invention;

[0028] Figure 2 The segmentation network framework diagram based on multi-source cross attention fusion provided by the present invention;

[0029] Figure 3 The segmentation network framework diagram based on light and SAR cross-attention fusion provided in this embodiment;

[0030] Figure 4Diagram of the optical and SAR data preprocessing process provided for this implementation case;

[0031] Figure 5 The segmentation result diagram provided in this embodiment;

[0032] Figure 6 The segmentation network framework diagram based on light and OCTA cross-attention fusion provided in this embodiment. DETAILED DESCRIPTION

[0033] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] Implementation Case 1: The target segmentation method based on multi-source cross-attention fusion provided by the present invention is used in the optical and SAR object classification tasks in the remote sensing field. First, the input optical and SAR image data are preprocessed, and then the data and labels in the training set are input into the segmentation network for training. The segmentation network weights for object classification in the training image are trained. After a certain number of training iterations, the average accuracy of the network reaches fitting, and the network weights are saved. The optical and SAR image data in the test set are input to obtain the object classification result map of the corresponding image, and the object classification of the multi-source image is completed. Figure 3 As shown in Figure 1, the object classification method based on light and SAR cross-attention fusion mainly includes the following steps:

[0035] Step 1: The WHU-OPT-SAR dataset, which provides a wide range of remote sensing images of different terrains and vegetation in a province, a total of 100 5556×3704 pixel optical images (OPT) covering an area of ​​50,000 square kilometers, synthetic aperture radar (SAR) images of the same area, and image labels with pixel-level annotations corresponding to background, farmland, city, village, water body, forest, road and other 8 categories, is used as the dataset for this study. In order to prevent the model from overfitting and improve the generalization performance, the dataset is preprocessed. The preprocessing process is as follows: Figure 4 shown.

[0036] Firstly, the single-channel SAR image data is read. Considering that the single-channel SAR image only has phase and amplitude features, but in practice the phase of the single-channel SAR system does not provide useful information, and the amplitude (or intensity) is the only useful information, the absolute value function is used to extract the amplitude feature of the SAR image; the optical image data is read and converted into a grayscale image, and the edge detection factor is used to extract the contour of the ground object in the optical image from the grayscale image, and the edge information is enhanced to the original optical image, thereby improving the distinction between different ground objects and facilitating feature learning; the data set is divided into training set, validation set, and test set in a ratio of 8:1:1, and finally random horizontal rotation and random vertical flipping are used for data enhancement.

[0037] Step 2: Model construction. The model of the present invention includes three parts: a dual-branch encoder (Encoder), a self-built CAFFM (Cross Attention Feature Fusion Module) and a feature decoder CU2C module (Decoder). First, considering that the single-source-based object classification method lacks consideration of many detailed features of the image, thus affecting the classification accuracy, a dual-branch encoder is used to extract features from optical images and SAR images, such as Figure 3 As shown in Encoder.

[0038] The Encoder module uses ResNet-50 as the backbone network, and selects shallower and deeper feature layers from it. For low-level features, the output of the shallow Stage2 of the corresponding network is selected, and the low-level features of optical and SAR are marked as and For high-level features, the output of the deepest layer Stage 4 of the corresponding network is selected and marked as the high-level features of optical and SAR respectively. and In order to deal with the complex background problem of optical images, the pre-trained Resnet50 weights pre-trained on the ImageNet benchmark dataset are loaded into the optical feature extraction branch in the Encoder. The previously learned feature representation can be used to accelerate the model learning of the ground feature information in the optical image and the convergence process of the ground feature. Since SAR images do not have optical features, the pre-trained weights are not processed. In order to make full use of the prominent features and complementary features of optical and SAR images to make the ground feature classification more accurate, it is decided to use the self-built CAFFM module, whose modules are as follows Figure 3 As shown in CAFFM, this module converts the optical and SAR low-level feature maps output by stage2 into and Optical and SAR high-level feature maps output by stage4 and CAFFM feature fusion is performed separately. The CAFFM module is improved into a cross-attention feature fusion module based on the attention fusion model. First, the optical Q feature and the SAR K feature are subjected to similarity calculation, that is, the similarity between each element in the optical feature and the corresponding element in the SAR feature is compared, and the output of the SAR V feature is determined accordingly. At the same time, the SAR Q feature and the optical K feature are subjected to similarity calculation, that is, the similarity between each element in the SAR feature and the corresponding element in the optical feature is compared, and the output of the optical V feature is determined accordingly. The features of the two are subjected to Hadamard product for feature fusion, so as to obtain and In short, the cross-modal attention fusion module quantifies the similarities between features of different modalities and then adjusts the weights of features based on these similarities to achieve interaction and fusion of cross-modal information. By comparing with optical and SAR low-level feature maps and Perform concatenation and convolution to obtain low-level fusion feature maps of optical and SAR Through the advanced feature map of optics and SAR and Perform stitching, ASPP, and upsampling to obtain optical and SAR advanced fusion feature maps Finally, in order to make the final output classification result map better display the learned features, it is decided to use the CU2C decoder module, which consists of a splicing, an upsampling layer, and two convolutional layers. First, the optical SAR low-level fusion feature map obtained in the Encoder Heguang SAR advanced fusion feature map Channel splicing is performed, and then the feature map size is restored to the original image size through the upsampling layer. Finally, the feature map is convolved twice to obtain the output result map of the object classification.

[0039] Step 3: training model. The present invention is carried out on a workstation equipped with an NVID IA-RTX4070 GPU, the programming language is Python, the PyTorch deep learning framework is used, Adam is used as the optimizer for model training, the comprehensive loss function of CrossEntropyLoss and Di ceLoss is used as the loss function for model training, α and β are set to 1.2 and 0.8 respectively, the batch size is 16, and the learning rate is 0.001. Before model training, the initial weights of background, farmland, city, village, water body, forest, road and other 8 categories are set according to the proportion of the ground object category in the data [0.0, 0.0162, 0.1189, 0.0959, 0.0392, 0.01487, 0.5864, 0.3186], and the category with a lower proportion is given a higher degree of attention. The optical and SAR images of the training data are input into the segmentation network based on the cross-attention fusion of light and SAR for training. After 40 rounds of training, the model is gradually fitted, and the model parameters with the best training effect are saved in each iteration during training.

[0040] Step 4: Evaluate the model. In order to verify the performance of the model, the model is tested on the test set. The optical and SAR images of the test set are also preprocessed and sent to the optimal weight network to output the predicted segmentation results. The segmentation results are shown in the figure below. Figure 5 shown.

[0041] Implementation Case 2: The segmentation method based on multi-source cross-attention fusion provided by the present invention is used in the retinal arteriovenous segmentation task in the medical field. First, the input optical and OCTA image data are preprocessed, and then the data and labels in the training set are input into the segmentation network for training. The segmentation network weights for classifying objects in the training image are trained. After a certain number of training iterations, the average accuracy of the network reaches fitting, and the network weights are saved. The optical and OCTA images in the test set are input to obtain the retinal arteriovenous segmentation result map of the corresponding image, completing the arteriovenous segmentation task of the multi-source image. Figure 3 As shown in the figure, the retinal arteriovenous segmentation method based on optical and OCTA cross-attention fusion mainly includes the following steps:

[0042] Step 1: 40 retinal fundus optical images and corresponding OCTA image data are used as the data set for this study. In order to prevent the model from overfitting and improve the generalization performance, the data set is preprocessed. Considering that the vascular contrast in the optical fundus image is high, it is helpful to identify the rough morphology of the blood vessels, and the detailed information in the OCTA image helps to accurately distinguish between arteries and veins. Therefore, the optical fundus image is first grayed, the CLAHE contrast is enhanced, and the OCTA image is denoised and data enhanced. The number of data sets is increased by randomly cropping the optical and corresponding OCTA fundus images to improve the generalization ability of the model; the data set is divided into training set, validation set, and test set in a ratio of 8:1:1, and finally random horizontal rotation and random vertical flipping are used for data enhancement.

[0043] Step 2: Model construction. The model of the present invention includes three parts: a dual-branch encoder (Encoder), a self-built CAFFM (Cross Attention Feature Fusion Module) and a feature decoder CU2C module (Decoder). First, considering that the target segmentation method based on a single source lacks consideration of many detailed features of the image, thus affecting the classification accuracy, a dual-branch encoder is used to extract features of optical images and OCTA images, such as Figure 6 The encoder module uses ResNet-50 as the backbone network, and selects shallower and deeper feature layers from it. For low-level features, the output of the shallow Stage2 of the corresponding network is selected, and the low-level features of optics and OCTA are marked as and For high-level features, the output of the deepest layer Stage 4 of the corresponding network is selected and labeled as the high-level features of optical and OCTA respectively. and In order to deal with the complex background problem of fundus retinal images, the features of fundus images are different from those of ordinary optical images, and the pre-trained weights are not processed; in order to make full use of the prominent and complementary features of optical and OCTA images to make arteriovenous segmentation more accurate, it is decided to use the self-built CAFFM module, whose modules are as follows Figure 3 As shown in CAFFM, this module converts the optical and OCTA low-level feature maps output by stage2 into and Optical and OCTA advanced feature maps output by stage4 and CAFFM feature fusion is performed separately. The CAFFM module is improved into a cross-attention feature fusion module based on the attention fusion model. First, the optical Q feature and the OCTA K feature are similarity calculated, that is, the similarity between each element in the optical feature and the corresponding element in the OCTA feature is compared, and the output of the OCTA V feature is determined accordingly. At the same time, the OCTA Q feature and the optical K feature are similarity calculated, that is, the similarity between each element in the OCTA feature and the corresponding element in the optical feature is compared, and the output of the optical V feature is determined accordingly. The Hadamard product of the two features is performed for feature fusion to obtain and In short, the cross-modal attention fusion module quantifies the similarities between features of different modalities and then adjusts the weights of features based on these similarities to achieve interaction and fusion of cross-modal information. By comparing with optical and OCTA low-level feature maps and Perform concatenation and convolution to obtain low-level fusion feature maps of optical and SAR Through the advanced features of optical and OCTA and Perform stitching, ASPP, and upsampling to obtain optical and OCTA advanced fusion feature maps Finally, in order to make the final output classification result map better display the learned features, it is decided to use the CU2C decoder module, which consists of a splicing, upsampling layer, and two convolutional layers. First, the optical OCTA low-level fusion feature map obtained in the Encoder Heguang OCTA advanced fusion feature map Channel splicing is performed, and then the feature map size is restored to the original image size through the upsampling layer. Finally, the feature map is convolved twice to obtain the output result map of retinal arteriovenous segmentation.

[0044] Step 3: training model. The present invention is carried out on a workstation equipped with an NVID IA-RTX4070 GPU, the programming language is Python, the PyTorch deep learning framework is used, Adam is used as the optimizer for model training, the comprehensive loss function of CrossEntropyLoss and Di ceLoss is used as the loss function for model training, α and β are set to 0.8 and 1.2 respectively, the batch size is 16, and the learning rate is 0.001. Before model training, the initial weights of the three categories of background, artery, and vein are set according to the proportion of arteriovenous blood vessel categories in the data [0.0, 0.6674, 0.5372]. The lower the proportion of the category, the higher the attention is given. The optical and OCTA images of the training data are input into the segmentation network based on the cross-attention fusion of light and OCTA for training. After 75 rounds of training, the model is gradually fitted, and the model parameters with the best training effect are saved in each iteration during training.

[0045] Step 4: Evaluate the model. In order to verify the performance of the model, the model was tested on the test set. The optical and OCTA images of the test set were also preprocessed and sent to the optimal weight network to output the predicted segmentation results. Its multi-source cross-attention fusion retinal artery and vein segmentation algorithm has better effect and higher accuracy than the segmentation algorithm of a single optical image.

[0046] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A target segmentation method based on multi-source cross attention fusion, obtaining multi-source image data and performing corresponding preprocessing operations on the data set, characterized in that: include: Construct a segmentation network based on multi-source image data fusion. By introducing a dual-branch encoder, a self-built CAFFM module and a feature decoder CU2C module, a segmentation network based on multi-source cross-attention fusion is constructed for training and prediction of multi-source data target segmentation in different fields. The data in the training set and the corresponding labels are sent to the segmentation network to train the segmentation network for the multi-source data target segmentation task. By calculating the loss function of the segmentation network and iteratively updating the parameters, after a certain number of training rounds, the accuracy of the network test set no longer improves and the training is stopped.

2. The target segmentation method based on multi-source cross attention fusion according to claim 1, characterized in that: The preprocessing operations are as follows: before training the segmentation model for general segmentation tasks, the multi-source data are first preprocessed. The absolute value function is used to extract the amplitude characteristics of the SAR image, the optical image data is read and converted into a grayscale image, and the edge detection factor is used to extract the contour of the ground objects in the optical image from the grayscale image, and the edge information is enhanced to the original optical image; after image preprocessing, the data set is divided into training set, validation set, and test set in a ratio of 8:1:1, and finally random horizontal rotation and random vertical flipping are used for data enhancement.

3. The target segmentation method based on multi-source cross attention fusion according to claim 1, characterized in that: In the retinal arteriovenous segmentation task in the medical field, the optical fundus images are first grayscaled and CLAHE contrast enhanced, the OCTA images are denoised and data enhanced, and the number of data sets is increased by randomly cropping the optical and corresponding OCTA fundus images; after image preprocessing, the data set is divided into training set, validation set, and test set in a ratio of 8:1:1, and finally random horizontal rotation and random vertical flipping are used for data enhancement.

4. The target segmentation method based on multi-source cross attention fusion according to claim 1, characterized in that: The segmentation network based on multi-source image data fusion includes a dual-branch encoder module, a self-built CAFFM module and a feature decoder CU2C module; a dual-branch encoder is used to extract features from multi-source image data of the same target; The dual-branch encoder module uses ResNet-50 as the backbone network, and selects shallower and deeper feature layers from it respectively; For low-level features, we select the output of the shallow Stage 2 of the corresponding network and mark the low-level features of data source D1 and data source D2 as and For high-level features, the output of the deepest layer Stage4 of the corresponding network is selected and marked as the high-level features of data source D1 and data source D2 respectively. and The CAFFM module converts the low-level feature maps of data source D1 and data source D2 output by stage2 and High-level feature graphs of data sources D1 and D2 output by stage4 and Perform CAFFM feature fusion respectively, the CAFFM module is improved on the basis of the attention fusion model; The specific improvements are as follows: First, perform a similarity operation on the Q feature of data source D1 and the K feature of data source D2, that is, compare the similarity between each element in the D1 feature and the corresponding element in the D2 feature, and determine the output of the V feature of D2 accordingly; At the same time, the similarity operation is performed on the Q feature of D2 and the K feature of D1, that is, the similarity between each element in the D2 feature and the corresponding element in the D1 feature is compared, and the output of the V feature of D1 is determined accordingly; the Hadamard product of the two features is performed to fuse the features, so as to obtain and at last, By comparing with D1 and D2 low-level feature maps and Perform concatenation and convolution to obtain low-level fusion feature maps of D1 and D2 By comparing with D1 and D2 high-level feature maps and Perform splicing, ASPP, and upsampling to obtain D1 and D2 high-level fusion feature maps The CU2C decoder module is used, which consists of a concatenation layer, an upsampling layer, and two convolutional layers. First, the D1 and D2 low-level fusion feature maps obtained in the Encoder are And D1, D2 advanced fusion feature map Channel splicing is performed, and then the feature map size is restored to the original image size through the upsampling layer. Finally, the feature map is convolved twice to obtain the output result map of the target segmentation.

5. The target segmentation method based on multi-source cross attention fusion according to claim 1, characterized in that: The segmentation network achieves the optimal result through the comprehensive loss function of CrossEntropyLoss and DiceLoss. The comprehensive loss function is as follows: TotalLoss=α×CrossEntropyLoss+β×DiceLoss; In the formula, CrossEntropyLoss is the cross entropy loss, DiceLoss is the Dice loss, and α and β are weight parameters.

Images