A cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching

Abstract

The present invention provides a cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching, including: extracting regions of interest using a semantic segmentation algorithm through an image preprocessing module, segmenting remote sensing images into multiple regions and generating image blocks; extracting global features, regional features, and pixel-level features of the image respectively, and performing fine-grained encoding on key areas such as fine-grained ground feature edges; a text multi-level encoding module performs feature encoding on the document, sentence, and word levels of the text based on a pre-trained language model to ensure multi-level understanding of the text; in a multi-level matching and fusion module, the similarity between the remote sensing image and the text description is calculated through a cross-attention mechanism, and weighted fusion of features at all levels is performed to finally output a retrieval score. This method not only improves the accuracy and robustness of image and text retrieval, but can also be widely used in remote sensing monitoring, environmental change identification, geographic information systems and other fields.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching. Background Art

[0002] Remote sensing image and text retrieval is a multimodal information retrieval technology that combines remote sensing imagery with natural language text descriptions. With the advancement of remote sensing technology, the application of remote sensing imagery is becoming increasingly broad, encompassing environmental monitoring, urban planning, disaster warning, agricultural assessment, and other areas. However, the high dimensionality, complexity, and specialized nature of remote sensing imagery make traditional image retrieval methods challenging in practical applications. Therefore, how to effectively extract useful information from remote sensing imagery and match it with relevant text descriptions has become a key issue.

[0003] The significance of remote sensing image and text retrieval lies in its ability to efficiently retrieve remote sensing data by leveraging the semantic associations between text and imagery. For example, a natural language query input by a user can quickly locate images relevant to the query within a vast amount of remote sensing imagery, significantly improving information retrieval efficiency. Furthermore, remote sensing image and text retrieval supports cross-modal learning. Even if the feature representations of remote sensing images and text differ, the system can still achieve a comprehensive understanding of data from different modalities through effective feature alignment and matching.

[0004] Many existing methods rely on simple keyword-based matching or crude feature similarity calculations, failing to fully understand the deep semantics in images and text. For example, traditional methods based on convolutional networks, due to the limitations of their receptive field, can often only capture information in local areas, neglecting the global information and long-range dependencies in remote sensing images. Existing methods also typically focus on macro-level image and text features, while insufficiently addressing fine-grained features in remote sensing images. This results in poor robustness of retrieval results when dealing with fine-grained problems. Therefore, developing a practical and efficient remote sensing image and text retrieval method has become an urgent need. Summary of the Invention

[0005] The purpose of the present invention is to provide a cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching.

[0006] To achieve the above objectives, the present invention provides a cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching, comprising the following steps:

[0007] S1, extract the region of interest using the semantic segmentation algorithm through the image preprocessing module, segment the remote sensing image into multiple regions and generate image blocks;

[0008] S2: A hybrid architecture of a convolutional neural network and a Transformer model combined with a multi-level visual attention mechanism is used to extract global, regional, and pixel-level features of the image in a multi-level feature extraction module, and to perform fine-grained encoding of key areas such as fine-grained object edges.

[0009] S3. Use a multi-level text encoding module to encode text features at the document, sentence, and word levels based on a pre-trained language model, improving the model's multi-level understanding of text;

[0010] S4. In the multi-level matching and fusion module, the similarity between remote sensing images and text descriptions is calculated through the cross-attention mechanism and dynamic fusion mechanism, and the features at each level are weighted and fused;

[0011] S5. Based on multi-level feature matching and fusion, output the image and text retrieval score.

[0012] Furthermore, step S1 is specifically as follows:

[0013] S1.1-1. Use a filter-based denoising method (Gaussian filtering) to denoise the image. Gaussian filtering is a commonly used image smoothing technique used to remove noise from an image while preserving the image's edge information. Suppose the original input image is I(x,y), the Gaussian filter is G(x,y), and the mathematical expression of the two-dimensional Gaussian function is:

[0014] ;

[0015] Where G(x,y) is the value of the Gaussian function at position (x,y), σ is the standard deviation of the Gaussian function, and x,y are the coordinates of the pixel, the offset relative to the current pixel;

[0016] S1.1-2. Fill the edges of the input image I (x, y) and then perform convolution on each pixel with the Gaussian kernel to generate a filtered image. , calculate the output value of each pixel The formula is as follows:

[0017] ;

[0018] Where I(xi,yi) is the pixel value of the original image at position (xi,yi); G(x,y) is the weight of the position (x,y) in the Gaussian kernel, and k represents the radius of the kernel.

[0019] S1.1-3. The Gaussian kernel needs to be normalized so that the sum of all weights is equal to 1. The normalization formula is as follows:

[0020] ;

[0021] Where G(x,y) is the weight of the position (x,y) in the Gaussian kernel, k represents the radius of the kernel, and x,y are the coordinates of the pixel;

[0022] S1.2-1. Use semantic segmentation algorithms to extract regions of interest. Semantic segmentation algorithms use computer vision technology to assign specific semantic labels to each pixel in an image, achieving accurate segmentation of different regions or objects in the image. Remote sensing images are divided into multiple regions and image blocks are generated. The network performs feature extraction on the image through convolution operations, extracting low-level (such as edges, textures, etc.) and high-level (such as objects, regions, etc.) features. The formula for the convolution operation is:

[0023] ;

[0024] Where p is the input image, is the convolution kernel, f is the output after convolution;

[0025] S1.2-2. Pooling operation is used to reduce the spatial resolution of the image, reduce the amount of calculation, and retain the features. The formula of the pooling operation is:

[0026] ;

[0027] Where R is the pooling window, p is the input image, and Z is the output after pooling;

[0028] S1.2-3. After multiple convolutions and pooling, the image resolution will be reduced, so it is necessary to restore the spatial resolution of the image through upsampling (deconvolution). The formula for the deconvolution operation is:

[0029] ;

[0030] Among them, I(x,y) is the upsampled image pixel, G(i,j) is the deconvolution kernel;

[0031] S1.2-4. The output of each pixel is the probability that it belongs to each category. Normalized exponential function is usually used to convert these scores into probability values. The formula of normalized exponential function is:

[0032] ;

[0033] in, It's a pixel belong The score, It's a pixel Belong to category The probability of , C is the total number of categories;

[0034] S1.2-5. After classification, each pixel of the image will be assigned a category label. Based on the category label, the area of specific category can be extracted to form the area of interest (AOI). The formula for extracting AOI is:

[0035] ;

[0036] in, AOI is the region of interest, Is the pixel (x,y) belongs to the category The probability of , threshold is the set threshold;

[0037] Furthermore, step S2 is specifically as follows:

[0038] S2.1-1. In global feature extraction, convolutional layers with larger kernels (e.g., 5*5 or 7*7) are typically used to process images. This helps the network capture a wider range of contextual information, thereby gaining a global understanding of the image. The formula for the convolution operation is shown in S1.2-1.

[0039] S2.1-2. When extracting global features, the Transformer model uses the self-attention mechanism to model the global dependencies in the image. The self-attention mechanism is the core component of the Transformer model. It is used to calculate the correlation between elements in the input sequence and help the model capture the dependencies between elements at different positions in the sequence, no matter how far apart they are. Suppose a feature matrix is input ,in , is a set of real numbers. The attention output is calculated by query, key, and value. The formula is as follows:

[0040] ;

[0041] ;

[0042] Among them, Q, K, V, T are query, key, value matrix and transpose respectively. is the dimension of the key vector, A is the calculated attention weight matrix, It is the weighted global feature output, and softmax is a normalized exponential function. Its core function is to convert a set of input values into a probability distribution.

[0043] S2.1-3. Calculate the attention weight through the visual attention mechanism to focus on the key areas of the image. The formula is as follows:

[0044] ;

[0045] in, is the weighted global feature output, is the global attention weight, and softmax is a normalized exponential function whose core function is to convert a set of input values into a probability distribution;

[0046] S2.2-1. In regional feature extraction, a small convolution kernel (e.g., 3*3 or 5*5) is usually used to extract features from the image using a sliding window method. By extracting regional features, the network can focus on the features of different parts of the image. The formula for the convolution operation is shown in S1.2-1.

[0047] S2.2-2. Use pooling to unify the size of the region and provide a fixed-size regional feature map for subsequent processing. Then calculate the weighted output of the regional feature. The formula is as follows:

[0048] ;

[0049] Among them, m is the input region feature map, It is regional pooling, is the maximum pooling, is average pooling;

[0050] S2.2-3. The network can adaptively focus attention on relevant areas of the image, especially areas with key semantic information. The cross-attention mechanism formula is as follows:

[0051] ;

[0052] in, is the regional feature extracted by the network, is the regional attention weight, and softmax is a normalized exponential function, whose core function is to convert a set of input values into a probability distribution;

[0053] S2.3-1. In pixel-level feature extraction, we usually focus on fine-grained features of the image, such as edges of objects and detailed textures. The formula for the convolution operation is shown in S1.2-1. The Sobel operator is used to calculate the gradient of the image and extract edges. Its convolution kernel and the calculated image gradient strength are as follows:

[0054] ; ;

[0055] ;

[0056] S2.3-2. In pixel-level feature extraction, the cross-attention mechanism assigns weights to each pixel, as follows:

[0057] ;

[0058] in, is the pixel feature extracted by the network, is the pixel attention weight, and softmax is the normalized exponential function, whose core function is to convert a set of input values into a probability distribution.

[0059] Furthermore, step S3 is specifically as follows:

[0060] S3.1-1. Assume that the input text T is segmented into document level (M sentences), each sentence Further word segmentation , using BERT as the basic encoder to generate word, sentence, and document-level features through pixel features extracted by the network. It achieves a deep semantic understanding of text through a bidirectional self-attention mechanism, significantly improving the performance of multiple natural language processing tasks. The word-level feature formula is as follows:

[0061] ;

[0062] in, is the set of real numbers, Represents a dimension The real matrix space of , L is the total length of the sequence, and d is the BERT hidden layer dimension (such as 768);

[0063] S3.1-2, for each sentence Word-level features Perform average pooling, the formula is as follows:

[0064] ;

[0065] ;

[0066] Among them, N is the number of words in each sentence, It is a characteristic of the sentence. is the set of all sentence features;

[0067] S3.1-3. Sentence-level features Use self-attention aggregation and output as global document vector , the formula is as follows:

[0068] ;

[0069] Furthermore, step S4 is specifically as follows:

[0070] The dynamic fusion mechanism is a strategy for multimodal or multi-level feature fusion. Its core idea is to adaptively allocate weights between different features or information sources based on specific input and task requirements to achieve more flexible and effective feature integration. It performs weighted summation of document-level, sentence-level, and word-level features, and weights the contribution of each layer through learned weights. The weighted fusion formula is:

[0071] ;

[0072] in, is a document-level feature, is a sentence feature, It is a word-level feature.

[0073] Beneficial effects: The present invention can capture the global semantics and local detail information of images and texts through multi-level feature extraction and alignment. At the same time, it utilizes the semantic collaborative matching mechanism to improve the semantic consistency between image and text features, enhance the accuracy of image-text matching, and through fine-grained encoding, it can effectively capture tiny features such as edges and textures of objects in remote sensing images, thereby improving the precision of retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic flow diagram of the present invention;

[0075] Figure 2 The pre-processing module of the present invention adopts the principle diagram of Gaussian filtering;

[0076] Figure 3 It is a schematic diagram of the principle of the multi-level visual attention mechanism adopted by the present invention;

[0077] Figure 4 This is a flow chart of the present invention using the Transformer model in the multi-level feature extraction module. DETAILED DESCRIPTION

[0078] The present invention will be further described below with reference to the accompanying drawings.

[0079] like Figure 1 As shown in FIG, a cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching includes the following steps:

[0080] S1, extract the region of interest using the semantic segmentation algorithm through the image preprocessing module, segment the remote sensing image into multiple regions and generate image blocks;

[0081] S2: A hybrid architecture of a convolutional neural network and a Transformer model combined with a multi-level visual attention mechanism is used to extract global, regional, and pixel-level features of the image in a multi-level feature extraction module, and to perform fine-grained encoding of key areas such as fine-grained object edges.

[0082] S3. Use a multi-level text encoding module to encode text features at the document, sentence, and word levels based on a pre-trained language model, improving the model's multi-level understanding of text;

[0083] S4. In the multi-level matching and fusion module, the similarity between remote sensing images and text descriptions is calculated through the cross-attention mechanism and dynamic fusion mechanism, and the features at each level are weighted and fused;

[0084] S5. Based on multi-level feature matching and fusion, output the image-text retrieval score;

[0085] As a preferred implementation, step S1 specifically includes:

[0086] S1.1-1, use the filter-based denoising method (Gaussian filtering) to denoise the image, such as Figure 1 As shown in the figure, Gaussian filtering is a commonly used image smoothing technique used to remove noise from an image while retaining the edge information of the image. Suppose the original input image is I (x, y), the Gaussian filter is G (x, y), and the mathematical expression of the two-dimensional Gaussian function is:

[0087] ;

[0088] Where G(x,y) is the value of the Gaussian function at position (x,y), σ is the standard deviation of the Gaussian function, and x,y are the coordinates of the pixel, the offset relative to the current pixel;

[0089] S1.1-2. Fill the edges of the input image I (x, y) and then perform convolution on each pixel with the Gaussian kernel to generate a filtered image. , calculate the output value of each pixel The formula is as follows:

[0090] ;

[0091] Where I(xi,yi) is the pixel value of the original image at position (xi,yi); G(x,y) is the weight of the position (x,y) in the Gaussian kernel, and k represents the radius of the kernel.

[0092] S1.1-3. The Gaussian kernel needs to be normalized so that the sum of all weights is equal to 1. The normalization formula is as follows:

[0093] ;

[0094] Where G(x,y) is the weight of the position (x,y) in the Gaussian kernel, k represents the radius of the kernel, and x,y are the coordinates of the pixel;

[0095] S1.2-1. Use semantic segmentation algorithms to extract regions of interest. Semantic segmentation algorithms use computer vision technology to assign specific semantic labels to each pixel in an image, achieving accurate segmentation of different regions or objects in the image. Remote sensing images are divided into multiple regions and image blocks are generated. The network performs feature extraction on the image through convolution operations, extracting low-level (such as edges, textures, etc.) and high-level (such as objects, regions, etc.) features. The formula for the convolution operation is:

[0096] ;

[0097] Where p is the input image, is the convolution kernel, f is the output after convolution;

[0098] S1.2-2. Pooling is used to reduce the spatial resolution of an image, reduce the amount of computation, and retain features. The formula for the pooling operation is:

[0099] ;

[0100] Where R is the pooling window, p is the input image, and Z is the output after pooling;

[0101] S1.2-3. After multiple convolutions and pooling, the image resolution will be reduced, so it is necessary to restore the spatial resolution of the image through upsampling (deconvolution). The formula for the deconvolution operation is:

[0102] ;

[0103] Among them, I(x,y) is the upsampled image pixel, G(i,j) is the deconvolution kernel;

[0104] S1.2-4. The output of each pixel is the probability that it belongs to each category. Normalized exponential function is usually used to convert these scores into probability values. The formula of normalized exponential function is:

[0105] ;

[0106] in, It's a pixel belong The score, It's a pixel Belong to category The probability of , C is the total number of categories;

[0107] S1.2-5. After classification, each pixel of the image will be assigned a category label. Based on the category label, the area of specific category can be extracted to form the area of interest (AOI). The formula for extracting AOI is:

[0108] ;

[0109] in, AOI is the region of interest, Is the pixel (x,y) belongs to the category The probability of , threshold is the set threshold;

[0110] As a preferred implementation, step S2 is specifically as follows:

[0111] S2.1-1. In global feature extraction, convolutional layers with larger kernels (e.g., 5*5 or 7*7) are typically used to process images. This helps the network capture a wider range of contextual information, thereby gaining a global understanding of the image. The formula for the convolution operation is shown in S1.2-1.

[0112] S2.1-2, when extracting global features, the Transformer model uses the self-attention mechanism to model the global dependencies in the image, such as Figure 4 As shown in , the self-attention mechanism is a core component of the Transformer model, which is used to calculate the correlation between elements within the input sequence and help the model capture the dependencies between elements at different positions in the sequence, no matter how far apart they are, such as Figure 3 As shown, assuming that a feature matrix is input ,in , is a set of real numbers. The attention output is calculated by query, key, and value. The formula is as follows:

[0113] ;

[0114] ;

[0115] Among them, Q, K, V, T are query, key, value matrix and transpose respectively. is the dimension of the key vector, A is the calculated attention weight matrix, It is the weighted global feature output, and softmax is a normalized exponential function. Its core function is to convert a set of input values into a probability distribution.

[0116] S2.1-3. Calculate the attention weight through the visual attention mechanism to focus on the key areas of the image. The formula is as follows:

[0117] ;

[0118] in, is the weighted global feature output, is the global attention weight, and softmax is a normalized exponential function whose core function is to convert a set of input values into a probability distribution;

[0119] S2.2-1. In regional feature extraction, a small convolution kernel (e.g., 3*3 or 5*5) is typically used to extract features from an image using a sliding window method. This allows the network to focus on features in different parts of the image. The formula for the convolution operation is shown in S1.2-1.

[0120] S2.2-2. Use pooling to unify the size of the region and provide a fixed-size regional feature map for subsequent processing. Then calculate the weighted output of the regional feature. The formula is as follows:

[0121] ;

[0122] in, is the input region feature map, It is regional pooling, is the maximum pooling, is average pooling;

[0123] S2.2-3. The network can adaptively focus attention on relevant areas of the image, especially areas with key semantic information. The cross-attention mechanism formula is as follows:

[0124] ;

[0125] in, is the regional feature extracted by the network, is the regional attention weight, and softmax is a normalized exponential function, whose core function is to convert a set of input values into a probability distribution;

[0126] S2.3-1. In pixel-level feature extraction, we typically focus on fine-grained image features, such as object edges and texture details. The formula for the convolution operation is shown in S1.2-1. The Sobel operator is used to calculate the gradient of the image and extract edges. Its convolution kernel and the calculated image gradient strength are as follows:

[0127] ; ;

[0128] ;

[0129] S2.3-2. In pixel-level feature extraction, the cross-attention mechanism assigns weights to each pixel, as follows:

[0130] ;

[0131] in, is the pixel feature extracted by the network, is the pixel attention weight, and softmax is the normalized exponential function, whose core function is to convert a set of input values into a probability distribution.

[0132] As a preferred implementation, step S3 is specifically as follows:

[0133] S3.1-1, Assume that the input text T is segmented into document level, (M sentences), each sentence Further word segmentation , using BERT as the basic encoder to generate word, sentence, and document-level features through pixel features extracted by the network. It achieves a deep semantic understanding of text through a bidirectional self-attention mechanism, significantly improving the performance of multiple natural language processing tasks. The word-level feature formula is as follows:

[0134] ;

[0135] in, is the set of real numbers, Represents a dimension The real matrix space of , L is the total length of the sequence, and d is the BERT hidden layer dimension (such as 768);

[0136] S3.1-2, for each sentence Word-level features Perform average pooling, the formula is as follows:

[0137] ;

[0138] ;

[0139] Among them, N is the number of words in each sentence, It is a characteristic of the sentence. is the set of all sentence features;

[0140] S3.1-3. Sentence-level features Use self-attention aggregation and output as global document vector , the formula is as follows:

[0141] ;

[0142] As a preferred implementation, step S4 is specifically as follows:

[0143] The dynamic fusion mechanism is a strategy for multimodal or multi-level feature fusion. Its core idea is to adaptively allocate weights between different features or information sources based on specific input and task requirements to achieve more flexible and effective feature integration. It performs weighted summation of document-level, sentence-level, and word-level features, and weights the contribution of each layer through learned weights. The weighted fusion formula is:

[0144] ;

[0145] in, is a document-level feature, is a sentence feature, It is a word-level feature.

Claims

1. A cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching, characterized by: The steps include: S1, extracting the region of interest using the semantic segmentation algorithm through the image preprocessing module, segmenting the remote sensing image into multiple regions and generating image blocks; S2. A hybrid architecture of a convolutional neural network and a Transformer model combined with a multi-level visual attention mechanism is used to extract global, regional, and pixel-level features of the image in a multi-level feature extraction module, and fine-grained encoding of key areas of the edge of fine-grained objects is performed. S3: Use the text multi-level encoding module to encode the text at the document, sentence, and word levels based on the pre-trained language model to improve the model's multi-level understanding of the text. Specifically: S3.1-1. Assume that the input text T is segmented into document level , that is, M sentences, each sentence Further segmentation into , using BERT as the basic encoder to generate word, sentence, and document level features through the pixel features extracted by the network. The word level feature formula is as follows: ； in, is the set of real numbers, Represents a dimension The real matrix space of , L is the total length of the sequence, and d is the dimension of the BERT hidden layer; S3.1-2, for each sentence Word-level features Perform average pooling, the formula is as follows: ； ； Among them, N is the number of words in each sentence, It is a characteristic of the sentence. is the set of all sentence features; S3.1-3. Sentence-level features Use self-attention aggregation and output as global document vector , the formula is as follows: ； S4. In the multi-level matching and fusion module, the similarity between the remote sensing image and the text description is calculated through the cross-attention mechanism and the dynamic fusion mechanism, and the features at each level are weighted and fused. The specific steps are as follows: The document-level, sentence-level, and word-level features are weighted and summed, and the contribution of each layer is weighted by the learned weights. The weighted fusion formula is: ； in, is a document-level feature, is the set of all sentence features, is a word-level feature; S5. Based on multi-level feature matching and fusion, output the image and text retrieval score.

2. The cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching according to claim 1 is characterized in that: Step S1 is specifically as follows: S1.1-1. Use a filter-based denoising method to denoise the image. Suppose the original input image is I(x,y), the Gaussian filter is G(x,y), and the mathematical expression of the two-dimensional Gaussian function is: ; Where G(x,y) is the value of the Gaussian function at position (x,y), σ is the standard deviation of the Gaussian function, and x,y are the coordinates of the pixel; S1.1-2. Fill the edges of the input image I (x, y) and then perform a convolution operation on each pixel with the Gaussian kernel to generate a filtered image. , calculate the output value of each pixel The formula is as follows: ； Where I(xi,yi) is the pixel value of the original image at position (xi,yi); G(x,y) is the weight of the position (x,y) in the Gaussian kernel, and k represents the radius of the kernel. S1.1-3. The Gaussian kernel needs to be normalized so that the sum of all weights is equal to 1. The normalization formula is as follows: ； Where G(x,y) is the weight of the position (x,y) in the Gaussian kernel, k represents the radius of the kernel, and x,y are the coordinates of the pixel; S1.2-1. Use semantic segmentation algorithm to extract the region of interest, divide the remote sensing image into multiple regions and generate image blocks. The network extracts features from the image through convolution operation, extracting low-level features and high-level features. The formula of convolution operation is: ； Where p is the input image, is the convolution kernel, f is the output after convolution; S1.2-2. Pooling operation is used to reduce the spatial resolution of the image, reduce the amount of calculation, and retain the features. The formula of the pooling operation is: ； Where R is the pooling window, p is the input image, and Z is the output after pooling; S1.2-3. After multiple convolutions and pooling, the image resolution will be reduced. The spatial resolution of the image is restored by upsampling. The formula for the upsampling operation is: ； Among them, I(x,y) is the upsampled image pixel, G(i,j) is the deconvolution kernel; S1.2-4. The output of each pixel is the probability that it belongs to each category. These scores are converted into probability values using the normalized exponential function. The formula of the normalized exponential function is: ； in, It's a pixel belong The score, It's a pixel Belong to category The probability of , C is the total number of categories; S1.2-5. After classification, each pixel of the image is assigned a category label. Based on the category label, the region of interest is extracted. The formula for extracting the region of interest is: ； in, AOI is the region of interest, Is the pixel (x,y) belongs to the category The probability of , threshold is the set threshold.

3. The cross-modal remote sensing image and text retrieval method based on multi-level semantic collaborative matching according to claim 2 is characterized in that: The step S2 is specifically as follows: S2.1-1. In global feature extraction, a convolution layer is used to process the image. The formula for the convolution operation is shown in step S1.2-1. S2.1-2, when extracting global features, the Transformer model uses the self-attention mechanism to model the global dependencies in the image. Assuming that a feature matrix is input ,in , is a set of real numbers, and the attention output is calculated by query, key, and value. The formula is as follows: ； ； Among them, Q, K, V, T are query, key, value matrix and transpose respectively. is the dimension of the key vector, A is the calculated attention weight matrix, It is the weighted global feature output, and softmax is a normalized exponential function. Its core function is to convert a set of input values into a probability distribution. S2.1-3. Calculate the attention weight through the visual attention mechanism to focus on the key areas of the image. The formula is as follows: ； in, is the weighted global feature output, is the global attention weight; S2.2-1. In regional feature extraction, convolution kernels are used to extract features from images using a sliding window method. Through regional feature extraction, the network focuses on the features of different parts of the image. The formula for the convolution operation is shown in S1.2-1. S2.2-2. Pool the region of interest to unify the size of the region and provide a fixed-size regional feature map for subsequent processing. Then calculate the weighted output of the regional feature. The formula is as follows: ； Among them, m is the input region feature map, It is regional pooling, is the maximum pooling, is average pooling; S2.2-3. The network can adaptively focus attention on the relevant areas of the image, that is, the areas with key semantic information. The cross-attention mechanism formula is as follows: ； in, is the regional feature extracted by the network, is the regional attention weight; S2.3-1. In pixel-level feature extraction, we focus on fine-grained features of the image. The formula for the convolution operation is shown in S1.2-1. The Sobel edge detection operator is used to calculate the gradient of the image and extract the edge. Its convolution kernel and the calculated gradient strength of the image are as follows: ； ； ； S2.3-2. In pixel-level feature extraction, the cross-attention mechanism assigns a weight to each pixel, as follows: ； in, is the pixel feature extracted by the network, is the pixel attention weight.

Images