Deep learning tropical cyclone center positioning method based on multiband fusion

Abstract

The invention discloses a deep learning tropical cyclone center positioning method based on multiband fusion. The method comprises the following steps: preprocessing a collected satellite image; constructing a deep learning model which comprises a backbone network feature extraction module, a feature fusion module and a feature enhancement module; extracting features of each tropical cyclone image through a backbone network feature extraction module; carrying out feature fusion on the extracted feature maps through a feature fusion module; the fused features are enhanced through a feature enhancement module; and the enhanced features are sent to ViT, global feature dependence in the image is captured through a self-attention mechanism, and a final output result is generated. According to the method, the information of the tropical cyclone on different wave band images is fully utilized, the spatial features are extracted, the spatial features extracted on the different wave band images are fully fused, the features of the different wave band images neglected by an existing model can be made up, and the positioning precision of the tropical cyclone center is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and relates to tropical cyclone center positioning technology, and specifically relates to a tropical cyclone center positioning method based on deep learning of multi-band fusion. Background Art

[0002] A tropical cyclone is a strong weather system originating from tropical or subtropical oceans, which can cause huge economic and property losses, as well as significant casualties. Accurately locating the center of a tropical cyclone is crucial to disaster prevention and mitigation. Better center positioning can better initialize numerical models and more accurately predict TC intensity and trajectory, helping to reduce losses and damage. Benefiting from extensive satellite observations, meteorological satellites are widely used to monitor tropical cyclones. Many methods have been proposed to locate the center of a tropical cyclone. The most commonly used method is to develop a certain number of cloud templates based on the morphological characteristics of tropical cyclones at various stages of development. Given a tropical cyclone cloud map, it can be classified according to the template, and each template has a corresponding center positioning template and intensity size estimate. However, this method relies heavily on subjective judgment by experts. Specifically, it requires manual determination of the center of a tropical cyclone, which is not conducive to the automation of center positioning.

[0003] With the development of artificial intelligence technology, more and more researchers are trying to combine artificial intelligence with meteorology. As a representative method of artificial intelligence, convolutional neural networks (CNN) can learn high-level semantic features directly from input images, so they are widely used in tropical cyclone intensity estimation tasks. Yang et al. used a deep learning model to identify and locate tropical cyclones. The model first uses a classification network to determine whether there is a tropical cyclone in the input cloud image; if so, the positioning network is used to locate the center. Wang et al. developed a physically enhanced deep convolutional neural network to determine the center of low-intensity tropical cyclones. The average distance between the center identified by the model and the center recorded in the best tracking datasets of tropical depressions and tropical storms is 20.1 km and 19.1 km, respectively.

[0004] As mentioned above, the CNN-based model has achieved excellent results in locating the center of tropical cyclones. Although deep learning methods have achieved good positioning accuracy, they usually only use one or two satellite band images and do not fully explore the impact of various bands and their combinations on positioning. In addition, the existing deep models do not distinguish between tropical cyclone clouds and nearby convective systems, which will also cause model deviations. Summary of the invention

[0005] Purpose of the invention: In order to overcome the shortcomings of the prior art, a deep learning method for locating the center of a tropical cyclone based on multi-band fusion is provided, which makes full use of the information of the tropical cyclone on images of different bands, extracts spatial features, and fully integrates the spatial features extracted from images of different bands. It can make up for the features of images of different bands ignored by the existing models and effectively improve the positioning accuracy of the center of the tropical cyclone.

[0006] Technical solution: To achieve the above purpose, the present invention provides a tropical cyclone center positioning method based on deep learning of multi-band fusion, comprising the following steps:

[0007] S1: preprocessing the collected satellite images;

[0008] S2: Build a deep learning model, including a backbone network feature extraction module, a feature fusion module, and a feature enhancement module;

[0009] S3: The preprocessed tropical cyclone images in different bands are used as the input of the deep learning model, and the features of each tropical cyclone image are extracted through the backbone network feature extraction module;

[0010] S4: feature fusion is performed on the extracted feature maps through the feature fusion module;

[0011] S5: Enhance the fused features through the feature enhancement module;

[0012] S6: The enhanced features are fed into ViT, which uses its self-attention mechanism to capture the global feature dependencies in the image and generate the final output results.

[0013] Furthermore, the backbone network feature extraction module in step S2 uses parallel multi-branch RepVGG as the backbone network.

[0014] Furthermore, the backbone network in the backbone network feature extraction module includes three feature layers, and a CBAM attention module is arranged behind each feature layer.

[0015] Furthermore, the parallel multi-branch RepVGG uses a fusion of 3×3 convolution, 1×1 convolution and identity connection to extract high-level features and obtain the feature map M∈X C×64×H′×W′ .

[0016] Furthermore, in step S2, the feature fusion module multiplies and fuses the feature maps extracted by different branches, and then integrates the features from the infrared, near-infrared and water vapor images on the channel through the CBAM attention, and focuses on the spatial position to identify the key features, directing the network's attention to important channels and spatial positions, thereby improving the feature representation.

[0017] Furthermore, the feature enhancement module in step S2 includes a CNN network and a ViT self-attention module.

[0018] Furthermore, the step S5 includes: using a convolutional network to further compress and interpret the fused features, and then performing patch flattening, linear mapping and position encoding processing.

[0019] Furthermore, the step S5 is specifically as follows: first, downsampling is performed using a four-layer CNN to reduce the size of the feature map; before inputting the encoder module, category labeling, i.e., adding class[token] and position encoding, is required; each sub-graph after convolution is subjected to a linear mapping layer and flattened dimension to obtain a vector; and then a new vector, i.e., the category label, is initialized in front of each vector; this new vector will interact with other vectors after calculation and can fuse the information of other vectors; position encoding is to give a set of new learnable parameters to allow it to learn the positional relationship between sub-graphs; the encoder module is composed of a normalization layer, a multi-layer perceptron, etc., and the flattened one-dimensional vector is first normalized, and the method used is layer normalization, which is different from batch normalization in that its normalization method directly processes individual image data and has nothing to do with the number of images.

[0020] Furthermore, the normalized data in step S6 enters the self-attention mechanism of ViT, and the degree of attention of each tile to other tiles is determined by calculating the similarity between the query, key and value vectors. Then, the value vectors are weighted and summed according to these attention degrees to obtain a comprehensive representation of each tile, thereby capturing the key information and contextual dependencies in the image, and then the input data before processing is superimposed with the output data after random inactivation; the output result is then sent to the normalization layer to complete further normalization processing; the data is then sent to the multi-layer perceptron for regularization by random inactivation; finally, the input data before normalization and the output data after random inactivation are added again, and the result is used as the output of the final encoder module.

[0021] Beneficial effects: Compared with the prior art, the present invention innovatively proposes a multi-band adaptive fusion mechanism, fully utilizes the image features of the three channels of near infrared, water vapor and infrared, adopts the deep learning architecture of RepVGG+CBAM+ViT, realizes the effective extraction and enhancement of features, introduces the product fusion strategy, and improves the model performance compared with the additive fusion. It fully utilizes the information of tropical cyclones on images of different bands, extracts spatial features, and fully integrates the spatial features extracted from images of different bands, which can make up for the features of images of different bands ignored by the existing models, and effectively improve the positioning accuracy of the center of tropical cyclones. The deep learning framework proposed in the present invention provides a real-time and automated tropical cyclone center positioning service, which has broad application prospects. It can not only be extended to other tasks of tropical cyclones (such as intensity estimation), but also can be extended to tropical cyclone monitoring in other sea areas, and can assist forecasters in tropical cyclone monitoring and early warning, providing strong technical support for meteorological departments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the method of the present invention;

[0023] Figure 2 A framework diagram for locating the model's tropical cyclone centre;

[0024] Figure 3 Schematic diagram of tropical cyclone image processing. DETAILED DESCRIPTION

[0025] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0026] like Figure 1 As shown, the present invention provides a tropical cyclone center positioning method based on deep learning of multi-band fusion, comprising the following steps:

[0027] S1: Data cropping preprocessing of collected satellite images;

[0028] A tropical cyclone dataset for center positioning was built, in which infrared cloud images included 8.6 μm, 9.6 μm, 10.4 μm, 11.2 μm, 12.3 μm and 13.3 μm, and water vapor cloud images included 6.2 μm, 6.9 μm and 7.3 μm. The data were combined in the channel direction and annotated with the best path set provided by the Meteorological Agency as the current center position. The annotation types included tropical cyclone date, category, tropical cyclone name, longitude, latitude and intensity.

[0029] S2: Build a deep learning model, including a backbone network feature extraction module, a feature fusion module, and a feature enhancement module;

[0030] Reference Figure 2 The backbone network feature extraction module uses parallel multi-branch RepVGG as the backbone network. The backbone network in the backbone network feature extraction module includes three feature layers, and each feature layer is followed by a CBAM attention module. In the parallel multi-branch RepVGG, a fusion of 3×3 convolution, 1×1 convolution and identity connection is used to extract high-level features, and the feature map M∈X is obtained. C×64×H′×W′ .

[0031] In existing CNN networks, performance tends to improve as the network goes deeper. However, as the network continues to go deeper, deeper models may experience performance degradation due to issues such as gradient vanishing. To address this problem, RepVGG introduces parallel branches, which can effectively avoid gradient vanishing. The multi-branch structure imposes constraints on system specifications. The RepVGG network allows each convolutional layer to be freely configured as needed, and also supports channel pruning, so as to achieve a better balance between performance and efficiency.

[0032] The feature fusion module multiplies and fuses the feature maps extracted by different branches, and then integrates the features from infrared, near-infrared and water vapor images through the CBAM attention mechanism. It combines the channel attention and spatial attention modules. The channel attention module performs maximum pooling and average pooling on the input feature map in the channel dimension, and then inputs the two pooling results into a fully connected layer, and finally outputs a channel attention weight vector. This weight vector can be used to weight each channel of the input feature map to enhance important channel features and suppress unimportant channel features. The spatial attention module usually uses global average pooling to obtain the feature vector of each pixel, and then outputs the weight of each pixel through a fully connected layer. These weights can be used to weight each pixel of the input feature map to emphasize important areas in the image and suppress unimportant areas. By connecting the two modules in series, a complete CBAM module is obtained, which is inserted into the convolutional neural network to improve model performance.

[0033] The feature enhancement module includes a CNN network and a ViT self-attention module.

[0034] S3: The preprocessed tropical cyclone images of different bands are used as the input of the deep learning model, and the high-level features of each tropical cyclone image are extracted through the backbone network feature extraction module; I∈X 10×H×W , H×W represents the length and width of the image, and the output is the tropical cyclone center (x, y);

[0035] S4: feature fusion is performed on the extracted feature maps through the feature fusion module;

[0036] S5: Enhance the fused features through the feature enhancement module, specifically:

[0037] First, four-layer CNN is used for downsampling to reduce the size of the feature map; before entering the encoder module, category labeling, i.e., adding class[token] and position encoding, is required. After convolution, each sub-graph is passed through a linear mapping layer and flattened to obtain a vector, and then a new vector, i.e., the category label, is initialized in front of each vector. After calculation, this new vector will interact with other vectors and can integrate the information of other vectors; position encoding gives a new set of learnable parameters to allow it to learn the positional relationship between sub-graphs; the encoder module consists of a normalization layer, a multi-layer perceptron, etc. The flattened one-dimensional vector is first normalized using layer normalization, which is different from batch normalization in that its normalization method directly processes individual image data and has nothing to do with the number of images.

[0038] S6: The enhanced features are fed into ViT, which uses its self-attention mechanism to capture the global feature dependencies in the image and generate the final output results;

[0039] The normalized data enters the self-attention mechanism of ViT, and the similarity between the query, key, and value vectors is calculated to determine the degree of attention of each tile to other tiles. Then, the value vectors are weighted and summed according to these attention levels to obtain a comprehensive representation of each tile, thereby capturing the key information and contextual dependencies in the image, and then the input data before processing is superimposed with the output data after random inactivation; then the output result is sent to the normalization layer for further normalization processing; then the data is sent to the multi-layer perceptron for random inactivation regularization; finally, the input data before normalization and the output data after random inactivation are added again, and the result is used as the output of the final encoder module.

[0040] In order to verify the effectiveness of the method of the present invention, the method of the present invention was applied to an example and verified through experiments, as follows:

[0041] (1) Experimental configuration and parameter setting.

[0042] The models involved in this embodiment are all trained on the Pytorch framework (Python 3.6 version) on NVIDIA TITAN Xp. The batch size is set to 8, the SGD (Stochastic Gradient Descent) gradient descent method is used, the learning rate is 0.001, and 50 epochs are trained.

[0043] The task of the present invention is regression, so the selected loss function is the root mean square error (RMSE). Given a training set, the loss function trains the model by minimizing the difference between the true center position of the tropical cyclone and the estimated position, and the total loss is shown in the following formula.

[0044]

[0045] l total = l (x)RMSE +l (y)RMSE

[0046] Dataset and experimental parameter settings:

[0047] The training process selected the L1 data of Sunflower-8, the data format is NetCDF, and the resolution is 5km and 2km. Considering that the research area of ​​this embodiment is the northwest Pacific, only the images in the range of 0°-50°N, 100°-180°E in the L1 data with a spatial resolution of 5km (the latitude and longitude resolution in the best path set is 0.1°, about 10km) are intercepted (the image size is 1400×1000), and the time resolution is 1 hour, from August 1, 2015 to December 31, 2019. According to the typical applications of each band listed in Table 1, this embodiment selects 10 channels to realize tropical cyclone positioning, which are 3.9 microns, 6.2 microns, 6.9 microns, 7.3 microns, 8.6 microns, 9.6 microns, 10.4 microns, 11.2 microns, 12.3 microns and 13.3 microns.

[0048] Table 1 Sunflower-8 band and its use

[0049]

[0050] The optimal path set provides the longitude and latitude of the center of the tropical cyclone with a resolution of 6 hours. In order to match the resolution of the cloud map, this embodiment linearly interpolates the path set to 1 hour. The original cloud map is cropped according to the obtained longitude and latitude, and only the image containing the tropical cyclone cloud system is intercepted. The intercepted image size is 512×512 pixels, covering a range of more than 5000km, which is sufficiently complete to contain the tropical cyclone cloud system. In order to achieve the positioning task, the center of the tropical cyclone is not at the center of the intercepted image. This embodiment adds random noise to the center point so that the center of the tropical cyclone has an offset of 20-40 pixels (about 110-220km) relative to the center of the image. The task of the model is to estimate the number of pixels offset on the x-axis and y-axis respectively. In order to expand the amount of training data, this embodiment performs data enhancement processing on the image, that is, the original image is horizontally flipped, vertically flipped, and horizontally and vertically flipped simultaneously in the clockwise direction, and the label information of the position center is also flipped accordingly. The enhanced image is as follows Figure 3 As shown in the figure, a is a 512×512 tropical cyclone image, b is a vertically flipped image, c is a horizontally flipped image, and d is a horizontally and vertically flipped image; the number of training set images increased from 10573 to 42292 images (2015-2018), and the test set images were not data enhanced, with a total of 3178 images (2019). In addition, in order to make the model converge faster, the input images and labels were normalized to keep them between 0 and 1.

[0051] (2) Cyclone center positioning results:

[0052] In order to verify the effect of model center positioning, this embodiment uses root mean square error (RMSE) and mean absolute error (MAE) as evaluation indicators. The result of the model output is the amount of offset pixels. By comparing with the label, the longitude and latitude errors of model positioning can be obtained. In order to convert the longitude and latitude errors into kilometer errors, this embodiment uses the haversine formula.

[0053] In order to facilitate understanding of the technical solution of the present invention, three groups of simulation experiments are given below.

[0054] Experiment 1: Explore the impact of different channel images on center positioning

[0055] The experimental results of this embodiment are as follows:

[0056] The accuracy of center positioning is related to the characteristics of tropical cyclone images. Different channel images provide different characteristics. Infrared images mainly reflect the thermal radiation information of objects, and water vapor images mainly reflect the water vapor content and distribution information in the atmosphere, which will affect the accuracy of center positioning. In order to explore the impact of different channel images on center positioning, this section will use three channel images of the sunflower-8 satellite image X H×W×10, where IR represents infrared image, SIR represents near infrared image, WV represents water vapor image, and other branches are deleted when the single channel image is tested. In order to reduce the impact of model structure changes on the experiment, we keep other parts unchanged. Four models are trained for different scenes to analyze the impact of different channel images on center positioning.

[0057] Table 2 The impact of different channel images on center positioning

[0058]

[0059] The final experimental results are shown in Table 2. These results differ only in the image channels of the model input. According to the table, when the three channel images are input as a branch at the same time, the center positioning effect is the best, with an RMSE of 32.42 km. This is because the model can extract richer feature information by using different channel images as combined input, which helps to output better results. When a single branch structure is used as input, the infrared channel image has the best effect, with an RMSE of 34.35 km, and the water vapor channel has the worst effect. It can be seen from the table that infrared images play an important role in center positioning. Good results have been achieved by using only infrared images. This is because infrared images usually contain information such as cloud top temperature and cloud structure of tropical cyclones, which are often closely related to their formation process. Therefore, it can be concluded that the best results are obtained when all channel images are used as input together, and water vapor and near-infrared images can supplement information that infrared images may not be able to capture, which helps to understand tropical cyclones more comprehensively through models.

[0060] Experiment 2 conducts an ablation experiment on the model to analyze the impact of different parts of the model on center positioning.

[0061] The experimental results of this embodiment are as follows:

[0062] This section will conduct an ablation experiment on the model. In the table, "Baseline" represents a standard three-branch structure feature extraction network, and "CBAM" represents a convolutional attention module, where the channel attention module can adaptively adjust the channel weights to improve the correlation of different channels in the feature map, and the spatial attention module can learn the importance of each position based on the spatial position relationship between pixels to suppress irrelevant information. Visual Transformer (ViT) consists of a multi-head self-attention mechanism and a fully connected feedforward network, which can effectively capture the global information in the image and extract global features.

[0063] Table 3 Ablation experiment results

[0064]

[0065] From the results in Table 3, we can see that for the input image XH×W×10 Directly using a single RepVGG network to extract features without dividing the 10-band input image into 3 groups, the RMSE of the center positioning result is 33.59km, which is slightly lower than the 3-branch feature extraction method, indicating that dividing the input image into 3 groups according to the channel category and extracting features separately can better extract the key information of the corresponding channel image; and under the premise of grouping feature extraction of the input image, the RMSE after adding only the CMAM module is 34.15km, and the RMSE after adding only the ViT module is 33km, which indicates that compared with the CBAM attention module, the ability of ViT to capture global information and extract global features has better performance improvement for the model. When the two are used at the same time, the RMSE result is 32.42km, which is 5.1% and 1.8% lower than the results when the two are used alone, respectively, which indicates that inserting different attention modules in the model is effective and there is no conflict, and they can cooperate well with each other, and the two modules work together to enable the neural network to extract important features more accurately, thereby improving the model performance. The effectiveness of the method of the present invention in locating the center of tropical cyclones using multi-band images is verified.

[0066] Experiment 3: Explore the impact of different feature fusion methods on center positioning

[0067] The experimental results of this embodiment are as follows:

[0068] How to fuse the features extracted from images of different channels is the key to the accuracy of center positioning. Feature fusion methods mainly include addition, splicing and product. Feature addition refers to adding multiple feature maps or feature vectors element by element to fuse their information. Feature splicing refers to directly connecting different feature maps or feature vectors in a certain dimension (usually the channel dimension) to form a wider feature map or feature vector. Feature product is to multiply multiple feature maps or feature vectors element by element to fuse their information. This experiment uses both addition and multiplication methods.

[0069] Table 4 Effects of different feature fusion methods on center positioning

[0070]

[0071]

[0072] As shown in the results in Table 4, when the three feature maps obtained by the three-branch feature extraction network are added and fused, the RMSE of the center positioning result is 33.31km, while when product fusion is used, the RMSE is 32.42km. This shows that although the feature addition method can enhance important features and reduce noise to a certain extent, feature product extraction can enhance co-occurring features and reduce unimportant features, and improve the stability and generalization ability of the network on this model. On the basis of using product fusion, the stepwise product method is further used to improve the model, but the RMSE obtained is 34.15km, and the effect is significantly worse.

[0073] After the above experimental verification and analysis, the following conclusions are drawn:

[0074] 1. The present invention innovatively proposes a multi-band adaptive fusion mechanism, which fully utilizes the image features of the three channels: near infrared, water vapor and infrared.

[0075] 2. The present invention adopts the deep learning architecture of RepVGG+CBAM+ViT to achieve effective feature extraction and enhancement.

[0076] 3. The introduction of product fusion strategy improves model performance compared with additive fusion.

[0077] In addition, the deep learning framework proposed in the present invention provides a real-time automated tropical cyclone center positioning service, which has broad application prospects. It can not only be extended to other tropical cyclone tasks (such as intensity estimation), but also be extended to tropical cyclone monitoring in other sea areas. It can assist forecasters in tropical cyclone monitoring and early warning, and provide strong technical support for meteorological departments.

Claims

1. A tropical cyclone center positioning method based on deep learning of multi-band fusion, characterized in that: The steps include: S1: preprocessing the collected satellite images; S2: Build a deep learning model, including a backbone network feature extraction module, a feature fusion module, and a feature enhancement module; S3: The preprocessed tropical cyclone images in different bands are used as the input of the deep learning model, and the features of each tropical cyclone image are extracted through the backbone network feature extraction module; S4: feature fusion is performed on the extracted feature maps through the feature fusion module; S5: Enhance the fused features through the feature enhancement module; S6: The enhanced features are fed into ViT, which uses its self-attention mechanism to capture the global feature dependencies in the image and generate the final output results.

2. According to claim 1, a tropical cyclone center positioning method based on deep learning of multi-band fusion is characterized in that: In step S2, the backbone network feature extraction module uses parallel multi-branch RepVGG as the backbone network.

3. The method for locating the center of a tropical cyclone based on deep learning of multi-band fusion according to claim 2, characterized in that: The backbone network in the backbone network feature extraction module includes three feature layers, and a CBAM attention module is arranged behind each feature layer.

4. The method for locating the center of a tropical cyclone based on deep learning of multi-band fusion according to claim 2, characterized in that: The parallel multi-branch RepVGG uses a fusion of 3×3 convolution, 1×1 convolution and identity connection to extract high-level features and obtain the feature map M∈X C×64×H′×W′ .

5. The method for locating the center of a tropical cyclone based on deep learning of multi-band fusion according to claim 1, characterized in that: In step S2, the feature fusion module multiplies and fuses the feature maps extracted by different branches, and then integrates the features from infrared, near-infrared and water vapor images through the CBAM attention mechanism. It combines the channel attention and spatial attention modules, wherein the channel attention module performs maximum pooling and average pooling on the input feature map in the channel dimension, and then inputs the two pooling results into a fully connected layer, and finally outputs a channel attention weight vector; the spatial attention module uses global average pooling to obtain the feature vector of each pixel, and then outputs the weight of each pixel through a fully connected layer; by connecting the two modules in series, a complete CBAM module is obtained, which is used to be inserted into the convolutional neural network to improve the model performance.

6. The method for locating the center of a tropical cyclone based on deep learning and multi-band fusion according to claim 1, characterized in that: The feature enhancement module in step S2 includes a CNN network and a ViT self-attention module.

7. The method for locating the center of a tropical cyclone based on deep learning of multi-band fusion according to claim 6, characterized in that: The step S5 includes: using a convolutional network to further compress and interpret the fused features, and then performing patch flattening, linear mapping and position encoding processing.

8. The method for locating the center of a tropical cyclone based on deep learning of multi-band fusion according to claim 7, characterized in that: The step S5 is specifically as follows: first, downsampling is performed using a four-layer CNN to reduce the size of the feature map; before inputting the encoder module, category marking is required, i.e., adding class[token] and position encoding, and each sub-graph after convolution is passed through a linear mapping layer and flattened to obtain a vector, and then a new vector, i.e., a category label, is initialized before each vector. After calculation, this new vector will interact with other vectors and can fuse the information of other vectors; Position encoding gives a new set of learnable parameters to allow it to learn the positional relationship between sub-graphs; the encoder module includes a normalization layer and a multi-layer perceptron, and the flattened one-dimensional vector is first normalized.

9. The method for locating the center of a tropical cyclone based on deep learning of multi-band fusion according to claim 8, characterized in that: The normalized data in step S6 enters the self-attention mechanism of ViT, and the degree of attention of each tile to other tiles is determined by calculating the similarity between the query, key and value vectors. Then, the value vectors are weighted and summed according to these attention degrees to obtain a comprehensive representation of each tile, thereby capturing the key information and contextual dependencies in the image, and then the input data before processing is superimposed with the output data after random inactivation; the output result is then sent to the normalization layer to complete further normalization processing; the data is then sent to the multi-layer perceptron for regularization by random inactivation; finally, the input data before normalization and the output data after random inactivation are added again, and the result is used as the output of the final encoder module.

Images