Hybrid convolution-transformer architecture and self-supervision method based on window mask strategy

Abstract

This paper discloses a hybrid convolution-transformer architecture and self-supervised method based on a window mask strategy. This strategy is used to divide an image into visible image patches and mask image patches. The visible image patches are then reassembled and fed into a hybrid convolution-transformer pyramid backbone network designed to balance efficiency and performance for feature learning. Features at different scales are fused and aligned using a multi-scale fusion module and a feature alignment module, respectively. These features are then concatenated and aligned with a learnable mask feature vector, which serves as input to a lightweight decoder for image reconstruction and prediction of mask image patches. When labeled data is limited, pre-training can be performed using a large amount of unlabeled industrial data, further improving the performance of supervised learning tasks while significantly reducing labeling and time costs. This paper is suitable for industrial inspection scenarios with rigid requirements such as real-time performance and high precision.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a hybrid convolution-transformer architecture based on a window masking strategy and a self-supervision method thereof. Background Art

[0002] Inspired by the tremendous success of Masked Language Modeling (MLM) in natural language processing and the rapid development of Vision Transformer (ViT) in computer vision, Masked Image Modeling (MIM) has achieved outstanding results in computer vision. Mask Autoencoders (MAEs) are a representative self-supervised method within MIM and have gradually become a leading self-supervised pre-training paradigm in computer science. By applying a random masking strategy to the original image, MAEs only take visible image patches as input and predict masked image patches. They expect the encoder network to learn features containing rich semantic information by restoring the pixels of the masked image patches.

[0003] Essentially, the most outstanding design of the MAE is its asymmetric encoder-decoder architecture, where the encoder operates only on visible image patches, while the decoder aims to recover all patches. This approach not only improves pre-training speed and reduces GPU memory usage, but also achieves excellent performance on downstream tasks. On the other hand, ViT, as its encoder network, faces significant obstacles in industrial inspection applications due to its high computational cost and large number of parameters. ViT's self-attention module can learn long-term dependencies between features, giving it a stronger ability to model global context than convolutional neural networks. In fact, local inductive biases and a hierarchical architecture are crucial to enhancing ViT's performance. Many recent works have explored combining convolutional neural networks with the Transformer. Hybrid convolutional-transformer networks have demonstrated excellent performance on vision tasks such as image classification, object detection, and instance segmentation. However, their significant computational cost and slow inference speed have hindered their industrial application.

[0004] Therefore, designing a hybrid convolution-transformer pyramid backbone network that leverages the self-attention mechanism to maximize the performance of convolutional neural networks (CNNs) has become a pressing technical challenge. This requires not only achieving good performance on public datasets but also achieving good performance and computational efficiency on industrial instance segmentation data.

[0005] On the other hand, the mask auto-encoding strategy in MAE is highly effective compared to many self-supervised methods. However, it is limited because it only supports isotropic ViT structures. Many current methods exploit this mask auto-encoding strategy by padding the masked regions with zeros to restore the entire image. While effective, this approach only yields a suboptimal model and sacrifices significant training efficiency.

[0006] Therefore, considering designing a self-supervised learning method that applies the mask automatic encoding strategy to the hybrid convolution-transformer pyramid backbone network to further improve the detection performance of industrial data and reduce the time cost is one of the technical problems to be solved by the present invention. Summary of the Invention

[0007] The present invention aims to extend the masked autoencoding strategy and asymmetric encoder-decoder architecture of MAE to a hybrid convolutional-transformer pyramid network, thereby solving the problem that the masked autoencoding strategy is not applicable to the hybrid convolutional-transformer pyramid network. Therefore, the present invention proposes a simple and effective convolutional context transformer masked autoencoder method and a hybrid convolutional-transformer architecture based on a window masking strategy to achieve the purpose of improving industrial inspection efficiency and utilizing large amounts of unlabeled data for self-supervised learning to enhance performance.

[0008] The present invention is achieved by adopting the following technical solutions.

[0009] In one aspect, the present invention discloses a self-supervised method for a hybrid convolution-transformer architecture based on a window masking strategy, comprising:

[0010] The input image is divided and reorganized into windows to perform window mask sampling and image restoration to obtain the original resolution. Figure 1 Half of the reconstructed image;

[0011] The encoder based on the hybrid convolution-transformer pyramid network is divided into stages, and the input feature reconstructed image is scaled down in sequence, and the local and global features of the reconstructed image are fully integrated;

[0012] The image features of different scales after the fusion of local features and global features are fused in the same dimension at multiple scales. The fused feature images are aligned and used as the input of the decoder.

[0013] The decoder based on the hybrid convolution-transformer pyramid network maps the dimensions of the fused and aligned image features to the same number of dimensions as the decoder input image features. The decoder input image is reconstructed through the visual transformer module, and the mean square loss function between the output features of the base network and the original image is solved.

[0014] The loss function formula is used to calculate the total error of the hybrid convolution-transformer pyramid network, and then the network parameters are updated through error backpropagation and stochastic gradient descent. The iteration is continued until the loss function converges, completing the overall training process of self-supervised learning.

[0015] In an embodiment of the present invention, the input image is subjected to window mask sampling and image restoration through image division and window reorganization, including inputting a three-channel image, dividing the image into image patches of equal size, and reorganizing all the patches into different windows, extracting the visible patches in each window, and reorganizing them into an image as the input of the encoder.

[0016] In an embodiment of the present invention, the encoder includes a convolution prior module, a large kernel convolution module and a transformer-convolution fusion module.

[0017] In this embodiment of the present invention, the encoder based on the hybrid convolution-transformer pyramid network is divided into four stages:

[0018] The convolutional prior module is located at the front end of the hybrid convolution-transformer pyramid network;

[0019] The first and second stages are composed of a downsampling module and a large kernel convolution module, respectively;

[0020] The third and fourth stages consist of the downsampling module and the transformer-convolution fusion module, respectively;

[0021] The input feature map is reduced in size through four stages.

[0022] In this embodiment of the present invention, the convolution prior module includes a 7×7 convolution block and two 3×3 convolution blocks.

[0023] In this embodiment of the present invention, the large kernel convolution module includes a 7×7 depthwise convolution block and two linear layers.

[0024] In an embodiment of the present invention, the downsampling module is composed of a convolution block with a convolution kernel size of 3, a stride of 2, and a supplementary boundary of 1, which reduces the image size to 1 / 2 of the original size and doubles the channel, and also includes a normalization layer.

[0025] In an embodiment of the present invention, the transformer-convolution fusion module stacks the self-attention module and the large-kernel convolution module in a vertical manner to achieve full fusion of local features and global features; the self-attention module includes a self-attention module, a residual module and a multi-layer perceptron module.

[0026] In an embodiment of the present invention, image features of different scales after fusion of local features and global features are fused in the same dimension for multi-scale fusion, and the feature maps of the second, third and fourth stages are output to the same dimension for multi-scale fusion using a downsampling module, a linear layer and an upsampling module.

[0027] On the other hand, the present invention discloses a hybrid convolution-transformer architecture based on a window mask strategy of the method, comprising: an encoder and a decoder.

[0028] The encoder includes a convolution prior module, a large kernel convolution module, a downsampling module, and a transformer-convolution fusion module, where:

[0029] Convolutional prior module, the convolutional prior module is at the beginning of the hybrid convolution-transformer pyramid network, using overlapping convolutions with 7×7 convolution kernels to increase the receptive field and increase feature interaction;

[0030] The downsampling module consists of convolution blocks with convolution kernels to reduce the image size; the reorganized feature map is normalized;

[0031] The large kernel convolution module and the downsampling module constitute the first two stages of the encoder, respectively. They use a large 7×7 convolution kernel to increase the receptive field. They use a deep convolution operation and use two linear layers to interact with the feature information of different channels.

[0032] The transformer-convolution fusion module and the downsampling module respectively constitute the last two stages of the encoder. The transformer-convolution fusion module includes depthwise convolution and two linear layers. It is used to improve the receptive field by using 7×7 convolution operations. The self-attention module and the large-kernel convolution module are stacked in a vertical manner to achieve a full fusion of local and global features.

[0033] An upsampling module, including a linear layer and a pixel deconvolution module; the linear layer is used to expand the feature channel dimension, and the pixel deconvolution module is used to upsample to the same dimension as the decoder input;

[0034] The decoder includes a transformer module, which is used to map the fused and aligned image feature dimensions to the same number as the decoder input image feature dimensions. The decoder input image is reconstructed through the visual transformer module to solve the mean square loss function between the output features of the basic network and the original image.

[0035] The present invention adopts the above technical solution, which has the following beneficial effects:

[0036] The present invention addresses the current lack of a self-supervised method based on a masked autoencoder suitable for a hybrid convolution-transformer pyramid network. A window mask strategy is used to divide an image into visible image patches and masked image patches. The visible image patches are recombined and input into a designed hybrid convolution-transformer pyramid backbone network that takes both efficiency and performance into consideration for feature learning. Features of different scales are fused and aligned respectively through a multi-scale fusion module and a feature alignment module, and then concatenated and aligned with a learnable masked feature vector. This is used as the input of a lightweight decoder for image reconstruction tasks and predicting masked image patches. When labeled data is limited, a large amount of industrial unlabeled data can be used for pre-training, further improving the performance of supervised learning tasks and significantly reducing labeling costs and time costs.

[0037] This paper proposes a window masking strategy to process raw images. It utilizes a hybrid convolutional-transformer pyramid network, which balances performance and efficiency, as the encoder to learn image features. The addition of a multi-scale fusion module enables the decoder to focus on image reconstruction while simultaneously supervising the encoder's features at different scales. Image reconstruction by the decoder forces the encoder to learn better features, a mutually dependent process. This self-supervised approach, based on a hybrid convolutional-transformer architecture with a window masking strategy, offers the advantages of simplicity, efficiency, low time and computational costs, and strong performance for downstream tasks.

[0038] The present invention has strong generalization performance. Using this method for self-supervised learning greatly shortens the training time. Extracting the encoder features of self-supervised learning as a pre-training model can achieve better performance than supervised learning in downstream instance segmentation tasks.

[0039] This invention achieves real-time robotic grasping in densely packed target scenes, achieving an inference time of 125ms for a single 1640×1250 image. With its high accuracy, high efficiency, and low computational cost, this method effectively addresses the challenges of long supervised learning training times, limited targeting, and low generalization performance. It fully utilizes the vast amount of unlabeled data in industrial scenarios, saving computational costs and making it suitable for direct application in tasks such as robotic grasping and drone scene segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute an improper limitation of the present invention. In the drawings:

[0041] Figure 1 A schematic diagram showing the principle of the window mask adoption strategy of the present invention;

[0042] Figure 2 Schematic diagram of the principle of the encoder of the present invention;

[0043] Figure 3 Schematic diagram of the overall architecture of the self-supervision method of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The exemplary embodiments and descriptions of the present invention are used to explain the present invention but are not intended to limit the present invention.

[0045] like Figure 1 As shown, the present invention provides a self-supervised method for a hybrid convolution-transformer architecture based on a window mask strategy, comprising the following steps:

[0046] Step S101, a window mask strategy is applied to the input image: a three-channel image is input, the image is divided into equal-sized image patches, and all patches are reorganized into different windows. A random mask strategy is used within each window to extract the visible patches within each window, and the patches are reorganized into an image as the input of the encoder.

[0047] Among them, image division and window reorganization:

[0048] The three-channel image (H×W×3) is divided into image patches of equal size, for example, the image patch size is 16×16, and the image size is 256×256×3; the original image is divided into an appropriate number of windows. The present invention uses 4 windows, that is, four windows, each window contains an equal number of image patches, totaling 64.

[0049] Among them, window mask sampling and image restoration:

[0050] A random masking strategy is used in each window with a mask ratio of 75%, that is, 75% of the image patches in each window are randomly selected as mask image patches, which is 48 in number, and the remaining 25% of the image patches are used as visible image patches, which is 16 in number; the 16 visible image patches in each window are extracted and randomly arranged according to the position of the window, that is, the 16 image patches in each window are rearranged into a 4×4 square image block, and finally different windows are reorganized into an image according to the original division order. Generally, the reorganized image has half the resolution of the original image.

[0051] S102, encoder learning features: The encoder is a hybrid convolution-transformer pyramid network that effectively combines convolution and self-attention mechanisms. Its input size is the feature map 128×128×3 obtained in step S101. The encoder is divided into four stages, including a convolution prior module, a large kernel convolution block, and a transformer-convolution fusion module. The convolution prior module is at the beginning of the network, using overlapping windows to improve performance and reduce input resolution. The large kernel convolution block constitutes the first two stages of the encoder, and the transformer-convolution fusion module constitutes the last two stages. Each stage contains a downsampling module, which reduces the input feature map to 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively. The overall network framework of the encoder is as follows: Figure 2 shown.

[0052] The encoder is divided into four stages:

[0053] The convolutional prior module is located at the front end of the hybrid convolution-transformer pyramid network;

[0054] The first and second stages are composed of a downsampling module and a large kernel convolution module, respectively;

[0055] The third and fourth stages are composed of the downsampling module and the transformer-convolution fusion module, respectively.

[0056] Among them, the convolution prior module:

[0057] like Figure 2 As shown in the convolutional prior module in Figure 3, the convolutional prior module consists of a 7×7 convolutional block and two 3×3 convolutional blocks. Overlapping convolutions with 7×7 kernels increase the receptive field, enhance feature interaction, and improve performance. Continuous 3×3 convolutions introduce sufficient convolutional prior knowledge. Meanwhile, the image resolution is reduced to 64×64, and the channel dimension is increased to 64, reducing the computational cost of subsequent modules.

[0058] Among them, the large kernel convolution module:

[0059] like Figure 2 As shown in the large kernel convolution module in the figure, the large kernel convolution module includes a 7×7 depth convolution block and two linear layers, which constitute the first two stages of the encoder. The 7×7 large convolution kernel is used to improve the receptive field; the depth convolution operation is used to reduce the computational complexity; after the depth convolution, two linear layers are used to interact with the feature information of different channels.

[0060] Among them, the downsampling module:

[0061] The downsampling module consists of a convolution block with a kernel size of 3, a stride of 2, and a supplementary margin of 1. Its function is to reduce the image size to 1 / 2 of the original and double the channels. It also includes a normalization layer to normalize the reorganized feature map and stabilize the training.

[0062] Among them, transformer-convolution fusion module:

[0063] like Figure 2 As shown in the Transformer-Convolution Fusion Module in

[15] , this module vertically stacks the Self-Attention Module and the Large-Kernel Convolution Module to fully integrate local and global features. The Self-Attention Module consists of a Self-Attention Module, a Residual Module, and a Multi-Layer Perceptron Module. Deep convolution is introduced in the Multi-Layer Perceptron layer of the Self-Attention Module to achieve a smooth transition between the features of the two modules. The Large-Kernel Convolution Module also uses 7×7 convolution operations to increase the receptive field and consists of deep convolution and two linear layers.

[0064] S103, the multi-scale fusion module constitutes the encoder input, and the feature alignment module aligns the fused features: the downsampling module, the linear layer and the upsampling module are used to output the feature maps of the second, third and fourth stages in step S102 to the same dimension for multi-scale fusion. The fused feature maps are passed through the feature alignment module as the input of the decoder. The schematic diagram of the multi-scale fusion module is shown in FIG. Figure 3 As shown in the multi-scale fusion module in .

[0065] Among them, the multi-scale fusion module:

[0066] This module fuses features from different encoder scales. Specifically, the features from the second encoder stage (16×16×192) are downsampled using 2×2 convolutions, and the channels are mapped to the same feature dimension of 512 as the visible image patch. The features from the third encoder stage (8×8×384) are mapped to the same feature dimension using 1×1 convolutions. The features from the fourth encoder stage (4×4×512) are mapped to the same feature dimension using an upsampling module. Finally, the transformed features from different scales are fused (feature vectors are added).

[0067] Among them, the upsampling module:

[0068] The upsampling module consists of a linear layer and a pixel deconvolution module. The linear layer is used to expand the feature channel dimension by 4 times, and then the pixel deconvolution module is used to upsample to the same dimension as the decoder input.

[0069] Among them, the feature alignment module:

[0070] This module aligns the fused features with the encoder input features by recording the patch number of the encoder input features and then reassembling the fused features according to that number. The features are then converted to a one-dimensional vector and concatenated with a learnable mask vector. The concatenated features are then reassembled according to the recorded patch number of the original image.

[0071] S104, decoder reconstruction image: This part processes the features of the feature map after fusion and alignment in step S103 and the combination of the learnable mask vector, and reconstructs the decoder input image through a lightweight visual transformer module. The decoder diagram is shown in the figure below. Figure 3 The decoder in .

[0072] Among them, the decoder basic network:

[0073] The decoder uses a linear layer to map the fused and aligned feature dimensions to the same number as the decoder input feature dimensions, i.e., 512. This is then passed through a lightweight (depth 8) network consisting of self-attention modules for feature learning. By comparing the output features of the basic network with the original image, the MSE-Loss of the original image is calculated, thereby achieving the goal of encouraging the encoder to learn better input features by supervising the decoder reconstruction effect.

[0074] S105, network training:

[0075] The network input is a three-channel image, such as an image of an industrial dense commodity scene. It is trained for self-supervised learning tasks using a data-driven approach, using MSE-Loss as the loss function:

[0076]

[0077] Where m is the set of mask patches, i is the index of all image patches. I(i) is the normalized pixel value of the input image, is the output of the decoder.

[0078] The total network error is calculated using the loss function formula, and then the network parameters are updated through error back propagation and stochastic gradient descent method. The iteration is continued until the loss function converges, completing the overall training process of self-supervised learning.

[0079] like Figure 3 As shown, the present invention provides a hybrid convolution-transformer architecture based on a window mask strategy, including: an encoder and a decoder.

[0080] The encoder includes a convolution prior module, a large kernel convolution module, a downsampling module, and a transformer-convolution fusion module, where:

[0081] Convolutional prior module,The convolutional prior module starts at the hybrid convolution-transformer pyramid network and uses overlapping convolutions containing 7×7 convolution kernels to improve the receptive field and increase feature interaction.

[0082] The downsampling module consists of a convolution block with a convolution kernel to reduce the image size; the reorganized feature map is normalized.

[0083] The large kernel convolution module and the downsampling module constitute the first and second stages of the encoder respectively, using a large 7×7 convolution kernel to enhance the receptive field; a deep convolution operation is adopted, using two linear layers to interact with the feature information of different channels.

[0084] The transformer-convolution fusion module and the downsampling module constitute the fourth stage of the encoder respectively. The transformer-convolution fusion module includes depthwise convolution and two linear layers. It is used to enhance the receptive field by using 7×7 convolution operation. The self-attention module and the large-kernel convolution module are stacked vertically to achieve full fusion of local features and global features.

[0085] The upsampling module includes a linear layer and a pixel deconvolution module; the linear layer is used to expand the feature channel dimension, and the pixel deconvolution module is used to upsample to the same dimension as the decoder input.

[0086] The decoder includes a transformer module, which is used to map the fused and aligned image feature dimensions to the same number as the decoder input image feature dimensions. The decoder input image is reconstructed through the visual transformer module to solve the mean square loss function between the output features of the basic network and the original image.

[0087] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments.

[0088] Example

[0089] The present invention provides a self-supervised method for a hybrid convolution-transformer architecture based on a window mask strategy, which includes two parts: model training and model use. The specific implementation methods are as follows:

[0090] 1. Network model training

[0091] This self-supervised pre-training model, based on a hybrid convolutional-transformer architecture with a window mask strategy, requires sufficient unlabeled data from industrial scenarios. Each image is captured and stored using an industrial camera, eliminating the need for deduplication and filtering. These images are typically three-channel color images, which contain richer information than single-channel images and help enhance the performance of downstream tasks.

[0092] After obtaining enough industrial inspection data, each image needs to be masked and reorganized using a window mask strategy. The window mask sampling method is as follows: Figure 1 As shown; the visible image patch is input into the encoder for feature learning, and the network structure of the encoder is as follows Figure 2; Then, the multi-scale fusion module and feature alignment module are used to fuse and align the features of the last three stages of the encoder respectively; the aligned features are linearly mapped, spliced and reorganized with the learnable mask vector, and used as the input of the decoder to reconstruct the image using a lightweight network. The overall network architecture is as follows Figure 3 As shown in Figure 1, it includes a window mask, an encoder, a multi-scale fusion module, and a decoder. Finally, the loss between the reconstructed image and the original image is calculated using Equation (1), and the parameters are iteratively updated according to the gradient backpropagation method. The GPU is used for accelerated training until the network error is reduced to within the set threshold or the number of network iterations meets the requirements.

[0093] 2. Training model usage

[0094] For industrial robot grasping scenarios, the encoder is extracted as the backbone network for the downstream instance segmentation task, and the encoder weights in the self-supervised pre-trained model are extracted. Using an industrially labeled instance segmentation dataset for object instance segmentation, loading the self-supervised pre-trained model before proceeding with this supervised learning task can significantly improve instance segmentation performance. This method is applicable to any industrial data and offers the advantages of high efficiency, high accuracy, and low computational and time costs.

[0095] The present invention can be directly applied to all tasks including robot grasping, drone scene segmentation and industrial defect detection. Because the method has the advantages of high real-time performance, high performance, low overhead, and replaceable modular design, it can obtain a high-performance and generalized pre-trained model while being widely applicable to any real-time network model. It is also more suitable for industrial inspection scenarios with hard requirements such as real-time and high precision.

[0096] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A self-supervised method for a hybrid convolution-transformer architecture based on a window mask strategy, characterized in that include: The input image is subjected to window mask sampling and image restoration through image division and window reorganization to obtain a reconstructed image with half the resolution of the original image; The encoder based on the hybrid convolution-transformer pyramid network is divided into stages, and the input feature reconstructed image is scaled down in sequence, and the local and global features of the reconstructed image are fully integrated; The image features of different scales after the fusion of local features and global features are fused in the same dimension at multiple scales. The fused feature images are aligned and used as the input of the decoder. The decoder based on the hybrid convolution-transformer pyramid network maps the fused and aligned image feature dimensions to the same number as the decoder input image feature dimensions. The decoder input image is reconstructed through the visual transformer module, and the mean square loss function between the output features of the base network and the original image is solved. The loss function formula is used to calculate the total error of the hybrid convolution-transformer pyramid network, and then the network parameters are updated through error backpropagation and stochastic gradient descent. The iteration is continued until the loss function converges, completing the overall training process of self-supervised learning.

2. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 1, characterized in that The input image is subjected to window mask sampling and image restoration through image division and window reorganization, including inputting a three-channel image, dividing the image into equal-sized image patches, and reorganizing all patches into different windows. The visible patches in each window are extracted and reorganized into an image as the input of the encoder.

3. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 1, characterized in that The encoder consists of a convolutional prior module, a large kernel convolution module, and a transformer-convolution fusion module.

4. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 3, characterized in that The encoder is divided into four stages based on the hybrid convolution-transformer pyramid network: The convolutional prior module is located at the front end of the hybrid convolution-transformer pyramid network; The first and second stages are composed of a downsampling module and a large kernel convolution module, respectively; The third and fourth stages consist of the downsampling module and the transformer-convolution fusion module, respectively; The input feature map is reduced in size through four stages.

5. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 3, characterized in that The convolutional prior module consists of a 7×7 convolution block and two 3×3 convolution blocks.

6. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 3, characterized in that The large kernel convolution module consists of a 7×7 depthwise convolution block and two linear layers.

7. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 4, characterized in that The downsampling module consists of a convolution block with a kernel size of 3, a stride of 2, and a padding of 1, which reduces the image size to 1 / 2 of the original and doubles the channels, and also includes a normalization layer.

8. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 3, characterized in that The transformer-convolution fusion module stacks the self-attention module and the large-kernel convolution module in a vertical manner to achieve full fusion of local features and global features; the self-attention module includes the self-attention module, the residual module and the multi-layer perceptron module.

9. The self-supervised method of hybrid convolution-transformer architecture based on window mask strategy according to claim 4, characterized in that The image features of different scales after the fusion of local features and global features are fused in the same dimension for multi-scale fusion. The feature maps of the second, third and fourth stages are output to the same dimension for multi-scale fusion using the downsampling module, linear layer and upsampling module.

10. A hybrid convolution-transformer system based on a window mask strategy according to any one of claims 1 to 9, characterized in that: Includes: encoder and decoder; The encoder includes a convolution prior module, a large kernel convolution module, a downsampling module, and a transformer-convolution fusion module, where: Convolutional prior module, the convolutional prior module is at the beginning of the hybrid convolution-transformer pyramid network, using overlapping convolutions with 7×7 convolution kernels to increase the receptive field and increase feature interaction; The downsampling module consists of convolution blocks with convolution kernels to reduce the image size; the reorganized feature map is normalized; The large kernel convolution module and the downsampling module constitute the first two stages of the encoder, respectively. They use a large 7×7 convolution kernel to increase the receptive field; they use deep convolution operations to reduce the amount of computation, and use two linear layers to interact with the feature information of different channels; The transformer-convolution fusion module and the downsampling module respectively constitute the last two stages of the encoder. The transformer-convolution fusion module includes depthwise convolution and two linear layers. It uses 7×7 convolution operations to increase the receptive field. The self-attention module and the large-kernel convolution module are stacked vertically to achieve a full fusion of local and global features. An upsampling module, including a linear layer and a pixel deconvolution module; the linear layer is used to expand the feature channel dimension, and the pixel deconvolution module is used to upsample to the same dimension as the decoder input; The decoder includes a transformer module, which is used to map the fused and aligned image feature dimensions to the same number as the decoder input image feature dimensions. The decoder input image is reconstructed through the visual transformer module to solve the mean square loss function between the output features of the basic network and the original image.

Images