A visual transformer self-supervised learning method and system based on multi-dimensional relationship modeling

Abstract

The application belongs to the technical field of computer vision, and provides a visual Transformer self-supervised learning method and system based on multi-dimensional relationship modeling; the method uses self-relationship modeling on spatial dimensions and channel dimensions, uses different image transformation to process images to obtain different views of the images; the different views of the images are processed by a teacher network and a student network respectively to obtain feature maps; the feature map extracted by the student network is further processed by a convolution layer; a self-relationship matrix of the feature map in the spatial dimensions and the channel dimensions is calculated by point multiplication; the difference between the relationship matrices extracted by the teacher network and the student network is calculated as a loss function, and the derivative of the loss function with respect to the network parameters is used to update the network parameters; compared with the existing self-supervised learning method which only considers the features of visual patterns, the method simultaneously considers the correlation of visual patterns in the spatial and channel dimensions, and can significantly improve the accuracy of image recognition, semantic segmentation, target detection, instance detection and the like.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a visual Transformer self-supervised learning method and system based on multi-dimensional relationship modeling. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Self-supervised learning has gradually become a research hotspot in the field of computer vision, especially self-supervised models based on Transformers, which have been proven to extract high-quality image features without being limited by annotation costs. Improving the model's ability to represent visual pattern features is key to self-supervised learning.

[0004] In real-world scenarios, different visual patterns are related; similar visual patterns are strongly correlated, while different visual patterns are weakly correlated. For Transformer networks, the extracted image features contain multiple dimensions, such as spatial and channel dimensions. Spatially, each pixel belongs to a category, and pixels belonging to the same category are strongly correlated. In the channel dimension, different channels represent different visual models, and the patterns represented by different channels also have certain relationships. The ability to accurately capture the relationships between visual patterns is a necessary condition for neural networks to have powerful feature extraction capabilities.

[0005] The inventors discovered that current self-supervised learning methods for Transformer networks primarily rely on image features, neglecting the relationships between visual patterns. Chen et al. constructed positive and negative samples through contrastive learning, enhancing representation quality by bringing positive samples closer and pushing negative samples further apart in the feature space. Caron et al. assigned image category labels through clustering, improving the model's ability to recognize image semantics by strengthening category consistency across different viewpoints. Building upon Caron et al.'s method, Zhou et al. introduced category consistency at a finer granular dimension—pixels—further enhancing the model's representational capabilities. While these methods primarily focus on the feature level and have achieved good results on many visual tasks, they neglect the relationships between visual patterns, thus failing to fully leverage the potential of self-supervised models to express visual patterns. Summary of the Invention

[0006] To address at least one of the technical problems in the background art, this invention provides a visual Transformer self-supervised learning method and system based on multi-dimensional relationship modeling. This method explicitly enhances the model's ability to model visual relationships during Transformer self-supervised training, solving the problem that existing self-supervised learning methods only consider features and ignore the relationships between visual patterns.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] The first aspect of this invention provides a self-supervised learning method for visual Transformers based on multi-dimensional relationship modeling, comprising the following steps:

[0009] The training image sample data is preprocessed to obtain different views corresponding to each image, and the feature maps of the corresponding views are extracted.

[0010] Semantic features are obtained through self-supervised learning using feature maps from different views corresponding to each image and a Transformer network; wherein, the self-supervised learning process includes:

[0011] Based on the feature maps of different views, the autorelation matrix of all views in the channel dimension is obtained by calculating the dot product. The feature map of each view is divided into multiple subsets along the channel dimension, and the autorelation matrix between pixels in the spatial dimension of each subset feature map is calculated by the dot product.

[0012] The difference between the autorelation matrices in the channel dimension and the spatial dimension is used as the loss function, and the parameters of the Transformer network are updated by the gradient of the loss function.

[0013] A second aspect of the present invention provides a visual Transformer self-supervised learning system based on multi-dimensional relationship modeling, comprising:

[0014] The feature extraction module is used to preprocess the training image sample data to obtain different views corresponding to each image and extract the feature map of the corresponding view;

[0015] The self-supervised learning module is used to obtain semantic features through self-supervised learning based on the feature maps of different views corresponding to each image and the Transformer network; wherein, the self-supervised learning process includes:

[0016] Based on the feature maps of different views, the autorelation matrix of all views in the channel dimension is obtained by calculating the dot product. The feature map of each view is divided into multiple subsets along the channel dimension, and the autorelation matrix between pixels in the spatial dimension of each subset feature map is calculated by the dot product.

[0017] The difference between the autorelation matrices in the channel dimension and the spatial dimension is used as the loss function, and the parameters of the Transformer network are updated by the gradient of the loss function.

[0018] A third aspect of the present invention provides a computer-readable storage medium.

[0019] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of a visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described above.

[0020] A fourth aspect of the present invention provides a computer device.

[0021] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described above.

[0022] Compared with the prior art, the beneficial effects of the present invention are:

[0023] This invention improves the representation quality of self-supervised models by utilizing self-relation modeling in both spatial and channel dimensions. Different image transformations are used to process the image, resulting in different views. These different views are then processed by teacher and student networks to obtain feature maps. The self-relation matrices of the feature maps in both spatial and channel dimensions are calculated using dot products. The difference between the relationship matrices extracted by the teacher and student networks is used as the loss function, and its derivative with respect to the network parameters is used to update the network parameters. Compared to existing self-supervised learning methods that only consider visual pattern features, this method considers the correlation of visual patterns in both spatial and channel dimensions. The self-supervised pre-trained model based on this method can provide high-quality semantic features, significantly improving the accuracy of tasks such as image recognition, semantic segmentation, object detection, and instance detection.

[0024] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0026] Figure 1 This is a flowchart of the visual Transformer self-supervised learning method based on multi-dimensional relationship modeling in this invention;

[0027] Figure 2 This is a framework diagram of the visual Transformer self-supervised learning method based on multi-dimensional relationship modeling in this invention;

[0028] Figure 3 This is a schematic diagram illustrating the main differences between the present invention and existing methods. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0031] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0032] To address the problem mentioned in the background section that existing self-supervised learning methods only consider features and ignore the relationships between visual patterns, this invention uses different image transformations to process images and obtain different views of the image. These different views are processed by a teacher network and a student network, respectively, to obtain feature maps. The feature maps extracted by the student network are further processed through convolutional layers. The self-relation matrix of the feature maps in the spatial and channel dimensions is calculated using dot products. The difference between the relationship matrices extracted by the teacher network and the student network is calculated as the loss function, and its derivative with respect to the network parameters is used to update the network parameters. This invention proposes a method to explicitly enhance the model's ability to model visual relationships during Transformer self-supervised training. Compared to other methods that only consider features to achieve self-supervised learning, this invention's method can significantly improve the ability of self-supervised models to model visual relationships, thereby improving the feature representation ability of self-supervised models. The model trained based on this method can be used as a pre-training model for various visual tasks, while simultaneously improving accuracy.

[0033] Example 1

[0034] like Figures 1-2 As shown, this embodiment provides a self-supervised learning method for visual Transformers based on multi-dimensional relationship modeling, including the following steps:

[0035] Step 1: Obtain the image sample data to be trained;

[0036] Step 2: Preprocess the training image sample data;

[0037] The preprocessing process includes:

[0038] For the large-scale dataset to be trained, the images are preprocessed by random cropping, random flipping, random blurring, random grayscale conversion, etc., and the preprocessed images are transformed to a uniform size.

[0039] It is understandable that the specific size of the image can be set according to the requirements. In this implementation, the size of the preprocessed image is uniformly transformed to 224×224.

[0040] Step 3: Perform random transformations on each preprocessed image using different image transformation methods to obtain different views corresponding to each image;

[0041] Step 4: Perform self-supervised learning to obtain semantic features based on different views corresponding to each image and the Transformer network. The self-supervised learning process includes:

[0042] Feature maps for the corresponding views are obtained by extracting features using both teacher and student networks;

[0043] Based on the feature maps of different views, the autorelation matrix of all views in the channel dimension is obtained by calculating the dot product. The feature map of each view is divided into multiple subsets along the channel dimension. The autorelation matrix between pixels in the spatial dimension of each sub-feature map is calculated by the dot product. The difference between the autorelation matrices in the channel dimension and the spatial dimension is used as the loss function. The parameters of the student network and the teacher network are updated by the gradient of the loss function.

[0044] As one or more embodiments, in step 4, the Transformer network includes a teacher network and a student network, wherein the student network updates its parameters through gradient descent, and the parameters of the teacher network are updated based on the parameters of the student network according to momentum.

[0045] To better understand the technical solution of the present invention, this embodiment takes the case where each image has two views, denoted as x1 and x2 respectively. If the number of views is greater than two, the extra views are processed by the student network. During self-supervised learning, the processing method of these views is the same as the processing method of x1.

[0046] Specifically, the technical solutions include the following:

[0047] Step 401: Feature extraction is performed using the teacher network and student network respectively to obtain feature maps for the corresponding views;

[0048] z1 = S(x1)

[0049] z2 = T(x2)

[0050] Where S represents the student network and T represents the teacher network.

[0051] Step 402: The feature maps extracted from the student network are further processed using convolutional layers, normalization layers, and activation layers;

[0052] z'1=M(z1)

[0053] In this embodiment, M represents a function consisting of convolutional layers, normalization layers, and activation layers, representing further processing of features extracted by the student network; the normalization layer is batch normalization; the activation layer is ReLU; the input dimension of the convolutional layer in M ​​is 384, and the output dimension is 384.

[0054] The feature dimension is B×N×C, where B represents the number of images in the current training batch, N represents the number of pixels in the image, and C represents the feature dimension.

[0055] Step 403: Based on the feature maps obtained in Steps 401 and 402, calculate the autorelation matrix in all view channel dimensions by dot product, divide the feature map into multiple subsets along the channel dimensions, and calculate the autorelation matrix in the spatial dimension between pairs of pixels in each feature map subset by dot product.

[0056] Specifically, it includes:

[0057] (1) In terms of spatial dimension, the feature map is first divided into several subsets along the channels:

[0058]

[0059] The `split` function represents the operation of dividing channels. The dimension is B×H×N / H×C, where H represents the number of subsets.

[0060] Then, the autocorrelation matrix in the spatial dimension between each pair of pixels in each subset is calculated by dot product:

[0061]

[0062] Here, matrix P1 represents the relationship between pairs of pixels in each subset, and transpose represents the last two dimensions of the transpose matrix;

[0063] It is understandable that the above example uses one view as an example, and the self-relation matrix P2 in the spatial dimension of the other view is obtained by the same method, so it will not be repeated here.

[0064] (2) Calculate the autorelation matrix of all views in the channel dimension using the feature maps of the corresponding views:

[0065] C1 = transpose(z′1) * z′1

[0066] C2 = transpose(z2) * z2

[0067] Step 404: Update the parameters of the student and teacher networks based on the differences in the self-relation matrices in the channel and spatial dimensions. The difference between the relation matrices of different views of the same image is measured using cross-entropy loss as the loss function, and the derivative of the loss function with respect to the network parameters is calculated as the gradient to update the network parameters; specifically including:

[0068] (1) In the channel dimension, the cross-entropy loss function is used to measure the difference in the spatial dimension relationship matrix between the two views:

[0069]

[0070] in, This represents the relationship between pixels i and j in the first view channel dimension. This represents the relationship between pixel i and pixel j in the second view channel dimension.

[0071] The greater the difference in the relation matrix, the larger the value of the loss function.

[0072] (2) In terms of spatial dimension, the difference in the channel dimension relationship matrix between the two views is measured using the cross-entropy loss function:

[0073]

[0074] Where, in the formula, This represents the relationship between pixels i and j in the first view space dimension. This represents the relationship between pixel i and pixel j in the second view space dimension.

[0075] (3) Add the two loss functions in the channel dimension and the spatial dimension to obtain the final loss function:

[0076] L = L c +L p

[0077] (4) Calculate the reciprocal of the loss with respect to the network parameters, and use it as the gradient to update the parameters of the student network using backpropagation:

[0078]

[0079] in, These are the student network parameters at time t, where... The gradient is the derivative of the loss function with respect to the network parameters, where α is the learning rate. This represents the parameters of the updated model.

[0080] Parameters of the teacher network Update based on the parameter momentum of the student network:

[0081]

[0082] Where m represents momentum. The parameters of the teacher network at time t are gradually increased from 0.996 to 1 as training progresses.

[0083] The self-supervised learning method described in the above embodiments can be applied to tasks such as object detection, semantic segmentation, semi-supervised semantic segmentation, and image classification after training.

[0084] To demonstrate the effectiveness of this invention, the performance of the proposed pre-training method was tested on multiple tasks. As shown in Table 1, compared to the existing method DINO, the self-supervised pre-training method proposed in this invention achieves superior results in tasks such as object detection, semantic segmentation, semi-supervised semantic segmentation, and image classification.

[0085] Table 1 Comparison of training effects between the present invention and existing methods.

[0086] Task DINO This invention Target detection 46.0 46.6 VOC semantic segmentation / mIoU 77.1 78.5 Semi-supervised semantic segmentation / mIoU 22.3 25.9 Image classification / % 79.7 80.4

[0087] The advantages of the above technical solution are, for example, Figure 3 As shown, where, Figure 3 In this context, (a) indicates that existing methods primarily focus on characterization; and (b) indicates that the present invention focuses on the relationships between visual patterns.

[0088] Existing methods mainly focus on representation, while this invention focuses on the relationship between visual patterns. The self-supervised learning method proposed in this invention considers the relationship between visual patterns in the features extracted by the neural network. Compared with other methods that only consider features to achieve self-supervised learning, the method of this invention can significantly improve the ability of self-supervised models to model visual relationships, thereby improving the feature representation ability of self-supervised models. The model trained based on this method can be used as a pre-training model for various visual tasks, while improving accuracy.

[0089] Example 2

[0090] This embodiment provides a visual Transformer self-supervised learning system based on multi-dimensional relationship modeling, including:

[0091] The sample data acquisition module is used to acquire sample data of the images to be trained.

[0092] The feature extraction module is used to preprocess the training image sample data to obtain different views corresponding to each image and extract the feature map of the corresponding view;

[0093] The self-supervised learning module is used to obtain semantic features through self-supervised learning based on the feature maps of different views corresponding to each image and the Transformer network; wherein, the self-supervised learning process includes:

[0094] Based on the feature maps of different views, the autorelation matrix of all views in the channel dimension is obtained by calculating the dot product. The feature map of each view is divided into multiple subsets along the channel dimension, and the autorelation matrix between pixels in the spatial dimension of each subset feature map is calculated by the dot product.

[0095] The difference between the autorelation matrices in the channel dimension and the spatial dimension is used as the loss function, and the parameters of the Transformer network are updated by the gradient of the loss function.

[0096] Example 3

[0097] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described above.

[0098] Example 4

[0099] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described above.

[0100] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0101] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0104] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0105] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A self-supervised learning method for visual Transformers based on multi-dimensional relationship modeling, characterized in that, Includes the following steps: The training image sample data is preprocessed to obtain different views corresponding to each image, and the feature maps of the corresponding views are extracted. Semantic features are obtained through self-supervised learning using feature maps from different views corresponding to each image and a Transformer network; wherein, the self-supervised learning process includes: Based on the feature maps of different views, the autorelation matrix of all views in the channel dimension is obtained by calculating the dot product. The feature map of each view is divided into multiple subsets along the channel dimension, and the autorelation matrix between pixels in the spatial dimension of each subset feature map is calculated by the dot product. The difference between the autorelation matrices in the channel dimension and the spatial dimension is used as the loss function, and the parameters of the Transformer network are updated by the gradient of the loss function.

2. The visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in claim 1, characterized in that, The step of using the difference between the autorelation matrices in the channel dimension and the spatial dimension as the loss function, and updating the parameters of the Transformer network through the gradient of the loss function, includes: Based on the autorelation matrices in the channel dimension and the spatial dimension, the difference between the autorelation matrices in the channel dimension and the spatial dimension is calculated using the cross-entropy loss function to obtain the corresponding loss function; The loss function is used as the gradient to update the parameters of the student network using backpropagation; The parameters of the teacher network are obtained by updating the parameter momentum of the student network.

3. The visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in claim 1, characterized in that, For each image, different views are represented by feature maps obtained through feature extraction using teacher and student networks: in, Indicates student network, Indicates a teacher network, and Represents different views.

4. The visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in claim 1, characterized in that, After acquiring the training image sample data, the data is preprocessed. Each image is randomly transformed using different image transformation methods to obtain different views corresponding to each image.

5. The visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in claim 3, characterized in that, Feature maps for the corresponding views are obtained by extracting features using both teacher and student networks. The feature maps extracted by the student network are further processed using convolutional layers, normalization layers, and activation layers.

6. The visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in claim 2, characterized in that, The loss function is obtained by calculating the difference between the self-relation matrices in the channel dimension and the spatial dimension using the cross-entropy loss function respectively. The loss function in the channel dimension is: In the formula, Represents pixels in the first view channel dimension and pixels The relationship between them Represents pixels in the second view channel dimension and pixels The relationship between them; The loss function in the spatial dimension is: In the formula, Represents pixels in the first view space dimension and pixels The relationship between them Represents pixels in the second view space dimension and pixels The relationship between them.

7. The visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in claim 2, characterized in that, The expression for updating the parameters of the student network using the loss function as the gradient through backpropagation is as follows: in, It is a moment Student network parameters It is the gradient, which is the derivative of the loss function with respect to the network parameters. It's the learning rate. Indicates the parameters of the updated model; The expression for obtaining the parameters of the teacher network according to the parameter momentum update of the student network is as follows: in, Indicates momentum. It is a moment Teacher network parameters.

8. A visual Transformer self-supervised learning system based on multi-dimensional relationship modeling, characterized in that, include: The feature extraction module is used to preprocess the training image sample data to obtain different views corresponding to each image and extract the feature map of the corresponding view; The self-supervised learning module is used to obtain semantic features through self-supervised learning based on the feature maps of different views corresponding to each image and the Transformer network; wherein, the self-supervised learning process includes: Based on the feature maps of different views, the autorelation matrix of all views in the channel dimension is obtained by calculating the dot product. The feature map of each view is divided into multiple subsets along the channel dimension, and the autorelation matrix between pixels in the spatial dimension of each subset feature map is calculated by the dot product. The difference between the autorelation matrices in the channel dimension and the spatial dimension is used as the loss function, and the parameters of the Transformer network are updated by the gradient of the loss function.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the visual Transformer self-supervised learning method based on multi-dimensional relationship modeling as described in any one of claims 1-7.

Images