A self-supervised learning method and system based on heterogeneous networks
By using a self-supervised learning method based on heterogeneous networks to update parameters based on differences between features, the problem of low efficiency and poor adaptability in network architecture design in existing methods is solved, thereby improving the image feature extraction capability and the accuracy of computer vision tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NANKAI UNIV
- Filing Date
- 2023-08-01
- Publication Date
- 2026-05-26
AI Technical Summary
Existing self-supervised learning methods are inefficient and poorly adaptable when designing new network architectures, and cannot effectively utilize the complementarity between different networks.
By connecting heterogeneous base and auxiliary networks in series, teacher and student models with identical structures are constructed. Unlabeled image datasets are used for preprocessing and self-supervised learning. The differences between features are used as a loss function to update network parameters, thus achieving self-supervised learning of heterogeneous networks.
It improves the high-quality image feature representation capability of the basic network, and enhances the performance of computer vision tasks such as image classification, unsupervised semantic segmentation, semantic segmentation, and object detection.
Smart Images
Figure CN117036854B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and in particular relates to a self-supervised learning method and system based on heterogeneous networks. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] Recently, self-supervised representation learning has been shown to learn rich image representations from large amounts of unlabeled data, thereby improving the accuracy of neural network models in tasks such as image classification, unsupervised semantic segmentation, semantic segmentation, object detection, and instance detection. Due to different computational mechanisms, different networks learn image representations with different features and advantages, thus different networks can complement each other.
[0004] Existing methods, such as Conformer, Convformer, and Swin, mainly utilize this complementarity by combining different networks when designing new neural network architectures. However, designing new neural network architectures requires careful consideration and extensive experimentation, and the resulting models often require more parameters. Furthermore, once the design is complete, the characteristics and advantages of the newly designed structure are fixed and cannot continue to adapt to new characteristics and advantages.
[0005] Therefore, existing network complementarity schemes have two problems: (1) efficiency problems caused by the cumbersome design of new network architectures, and (2) adaptability problems of new architectures that only consider learning the representation of a single architecture. These two problems lead to many limitations on mutual learning between networks. Summary of the Invention
[0006] To overcome the shortcomings of the prior art, this invention provides a self-supervised learning method and system based on heterogeneous networks. It utilizes heterogeneous auxiliary networks to guide the representation learning of the base network, enabling the base network to learn the advantages of different network architectures and improve the base network's ability to represent high-quality image features.
[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:
[0008] The first aspect of this invention provides a self-supervised learning method based on heterogeneous networks.
[0009] A self-supervised learning method based on heterogeneous networks includes:
[0010] By connecting heterogeneous base and auxiliary networks, teacher and student models with identical structures are constructed.
[0011] Preprocess the unlabeled image dataset to obtain two different views corresponding to each image;
[0012] Based on the preprocessed unlabeled image dataset, self-supervised learning is performed on the teacher model and the student model. The learned base network is used as a self-supervised model for feature extraction of the image to be processed.
[0013] The self-supervised learning is as follows:
[0014] Two different views corresponding to the same image in the unlabeled image dataset are input into the teacher model and the student model respectively to obtain the features output by the base network and auxiliary network of the teacher model and the student model respectively; the difference between the features is used as the loss function to update the parameters of the base network and the auxiliary network.
[0015] Furthermore, the construction of teacher and student models with identical structures specifically includes:
[0016] The basic network and the auxiliary network are connected in sequence. The basic network takes the image as input, and the auxiliary network takes the features output by the basic network as input.
[0017] The concatenated network is copied twice, as a teacher model and a student model.
[0018] Furthermore, the preprocessing specifically includes:
[0019] For each image in the unlabeled image dataset, perform image transformation to obtain two different views corresponding to each image.
[0020] Furthermore, the image transformation includes random cropping, random scaling, random color changes, and random blurring.
[0021] Furthermore, the features obtained from the base network and auxiliary network outputs of the teacher model and the student model respectively include four features: features of the base network output of the teacher model, features of the auxiliary network output of the teacher model, features of the base network output of the student model, and features of the auxiliary network output of the student model.
[0022] Furthermore, the step of using the difference between features as the loss function to update the parameters of the base network and the auxiliary network specifically involves:
[0023] The parameters of the auxiliary network are updated using the difference between the features output by the auxiliary network of the teacher model and the features output by the auxiliary network of the student model as the loss function.
[0024] The parameters of the base network are updated using the difference between the features output by the auxiliary network of the teacher model and the features output by the base network of the student model as the loss function.
[0025] Furthermore, the specific formula for the loss function is as follows:
[0026]
[0027]
[0028] in, These are the features output by the auxiliary networks of the student model and the teacher model, respectively, z1, These are the features output by the base network of the student model and the features output by the auxiliary network of the teacher model, respectively, p1, and It is a multilayer perceptron used for image classification based on the features output by the network. Softmax is a normalized exponential function used to calculate the probability of image classification.
[0029] A second aspect of the present invention provides a self-supervised learning system based on heterogeneous networks.
[0030] A self-supervised learning system based on heterogeneous networks includes a model building module, a preprocessing module, and a self-supervised learning module.
[0031] The model building module is configured to: connect heterogeneous base networks and auxiliary networks in series to build teacher and student models with identical structures;
[0032] The preprocessing module is configured to preprocess the unlabeled image dataset to obtain two different views corresponding to each image;
[0033] The self-supervised learning module is configured to: perform self-supervised learning on the teacher model and student model based on the preprocessed unlabeled image dataset, and use the learned base network as a self-supervised model for feature extraction of the image to be processed.
[0034] The self-supervised learning is as follows:
[0035] Two different views corresponding to the same image in the unlabeled image dataset are input into the teacher model and the student model respectively to obtain the features output by the base network and auxiliary network of the teacher model and the student model respectively; the difference between the features is used as the loss function to update the parameters of the base network and the auxiliary network.
[0036] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a self-supervised learning method based on heterogeneous networks as described in the first aspect of the present invention.
[0037] A fourth aspect of the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a self-supervised learning method based on heterogeneous networks as described in the first aspect of the present invention.
[0038] The above one or more technical solutions have the following beneficial effects:
[0039] This invention constructs a loss function based on the difference between the features extracted by the base network and the auxiliary network. While ensuring that the auxiliary network learns high-quality representations, it transfers the characteristics and advantages of the auxiliary network to the base network, enabling the base network to learn the characteristics and advantages of the auxiliary network, thereby improving the base network's ability to represent high-quality image features.
[0040] Compared to other methods that rely on cumbersome design of new network architectures to leverage the complementarity between different neural network architectures, this invention proposes a self-supervised learning method that uses representation learning to enable a network to learn features of other architectures without modifying the network structure, thus improving efficiency.
[0041] Compared to existing self-supervised learning methods that only consider learning representations of a single architecture, this invention utilizes the complementarity of different architectures to improve the quality of representations extracted by the self-supervised model. The self-supervised model trained based on this method can be used for various computer vision tasks and improve performance, including image classification, unsupervised semantic segmentation, semantic segmentation, object detection, instance detection, etc.
[0042] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0043] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0044] Figure 1 This is a flowchart of the method in the first embodiment.
[0045] Figure 2 This is a method framework diagram for the first embodiment. Detailed Implementation
[0046] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0047] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0048] Example 1
[0049] In one or more embodiments, a self-supervised learning method based on heterogeneous networks is disclosed. Figure 1 The flowchart for the self-supervised learning method is as follows. Figure 2 A framework diagram for self-supervised learning methods, such as Figure 1 , Figure 2 As shown, it includes the following steps:
[0050] Step S1: Connect the heterogeneous base network and auxiliary network to construct teacher and student models with the same structure.
[0051] To enable the base network to learn the characteristics and advantages of the auxiliary network architecture, the base network and auxiliary network are first concatenated in sequence, i.e., the base network takes the image as input and the auxiliary network takes the output features of the base network as input. Then, two copies of the concatenated network are made, namely the teacher model and the student model. The base network and auxiliary network of the student model are represented by f1 and h1, respectively, and the base network and auxiliary network of the teacher model are represented by f2 and h2, respectively.
[0052] Step S2: Preprocess the unlabeled image dataset to obtain two different views corresponding to each image.
[0053] Specifically, for each image in the unlabeled image dataset, the image is transformed using different methods such as random cropping, random scaling, random color changes, and random blurring to obtain two different views of the image, denoted as x1 and x2 respectively.
[0054] Step S3: Based on the preprocessed unlabeled image dataset, perform self-supervised learning on the teacher model and student model, and use the learned base network as a self-supervised model for feature extraction of the image to be processed.
[0055] Using an unlabeled image dataset as the training set, both the teacher and student models are trained. During training, the differences in features extracted by different networks are used as the loss function to continuously update the parameters of the base and auxiliary networks until the training stops. The specific process is as follows:
[0056] Step S301: Input two different views corresponding to the same image in the unlabeled image dataset into the teacher model and the student model respectively, and obtain the features output by the basic network and auxiliary network of the teacher model and the student model respectively.
[0057] The two views are processed by the teacher model and the student model respectively, yielding the feature z1 = f1(x1) output by the base network in the student model, and the feature output by the auxiliary network in the student model. The features z2 = f2(x2) output by the base network in the teacher model are obtained, and the features output by the auxiliary network in the teacher model are also obtained.
[0058] Step S301: Using the difference between features as the loss function, update the parameters of the base network and the auxiliary network, specifically divided into:
[0059] (1) Design the loss function, including:
[0060] Based on the differences in the output features of the auxiliary networks of the teacher model and the student model, i.e. and The difference is represented by the loss function:
[0061]
[0062] in, and It is a multilayer perceptron used for image classification based on the features output by the network. Softmax is a normalized exponential function used to calculate the probability of image classification. Overall, it transforms the features output by the network into a probability distribution of image categories. The parameters of the auxiliary network in the student model are updated with this loss function as the target, ensuring that the auxiliary network learns high-quality image representations.
[0063] The difference between the output features of the teacher model auxiliary network and the output features of the student model base network, i.e. The difference between z1 and z2 is represented by the loss function:
[0064]
[0065] Among them, p1 and It is a multilayer perceptron, which has the same function as the perceptron in formula (1). It converts the features output by the network into a probability distribution of the image category as a whole. It updates the parameters of the base network in the student model with the loss function as the target, and transfers the characteristics and advantages of the auxiliary network to the base network, so that the base network learns the characteristics and advantages of the auxiliary network.
[0066] Adding the two loss functions together yields the total loss function, the specific formula of which is:
[0067]
[0068] (2) Express the parameters of the student model and the classroom model as θ respectively. student and θ teacher Based on the overall loss function, the parameters θ of the student model are updated using the backpropagation algorithm. student In the student model, the parameters of the base network and auxiliary network are defined. The first loss function term ensures that the auxiliary network learns high-quality representations, while the second loss function term transfers the characteristics and advantages of the auxiliary network to the base network, enabling the base network to learn these characteristics and advantages. The parameters of the teacher model are updated according to the parameter momentum of the student model. It is assumed that... and Let be the parameters of the teacher model and the student model after the t-th iteration, respectively. Then, the parameters of the teacher model are updated as follows:
[0069]
[0070] Where λ represents the update speed of the teacher model, which is a number between 0.0 and 1.0.
[0071] After training, the base networks of both the teacher model and the student model can be used as the final self-supervised models for feature extraction in tasks such as image classification, unsupervised semantic segmentation, semantic segmentation, object detection, and instance detection.
[0072] A more specific example is to apply the self-supervised model obtained by the above method to image classification, specifically including:
[0073] (1) Obtain the image to be classified;
[0074] (2) Input the image to be classified into the self-supervised model trained by the above self-supervised learning method and extract features;
[0075] (3) Based on the extracted image features, the image is classified using a multilayer perceptron to obtain the classification results.
[0076] To demonstrate the effectiveness of this invention, the self-supervised learning method proposed in this invention was tested on multiple computer vision tasks. As shown in Table 1, compared with the existing method iBOT, the self-supervised training method proposed in this invention can achieve superior results in tasks such as object detection, semantic segmentation, unsupervised semantic segmentation, and image classification.
[0077] Table 1 Comparison of training effects between the present invention and existing methods.
[0078] iBOT This invention Object detection / mAP 45.9 46.3 VOC semantic segmentation / mIoU 78.8 80.1 ImageNet-S Unsupervised Semantic Segmentation / mIoU 14.0 16.1 ImageNet-1K image classification / % 71.3 72.6
[0079] This invention utilizes heterogeneous auxiliary networks to guide the representation learning of a base network, enabling the base network to learn the advantages of different network architectures and improving the representation quality of the base network. Compared to existing self-supervised representation learning methods that only consider the learning of representations of a single architecture, this method allows the model to learn the advantages of different architectures. The self-supervised representation learning method based on this method can provide high-quality image features and can significantly improve the accuracy of tasks such as image classification, unsupervised semantic segmentation, semantic segmentation, object detection, and instance detection.
[0080] Example 2
[0081] In one or more embodiments, a self-supervised learning system based on heterogeneous networks is disclosed, including a model building module, a preprocessing module, and a self-supervised learning module:
[0082] The model building module is configured to: connect heterogeneous base networks and auxiliary networks in series to build teacher and student models with identical structures;
[0083] The preprocessing module is configured to preprocess the unlabeled image dataset to obtain two different views corresponding to each image;
[0084] The self-supervised learning module is configured to: perform self-supervised learning on the teacher model and student model based on the preprocessed unlabeled image dataset, and use the learned base network as a self-supervised model for feature extraction of the image to be processed.
[0085] The self-supervised learning is as follows:
[0086] Two different views corresponding to the same image in the unlabeled image dataset are input into the teacher model and the student model respectively to obtain the features output by the base network and auxiliary network of the teacher model and the student model respectively; the difference between the features is used as the loss function to update the parameters of the base network and the auxiliary network.
[0087] Example 3
[0088] The purpose of this embodiment is to provide a computer-readable storage medium.
[0089] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a self-supervised learning method based on heterogeneous networks as described in Embodiment 1 of this disclosure.
[0090] Example 4
[0091] The purpose of this embodiment is to provide an electronic device.
[0092] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of a self-supervised learning method based on heterogeneous networks as described in Embodiment 1 of this disclosure.
[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A self-supervised learning method based on heterogeneous networks, characterized in that, include: By connecting heterogeneous base and auxiliary networks, teacher and student models with identical structures are constructed. The construction of teacher and student models with identical structures is specifically as follows: The basic network and the auxiliary network are connected in sequence. The basic network takes the image as input, and the auxiliary network takes the features output by the basic network as input. Duplicate the concatenated network twice, creating a teacher model and a student model respectively. Preprocess the unlabeled image dataset to obtain two different views corresponding to each image; Based on the preprocessed unlabeled image dataset, self-supervised learning is performed on the teacher model and the student model. The learned base network is used as a self-supervised model for feature extraction of the image to be processed. The self-supervised learning is as follows: Two different views corresponding to the same image in the unlabeled image dataset are input into the teacher model and the student model respectively to obtain the features output by the base network and the auxiliary network of the teacher model and the student model respectively; the difference between the features is used as the loss function to update the parameters of the base network and the auxiliary network. The features obtained from the basic network and auxiliary network outputs of the teacher model and the student model include four features: features of the basic network output of the teacher model, features of the auxiliary network output of the teacher model, features of the basic network output of the student model, and features of the auxiliary network output of the student model. The method of using the difference between features as the loss function to update the parameters of the base network and the auxiliary network is as follows: The parameters of the auxiliary network are updated using the difference between the features output by the auxiliary network of the teacher model and the features output by the auxiliary network of the student model as the loss function. The parameters of the base network are updated using the difference between the features output by the auxiliary network of the teacher model and the features output by the base network of the student model as the loss function.
2. The self-supervised learning method based on heterogeneous networks as described in claim 1, characterized in that, The preprocessing specifically includes: For each image in the unlabeled image dataset, perform image transformation to obtain two different views corresponding to each image.
3. The self-supervised learning method based on heterogeneous networks as described in claim 2, characterized in that, The image transformation includes random cropping, random scaling, random color change, and random blurring.
4. The self-supervised learning method based on heterogeneous networks as described in claim 1, characterized in that, The specific formula for the loss function is as follows: in, , These are the features output by the auxiliary networks of the student model and the teacher model, respectively. , These are the features output by the base network of the student model and the features output by the auxiliary network of the teacher model, respectively. , and It is a multilayer perceptron used for image classification based on the features output by the network. softmax It is a normalized exponential function used to calculate the probability of image classification.
5. A self-supervised learning system based on heterogeneous networks, characterized in that, It includes a model building module, a preprocessing module, and a self-supervised learning module: The model building module is configured to: connect heterogeneous base networks and auxiliary networks in series to build teacher and student models with identical structures; The construction of teacher and student models with identical structures is specifically as follows: The basic network and the auxiliary network are connected in sequence. The basic network takes the image as input, and the auxiliary network takes the features output by the basic network as input. Duplicate the concatenated network twice, creating a teacher model and a student model respectively. The preprocessing module is configured to preprocess the unlabeled image dataset to obtain two different views corresponding to each image; The self-supervised learning module is configured to: perform self-supervised learning on the teacher model and student model based on the preprocessed unlabeled image dataset, and use the learned base network as a self-supervised model for feature extraction of the image to be processed. The self-supervised learning is as follows: Two different views corresponding to the same image in the unlabeled image dataset are input into the teacher model and the student model respectively to obtain the features output by the base network and the auxiliary network of the teacher model and the student model respectively; the difference between the features is used as the loss function to update the parameters of the base network and the auxiliary network. The features obtained from the basic network and auxiliary network outputs of the teacher model and the student model include four features: features of the basic network output of the teacher model, features of the auxiliary network output of the teacher model, features of the basic network output of the student model, and features of the auxiliary network output of the student model. The method of using the difference between features as the loss function to update the parameters of the base network and the auxiliary network is as follows: The parameters of the auxiliary network are updated using the difference between the features output by the auxiliary network of the teacher model and the features output by the auxiliary network of the student model as the loss function. The parameters of the base network are updated using the difference between the features output by the auxiliary network of the teacher model and the features output by the base network of the student model as the loss function.
6. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-4.
7. A storage medium characterized in that it non-transitory stores computer-readable instructions, wherein, When the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-4 are executed.