A method and apparatus for generalized supervision representation learning

Abstract

The application discloses a kind of pan supervision representation learning method and device, the method includes: obtaining training data;Wherein, training data includes image data and the label information corresponding to image data;Image data corresponding first spatial feature is obtained by the feature extraction of visual network model to the training data input, and second spatial feature is obtained based on first spatial feature mapping, and third spatial feature is obtained based on second spatial feature mapping;First loss function value of first spatial feature and second spatial feature is calculated, and the second loss function value of third spatial feature is calculated according to label information;The parameter of the visual network model is updated based on first loss function value and second loss function value, to obtain the visual network model after training.The application can make that the representation learned not only can obtain superior performance on training data set, but also can obtain good transfer performance on other visual tasks, can realize image detection and segmentation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning technology, and in particular to a generalized supervised representation learning method and apparatus. Background Technology

[0002] In the development of deep learning technology, learning a good representation from data is an important and challenging task, and a fundamental step in many research fields, such as word embedding learning and deep metric learning. In computer vision, existing methods generally employ supervised or unsupervised learning algorithms, using deep neural networks to extract meaningful representational information from raw images, and then using these learned representations for various downstream tasks, such as image classification, semantic segmentation, and object detection.

[0003] As an important branch of unsupervised learning, self-supervised learning aims to automatically generate labels for training model representations through multiple different agent tasks. Existing methods include those based on image restoration and image rotation prediction. However, self-supervised methods often achieve relatively poor classification performance because they forgo utilizing large amounts of labeled data, which could provide sufficient high-level semantic information for the representation. Fully supervised deep representation learning algorithms tend to achieve better results. On the ImageNet dataset, fully supervised algorithms can achieve over 80% accuracy, demonstrating their superiority. Fully supervised learning focuses on network architecture design, data augmentation, and loss function structure. First, foundational models proposed in recent years, such as ResNet and Visual Transformer, have improved benchmark results on various datasets. For data processing, related research aims to fully utilize raw images for higher training efficiency and better generalization performance, such as the widely used data augmentation method Mixup and many other data augmentation algorithms. For loss function design, classification loss functions are generally based on cross-entropy loss functions, including ArcFace and CosFace. In addition, some loss functions are used in deep metric learning to construct more discriminative embedding spaces. Among them, the common triplet loss function and surrogate loss function have been applied in a variety of vision tasks.

[0004] However, recent research has focused not only on the performance of algorithms on single datasets but also on the transfer performance of representation learning. For example, researchers have transferred pre-trained models on ImageNet to datasets for tasks such as object detection and segmentation to test their performance. In recent years, contrastive learning algorithms have demonstrated that unsupervised learning methods can also learn meaningful data representations and achieve stronger transfer performance than fully supervised learning on downstream tasks. For example, models pre-trained with the MoCo algorithm have outperformed fully supervised learning models on detection and segmentation tasks. Generally speaking, contrastive learning aims to reduce the distance between different data augmentation results of the same input image in the representation space and increase the distance between data augmentation results of different input images. Therefore, contrastive learning tends to learn general instance-level information, while fully supervised learning focuses more on task-related category-level information. In this context, fully supervised learning and unsupervised learning have different advantages in terms of the learned representations, which naturally raises the question: is it possible to combine self-supervised learning and fully supervised learning to obtain a better model? Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems in the related art.

[0006] Therefore, the purpose of this invention is to propose a generalized supervised representation learning method. This method involves performing two random data augmentations on the original image, including random cropping, random Gaussian noise, and random flipping. A shared neural network model is then used to process the augmented data to obtain the corresponding data representation. A symmetric self-supervised contrastive loss function is used to constrain the data representation, where one data representation is projected onto another data space using a multilayer perceptron to prevent model collapse. After unsupervised constraints, another multilayer perceptron is added for projection, and the projected result is fully supervised using the data's true labels. The data representations trained under both unsupervised and fully supervised conditions are then transferred to downstream tasks such as detection and segmentation for transfer learning testing.

[0007] To achieve the above objectives, this invention proposes a generalized supervised representation learning method, comprising:

[0008] Acquire training data; wherein the training data includes image data and label information corresponding to the image data;

[0009] The training data is input into a visual network model to extract features to obtain the first spatial features corresponding to the image data, and the second spatial features are obtained based on the mapping of the first spatial features, and the third spatial features are obtained based on the mapping of the second spatial features.

[0010] Calculate the first loss function values ​​of the first spatial feature and the second spatial feature, and calculate the second loss function value of the third spatial feature based on the label information;

[0011] The parameters of the visual network model are updated based on the first loss function value and the second loss function value to obtain the trained visual network model.

[0012] The generalized supervised representation learning method according to embodiments of the present invention may also have the following additional technical features:

[0013] Furthermore, in one embodiment of the present invention, before inputting the training data into the visual network model for feature extraction, the method further includes: performing data preprocessing on the training data to obtain a first data augmentation result and a second data augmentation result; wherein the data preprocessing includes at least one of the following processing methods: random cropping, random Gaussian blurring, random flipping, and random grayscale processing.

[0014] Furthermore, in one embodiment of the present invention, the visual network model includes a first structural network, and the step of inputting training data into the visual network model for feature extraction to obtain the first spatial features corresponding to the image data includes: inputting the first data augmentation result and the second data augmentation result into the visual network model; and using the first structural network of the visual network model to perform feature extraction to obtain the first spatial features corresponding to the image data.

[0015] Furthermore, in one embodiment of the present invention, the visual network model further includes a second structural network, and the step of obtaining the second spatial feature based on the first spatial feature mapping and obtaining the third spatial feature based on the second spatial feature mapping includes: using the second structural network to perform feature mapping of the first spatial feature in a first preset data space to obtain the second spatial feature; and using the second structural network to perform feature mapping of the second spatial feature in a second preset data space to obtain the third spatial feature.

[0016] Furthermore, in one embodiment of the present invention, the step of calculating the first loss function value of the first spatial feature and the second spatial feature, and calculating the second loss function value of the third spatial feature based on the label information, includes: calculating the first loss function value of the first spatial feature and the second spatial feature using a self-supervised contrastive loss function; and calculating the second loss function value of the cross-entropy loss function of the third spatial feature using the label information.

[0017] To achieve the above objectives, another aspect of the present invention proposes a generalized supervised representation learning device, comprising:

[0018] A training data acquisition module is used to acquire training data; wherein, the training data includes image data and label information corresponding to the image data;

[0019] The feature data acquisition module is used to input the training data into a visual network model to extract features to obtain the first spatial features corresponding to the image data, and to obtain the second spatial features based on the first spatial features and the third spatial features based on the second spatial features.

[0020] The feature data learning module is used to calculate the first loss function value of the first spatial feature and the second spatial feature, and to calculate the second loss function value of the third spatial feature based on the label information;

[0021] The model parameter training module is used to update the parameters of the visual network model based on the first loss function value and the second loss function value to obtain the trained visual network model.

[0022] The generalized supervised representation learning and apparatus of this invention first performs data augmentation on the data to obtain two symmetrical samples; then, it uses a visual basic model to extract features to obtain data representations; it uses a self-supervised contrastive loss function to constrain the data representations of the two symmetrical samples; it uses a multilayer perceptron to project the data representations and uses real labels for fully supervised learning; finally, it transfers the pre-trained data representations to downstream tasks such as monitoring and segmentation, demonstrating good transfer performance.

[0023] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0025] Figure 1 A flowchart of the basis generalized supervised representation learning method according to an embodiment of the present invention;

[0026] Figure 2 This is an architecture diagram of the generalized supervised representation learning method according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram comparing the present invention with other representation learning methods according to embodiments of the present invention;

[0028] Figure 4 This is a schematic diagram of the structure of a generalized supervised representation learning device according to an embodiment of the present invention. Detailed Implementation

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] The following description, with reference to the accompanying drawings, describes a generalized supervised representation learning method and apparatus according to embodiments of the present invention.

[0032] Figure 1 This is a flowchart of a generalized supervised representation learning method according to an embodiment of the present invention.

[0033] like Figure 1 As shown, the method includes, but is not limited to, the following steps:

[0034] S1, Obtain training data; where the training data includes image data and the label information corresponding to the image data.

[0035] It is understandable that the acquired training data can be various types of image data, along with the corresponding label information for each image data. This data is then used as input to the subsequent model.

[0036] Specifically, for any input image x, this embodiment of the invention uses operations such as random cropping, random Gaussian blurring, random flipping, and random grayscale processing to obtain two data augmentation results, denoted as t1 and t2, corresponding to the first data augmentation result and the second data augmentation result, respectively. These two data augmented samples are used as positive samples in subsequent self-supervised learning.

[0037] S2, input the training data into the visual network model to extract features to obtain the first spatial features corresponding to the image data, and obtain the second spatial features based on the first spatial feature mapping, and obtain the third spatial features based on the second spatial feature mapping.

[0038] As an example, a mainstream visual foundational model framework is used to extract features from samples, including the ResNet structure based on convolutional neural networks and the Transformer structure based on self-attention mechanisms. The framework consists of a basic backbone network and a multilayer perceptron structure based on a fully connected network. The feature space is denoted as K, and the features obtained for t1 and t2 are denoted as k1 and k2, respectively, representing the first spatial feature and the second spatial feature.

[0039] Furthermore, using a multilayer perceptron structure, k1 is projected onto another data space Q of the same dimension, denoted as q1, which is the second spatial feature. Then, using the multilayer perceptron structure, the data space Q is projected onto another data space Y, with the feature denoted as y. i This is a feature of the third space.

[0040] S3, calculate the first loss function value of the first spatial feature and the second spatial feature, and calculate the second loss function value of the third spatial feature based on the label information.

[0041] Specifically, a self-supervised contrastive loss function is used to constrain the first spatial features and the second spatial features, as shown in the following formula:

[0042]

[0043] Where, q i k represents the feature corresponding to the i-th input image in the data space Q. i The feature corresponding to the i-th input image in the data space K is represented by the above contrast loss function, which can increase the similarity between different data augmentation results of the same input image, while reducing the similarity between data augmentations of different input images.

[0044] Furthermore, the data space Q is projected onto another data space Y using a multilayer perceptron structure, and the data features obtained in this space are denoted as y. i At the same time, using the corresponding real data labels i The cross-entropy loss function is then applied to the above features under supervision, and the specific formula is as follows:

[0045]

[0046] Among them, y i,j Represents y i The j-th component, where N represents the total number of samples in a batch of input images. The overall generalized supervised learning framework is a comprehensive structure combining the above self-supervised and fully supervised constraints.

[0047] S4. Update the parameters of the visual network model based on the first loss function value and the second loss function value to obtain the trained visual network model.

[0048] Understandably, after calculating the above loss function, the model parameters learn the image's feature data, allowing training to obtain a model with multiple features. The above framework, after self-supervised and fully supervised constraints, yields pre-trained data representations. These representations are then directly extracted and fine-tuned using the structures of existing object detection and semantic segmentation methods. The transfer learning performance of the generalized supervised representation learning is then tested. This performance is then used for subsequent image detection and segmentation.

[0049] Furthermore, the architecture diagram of the generalized supervised representation learning method of the present invention is as follows: Figure 2 As shown, unlike representation learning which utilizes labeled or unlabeled data separately, this invention proposes a unified end-to-end learning framework. Hierarchical surrogate representations are extracted from each image, and different levels of supervision are applied to these surrogate representations. Specifically, fully supervised learning imposes stricter constraints on the representation space compared to self-supervised learning. Therefore, self-supervised representations possess both instance-level and category-level information, while fully supervised learning removes instance-level information from the representation. Based on this, category-level representations are placed after instance-level representations, and category-level image information is extracted from them. Self-supervised learning and fully supervised learning with real-label information are applied to the corresponding surrogate representations respectively to ensure that our model can acquire both types of information conflict-free. The framework can be applied to existing contrastive learning methods by adding a multilayer perceptron structure to the instance-level surrogate representations and applying a cross-entropy loss function as supervision.

[0050] Furthermore, a comparative schematic diagram of the present invention with other representation learning methods is shown below. Figure 3 As shown, the effects of the present invention can be seen.

[0051] The generalized supervised representation learning method according to embodiments of the present invention can effectively combine self-supervised learning and fully supervised learning to obtain a training model with better generalization performance, and can also achieve good transfer performance on other visual tasks, enabling image detection and segmentation.

[0052] To achieve the above embodiments, such as Figure 4 As shown, this embodiment also provides a generalized supervised representation learning device 10, which includes: a training data acquisition module 100, a feature data acquisition module 200, a feature data learning module 300, and a model parameter training module 400.

[0053] The training data acquisition module 100 is used to acquire training data; wherein, the training data includes image data and label information corresponding to the image data;

[0054] The feature data acquisition module 200 is used to input training data into a visual network model to extract features to obtain the first spatial features corresponding to the image data, and to obtain the second spatial features based on the first spatial feature mapping, and to obtain the third spatial features based on the second spatial feature mapping.

[0055] The feature data learning module 300 is used to calculate the first loss function value of the first spatial feature and the second spatial feature, and to calculate the second loss function value of the third spatial feature based on the label information;

[0056] The model parameter training module 400 is used to update the parameters of the visual network model based on the first loss function value and the second loss function value to obtain the trained visual network model.

[0057] Furthermore, prior to the aforementioned feature data acquisition module 200, the following is also included:

[0058] The pre-training module is used to preprocess the training data to obtain a first data augmentation result and a second data augmentation result; wherein, the data preprocessing includes at least one of the following processing methods: random cropping, random Gaussian blurring, random flipping, and random grayscale processing.

[0059] Furthermore, the aforementioned visual network model includes a first structure network, and the aforementioned feature data acquisition module 200 is used for:

[0060] Input the first data augmentation result and the second data augmentation result into the visual network model;

[0061] The first spatial features corresponding to the image data are obtained by using the first structural network of the visual network model for feature extraction.

[0062] Furthermore, the aforementioned visual network model also includes a second structure network, and the aforementioned feature data acquisition module 200 is further used for:

[0063] The second spatial features are obtained by mapping the first spatial features into a first preset data space using a second structure network; and,

[0064] The second spatial features are mapped to the second preset data space using the second structure network to obtain the third spatial features.

[0065] Furthermore, the aforementioned feature data learning module 300 is also used for:

[0066] The first loss function values ​​of the first spatial features and the second spatial features are calculated using a self-supervised contrastive loss function; and,

[0067] The second loss function value of the cross-entropy loss function of the third spatial features is calculated using the label information.

[0068] The generalized supervised representation learning device according to embodiments of the present invention can effectively combine self-supervised learning and fully supervised learning to obtain a training model with better generalization performance, and can also achieve good transfer performance on other visual tasks, enabling image detection and segmentation.

[0069] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0070] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0071] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A generalized supervised representation learning method, characterized in that, Includes the following steps: Acquire training data; wherein the training data includes image data and label information corresponding to the image data; The training data is input into a visual network model for feature extraction to obtain a first spatial feature corresponding to the image data. The visual network model also includes a second structure network. The first spatial feature is mapped to a first preset data space using the second structure network to obtain a second spatial feature. The second spatial feature is also mapped to a second preset data space using the second structure network to obtain a third spatial feature. Calculate the first loss function values ​​of the first spatial feature and the second spatial feature, and calculate the second loss function value of the third spatial feature based on the label information; The parameters of the visual network model are updated based on the first loss function value and the second loss function value to obtain the trained visual network model.

2. The method according to claim 1, characterized in that, Before inputting the training data into the visual network model for feature extraction, the method further includes: The training data is preprocessed to obtain a first data augmentation result and a second data augmentation result; wherein the data preprocessing includes at least one of the following processing methods: random cropping, random Gaussian blurring, random flipping, and random grayscale processing.

3. The method according to claim 2, characterized in that, The visual network model includes a first structured network. The step of inputting training data into the visual network model for feature extraction to obtain the first spatial features corresponding to the image data includes: The first data augmentation result and the second data augmentation result are input into the visual network model; The first spatial features corresponding to the image data are obtained by using the first structural network of the visual network model to extract features.

4. The method according to claim 3, characterized in that, The step of calculating the first loss function values ​​of the first spatial features and the second spatial features, and calculating the second loss function value of the third spatial features based on the label information, includes: The first loss function values ​​of the first spatial feature and the second spatial feature are calculated using a self-supervised contrastive loss function; and, The second loss function value of the cross-entropy loss function of the third spatial feature is calculated using the label information.

5. A generalized supervised representation learning device, characterized in that, include: A training data acquisition module is used to acquire training data; wherein, the training data includes image data and label information corresponding to the image data; The feature data acquisition module is used to input the training data into a visual network model to extract features and obtain the first spatial features corresponding to the image data. The visual network model further includes a second structure network. The second structure network is used to map the first spatial features in a first preset data space to obtain the second spatial features. The second structure network is also used to map the second spatial features in a second preset data space to obtain the third spatial features. The feature data learning module is used to calculate the first loss function value of the first spatial feature and the second spatial feature, and to calculate the second loss function value of the third spatial feature based on the label information; The model parameter training module is used to update the parameters of the visual network model based on the first loss function value and the second loss function value to obtain the trained visual network model.

6. The apparatus according to claim 5, characterized in that, Before the feature data acquisition module, the following is also included: The pre-training module is used to perform data preprocessing on the training data to obtain a first data augmentation result and a second data augmentation result; wherein the data preprocessing includes at least one of the following processing methods: random cropping, random Gaussian blurring, random flipping, and random grayscale processing.

7. The apparatus according to claim 6, characterized in that, The visual network model includes a first structured network, and the feature data acquisition module is used for: The first data augmentation result and the second data augmentation result are input into the visual network model; The first spatial features corresponding to the image data are obtained by using the first structural network of the visual network model to extract features.

8. The apparatus according to claim 7, characterized in that, The feature data learning module is also used for: The first loss function values ​​of the first spatial feature and the second spatial feature are calculated using a self-supervised contrastive loss function; and, The second loss function value of the cross-entropy loss function of the third spatial feature is calculated using the label information.

Images