Open set recognition method based on representation learning and diffusion model

Abstract

The present invention discloses an open set recognition method based on representation learning and diffusion model, which mainly consists of two stages: model training and model detection: the model training stage mainly learns the representation learning network and the diffusion network, wherein the representation learning network is trained using supervised contrast loss and cross entropy loss, and then the compact representation learned by the feature extraction module is input into the diffusion network for training; in the model detection stage, the representation learning network is used to obtain feature representation and pseudo-label, and then the diffusion network is used to perform biased reconstruction on the test set data, and finally the known class and the unknown class are identified according to the reconstruction error. The proposed method uses representation learning and diffusion model for open set recognition. Compared with the existing open set recognition method based on reconstruction, the present invention essentially enhances the bias of the model to the known class from multiple levels, so that the model can simultaneously obtain a larger reconstruction error of the unknown class and a better reconstruction error of the known class, thereby obtaining a better open set recognition performance.

Description

Technical Field

[0001] The present invention relates to an open set recognition problem in the field of machine learning, and specifically to an open set recognition method based on representation learning and a diffusion model. Background Art

[0002] Open set recognition (OSR) proposes a more challenging recognition scenario compared to traditional (closed set) recognition tasks. It assumes that there is no prior knowledge of new classes during training (no training examples or related semantic information, etc.), but encounters new class examples that were not seen during training during testing. The learned classifier is required to not only accurately recognize known classes, but also effectively handle unknown new classes. At present, domain adaptation has been applied in many fields and has achieved success. For example, in medical diagnosis, due to objective conditions such as sampling bias, training examples may not cover all types of diseases. At the same time, due to factors such as pathological mechanisms, new variants also occur from time to time, making it very likely to encounter unknown new classes during testing. Therefore, researchers introduced open set recognition methods and achieved good results.

[0003] The main challenge of open set recognition is that the model has only seen known class samples during the training phase. When it encounters unknown class samples, it will inevitably identify the unknown class samples as known class samples, resulting in incorrect classification. This problem is called the class bias problem. Most existing open set recognition methods train the discriminant model to eliminate class bias by regularization, reserving placeholders, generating pseudo samples, etc., so that the model can distinguish between known and unknown classes. However, since unknown class samples are inaccessible during the training phase, these methods actually only estimate the distribution of unknown categories, which is not accurate and cannot fundamentally eliminate class bias. Therefore, another part of the methods choose to do it based on reconstruction. These methods assume that unknown class samples have a larger reconstruction error than known class samples, so that they can be identified based on the reconstruction error. However, due to the complexity and variability of data in actual scenarios, such an assumption cannot always hold true. In these methods, the reconstruction error is often regarded as a fixed attribute of the model. Once the model training is completed, the reconstruction error of the sample cannot be adjusted, which can easily cause misclassification. For this reason, an open set recognition method based on representation learning and diffusion model is proposed. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to address the deficiencies of the prior art and propose an open set recognition method based on representation learning and diffusion model, which uses class bias to enhance the reconstruction hypothesis during the reconstruction process and uses an adaptive method to adjust the reconstruction error at the sample level to solve the defects pointed out in the background technology. The method disclosed in the present invention can enhance the reconstruction error of unknown class samples and reduce the reconstruction error of known classes, thereby improving the accuracy of open set recognition.

[0005] Technical solution: Open set recognition method based on representation learning and diffusion model. The specific steps are as follows:

[0006] S1. Obtain a training set, input the images in the training set into the representation learning network, and obtain a trained representation learning network;

[0007] S2, obtain the diffusion network, and obtain the trained diffusion network through noise estimation loss training;

[0008] S3, reconstructing the model, connecting the trained representation learning network obtained in step S1 and the trained diffusion network obtained in step S2 in series to obtain a model for open set recognition, calculating the confidence score of each image through the model, and then performing biased reconstruction;

[0009] S4, open set recognition: According to the model for open set recognition obtained in step S3, the reconstruction errors of all inputs are calculated, and whether it is an unknown image is determined according to a threshold set by experience.

[0010] Preferably, in step S1, the images in the training set are normalized and preprocessed before being input into the representation learning network.

[0011] Preferably, in step S1, the representation learning network includes a feature extractor E, a projector P and a classifier C, wherein the feature extractor E is used to extract the feature from the input image. Extract images with feature representation i is the image index, N is the total number of images;

[0012] The projector P is used to extract the image with feature representation Extract the projection

[0013] The classifier C is used to classify images with feature representations Classify and obtain pseudo labels

[0014] Preferably, the feature extractor E and the projector P are trained by supervised contrast loss, expressed as follows:

[0015]

[0016]

[0017] in, represents the supervised contrast loss, N i represents the number of images with the same label as the i-th image, the conditional vector Indicates that the element that satisfies the condition cond in the brackets is 1, otherwise it is a vector of 0. i, j, k are all image indices, y i,y j Respectively represent the labels of the i-th and j-th images, τ is the temperature parameter, and sim(a,b) represents the similarity between variables a and b;

[0018] The classification head C is trained by the cross entropy loss, which is expressed as follows:

[0019]

[0020] Among them, c i represents the pseudo label of the i-th image, represents classification loss and CE() represents cross entropy loss.

[0021] Preferably, in step S2, the diffusion network is a U-Net network, and the diffusion network is trained as follows:

[0022]

[0023] where z t represents the image obtained by adding t steps of noise to the image z0 with feature representation but without noise, ∈ represents random Gaussian noise, ∈ θ represents the noise predicted by the diffusion network, c is the pseudo label, and t is the noise step size.

[0024] Preferably, in step S3, the confidence score of each image is calculated:

[0025] The confidence score of the i-th image is calculated as:

[0026] s i =δ(C(E(x i ))) T δ(C(E(x i ))) (5)

[0027] Where δ(·) represents the softmax function;

[0028] According to the confidence score, the guidance strength w of each image i and sampling step size T i The adaptive calculation is:

[0029]

[0030] Among them, the original step size T∈[100,200,...,1000], int() means rounding up to an integer;

[0031] The i-th image is T i The iterative calculation of the biased reconstruction is:

[0032]

[0033] in represents the noise scale, represents the image after adding t-1 steps of noise to the i-th image with feature representation but without noise. represents the noise predicted by the biased diffusion network and is calculated as:

[0034]

[0035] in, It represents the image after adding t steps of noise to the i-th image with feature representation but without noise.

[0036] Preferably, in step S4, the expression for calculating the reconstruction error of all inputs is:

[0037] p(know|z)=d(error(z,z rec ),η) (9)

[0038] Among them, p(know|z) is the probability that a given image z with the representation feature is a known image. If the reconstruction error is less than the threshold set according to experience, it is an unknown image, z rec It represents the biased reconstruction of the image. The step function d(x,η) means that the value is 1 when x≥η, otherwise it is 0. The error function error(x,y) represents the mean square error between variables x and y.

[0039] Beneficial effects: The open set recognition method based on representation learning and diffusion model of the present invention learns the compact feature representation of the input to enhance the model's bias towards known class samples and uses the supervision information obtained from representation learning to guide feature reconstruction. During the reconstruction process, the reconstruction error is adjusted at the sample level in an adaptive manner to detect unknown class samples. Compared with the existing open set recognition method based on reconstruction, this method can enhance the reconstruction error of unknown class samples and reduce the reconstruction error of known classes, thereby improving the accuracy of open set recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the open set recognition method based on representation learning and diffusion model;

[0041] Figure 2 This is a model structure diagram of the open set recognition method based on representation learning and diffusion model. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Example

[0044] See also Figure 1-2 The present invention provides a technical solution: an open set recognition method based on representation learning and diffusion model, comprising the following steps:

[0045] S1, representation learning network: obtained by supervised contrastive learning loss and cross entropy loss training, which can extract the feature representation of input samples and their pseudo labels;

[0046] S2, diffusion network: The diffusion network is trained by noise estimation loss, which can be reconstructed based on input and label guidance;

[0047] S3, reconstruction model: It is obtained by combining the trained representation learning network and the diffusion network. The sampling step size and the strength of the conditional guidance can be determined in an adaptive manner according to the pseudo-label, thereby achieving biased reconstruction of the input sample;

[0048] S4, open set recognition: The reconstruction error of all inputs can be calculated, and whether it is an unknown sample can be determined based on the threshold.

[0049] In this embodiment, specifically: in S1, the training set images are normalized and preprocessed before being input into S1 to represent the learning network.

[0050] In this embodiment, specifically: S1 indicates that the learning network is composed of a feature extractor E, a projector P and a classifier C;

[0051] The feature extractor E is used to extract the feature from the input image Extract feature representation

[0052] The projector P is used to represent the Extract the projection

[0053] The classifier C is used to represent the feature Classify and obtain pseudo labels

[0054] Preferably: the feature extractor E and the projector P are trained by supervised contrast loss as follows:

[0055]

[0056]

[0057] Among them, N i represents the number of samples with the same label as the i-th sample, and the conditional vector Indicates that the elements that meet the condition are 1 and the rest are 0. i represents the label of the i-th sample, τ is the temperature parameter;

[0058] Replacing the projector P with the classifier C and fixing the parameters of the feature extractor E, the network is retrained with the cross entropy loss as follows:

[0059]

[0060] in, represents the pseudo label of the i-th sample.

[0061] In this embodiment, specifically: the S2 diffusion network training is as follows:

[0062]

[0063] where z t It represents the image obtained by adding t steps of noise to the original image z0, ∈ and ∈ θ Represent the added noise and the noise predicted by the network respectively.

[0064] In this embodiment, specifically: the S3 reconstruction model obtains the feature representation and pseudo-label of the test set based on S1 and S2, so that the confidence score of each sample can be calculated to guide and supervise the biased reconstruction.

[0065] In this embodiment, specifically: the confidence score of the i-th sample is calculated as:

[0066] s i =δ(C(E(x i ))) T δ(C(E(x i ))) (5)

[0067] Where δ(·) represents the softmax function;

[0068] According to the confidence score, the guidance strength w of each sample i and sampling step size T i The adaptive calculation can be:

[0069]

[0070] Among them, the original step size T∈[100,200,...,1000].

[0071] In this embodiment, specifically: the i-th sample undergoes T i The iterative calculation of the biased reconstruction is:

[0072]

[0073] in Representing noise scale, biased noise prediction network Calculated as:

[0074]

[0075] In this embodiment, specifically: the S4 open set identification is calculated by the reconstruction error of the sample:

[0076] p(know|z)=d(error(z,z rec ),η) (9)

[0077] The step function d(x,η) indicates that the value is 1 when x≥η, otherwise it is 0, and the error function error(x,y) indicates the mean square error between x and y. By comparing the reconstruction error and the threshold, known and unknown class samples can be distinguished, and known class samples can be further classified using the trained classifier C.

[0078] Working principle or structural principle, when used, the present invention proposes an open set recognition method of representation learning and diffusion model, using representation learning to obtain a stronger representation biased towards known classes and a pseudo label of the sample, then training the diffusion network and using an adaptive method to obtain parameters such as reconstruction step size and guidance strength for the label to supervise the sample for reconstruction. When the sample is reconstructed, its reconstruction error is calculated and the threshold is used to determine whether the sample is a known class sample.

[0079] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An open set recognition method based on representation learning and diffusion model, characterized in that: The following steps are involved: S1. Obtain a training set, input the images in the training set into the representation learning network, and obtain a trained representation learning network; S2, obtain the diffusion network, and obtain the trained diffusion network through noise estimation loss training; S3, reconstructing the model, connecting the trained representation learning network obtained in step S1 and the trained diffusion network obtained in step S2 in series to obtain a model for open set recognition, calculating the confidence score of each image through the model, and then performing biased reconstruction; calculating the confidence score of each image: The confidence score of the i-th image is calculated as: s i =δ(C(E(x i ))) T δ(C(E(x i ))) (5) Where δ(·) represents the softmax function; According to the confidence score, the guidance strength w of each image i and sampling step size T i The adaptive calculation is: Among them, the original step size T∈[100,200,...,1000], int() means rounding up to an integer; The i-th image is T i The iterative calculation of the biased reconstruction is: in β i ∈(0,1) represents the noise scale, represents the image after adding t-1 steps of noise to the i-th image with feature representation but without noise. represents the noise predicted by the biased diffusion network and is calculated as: in, represents the image after adding t steps of noise to the i-th image with feature representation but without noise; S4, open set recognition: According to the model for open set recognition obtained in step S3, the reconstruction errors of all inputs are calculated, and whether it is an unknown image is determined according to a threshold set by experience.

2. The open set recognition method based on representation learning and diffusion model according to claim 1, characterized in that: In step S1, the images in the training set are normalized and preprocessed before being input into the representation learning network.

3. The open set identification method based on representation learning and diffusion model according to claim 1, characterized in that: In step S1, the representation learning network includes a feature extractor E, a projector P and a classifier C. The feature extractor E is used to extract the feature from the input image. Extract images with feature representation i is the image index, N is the total number of images; The projector P is used to extract the image with feature representation Extract the projection The classifier C is used to classify images with feature representations Classify and obtain pseudo labels 4. The open set recognition method based on representation learning and diffusion model according to claim 3, characterized in that: The feature extractor E and projector P are trained by supervised contrast loss, expressed as follows: in, represents the supervised contrast loss, N i represents the number of images with the same label as the i-th image, the conditional vector Indicates that the element that satisfies the condition cond in the brackets is 1, otherwise it is a vector of 0. i, j, k are all image indices, y i ,y j Respectively represent the labels of the i-th and j-th images, τ is the temperature parameter, and sim(a,b) represents the similarity between variables a and b; The classifier C is trained by the cross entropy loss, which is expressed as follows: Among them, c i represents the pseudo label of the i-th image, represents classification loss and CE() represents cross entropy loss.

5. The open set identification method based on representation learning and diffusion model according to claim 1, characterized in that: In step S2, the diffusion network is a U-Net network, and the training of the diffusion network is as follows: where z t represents the image obtained by adding t steps of noise to the image z0 with feature representation but without noise, ∈ represents random Gaussian noise, ∈ θ represents the noise predicted by the diffusion network, c is the pseudo label, and t is the noise step size.

6. The open set recognition method based on representation learning and diffusion model according to claim 1, characterized in that: In step S4, the expression for calculating the reconstruction error of all inputs is: p( know ∣z)=d(error(z,z rec ),η) (9) Among them, p(know|z) is the probability that a given image z with the representation feature is a known image. If the reconstruction error is less than the threshold set according to experience, it is an unknown image, z rec It represents the biased reconstruction of the image. The step function d(x,η) means that the value is 1 when x≥η, otherwise it is 0. The error function error(x,y) represents the mean square error between variables x and y.

Images