Image Feature Extraction Method and Device Based on Quadratic Inference and Adversarial Learning

Abstract

The application discloses an image feature extraction method and device based on secondary reasoning and adversarial learning, and relates to the technical field of image representation. The method comprises the following steps: preprocessing an original image dataset to construct a benchmark image dataset; cutting benchmark image data samples into a first picture set and a second picture set; reconstructing the second picture set according to the first picture set and the second picture set to generate a second reconstructed picture set; reconstructing the first picture set according to the second reconstructed picture set and the first picture set to generate a first reconstructed picture set; judging the sources of the second reconstructed picture set and the first picture set by a discriminator; iteratively training an encoder and the discriminator according to an adversarial neural network model to optimize the encoder and the discriminator; and inputting a target image into the encoder and the discriminator to extract features of the target image. The application realizes accurate extraction of image features by means of a secondary reconstruction mechanism and an adversarial neural network model which reconstructs blocks of uniform semantic information and improves the generalization of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image representation, in particular to an image feature extraction method and device based on secondary reasoning and adversarial learning. BACKGROUND

[0002] With the development of computer vision technology, feature extraction of images is the basis for various dense computing downstream tasks such as image detection and segmentation, and the accuracy of image feature extraction directly affects the accuracy of subsequent downstream tasks such as image detection and image segmentation.

[0003] At present, the method for extracting features of images is to reduce rich visual information to one of pre-defined object categories through a deep convolutional neural network, thereby realizing the extraction of image features. However, in deep learning, it is necessary to enable the deep learning model to automatically learn feature representation ability from massive data.

[0004] The existing image representation learning method is mainly based on a mask pre-training method, which infers the mask part through part of the content, so that the high-level semantic features learned by the model have reasoning ability and generalization.

[0005] However, the existing mask pre-training method has the following two limitations:

[0006] (1) The reasoning ability of the reconstructed mask block is insufficient, that is, the training target of the model is to reconstruct the mask block at the pixel level, which will cause the model to mainly learn the pixels of the mask block, and ignore the reasoning ability of the mask block;

[0007] (2) It is difficult to reconstruct blocks with uniform semantic information (such as similar RGB values), that is, the loss function of the model is to calculate the pixel difference between the reconstructed block and the target block, which causes blocks with similar RGB values to be predicted as the same RGB value, making it difficult to learn the texture information of blocks with uniform semantic information, and affecting the generalization of the model. SUMMARY

[0008] The technical problem to be solved by the present application is to provide an image feature extraction method and device based on secondary reasoning and adversarial learning, which can reconstruct blocks with uniform semantic information and improve the generalization of the model through a two-level reconstruction mechanism and an adversarial neural network model, and realize accurate extraction of image features.

[0009] To solve the above technical problems, the application provides an image feature extraction method based on secondary reasoning and adversarial learning, comprising: preprocessing an original image dataset to construct a benchmark image dataset, wherein the benchmark image dataset comprises at least one benchmark image data sample; cutting the benchmark image data sample into a first picture set and a second picture set respectively; taking the first picture set as the input of an encoder, taking the second picture set as the mask block of the first picture set, reconstructing the second picture set through a decoder to generate a second reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the second picture set; taking the second reconstructed picture set as the input of the encoder, taking the first picture set as the mask block of the second reconstructed picture set, reconstructing the first picture set through the decoder to generate a first reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the first picture set; taking the second reconstructed picture set and the first picture set as the input of a discriminator to judge the source of the second reconstructed picture set and the first picture set; iteratively training the encoder and the discriminator according to the adversarial neural network model, the pixel mean square error loss of the second picture set, the pixel mean square error loss of the first picture set, the source of the second reconstructed picture set and the first picture set, so as to optimize the encoder and the discriminator; and inputting a target image into the encoder and the discriminator to extract the features of the target image.

[0010] As an improvement of the above scheme, the step of taking the first picture set as the input of an encoder, taking the second picture set as the mask block of the first picture set, reconstructing the second picture set through a decoder to generate a second reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the second picture set comprises: converting each picture block in the first picture set into a first initial matrix, and expanding the feature dimension of the first initial matrix; splicing all expanded first initial matrices corresponding to the first picture set with a preset feature vector to generate a first benchmark matrix; combining the first benchmark matrix with position information to generate a first feature matrix; inputting the first feature matrix into the encoder to output a first high-level semantic feature; inputting the first high-level semantic feature and the second picture set into the decoder to reconstruct the second picture set to generate a second reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the second picture set.

[0011] As an improvement to the above scheme, the steps of using the first image set as input to the encoder, using the second image set as a mask block of the first image set, reconstructing the second image set through the decoder to generate a second reconstructed image set, and calculating the pixel mean square error loss of the reconstructed second image set include: converting each image block in the first image set into a two-dimensional first initial matrix of RGB three channels, and expanding the two-dimensional first initial matrix into a 1024-dimensional first initial matrix through a first linear layer; concatenating all the 1024-dimensional first initial matrices corresponding to the first image set with a preset feature vector to generate a first reference matrix; combining the first reference matrix with position information to generate a first feature matrix; inputting the first feature matrix into the encoder to output a first high-level semantic feature; restoring the first high-level semantic feature into a second reconstructed image set of RGB three channels through a second linear layer; and calculating the pixel mean square error loss of the reconstructed second image set based on the second reconstructed image set and the second image set.

[0012] As an improvement to the above scheme, the encoder includes an N-layer encoder module, which includes a multi-head attention layer, an Add&Norm layer, and a FeedForward layer. The Add&Norm layer includes an Add layer and a Layer Normalization layer. Add is a residual connection, and the Layer Normalization layer is used to normalize the hidden layers in the neural network to a standard normal distribution. The step of inputting the first feature matrix into the encoder to output the first high-level semantic feature includes: S1, inputting e l As input to the (l+1)th layer Encoder module, the first initial feature n is extracted through a multi-head attention layer and an Add&Norm layer. l ′, where l is a positive integer less than N, l is initially 0 and e0 is the first characteristic matrix, n′ l =LayerNorm(e l +MultiHeadAttention(e l MultiHeadAttention(·) is a multi-head attention layer, LayerNorm(·) is a layer normalization layer, and e l +MultiHeadAttention(e l S1 is the Add layer; S2, the first initial feature n′ is... l The input is fed into the Feed Forward layer and the Add & Norm layer to extract the first target feature e. l+1 , where e l+1 =LayerNorm(n′) l+FeedForward(n′ l FeedForward() consists of two fully connected layers; S3, update the current value of l to make l = l + 1, and return to step S1 until step S1 is run N times, then output the current first target feature as the first high-level semantic feature.

[0013] As an improvement to the above scheme, the steps of using the second reconstructed image set as input to the encoder, using the first image set as a mask block of the second reconstructed image set, generating a first reconstructed image set through the first image set, and calculating the pixel mean square error loss of reconstructing the first image set include: converting each image block in the second reconstructed image set into a two-dimensional second initial matrix with RGB three channels, and expanding the two-dimensional second initial matrix into a 1024-dimensional second initial matrix through a first linear layer; concatenating all the 1024-dimensional second initial matrices corresponding to the second reconstructed image set with a preset feature vector to generate a second reference matrix; combining the second reference matrix with position information to generate a second feature matrix; inputting the second feature matrix into the encoder to output a second high-level semantic feature; restoring the second high-level semantic feature into a first reconstructed image set with RGB three channels through a second linear layer; and calculating the pixel mean square error loss of reconstructing the first image set based on the first reconstructed image set and the first image set.

[0014] As an improvement to the above scheme, the encoder includes an N-layer encoder module, which includes a multi-head attention layer, an Add&Norm layer, and a FeedForward layer. The Add&Norm layer includes an Add layer and a Layer Normalization layer. Add is a residual connection, and the Layer Normalization layer is used to normalize the hidden layers in the neural network to a standard normal distribution. The step of inputting the second feature matrix into the encoder to output the second high-level semantic feature includes: S1, inputting e′ l As input to the (l+1)th layer Encoder module, the second initial feature n″ is extracted through a multi-head attention layer and an Add&Norm layer. l Where l is a positive integer less than N, l is initially 0 and e′0 is the second characteristic matrix, n″ l =LayerNorm(e′) l +MultiHeadAttention(e′ l MultiHeadAttention(·) is a multi-head attention layer, LayerNorm(·) is a layer normalization layer, and e l+MultiHeadAttention(e′ l S1 is the Add layer; S2, the second initial feature n″ is added. l The input is fed into the Feed Forward layer and the Add & Norm layer to extract the second target feature e′. l+1 , where e′ l+1 =LayerNorm(n″) l +FeedForward(n″ l FeedForward() consists of two fully connected layers; S3, update the current value of l to make l = l + 1, and return to step S1 until step S1 is run N times, then output the current second target feature as the second high-level semantic feature.

[0015] As an improvement to the above scheme, the objective function of the adversarial neural network model is:

[0016]

[0017] Where loss1(x2,M(x1)) is the pixel mean square error loss for reconstructing the second image set, loss2(x1,M(M(x1))) is the pixel mean square error loss for reconstructing the first image set, D(x1) is the probability that the discriminator will identify the first image set as the original image patch, and D(M(x1)) is the probability that the discriminator will identify the generated second reconstructed image set as the original image patch.

[0018] As an improvement to the above scheme, the step of preprocessing the original image dataset to construct a benchmark image dataset includes: performing rotation, cropping, and coloring operations on each original image data sample in the original image dataset to generate auxiliary image data samples; converting both the original image data samples and the auxiliary image data samples into benchmark image data samples of a preset size; and combining the benchmark image data samples into a benchmark image dataset.

[0019] As an improvement to the above scheme, the steps of cutting the reference image data samples into image sets and dividing the image sets into a first image set and a second image set respectively include: cutting each reference image data sample in the reference image dataset into image blocks of a preset size and a preset number; combining the image blocks corresponding to each reference image data sample into an independent image set; and randomly dividing the image sets into a first image set and a second image set according to a preset ratio.

[0020] Accordingly, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the steps of the above-described image feature extraction method based on quadratic reasoning and adversarial learning.

[0021] Implementing this invention has the following beneficial effects:

[0022] This invention increases the sample size of the original image dataset by preprocessing the original image data samples. Next, the original image data samples are uniformly and non-overlappingly segmented into image blocks, which are then randomly divided into a first image set and a second image set according to a certain ratio. Then, the first image set is used as the input to the encoder, and the second image set is used as the mask block of the first image set to reconstruct the corresponding mask block. Next, the reconstructed second image set is used as the input to the encoder, and the first image set is used as the mask block of the second reconstructed image set to reconstruct the corresponding mask block. Then, the second reconstructed image set and the corresponding mask block are used together as the input to the discriminator to determine the source of the image block. Finally, the encoder and discriminator are iteratively trained using GAN network principles to simultaneously optimize the model parameters of both the encoder and discriminator.

[0023] Therefore, based on image pre-training, this invention endows the reconstructed blocks with reasoning ability through a secondary reconstruction mechanism, introduces adversarial neural networks to reconstruct blocks with uniform semantic information and improve the generalization of the model, and provides a theoretical basis from the perspective of generative adversarial networks. Attached Figure Description

[0024] Figure 1 This is a flowchart of an embodiment of the image feature extraction method based on secondary reasoning and adversarial learning of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0026] See Figure 1 , Figure 1 The flowchart of an embodiment of the image feature extraction method based on quadratic inference and adversarial learning of the present invention is shown, which includes:

[0027] S101, Preprocess the original image dataset to construct a benchmark image dataset;

[0028] It should be noted that the sample size in the original image dataset can be increased by performing preprocessing operations on the original image data samples in the original image dataset. Specifically, step S101 includes:

[0029] (1) Rotate, crop and colorize each original image data sample in the original image dataset to generate auxiliary image data samples;

[0030] (2) Convert the original image data samples and auxiliary image data samples into reference image data samples of a preset size respectively;

[0031] (3) Combine the baseline image data samples into a baseline image dataset.

[0032] In other words, the benchmark image dataset includes at least one benchmark image data sample.

[0033] For example, each original image data sample in the original image dataset ImageNet-1K can be preprocessed by rotating, cropping, coloring, etc. to obtain the sample size of the original image dataset ImageNet-1K. Then, it can be uniformly converted into a reference image data sample with a size of 224×224 and named ImageNet-1K-1 reference image dataset.

[0034] S102, the baseline image data samples are cut into a first image set and a second image set respectively;

[0035] Specifically, step S102 includes:

[0036] (1) Cut each benchmark image data sample in the benchmark image dataset into image blocks of a preset size and a preset number;

[0037] (2) Combine the image blocks corresponding to each benchmark image data sample into an independent image set;

[0038] (3) The image set is randomly divided into a first image set and a second image set according to a preset ratio.

[0039] Preferably, the reference image data sample can be cut into image blocks using a uniform, non-overlapping cutting method, and then randomly divided into two parts (i.e., the first image set and the second image set) according to a certain ratio.

[0040] For example, each benchmark image data sample X in the ImageNet-1K-1 benchmark image dataset is cut into image blocks of the same size and number using a uniform, non-overlapping cutting method, and then randomly divided into a first image set X1 = {x1,…,x...} at a ratio of 50%. i ,…,x I} and the second image set X2={x2,…,x j ,…,x J From this, we can know that X = X1∪X2.

[0041] S103, the first image set is used as the input of the encoder, the second image set is used as the mask block of the first image set, the second image set is reconstructed by the decoder to generate the second reconstructed image set, and the pixel mean square error loss of the reconstructed second image set is calculated.

[0042] Specifically, step S103 includes:

[0043] (1) Convert each image block in the first image set into a first initial matrix and expand the feature dimension of the first initial matrix;

[0044] Preferably, each image block in the first image set is converted into a two-dimensional first initial matrix with RGB three channels, and the two-dimensional first initial matrix is ​​expanded into a 1024-dimensional first initial matrix through a first linear layer; for example, the image block x in the first image set... i ∈X1 expands the RGB three channels into 1024 dimensions through the first linear layer, i.e., x′ i =x i E;

[0045] (2) Concatenate all the expanded first initial matrices corresponding to the first image set with the preset feature vectors to generate the first reference matrix;

[0046] For example, initialize an additional feature vector x cla and the expanded first initial matrix x′ i Concatenate them into a first reference matrix, i.e., [x cla ,x′1,…,x′ I ].

[0047] (3) Combine the first reference matrix with the position information to generate the first feature matrix;

[0048] (4) Input the first feature matrix into the encoder to output the first high-level semantic feature;

[0049] For example, the feature vector x cla And the first initial matrix x′ after expansion of all the first image sets i Add the corresponding location information E to all pos Together they serve as the input to the encoder, i.e., the first feature matrix e0 = [x cla ,x′1,…,x′ I ]+E pos .

[0050] It should be noted that the encoder includes N layers of encoder modules; preferably, N=24, and the first feature matrix e0 is passed through the forward propagation of the 24 layers of encoder modules to obtain the first high-level semantic feature e. 23 .

[0051] Furthermore, the Encoder module includes a multi-head attention layer, an Add&Norm layer, and a FeedForward layer. The Add&Norm layer includes an Add layer and a Layer Normalization layer. The Add layer represents a residual connection, and the Layer Normalization layer is used to normalize the hidden layers in the neural network to a standard normal distribution. Furthermore, the step of inputting the first feature matrix into the encoder to output the first high-level semantic feature includes:

[0052] 4.1, e l As input to the (l+1)th layer Encoder module, the first initial feature n′ is extracted through a multi-head attention layer and an Add&Norm layer. l :

[0053] n′ l =LayerNorm(e l +MultiHeadAttention(e l ))

[0054] Where l is a positive integer less than N, and l is initially 0; the initial input of the Encoder module is e0 (i.e., the input of the first layer Encoder module is e0) and e0 is the first feature matrix; LayerNorm(·) is the Layer Normalization layer; e l +MultiHeadAttention(e l ) is the Add layer; MultiHeadAttention(·) is the multi-head attention layer, wherein MultiHeadAttention(e) is the Add layer. l It includes multiple self-attention (e) functions. l The Self-Attention(e) l )for:

[0055] Q l =e l *W l,Q ;

[0056] K l =e l *W l,K ;

[0057] V l =e l *W l，V ;

[0058]

[0059] Among them, Wl,Q W l,K W l,V These are the three trainable feature matrices in the l-th Encoder module, used to train feature e. l Perform a linear mapping operation, where d is a matrix Q. l and K l The number of columns; the MultiHeadAttention (e l ) will include all self-attention(e l The output features of each Self-Attention (e) are concatenated together. l The linear mapping matrix W corresponding to ) l,Q W l,K W l,V The difference indicates that the model judges the degree of correlation between input image features from different perspectives.

[0060] 4.2, the first initial feature n′ l The input is fed into the Feed Forward layer and the Add & Norm layer to extract the first target feature e. l+1 :

[0061] e l+1 =LayerNorm(n′) l +FeedForward(n′ l ))

[0062] FeedForward(·) consists of two fully connected layers. The first layer uses ReLU as its activation function, while the second layer does not use an activation function, i.e., max(0,n′). l W e,l1 +b e,l1 W e,l2 +b e,l2 ;

[0063] 4.3 Update the current value of l so that l = l + 1, and return to step 4.1. Repeat step 4.1 N times until the current first target feature is output as the first high-level semantic feature.

[0064] (5) Input the first high-level semantic features and the second image set into the decoder to reconstruct the second image set to generate the second reconstructed image set, and calculate the pixel mean square error loss of the reconstructed second image set.

[0065] It should be noted that the second image set is the mask block of the first image set and the training target of the encoder.

[0066] For example, when N=24, the first high-level semantic feature e is obtained. 23The first high-level semantic features of 1024 dimensions are restored to the second reconstructed image set M(X1) with RGB three channels through the second linear layer. The mean square error loss of the second reconstructed image set M(X1) and the image blocks of the second image set X2 is calculated by applying mean square error loss to the image blocks of the second image set M(X1) and M(X1) respectively. The pixel mean square error loss Loss1 = MSE(X2,M(X1)) can be calculated.

[0067] S104, the second reconstructed image set is used as the input of the encoder, the first image set is used as the mask block of the second reconstructed image set, the first image set is reconstructed by the decoder to generate the first reconstructed image set, and the pixel mean square error loss of the reconstructed first image set is calculated.

[0068] Specifically, step S104 includes:

[0069] (1) Convert each image block in the second reconstructed image set into a two-dimensional second initial matrix with RGB three channels, and expand the two-dimensional second initial matrix into a 1024-dimensional second initial matrix through the first linear layer;

[0070] For example, each image block in the second reconstructed image set This is transformed into a two-dimensional second initial matrix, and the RGB three channels are expanded to 1024 dimensions through the first linear layer. This allows for an effective expansion of the feature dimensions.

[0071] (2) The second initial matrix of all 1024 dimensions corresponding to the second reconstructed image set is concatenated with the preset feature vector to generate the second reference matrix;

[0072] For example, initialize an additional feature vector x cla And all 1024-dimensional second initial matrices Concatenate them to form the second reference matrix, that is

[0073] (3) Combine the second reference matrix with the position information to generate the second feature matrix;

[0074] (4) Input the second feature matrix into the encoder to output the second high-level semantic features;

[0075] For example, the feature vector x cla and all second initial matrices Add the corresponding location information E to all pos Together they serve as the input to the encoder, namely the second feature matrix.

[0076] Preferably, N=24, and the second feature matrix e′0 is passed through a 24-layer encoder module to obtain the first high-level semantic feature e′. 23 .

[0077] Furthermore, the step of inputting the second feature matrix into the encoder to output the second high-level semantic feature includes:

[0078] 4.1, e′ l As input to the (l+1)th layer Encoder module, the second initial feature n″ is extracted through a multi-head attention layer and an Add&Norm layer. l :

[0079] n″ l =LayerNorm(e′) l +MultiHeadAttention(e′ l ))

[0080] Where l is a positive integer less than N, and l is initially 0; the initial input of the Encoder module is e′0 (i.e., the input of the first layer Encoder module is e′0), and e′0 is the second feature matrix; LayerNorm(·) is the Layer Normalization layer, e′ l +MultiHeadAttention(e′ l ) is the Add layer; MultiHeadAttention(·) is the multi-head attention layer, wherein MultiHeadAttention(e′) is the multi-head attention layer. l It contains multiple self-attention (e′) l The Self-Attention(e′) l )for:

[0081] Q′ l =e′ l *W l,Q ;

[0082] K′ l =e′ l *W l,K ;

[0083] V′ l =e′ l *W l，V ;

[0084]

[0085] Among them, W l,Q W l,K W l,V These are the three trainable feature matrices in the l-th Encoder module, used to train feature e′. l Perform a linear mapping operation, where d is the matrix Q′.l and K′ l The number of columns; the MultiHeadAttention(e′) layer l ) will include all self-attention(e′) l The output features of each Self-Attention (e′) are concatenated together, where each Self-Attention (e′) is a concatenation of the output features of the Self-Attention (e′). l The linear mapping matrix W corresponding to ) l,Q W l,K W l,V The difference indicates that the model judges the degree of correlation between input image features from different perspectives.

[0086] 4.2, the second initial feature n″ l The input is fed into the FeedForward layer and the Add&Norm layer to extract the second target feature e′. l+1 :

[0087] e′ l+1 =LayerNorm(n″) l +FeedForward(n″ l ))

[0088] FeedForward(·) consists of two fully connected layers. The first layer uses ReLU as its activation function, while the second layer does not use an activation function, i.e., max(0,n″). l W e,l1 +b e,l1 W e,l2 +b e,l2 ;

[0089] 4.3 Update the current value of l so that l = l + 1, and return to step 4.1. Repeat step 4.1 N times until the current second target feature is output as the second high-level semantic feature.

[0090] (5) The second high-level semantic features are restored into the first reconstructed image set with RGB three channels through the second linear layer;

[0091] (6) Calculate the pixel mean square error loss for reconstructing the first image set based on the first reconstructed image set and the first image set.

[0092] It should be noted that the first image set is the mask block for the second reconstructed image set and the training target for the encoder.

[0093] For example, when N=24, the second high-level semantic feature e′ is obtained. 23The 1024-dimensional second high-level semantic features are restored to the first reconstructed image set M(M(X1)) of RGB three channels through the second linear layer. The mean square error loss of the first reconstructed image set M(M(X1)) and the first image set X1 is calculated by performing mean square error loss on the first image set. Loss2 = MSE(X1,M(M(X1))).

[0094] S105, using the second reconstructed image set and the first image set as input to the discriminator, determine the source of the second reconstructed image set and the first image set;

[0095] For example, the second reconstructed image set M(X1) and the corresponding mask block X1 (i.e. the image block of the first image set) are used together as the input of the discriminator, and the Softmax function is used to predict whether each image block is generated or comes from the original image block;

[0096] Furthermore, the discriminator consists of four encoder modules, and its accuracy is calculated using the cross-entropy loss function.

[0097] S106, The encoder and discriminator are iteratively trained based on the adversarial neural network model (i.e., CAN network model), the pixel mean square error loss of the second image set, the pixel mean square error loss of the first image set, and the sources of the second reconstructed image set and the first image set, in order to optimize the encoder and discriminator.

[0098] It should be noted that by iteratively training the encoder and discriminator using an adversarial neural network model (i.e., repeating steps S103-105), the model parameters of the encoder and discriminator can be optimized simultaneously. Specifically, the adversarial neural network model first trains the discriminator, then trains the encoder, and iterates in this order until the required number of training iterations is reached.

[0099] The objective function of the adversarial neural network model is:

[0100]

[0101] in,

[0102] loss1(x2,M(x1)) is the pixel mean square error loss for reconstructing the second image set;

[0103] loss2(x1,M(M(x1))) is the pixel mean square error loss for reconstructing the first image set;

[0104] D(x1) is the probability that the discriminator identifies the first image set as an original image patch;

[0105] D(M(x1)) is the probability that the discriminator will identify the generated second reconstructed image set as the original image patch.

[0106] This invention uses the encoder as a generator for an adversarial neural network model, with the aim of generating masked image patches, while the discriminator is used to determine whether the image patches are generated or not.

[0107] Accordingly, after completing the pre-training of the adversarial neural network model, the parameters of the encoder and subsequent linear layers of the adversarial neural network model can be retained. The encoder and linear layers can be fine-tuned by using linear probe or fine probe on the input dataset of the downstream task to fit the target of the downstream task.

[0108] S107, The target image is input into the encoder and discriminator to extract the features of the target image.

[0109] Therefore, after iteratively training the encoder and discriminator, inputting the target image into the trained encoder and discriminator can accurately extract the features of the target image with high precision.

[0110] Accordingly, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described image feature extraction method based on quadratic reasoning and adversarial learning.

[0111] In summary, this invention increases the sample size of the original image dataset by preprocessing the original image data samples. Secondly, the original image data samples are uniformly and non-overlappingly segmented into image blocks, and then randomly divided into a first image set and a second image set according to a certain ratio. Next, the first image set is used as the input to the encoder, and the second image set is used as the mask block of the first image set to reconstruct the corresponding mask block. Then, the reconstructed second image set is used as the input to the encoder, and the first image set is used as the mask block of the second reconstructed image set to reconstruct the corresponding mask block. Finally, the second reconstructed image set and the corresponding mask block are used together as the input to the discriminator to determine the source of the image block. Finally, the encoder and discriminator are iteratively trained using GAN network principles to simultaneously optimize the model parameters of both the encoder and discriminator. Therefore, based on image pre-training, this invention endows the reconstructed blocks with reasoning ability through a secondary reconstruction mechanism, introduces adversarial neural networks to reconstruct blocks with uniform semantic information and improve the generalization of the model, and provides a theoretical basis from the perspective of generative adversarial networks.

[0112] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. An image feature extraction method based on double reasoning and adversarial learning, characterized in that, The method comprises the following steps: preprocessing an original image dataset to construct a benchmark image dataset, wherein the benchmark image dataset comprises at least one benchmark image data sample; cutting the benchmark image data sample into a first picture set and a second picture set respectively; taking the first picture set as the input of an encoder, taking the second picture set as the mask block of the first picture set, reconstructing the second picture set by a decoder to generate a second reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the second picture set; specifically the steps comprise: converting each picture block in the first picture set into a first initial matrix, and expanding the feature dimension of the first initial matrix; splicing all the expanded first initial matrices corresponding to the first picture set with a preset feature vector to generate a first benchmark matrix; combining the first benchmark matrix with position information to generate a first feature matrix; inputting the first feature matrix into the encoder to output a first high-level semantic feature; inputting the first high-level semantic feature and the second picture set into a decoder to reconstruct the second picture set to generate a second reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the second picture set; taking the second reconstructed picture set as the input of an encoder, taking the first picture set as the mask block of the second reconstructed picture set, reconstructing the first picture set by the decoder to generate a first reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the first picture set; taking the second reconstructed picture set and the first picture set as the input of a discriminator to judge the source of the second reconstructed picture set and the first picture set; iteratively training the encoder and the discriminator according to the adversarial neural network model, the pixel mean square error loss of the second picture set, the pixel mean square error loss of the first picture set, the source of the second reconstructed picture set and the first picture set, so as to optimize the encoder and the discriminator; inputting a target image into the encoder and the discriminator to extract the features of the target image. 2.The image feature extraction method based on double reasoning and adversarial learning according to claim 1, wherein, The step of taking the first picture set as the input of an encoder, taking the second picture set as the mask block of the first picture set, reconstructing the second picture set by a decoder to generate a second reconstructed picture set, and calculating the pixel mean square error loss of reconstructing the second picture set comprises: converting each picture block in the first picture set into a two-dimensional first initial matrix of RGB three channels, and expanding the two-dimensional first initial matrix into a 1024-dimensional first initial matrix through a first linear layer; splicing all the 1024-dimensional first initial matrices corresponding to the first picture set with a preset feature vector to generate a first benchmark matrix; combining the first benchmark matrix with position information to generate a first feature matrix; inputting the first feature matrix into the encoder to output a first high-level semantic feature; restoring the first high-level semantic feature into a second reconstructed picture set of RGB three channels through a second linear layer; calculating the pixel mean square error loss of reconstructing the second picture set according to the second reconstructed picture set and the second picture set. 3.The image feature extraction method based on double reasoning and adversarial learning according to claim 1 or 2, characterized in that, The encoder comprises N-layer Encoder modules, the Encoder modules comprise a multi-head attention layer, an Add&Norm layer and a Feed Forward layer, the Add&Norm layer comprises an Add layer and a Layer Normalization layer, the Add is a residual connection, and the Layer Normalization layer is used for normalizing a hidden layer in a neural network to a standard normal distribution, and the step of inputting the first feature matrix into the encoder to output a first high-level semantic feature comprises: S1, to As the first l The input of the +1 layer Encoder module, and the first initial feature is extracted through the multi-head attention layer and the Add&Norm layer Wherein, l N is a positive integer less than N, l The initial value is 0 and The first feature matrix, , The multi-head attention layer, The Layer Normalization layer, The Add layer; S2, the first initial features are input to the Feed Forward layer and the Add & Norm layer to extract the first target features wherein, , are two fully connected layers; S3, Update l The current value makes l = l +1, and return to step S1. Repeat step S1 N times, then output the current first target feature as the first high-level semantic feature. 4.The image feature extraction method based on double reasoning and adversarial learning according to claim 1, wherein, The step of inputting the second reconstructed picture set as an input of the encoder, inputting the first picture set as a mask block of the second reconstructed picture set, generating a first reconstructed picture set through the first picture set, and calculating a pixel mean square error loss of reconstructing the first picture set comprises: Converting each picture block in the second reconstructed picture set into a two-dimensional second initial matrix of RGB three channels, and expanding the two-dimensional second initial matrix into a 1024-dimensional second initial matrix through a first linear layer; Splicing all 1024-dimensional second initial matrices corresponding to the second reconstructed picture set with a preset feature vector to generate a second reference matrix; Combining the second reference matrix with position information to generate a second feature matrix; Inputting the second feature matrix into the encoder to output a second high-level semantic feature; Restoring the second high-level semantic feature into a first reconstructed picture set of RGB three channels through a second linear layer; According to the first reconstructed picture set and the first picture set, a pixel mean square error loss of reconstructing the first picture set is calculated. 5.The image feature extraction method based on double reasoning and adversarial learning according to claim 4, wherein, The encoder comprises N-layer Encoder modules, the Encoder modules comprise a multi-head attention layer, an Add&Norm layer and a Feed Forward layer, the Add&Norm layer comprises an Add layer and a Layer Normalization layer, the Add is a residual connection, and the Layer Normalization layer is used for normalizing a hidden layer in a neural network to a standard normal distribution, and the step of inputting the second feature matrix into the encoder to output a second high-level semantic feature comprises: S1, to As the first l the input of the +1 layer Encoder module, and the second initial feature is extracted through the multi-head attention layer and the Add&Norm layer wherein, l is a positive integer less than N, l the initial value is 0 and is the second feature matrix, , is a multi-head attention layer, is a Layer Normalization layer, is an Add layer; S2, the second initial feature is input to the Feed Forward layer and the Add & Norm layer to extract the second target feature wherein, , are two full connection layers; S3, Update l The current value makes l = l +1, and return to step S1. Repeat step S1 N times, then output the current second target feature as the second high-level semantic feature. 6.The image feature extraction method based on double reasoning and adversarial learning according to claim 1, wherein, The objective function of the adversarial neural network model is: wherein loss1(x2, M(x1)) is a pixel mean square error loss of reconstructing the second picture set, loss2(x1, M(M(x1))) is a pixel mean square error loss of reconstructing the first picture set, D(x1) is a probability that the discriminator judges the first picture set as an original picture block, and D(M(x1)) is a probability that the discriminator judges the generated second reconstructed picture set as an original picture block. 7.The image feature extraction method based on double reasoning and adversarial learning according to claim 1, wherein, The step of preprocessing the original image data set to construct a reference image data set comprises: Respectively performing rotation, cropping and coloring operations on each original image data sample in the original image data set to generate auxiliary image data samples; Respectively converting the original image data samples and the auxiliary image data samples into reference image data samples of a preset size; Combining the reference image data samples into a reference image data set. 8.The image feature extraction method based on double reasoning and adversarial learning according to claim 1, wherein, The step of respectively cutting the reference image data samples into picture sets, and respectively dividing the picture sets into first picture sets and second picture sets comprises: Respectively cutting each reference image data sample in the reference image data set into picture blocks of a preset size and a preset number; Respectively combining the picture blocks corresponding to each reference image data sample into independent picture sets; The picture set is randomly divided into a first picture set and a second picture set according to a preset ratio respectively. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor, when executing the computer program, implements the steps of the method in any one of claims 1 to 8.

Images