Image feature extraction model training method, image feature extraction method and device
By combining decoding and reconstruction tasks and comparison learning, iterative training of the image feature extraction model is solved, and the problem of insufficient fineness and richness of image feature extraction is achieved, better visual feature extraction effect is achieved, and a variety of downstream tasks are supported.
Patent Information
- Application Number
- CN202111472879.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-30
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-11-30
AI Technical Summary
When the existing image feature extraction model reduces rich visual information into predefined object categories, it leads to insufficient feature fineness and richness, and cannot obtain ideal extraction effects.
By acquiring different processed images of the target sample image, feature extraction processing is performed separately, and the image feature extraction model is iteratively trained using reconstruction loss and contrast loss, combining decoding and reconstruction tasks and contrast learning to improve the extraction effect of image visual features.
It achieves better image visual feature extraction effect, overcomes the limitations of feature loss in traditional methods, improves the fineness and richness of the image feature extraction model, and supports a variety of intensive computing downstream tasks such as image detection and segmentation.
Smart Images

Figure CN114186622B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image feature extraction model training method, device, electronic device and storage medium. Background Art
[0002] With the development of computer vision technology, image feature extraction is the basis for various intensive computing downstream tasks such as image detection and segmentation. The accuracy of image feature extraction results directly affects the accuracy of subsequent downstream tasks such as image detection and image segmentation.
[0003] At present, most methods for extracting features from images use convolutional neural networks to reduce rich visual information to one of the predefined object categories, thereby extracting image features. However, the current image feature extraction model method reduces rich visual information to one of the predefined object categories, which will cause feature loss, thereby affecting the fineness and richness of the extracted image features, and failing to achieve ideal extraction effects. Summary of the invention
[0004] The present disclosure provides an image feature extraction model training method, device, electronic device and storage medium to at least solve the problem of insufficient fineness and richness of image features extracted in the related art. The technical solution of the present disclosure is as follows:
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image feature extraction model training method, comprising:
[0006] Acquire a first processed image and a second processed image of a target sample image; the target sample image is any image in a sample image training set; the first processed image and the second processed image are obtained based on different data enhancement methods;
[0007] Performing feature extraction processing on the first processed image, the second processed image and other sample images respectively to obtain a first image feature of the first processed image, a second image feature of the second processed image and image features of the other sample images; the other sample images are at least one sample image in the sample image training set except the target sample image, and the first image feature is extracted by a first image feature extraction model;
[0008] Reconstructing the first processed image according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image;
[0009] Obtaining a contrast loss according to the first image feature, the second image feature, and the image features of the other sample images; the contrast loss represents a difference between a feature distance between the first image feature and the second image feature and a feature distance between the first image feature and the image features of the other sample images;
[0010] Based on the contrast loss and the reconstruction loss, the first image feature extraction model is iteratively trained to obtain an image feature extraction model.
[0011] In an exemplary embodiment, the performing feature extraction processing on the first processed image, the second processed image, and other sample images in the sample image training set except the target sample image to obtain first image features of the first processed image, second image features of the second processed image, and image features of the other sample images respectively includes:
[0012] Performing feature extraction processing on the first processed image by using the first image feature extraction model to obtain a first image feature of the first processed image;
[0013] The second processed image and the other sample images are subjected to feature extraction processing respectively through the second image feature extraction model to obtain the second image features of the second processed image and the image features of the other sample images; the first image feature extraction model and the second image feature extraction model have the same model structure, but contain different numbers of multilayer perceptrons, and the model parameters of the second image feature extraction model are obtained by calculating the moving average of the model parameters of the first image feature extraction model.
[0014] In an exemplary embodiment, reconstructing the first processed image according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image includes:
[0015] The first image feature is decoded and reconstructed by a decoding model to obtain a reconstructed image corresponding to the first processed image; the decoding model is a decoder with a convolutional structure;
[0016] A feature comparison process is performed on the reconstructed image and the first processed image to obtain a feature difference between the reconstructed image and the first processed image as a reconstruction loss between the reconstructed image and the first processed image.
[0017] In an exemplary embodiment, obtaining the contrast loss according to the first image feature, the second image feature, and the image features of the other sample images includes:
[0018] Based on the first image feature and the second image feature, obtaining a positive sample pair;
[0019] Based on the first image feature and the image features of the other sample images, a plurality of negative sample pairs are obtained; each of the negative sample pairs includes the first image feature and the image feature of any one of the other sample images;
[0020] The difference between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs is obtained as the contrast loss.
[0021] In an exemplary embodiment, after performing feature extraction processing on the second processed image and the other sample images respectively through the second image feature extraction model to obtain the second image features of the second processed image and the image features of the other sample images, the method further includes:
[0022] According to the time sequence of obtaining the image features, the second image features of the second processed image and the image features of the other sample images are sequentially stored in a dynamic attenuation queue; the weights of the image features in the dynamic attenuation queue are decreased in order from the latest to the earliest storage time;
[0023] The sum of the feature distances of all positive and negative sample pairs is determined by the following method, including:
[0024] Acquire position information of the second image feature and image features of the other sample images in the dynamic attenuation queue;
[0025] Based on the position information, determining weights of the second image feature and image features of the other sample images;
[0026] Obtain the feature distances between the second image feature and the image features of the other sample images and the first image feature, and the weighted sum of the weights corresponding to the second image feature and the image features of the other sample images, as the sum of the feature distances of all positive and negative sample pairs.
[0027] In an exemplary embodiment, the iterative training of the first image feature extraction model based on the contrast loss and the reconstruction loss to obtain the image feature extraction model includes:
[0028] The contrast loss and the reconstruction loss are summed to obtain a loss sum;
[0029] When the loss sum does not meet a preset convergence condition, adjusting model parameters of the first image feature extraction model;
[0030] The first image feature extraction model after parameter adjustment is trained again using a new target sample image until the obtained loss sum meets the convergence condition, and the image feature extraction model is obtained based on the model parameters at the time of convergence.
[0031] According to a second aspect of an embodiment of the present disclosure, there is provided an image feature extraction method, comprising:
[0032] Get the image to be processed;
[0033] Inputting the image to be processed into an image feature extraction model to obtain initial image features of the image to be processed; the image feature extraction model is trained by any of the above-mentioned image feature extraction model training methods;
[0034] The initial image features are subjected to nonlinear transformation processing to obtain target image features as image features of the image to be processed.
[0035] According to a third aspect of an embodiment of the present disclosure, there is provided an image feature extraction model training device, comprising:
[0036] An acquisition unit is configured to execute acquisition of a first processed image and a second processed image of a target sample image; the target sample image is any image in a sample image training set; the first processed image and the second processed image are obtained based on different data enhancement methods;
[0037] an extraction unit, configured to perform feature extraction processing on the first processed image, the second processed image and other sample images respectively, to obtain a first image feature of the first processed image, a second image feature of the second processed image and image features of the other sample images; the other sample images are at least one sample image other than the target sample image in the sample image training set, and the first image feature is extracted by a first image feature extraction model;
[0038] a reconstruction unit, configured to perform reconstruction processing on the first processed image according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image;
[0039] a contrast unit configured to obtain a contrast loss according to the first image feature, the second image feature, and the image features of the other sample images; the contrast loss represents a difference between a feature distance between the first image feature and the second image feature and a feature distance between the first image feature and the image features of the other sample images;
[0040] The training unit is configured to perform iterative training on the first image feature extraction model based on the contrast loss and the reconstruction loss to obtain an image feature extraction model.
[0041] In an exemplary embodiment, the extraction unit is further configured to perform feature extraction processing on the first processed image through the first image feature extraction model to obtain the first image features of the first processed image; perform feature extraction processing on the second processed image and the other sample images through the second image feature extraction model to obtain the second image features of the second processed image and the image features of the other sample images; the first image feature extraction model and the second image feature extraction model have the same model structure, but contain different numbers of multilayer perceptrons, and the model parameters of the second image feature extraction model are obtained by calculating the moving average of the model parameters of the first image feature extraction model.
[0042] In an exemplary embodiment, the reconstruction unit is further configured to perform decoding and reconstruction processing on the first image features through a decoding model to obtain a reconstructed image corresponding to the first processed image; the decoding model is a decoder with a convolutional structure; and feature comparison processing is performed on the reconstructed image and the first processed image to obtain a feature difference between the reconstructed image and the first processed image as a reconstruction loss between the reconstructed image and the first processed image.
[0043] In an exemplary embodiment, the comparison unit is further configured to obtain a positive sample pair based on the first image feature and the second image feature; obtain multiple negative sample pairs based on the first image feature and the image features of the other sample images; each of the negative sample pairs includes the first image feature and the image features of any one of the other sample images; and obtain the difference between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs as the comparison loss.
[0044] In an exemplary embodiment, the device further includes a storage unit configured to store the second image features of the second processed image and the image features of the other sample images in a dynamic attenuation queue in sequence according to the time sequence of obtaining the image features; the weights of the image features in the dynamic attenuation queue decrease in order from the latest to the earliest storage time;
[0045] The comparison unit is further configured to execute the acquisition of position information of the second image feature and the image features of the other sample images in the dynamic attenuation queue; determine the weights of the second image feature and the image features of the other sample images based on the position information; obtain the feature distances between the second image feature and the image features of the other sample images and the first image feature, respectively, and the weighted sum of the weights corresponding to the second image feature and the image features of the other sample images, respectively, as the sum of the feature distances of all positive and negative sample pairs.
[0046] In an exemplary embodiment, the training unit is further configured to perform a summation process on the contrast loss and the reconstruction loss to obtain a loss sum; when the loss sum does not meet a preset convergence condition, adjust the model parameters of the first image feature extraction model; use a new target sample image to re-train the first image feature extraction model with adjusted parameters until the obtained loss sum meets the convergence condition, and obtain the image feature extraction model based on the model parameters at convergence.
[0047] According to a fourth aspect of an embodiment of the present disclosure, there is provided an image feature extraction device, the device comprising:
[0048] An image acquisition unit, configured to acquire an image to be processed;
[0049] A feature determination unit is configured to input the image to be processed into an image feature extraction model to obtain initial image features of the image to be processed; the image feature extraction model is trained by any of the above-mentioned image feature extraction model training methods;
[0050] The transformation unit is configured to perform nonlinear transformation processing on the initial image features to obtain target image features as image features of the image to be processed.
[0051] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:
[0052] processor;
[0053] a memory for storing instructions executable by the processor;
[0054] The processor is configured to execute the instructions to implement the method as described in any one of the above items.
[0055] According to a sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the methods described above.
[0056] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute any of the methods described above.
[0057] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:
[0058] After obtaining the first processed image and the second processed image of the target sample image, feature extraction is performed on the first processed image, the second processed image and other sample images respectively to obtain the first image feature, the second image feature and the image features of other sample images, wherein the first image feature is extracted by the first image feature extraction model, the first processed image is reconstructed based on the first image feature to obtain the reconstruction loss, the contrast loss is obtained based on the first image feature, the second image feature and the image features of other sample images, and the first image feature extraction model is iteratively trained based on the reconstruction loss and the contrast loss to obtain the image feature extraction model. The method obtains the reconstruction loss by using the decoding reconstruction task, obtains the contrast loss by using the contrast learning, and performs model training based on the combination of the reconstruction loss and the contrast loss, which can better realize the extraction of image visual features by unsupervised learning and improve the effect of the image visual features extracted by the obtained image feature extraction model.
[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.
[0061] Figure 1 The figure is a flowchart of a method for training an image feature extraction model according to an exemplary embodiment.
[0062] Figure 2 is a decoding detail diagram corresponding to the first processed image according to an exemplary embodiment.
[0063] Figure 3 It is a general framework diagram of an image feature extraction model training according to an exemplary embodiment.
[0064] Figure 4 The figure is a flowchart of a method for training an image feature extraction model according to another exemplary embodiment.
[0065] Figure 5It is a structural block diagram of an image feature extraction model training device according to an exemplary embodiment.
[0066] Figure 6 It is a structural block diagram of an image feature extraction device according to an exemplary embodiment.
[0067] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION
[0068] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.
[0069] It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0070] It should also be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein.
[0071] In an exemplary embodiment, if Figure 1 As shown, a method for training an image feature extraction model is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0072] In step 110, a first processed image and a second processed image of a target sample image are obtained. The target sample image is any image in a sample image training set. The first processed image and the second processed image are obtained based on different data enhancement methods.
[0073] The data enhancement method refers to a method for implementing data enhancement processing of an image, and the data enhancement method may include cropping, flipping, rotation, Gaussian noise, masking, color conversion, and filtering.
[0074] In a specific implementation, a sample image training set may be obtained first, and any sample image in the sample image training set may be used as a target sample image, and any two different data enhancement methods may be used to perform data enhancement processing on the target sample image to obtain a first processed image and a second processed image. For example, the target sample image x may be subjected to data enhancement processing using Gaussian noise and filter data enhancement methods to obtain a first processed image x1 and a second processed image x2.
[0075] In step 120, feature extraction processing is performed on the first processed image, the second processed image and other sample images respectively to obtain first image features of the first processed image, second image features of the second processed image and image features of other sample images; the other sample images are at least one sample image in the sample image training set except the target sample image, and the first image features are extracted by a first image feature extraction model.
[0076] The first image feature is extracted by a first image feature extraction model, and the second image feature and image features of other sample images are extracted by a second image feature extraction model.
[0077] In a specific implementation, the visual features of the first processed image, the second processed image, and other sample images can be extracted respectively through two first image feature extraction models and second image feature extraction models with the same architecture. Specifically, the first image feature of the first processed image is extracted through the first image feature extraction model, and the image features of the second processed image and other sample images are extracted through the second image feature extraction model. The first image feature extraction model and the second image feature extraction model can be encoders in the form of Transformers with the same structure, and the model parameters of the second image feature extraction model can be obtained by calculating the moving average of the model parameters of the first image feature extraction model.
[0078] In step 130, the first processed image is reconstructed according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image.
[0079] In a specific implementation, after obtaining the first image feature of the first processed image, the first image feature can be input into a decoding model, and the first image feature can be decoded and reconstructed by the decoding model to obtain a reconstructed image of the first processed image. The loss between the reconstructed image and the first processed image is calculated by a reconstruction loss function as the reconstruction loss.
[0080] More specifically, refer to Figure 2, which is a decoding detail image corresponding to the first processed image in an exemplary embodiment. The multi-layer attention feature map output by the first image feature extraction model is gradually integrated into the corresponding stage of the decoding process according to the extraction order of the encoding stage. The fusion is completed through the Concatation operation and the 3×3Conv operation. If the Transformer in the first image feature extraction model is in a multi-scale form, upsampling processing is also required during feature fusion.
[0081] In step 140, a contrast loss is obtained based on the first image feature, the second image feature and the image features of other sample images; the contrast loss represents the difference between the feature distance between the first image feature and the second image feature and the feature distance between the first image feature and the image features of other sample images.
[0082] It can be understood that contrastive learning is a discriminative self-supervised method that automatically constructs similar and dissimilar instances and uses a noise contrast estimation algorithm to make similar instances closer in the projected space, while dissimilar instances are farther apart in the projected space. The training goal of contrastive learning is to make the features sampled from the same image as close as possible in the representation space, and the features sampled from different images as far apart as possible.
[0083] In the present disclosure, the first processed image and the second processed image obtained based on the same target sample image represent the same image, and the target sample image and the other sample images represent different images. Therefore, the training purpose of comparative learning in the present disclosure is to make the features sampled from the first processed image and the second processed image as close as possible in the representation space, and the features sampled from the target sample image and the other sample images as far apart as possible.
[0084] In a specific implementation, the features obtained by sampling the same image can be combined into a positive sample pair, and the features obtained by sampling different images can be combined into a negative sample pair, and the cross entropy loss of the positive and negative samples can be calculated as the contrast loss. More specifically, the first image features of the first processed image and the second image features of the second processed image can be combined into a positive sample pair, and the first image features of the first processed image and each image in the other sample images can be combined into a negative sample pair, thereby obtaining multiple negative sample pairs. The distance between the two image features in the positive sample pair and each negative sample pair is obtained as the feature distance, and the difference between the feature distance of the positive sample pair and the sum of the feature distances of the positive and negative sample pairs is calculated as the contrast loss.
[0085] Furthermore, the image features of other sample images in this example may also be image features of processed images obtained after data enhancement processing is performed on other sample images, and the contrast loss is obtained based on the first image features, the second image features, and the image features of the processed images of other sample images.
[0086] In step 150, the first image feature extraction model is iteratively trained based on contrast loss and reconstruction loss to obtain an image feature extraction model.
[0087] In a specific implementation, the contrast loss and the reconstruction loss may be summed to obtain the loss sum, and the first image feature extraction model may be iteratively trained based on the loss sum. More specifically, the first image feature extraction model may be iterated by a gradient descent method, that is, when the loss sum does not meet a preset convergence condition, such as when the loss sum is greater than a preset accuracy value, the model parameters of the first image feature model may be adjusted by a gradient descent method to obtain a new first image feature extraction model, and steps 110 to 140 are re-executed on the new first image feature extraction model until the obtained loss sum meets the preset convergence condition, and the first image feature extraction model at the time of convergence is used as the image feature extraction model for actual visual feature extraction.
[0088] In the above-mentioned image feature extraction model training method, after obtaining the first processed image and the second processed image of the target sample image, feature extraction is performed on the first processed image, the second processed image and other sample images respectively to obtain the first image feature, the second image feature and the image features of the other sample images, wherein the first image feature is extracted by the first image feature extraction model, the first processed image is reconstructed based on the first image feature to obtain the reconstruction loss, the contrast loss is obtained based on the first image feature, the second image feature and the image features of the other sample images, and the first image feature extraction model is iteratively trained according to the reconstruction loss and the contrast loss to obtain the image feature extraction model. This method obtains the reconstruction loss by using the decoding and reconstruction task, obtains the contrast loss by using contrast learning, and performs model training based on the combination of the reconstruction loss and the contrast loss, which can better realize the extraction of image visual features by unsupervised learning and improve the effect of the image visual features extracted by the obtained image feature extraction model.
[0089] In an exemplary embodiment, in step 120, feature extraction processing is performed on the first processed image, the second processed image, and other sample images in the sample image training set except the target sample image, respectively, to obtain first image features of the first processed image, second image features of the second processed image, and image features of other sample images, including: performing feature extraction processing on the first processed image through a first image feature extraction model to obtain first image features of the first processed image; performing feature extraction processing on the second processed image and other sample images through a second image feature extraction model to obtain second image features of the second processed image and image features of other sample images; the first image feature extraction model and the second image feature extraction model have the same model structure, but different numbers of multilayer perceptrons included, and model parameters of the second image feature extraction model are obtained by calculating a moving average of model parameters of the first image feature extraction model.
[0090] In a specific implementation, the first image feature extraction model and the second image feature extraction model can be encoders in the form of Transformers with the same structure. The first image feature extraction model can be recorded as Guide Encoder (θ), and the second image feature extraction model can be recorded as Followup Encoder (ζ). The model parameters of the second image feature extraction model Followup Encoder (ζ) are obtained by calculating the moving average of the model parameters of the first image feature extraction model Guide Encoder (θ). The relationship between the model parameters of the two models can be expressed as:
[0091] ζ←mζ+(1-m)θ
[0092] Wherein, m represents the update weight parameter, m=0.99.
[0093] refer to Figure 3 , is a general framework diagram of image feature extraction model training in an exemplary embodiment, in which GuideEncoder(θ) represents the first image feature extraction model, and Followup Encoder(ζ) represents the first image feature extraction model. As shown in the figure, the number of multi-layer attention perceptrons of the first image feature extraction model and the second image feature extraction model are not the same. By setting the asymmetry of the multi-layer attention perceptron, the spatial independence of the positive and negative sample sets in the subsequent contrastive learning is enhanced.
[0094] In one iteration, the first image feature and the second image feature are calculated as follows:
[0095] q1=MLP{MLP{Guide Encoder(Augmented_view_1(x))}}
[0096] k1=MLP{MLP{Followup Encoder(Augmented_view_2(x))}}
[0097] (Augmented_view_1, Augmented_view_2∈Augment_method_set)
[0098] Wherein, Augmented_view_1(x) represents the first processed image x1, Augmented_view_2(x) represents the second processed image x2, and MLP (Multi-Layer Perception) represents a multi-layer perceptron. The calculation formula of the attention block in the Guide Encoder of the first image feature extraction model is:
[0099] E_x=Patch(Aug1_view(x))
[0100] Guide Encoder(x)=MLP{MSA(LN(E_x))}+MSA(LN(E_x))
[0101] Among them, E_x belongs to the Patch encoding function, which converts the input features of scale (H, W, C) into N*C encoding sequences through convolution and linear mapping operations, where N = (H / P)*(W / P), P∈N, LN represents normalization processing, and the MSA part is the multi-head attention information calculation module. The attention calculation formula in a single head is:
[0102]
[0103] Among them, Q, K, V are the coding sequences obtained by linear mapping of E_x, C is the number of input channels, and E is the position information.
[0104] In this embodiment, two Transformer-style encoders with the same structure are constructed as the first image feature extraction model and the second image feature extraction model respectively, so that comparative learning and training of one image feature extraction model can be performed on the other image feature extraction model to improve the feature extraction effect of the trained image feature extraction model.
[0105] In an exemplary embodiment, in step 130, the first processed image is reconstructed according to the first image feature to obtain a reconstruction loss between the reconstructed image of the first processed image and the first processed image, which can be achieved by: decoding and reconstructing the first image feature through a decoding model to obtain a reconstructed image corresponding to the first processed image; the decoding model is a decoder with a convolutional structure; and feature comparison is performed on the reconstructed image and the first processed image to obtain a feature difference between the reconstructed image and the first processed image as the reconstruction loss between the reconstructed image and the first processed image.
[0106] In the specific implementation, Figure 3 As shown in the overall framework diagram of the image feature extraction model training, the Decoder (w) in the figure represents the decoding model, which is a decoder with a convolutional structure. The first processed image x1 (i.e., Augmented_view_1 (x) in the figure) after the data enhancement processing of (H, W, C) is input, and the feature reconstruction result, i.e., the reconstructed image, is obtained by the first image feature extraction model Guide Encoder (θ) and the decoding model Decoder (w), which is denoted as r (H, W, C). If the first processed image x1 is denoted as A, the calculation process of the reconstruction loss between the reconstructed image r (H, W, C) and the first processed image x1 is:
[0107] A=Augmented_view_1(x)
[0108] Attention maps=feature_list(Guide Encoder(A))
[0109] r=Decoder(Attention maps, Guide Encoder(A))
[0110] As shown in the decoding details Figure 3 As shown in the figure, the multi-layer attention feature map output by the Guide Encoder (θ) of the first image feature extraction model is gradually fused to the corresponding stage of the decoding process according to the extraction order of the encoding stage. The fusion is completed through the Concatation operation and the 3×3Conv operation. If the Transformer in the first image feature extraction model is in a multi-scale form, upsampling processing is also required during feature fusion.
[0111] If the reconstructed image r(H, W, C) of the first processed image x1 is very close to the first processed image x1, then it can be considered that the two are close in content, because the layer closer to the output contains more content information. Therefore, the reconstruction loss function Reconstrction Loss can be obtained as:
[0112]
[0113] Among them, r represents the reconstructed image r(H, W, C), and A represents the first processed image Augmented_view_1(x), that is, x1.
[0114] In this embodiment, by decoding and reconstructing the first processed image, low-level features are densely combined to obtain finer image features, which can further enable the image feature extraction model to better learn the local position representation of the image, improve the fineness and richness of the extraction results, and overcome the defect of traditional image-level learning methods that reduce rich visual information to one of predefined object categories, resulting in feature loss that limits the performance of self-supervised learning.
[0115] In an exemplary embodiment, in step 140, a contrast loss is obtained based on the first image feature, the second image feature, and the image features of other sample images, including: obtaining a positive sample pair based on the first image feature and the second image feature; obtaining multiple negative sample pairs based on the first image feature and the image features of the other sample images; each negative sample pair includes the first image feature and the image feature of any one of the other sample images; and obtaining the difference between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs as the contrast loss.
[0116] In a specific implementation, for example, there are K sample images in the sample image training set, the target sample image obtained from the sample image training set is x, and the first processed image x1 and the second processed image x2 are obtained after data enhancement processing, and the other sample images in the sample image training set are denoted as y2…y K , for the first processed image x1, the second processed image x2 and other sample images y2…y K Perform feature extraction processing respectively to obtain the first image feature q1, the second image feature k1 and the image features k2…k of other sample images K The first image feature q1 and the second image feature k1 form a positive sample pair<q1,k1> , the first image feature q1 and the image features k2…k of other sample images are K Form K-1 negative sample pairs<q1,k2> … <q1,k K >. Calculate the distance between the two image features in each sample pair respectively, obtain the feature distance of 1 positive sample pair and the feature distance of K-1 negative sample pairs, calculate the sum of the feature distance of 1 positive sample pair and the feature distance of K-1 negative sample pairs, as the sum of the feature distances of all positive and negative sample pairs, and take the logarithm of the ratio between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs as the contrast loss.
[0117] In this embodiment, a positive sample pair is obtained based on the first image feature and the second image feature, and a plurality of negative sample pairs are obtained based on the first image feature and the image features of the other sample images. Then, the sum of the feature distances and the difference between the positive sample pair and the positive and negative sample pairs are calculated to obtain the contrast loss, so as to train the first image feature extraction model through the contrast loss. In this way, the image feature extraction model can learn the visual features of the image itself without relying on labels.
[0118] In an exemplary embodiment, after performing feature extraction processing on the second processed image and other sample images respectively through the second image feature extraction model to obtain the second image features of the second processed image and the image features of the other sample images, it also includes: storing the second image features of the second processed image and the image features of the other sample images in a dynamic attenuation queue in sequence according to the time sequence in which the image features are obtained; the weights of the image features in the dynamic attenuation queue decrease in order from late to early according to the storage time.
[0119] In the specific implementation, Figure 3 As shown in the overall framework diagram of the image feature extraction model training, the features extracted by the second image feature extraction model are all stored in the dynamic attenuation queue k. Specifically, the second image features of the second processed image and the image features of other sample images extracted by the second image feature extraction model can be stored in the dynamic attenuation queue in sequence according to the time sequence of obtaining the image features. The weights of the image features in the dynamic attenuation queue are different, and they decrease in order from late to early according to the storage time. That is, the earlier the image feature is stored in the dynamic attenuation queue, the smaller the weight, and the later the image feature is stored in the dynamic attenuation queue, the larger the weight.
[0120] The unequally weighted negative sample storage queue designed in this embodiment can reduce the encoding difference caused by the encoder momentum update weight by decaying the fixed weight proportionally according to the queue position number, so as to reduce the impact of data entering the storage queue earlier.
[0121] Furthermore, in an exemplary embodiment, the sum of the feature distances of all positive and negative sample pairs is determined in the following manner, including: obtaining position information of the second image feature and the image features of other sample images in the dynamic attenuation queue; determining the weights of the second image feature and the image features of the other sample images based on the position information; obtaining the feature distances of the second image feature and the image features of other sample images with the first image feature, respectively, and the weighted sum of the weights corresponding to the second image feature and the image features of other sample images, as the sum of the feature distances of all positive and negative sample pairs.
[0122] Among them, the weight represents the relative importance of each image feature.
[0123] In the specific implementation, the dynamic attenuation queue stores the features extracted by the second image feature extraction model. It can be seen that the negative sample pair is composed of the first image feature and other image features in the dynamic attenuation queue except the second image feature. Each image feature in the dynamic attenuation queue has a corresponding weight. Therefore, when calculating the contrast loss, the influence of the weight needs to be considered. Therefore, before calculating the contrast loss, the position information of the second image feature and the image features of other sample images in the dynamic attenuation queue can be obtained first. Based on the position information, the weights of the second image feature and the image features of other sample images are determined, and the product of the second image feature and the weight corresponding to the second image feature is calculated. The product of the image features of other sample images and the weights corresponding to the image features of other sample images is calculated. The two types of products are added to obtain a weighted sum, and the weighted sum is used as the sum of the feature distances of all positive and negative sample pairs. The logarithm of the ratio between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs is further calculated as the contrast loss.
[0124] For example, if the dynamic attenuation queue is quene, the first image feature is q i , the second image feature is k j , since the contrast loss is the cross entropy loss of positive and negative samples, the contrast loss can be expressed as:
[0125]
[0126] Among them, the molecule is the sum of the feature distances between the positive sample pairs, is the sum of the distances between the feature distances of the positive sample pair and the feature distances of the negative sample pair, and sim(*) is the similarity measurement algorithm. After introducing the decay queue, the above formula can be rewritten as:
[0127]
[0128] In this embodiment, by combining the weights of each image feature and calculating the sum of the feature distances of the positive and negative sample pairs, the encoding difference caused by the encoder momentum updating weights can be reduced to reduce the impact of data entering the storage queue earlier, thereby improving the accuracy of the obtained contrast loss.
[0129] In an exemplary embodiment, in step 150, the first image feature extraction model is iteratively trained based on the contrast loss and the reconstruction loss to obtain the image feature extraction model, including: summing the contrast loss and the reconstruction loss to obtain the loss sum; when the loss sum does not meet the preset convergence condition, adjusting the model parameters of the first image feature extraction model; using a new target sample image to re-train the first image feature extraction model after the parameter adjustment until the obtained loss sum meets the convergence condition, and obtaining the image feature extraction model based on the model parameters at convergence.
[0130] In a specific implementation, the preset convergence condition may be that the loss sum is less than a preset precision value. Then, after obtaining the contrast loss and the reconstruction loss, the contrast loss and the reconstruction loss may be added, and the obtained loss sum may be compared with the preset precision. When the loss sum is greater than the preset precision value, it is determined that the preset convergence condition is not met, and the model parameters of the first image feature extraction model are adjusted. Correspondingly, since the model parameters of the second image feature extraction model are obtained by calculating a moving average of the model parameters of the first image feature extraction model, the model parameters of the second image feature extraction model will also be updated accordingly. The first image feature extraction model after parameter adjustment is trained again using a new target sample image until the obtained loss sum meets the convergence condition. The model parameters at the time of convergence are saved, and the model parameters at the time of convergence are used as the model parameters of the image feature extraction model to obtain the image feature extraction model.
[0131] In this embodiment, the first image feature extraction model is trained by combining contrast loss and reconstruction loss. On the basis of contrast learning, combining the reconstruction task can enhance the network's general feature learning ability, better learn rich visual representations, and better support various intensive computing downstream tasks such as image detection and segmentation.
[0132] In an exemplary embodiment, it is a flowchart of an image feature extraction method shown in another exemplary implementation. In this embodiment, the method includes the following steps: acquiring an image to be processed; inputting the image to be processed into an image feature extraction model to obtain initial image features of the image to be processed; the image feature extraction model is trained by the method described in the above embodiment; performing nonlinear transformation processing on the initial image features to obtain target image features as image features of the image to be processed.
[0133] In a specific implementation, after the image feature extraction model is trained by the method described in the above embodiment, when extracting image features of the image to be processed in actual applications, the image to be processed can be input into the trained image feature extraction model to obtain the initial image features of the image to be processed, and then the initial image features are further extracted after two nonlinear changes of the Project layer to obtain the target image features as the image features of the image to be processed.
[0134] In this embodiment, after obtaining the initial image features of the image to be processed through the image feature extraction model, the initial image features are further subjected to nonlinear transformation processing, which can improve the generalization of the image features of the image to be processed and better apply them to other downstream tasks or applications.
[0135] In another exemplary embodiment, Figure 4 FIG. 1 is a flowchart of an image feature extraction model training method according to another exemplary embodiment. In this embodiment, the method includes the following steps:
[0136] Step 410, obtaining a first processed image and a second processed image of a target sample image; the target sample image is any image in a sample image training set;
[0137] Step 420, performing feature extraction processing on the first processed image by using a first image feature extraction model to obtain a first image feature of the first processed image;
[0138] Step 430, performing feature extraction processing on the second processed image and other sample images respectively through the second image feature extraction model to obtain second image features of the second processed image and image features of the other sample images;
[0139] Step 440, storing the second image features of the second processed image and the image features of other sample images in a dynamic attenuation queue in sequence according to the time sequence of obtaining the image features; the weights of the image features in the dynamic attenuation queue decrease in the order of storage time from the latest to the earliest;
[0140] Step 450, decoding and reconstructing the first image feature through a decoding model to obtain a reconstructed image corresponding to the first processed image; the decoding model is a decoder with a convolutional structure;
[0141] Step 460, performing feature comparison processing on the reconstructed image and the first processed image to obtain a feature difference between the reconstructed image and the first processed image as a reconstruction loss between the reconstructed image and the first processed image;
[0142] Step 470, obtaining a positive sample pair based on the first image feature and the second image feature, and obtaining a plurality of negative sample pairs based on the first image feature and image features of other sample images;
[0143] Step 480, obtaining the difference between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs as the comparison loss;
[0144] Step 490, summing the contrast loss and the reconstruction loss to obtain a loss sum, and iteratively training the first image feature extraction model based on the loss sum to obtain an image feature extraction model.
[0145] The training method of the image feature extraction model provided in this embodiment uses two encoders with Transformer structures to calculate the positive and negative sample contrast loss, in which one guide encoder updates parameters through gradients, and the other Followup Encoder (ζ) updates weights through the moving average momentum of the Guide Encoder (θ). A decoder with a convolutional structure is added after the guide encoder to strengthen the learning ability of the network by reconstructing the features encoded by the Transformer and calculating the content loss with the original data. Compared with the traditional method of realizing unsupervised learning based on the convolutional neural network algorithm, this method can obtain a more ideal visual feature expression and can better support various computer application tasks such as image classification, target detection and segmentation.
[0146] In an exemplary embodiment, in order to facilitate those skilled in the art to understand the embodiment of the present application, the following will be described in conjunction with specific examples of the accompanying drawings. Figure 3 , is a general framework diagram of the image feature extraction model training method in an application example. Guide Encoder (θ) and Followup Encoder (ζ) are encoders of Transformer structure, and Decoder (ω) is a decoder of convolution structure. The MLP module contains two layers of linear mapping and one layer of activation function, which maps the output of Encoder to the contrast space. The MLP modules with different numbers of branches in Guide Encoder (θ) and Followup Encoder (ζ) enhance the spatial independence of positive and negative sample sets in contrastive learning through asymmetry. The training process is as follows:
[0147] (1) Design positive and negative sample pairs:
[0148] Obtain a sample image training set, take each sample image in the sample image training set as a target sample image x in turn, select two different data augmentation methods from a data augmentation method set (Augment_method_set) to process the target sample image to obtain a first processed image Augmented_view_1(x) and a second processed image Augmented_view_2(x);
[0149] like Figure 3 In the overall framework diagram shown in FIG. 1 , in a batch (used to define the number of samples to be processed before updating the internal model parameters), the two processed images of the target sample image x are respectively processed by the first image feature extraction model GuideEncoder and the second image feature extraction model Followup Encoder to obtain the first image feature q1 and the second image feature k1, forming a positive sample pair<q1,k1> . Other sample images in the sample image training set are also feature extracted by the second image feature extraction model, and the image features extracted by the second image feature extraction model are stored in the dynamic decay queue (decayqueue). The first image feature and the image features of each other sample image in the dynamic decay queue (decay queue) form a negative sample pair. When the length of the dynamic decay queue is K, each sample in a batch has 1 positive sample pair and K-1 negative sample pairs, among which the negative samples in the weighted dynamic queue (weight moment queue) are updated in each iteration, and the weight ratio of the negative samples is: weight = 0.9999 positioni ,position i ∈{1,2,3,,,K} is the position information in the queue.
[0150] (2) Transformer extracts visual features
[0151] The first processed image Augmented_view_1(x) is subjected to feature extraction by the first image feature extraction model Guide Encoder, and the second processed image Augmented_view_2(x) and other sample images are subjected to feature extraction by the second image feature extraction model Followup Encoder.
[0152] (3) Calculate reconstruction loss
[0153] The target sample image is denoted as H, W, C, and the reconstructed image is denoted as r(H, W, C). If the first processed image Augmented_view_1(x) is denoted as A, the calculation process of the reconstruction loss between the reconstructed image r(H, W, C) and the first processed image is:
[0154] A=Augmented_view_1(x)
[0155] Attention maps=feature_list(Guide Encoder(A))
[0156] r=Decoder(Attention maps, Guide Encoder(A))
[0157] If the reconstructed image r(H, W, C) of the first processed image x1 is very close to the first processed image, then it can be considered that the two are close in content, because the layer closer to the output contains more content information. Therefore, the reconstruction loss function Reconstrction Loss can be obtained as:
[0158]
[0159] (4) Calculate contrast loss
[0160] The training goal of contrastive learning is to make the features sampled from the same image as close as possible in the representation space; the features sampled from different images as far apart as possible. The contrastive loss is the cross entropy loss of positive and negative samples:
[0161]
[0162] Among them, the molecule is the sum of the feature distances between the positive sample pairs, is the sum of the distances between the feature distances of the positive sample pair and the feature distances of the negative sample pair, and sim(*) is the similarity measurement algorithm. After introducing the decay queue, the above formula can be rewritten as:
[0163]
[0164] (5) Model training
[0165] Calculate the sum of contrast loss and reconstruction loss, and get the overall loss function as follows:
[0166] Loss=Contrastive Loss+Reconstrction Loss
[0167] The first image feature extraction model Guide Encoder is iteratively trained based on the total loss Loss to obtain an image feature extraction model.
[0168] After verification, in the model training phase, the loss function adopted by this scheme consists of two parts: the contrast loss function and the reconstruction loss function. The optimizer used by the model is Adamw, the learning rate strategy is Cosine Annealing, and the initial learning rate is 5e -4 , the warm up epoch is 5, and the warm up learning rate is 5e -7 After 300 epochs of training on the dataset, the loss is basically stable and the training is terminated. At this time, the Top1 acc of the model in IamgeNet linear verification is 75.70%; the verification results of detection based on Mask-rnn on the COCO dataset are: bbox_mAP: 44.0%, bbox_mAP_50: 66.7%, bbox_mAP_75: 48.0%; segm_mAP: 39.9%, segm_mAP_50: 63.4%, segm_mAP_75: 42.7% for detection and segmentation based on Mask-rcnn on the COCO dataset. The results of semantic segmentation based on the SERT segmentation framework on the Cityscapes dataset are: mIOU: 73.4%, mAcc: 80.95, which shows that this solution has a more obvious improvement compared with the existing unsupervised tasks.
[0169] The method proposed in this embodiment uses the decoding and reconstruction task to fuse the Transformer with the traditional convolution algorithm for contrastive learning to achieve unsupervised learning. By combining feature reconstruction with contrastive learning, the unsupervised learning method can be better used to extract the visual features of the image. Specifically, it has the following advantages: (1) It takes into account the modeling capabilities of spatial structure and global information, and proposes a new unsupervised training framework based on feature reconstruction. On the basis of the contrastive learning framework, the reconstruction task is combined to strengthen the general feature learning ability of the network. (2) The Transformer is combined with the convolution structure for feature extraction. The two can better learn rich visual representations through the constraints of reconstruction loss. Compared with CNN (convolutional neural network), Transformer has stronger parallel computing capabilities and global modeling capabilities, but it requires additional position encoding to compensate for the loss of spatial relative position, while the displacement invariance of the convolution algorithm in CNN can well compensate for the local positioning information. (3) In the decoding and reconstruction process, multi-scale and multi-level encoding attention features are used to obtain richer semantic information to better support various intensive computing downstream tasks such as image detection and segmentation. (4) A non-equal-weighted negative sample storage queue is designed. By decaying the fixed weight proportionally according to the queue position number, the encoding difference caused by the encoder momentum update weight is reduced, thereby reducing the impact of data that enters the storage queue earlier.
[0170] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0171] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.
[0172] Figure 5 is a structural block diagram of an image feature extraction model training device according to an exemplary embodiment. Figure 5 The device includes: an acquisition unit 510, an extraction unit 520, a reconstruction unit 530, a comparison unit 540 and a training unit 550, wherein:
[0173] An acquisition unit 510 is configured to execute acquisition of a first processed image and a second processed image of a target sample image; the target sample image is any image in a sample image training set; the first processed image and the second processed image are obtained based on different data enhancement methods;
[0174] The extraction unit 520 is configured to perform feature extraction processing on the first processed image, the second processed image and other sample images respectively, to obtain the first image feature of the first processed image, the second image feature of the second processed image and the image features of other sample images; the other sample images are at least one sample image other than the target sample image in the sample image training set, and the first image feature is extracted by the first image feature extraction model;
[0175] The reconstruction unit 530 is configured to perform reconstruction processing on the first processed image according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image;
[0176] The contrast unit 540 is configured to obtain a contrast loss according to the first image feature, the second image feature and the image features of other sample images; the contrast loss represents the difference between the feature distance between the first image feature and the second image feature and the feature distance between the first image feature and the image features of other sample images;
[0177] The training unit 550 is configured to perform iterative training on the first image feature extraction model based on contrast loss and reconstruction loss to obtain an image feature extraction model.
[0178] In an exemplary embodiment, the extraction unit 520 is further configured to perform feature extraction processing on the first processed image through a first image feature extraction model to obtain first image features of the first processed image; perform feature extraction processing on the second processed image and other sample images through a second image feature extraction model to obtain second image features of the second processed image and image features of other sample images; the first image feature extraction model and the second image feature extraction model have the same model structure, but contain different numbers of multilayer perceptrons, and the model parameters of the second image feature extraction model are obtained by calculating a moving average of the model parameters of the first image feature extraction model.
[0179] In an exemplary embodiment, the reconstruction unit 530 is further configured to perform decoding and reconstruction processing on the first image features through a decoding model to obtain a reconstructed image corresponding to the first processed image; the decoding model is a decoder with a convolutional structure; and feature comparison processing is performed on the reconstructed image and the first processed image to obtain the feature difference between the reconstructed image and the first processed image as the reconstruction loss between the reconstructed image and the first processed image.
[0180] In an exemplary embodiment, the comparison unit 540 is further configured to obtain a positive sample pair based on the first image feature and the second image feature; obtain multiple negative sample pairs based on the first image feature and the image features of other sample images; each negative sample pair includes the first image feature and the image feature of any one of the other sample images; obtain the difference between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs as the comparison loss.
[0181] In an exemplary embodiment, the apparatus further includes a storage unit configured to store the second image features of the second processed image and the image features of other sample images in a dynamic attenuation queue in sequence according to the time sequence of obtaining the image features; the weights of the image features in the dynamic attenuation queue decrease in order from the latest to the earliest storage time;
[0182] The comparison unit 540 is also configured to execute the acquisition of position information of the second image feature and the image features of other sample images in the dynamic attenuation queue; determine the weights of the second image feature and the image features of other sample images based on the position information; obtain the feature distances between the second image feature and the image features of other sample images and the first image feature, respectively, and the weighted sum of the weights corresponding to the second image feature and the image features of other sample images, respectively, as the sum of the feature distances of all positive and negative sample pairs.
[0183] In an exemplary embodiment, the training unit 550 is further configured to perform a summation process on the contrast loss and the reconstruction loss to obtain a loss sum; when the loss sum does not meet a preset convergence condition, adjust the model parameters of the first image feature extraction model; use a new target sample image to re-train the first image feature extraction model with adjusted parameters until the obtained loss sum meets the convergence condition, and obtain the image feature extraction model based on the model parameters at the time of convergence.
[0184] Figure 6 is a structural block diagram of an image feature extraction device according to an exemplary embodiment. Figure 6 The device includes: an image acquisition unit 610, a feature determination unit 620 and a transformation unit 630, wherein:
[0185] An image acquisition unit 610 is configured to acquire an image to be processed;
[0186] The feature determination unit 620 is configured to input the image to be processed into an image feature extraction model to obtain initial image features of the image to be processed; the image feature extraction model is trained by any of the above methods;
[0187] The transformation unit 630 is configured to perform nonlinear transformation processing on the initial image features to obtain target image features as image features of the image to be processed.
[0188] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0189] Figure 7 1 is a block diagram of an electronic device 700 for implementing an image feature extraction model training method according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0190] Reference Figure 7 , the electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .
[0191] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.
[0192] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, an optical disk, or a graphene memory.
[0193] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.
[0194] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0195] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting an audio signal.
[0196] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0197] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of the components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or the components of the electronic device 700, the presence or absence of contact between the user and the electronic device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0198] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0199] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0200] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the instructions can be executed by a processor 720 of the electronic device 700 to perform the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0201] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor 720 of the electronic device 700 to complete the above method.
[0202] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.
[0203] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.
[0204] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
[0205] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for training an image feature extraction model, characterized in that: include: Acquire a first processed image and a second processed image of the target sample image; The target sample image is any image in the sample image training set; The first processed image and the second processed image are obtained based on different data enhancement methods; Performing feature extraction processing on the first processed image, the second processed image and other sample images respectively to obtain a first image feature of the first processed image, a second image feature of the second processed image and image features of the other sample images; the other sample images are at least one sample image other than the target sample image in the sample image training set, the first image feature is extracted by a first image feature extraction model, the second image feature and image features of the other sample images are extracted by a second image feature extraction model, the first image feature extraction model and the second image feature extraction model have the same model structure, but different numbers of multilayer perceptrons included, and the model parameters of the second image feature extraction model are obtained by calculating a moving average of the model parameters of the first image feature extraction model; Reconstructing the first processed image according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image; Obtaining a contrast loss according to the first image feature, the second image feature, and the image features of the other sample images; the contrast loss represents a difference between a feature distance between the first image feature and the second image feature and a feature distance between the first image feature and the image features of the other sample images; Based on the contrast loss and the reconstruction loss, the first image feature extraction model is iteratively trained to obtain an image feature extraction model.
2. The method according to claim 1, characterized in that The reconstructing the first processed image according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image includes: The first image feature is decoded and reconstructed by a decoding model to obtain a reconstructed image corresponding to the first processed image; the decoding model is a decoder with a convolutional structure; A feature comparison process is performed on the reconstructed image and the first processed image to obtain a feature difference between the reconstructed image and the first processed image as a reconstruction loss between the reconstructed image and the first processed image.
3. The method according to claim 1, characterized in that The obtaining the contrast loss according to the first image feature, the second image feature and the image features of the other sample images includes: Based on the first image feature and the second image feature, obtaining a positive sample pair; Based on the first image feature and the image features of the other sample images, a plurality of negative sample pairs are obtained; each of the negative sample pairs includes the first image feature and the image feature of any one of the other sample images; The difference between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs is obtained as the contrast loss.
4. The method according to claim 3, characterized in that After performing feature extraction processing on the second processed image and the other sample images respectively through the second image feature extraction model to obtain the second image features of the second processed image and the image features of the other sample images, the method further includes: According to the time sequence of obtaining the image features, the second image features of the second processed image and the image features of the other sample images are sequentially stored in a dynamic attenuation queue; the weights of the image features in the dynamic attenuation queue are decreased in order from the latest to the earliest storage time; The sum of the feature distances of all positive and negative sample pairs is determined by the following method, including: Acquire position information of the second image feature and image features of the other sample images in the dynamic attenuation queue; Based on the position information, determining weights of the second image feature and image features of the other sample images; Obtain the feature distances between the second image feature and the image features of the other sample images and the first image feature, and the weighted sum of the weights corresponding to the second image feature and the image features of the other sample images, as the sum of the feature distances of all positive and negative sample pairs.
5. The method according to claim 1, characterized in that The iterative training of the first image feature extraction model based on the contrast loss and the reconstruction loss to obtain the image feature extraction model includes: The contrast loss and the reconstruction loss are summed to obtain a loss sum; When the loss sum does not meet a preset convergence condition, adjusting model parameters of the first image feature extraction model; The first image feature extraction model after parameter adjustment is trained again using a new target sample image until the obtained loss sum meets the convergence condition, and the image feature extraction model is obtained based on the model parameters at the time of convergence.
6. A method for extracting image features, characterized in that: The method comprises: Get the image to be processed; Inputting the image to be processed into an image feature extraction model to obtain initial image features of the image to be processed; the image feature extraction model is trained by the method according to any one of claims 1 to 5; The initial image features are subjected to nonlinear transformation processing to obtain target image features as image features of the image to be processed.
7. An image feature extraction model training device, characterized in that: include: An acquisition unit, configured to execute acquisition of a first processed image and a second processed image of a target sample image; The target sample image is any image in the sample image training set; the first processed image and the second processed image are obtained based on different data enhancement methods; an extraction unit configured to perform feature extraction processing on the first processed image, the second processed image and other sample images respectively, to obtain a first image feature of the first processed image, a second image feature of the second processed image and image features of the other sample images; the other sample images are at least one sample image other than the target sample image in the sample image training set, the first image feature is extracted by a first image feature extraction model, the second image feature and image features of the other sample images are extracted by a second image feature extraction model, the first image feature extraction model and the second image feature extraction model have the same model structure, but different numbers of multilayer perceptrons included, and model parameters of the second image feature extraction model are obtained by calculating a moving average of model parameters of the first image feature extraction model; a reconstruction unit, configured to perform reconstruction processing on the first processed image according to the first image feature to obtain a reconstruction loss between a reconstructed image of the first processed image and the first processed image; a contrast unit configured to obtain a contrast loss according to the first image feature, the second image feature, and the image features of the other sample images; the contrast loss represents a difference between a feature distance between the first image feature and the second image feature and a feature distance between the first image feature and the image features of the other sample images; The training unit is configured to perform iterative training on the first image feature extraction model based on the contrast loss and the reconstruction loss to obtain an image feature extraction model.
8. The device according to claim 7, characterized in that The reconstruction unit is further configured to perform decoding and reconstruction processing on the first image features through a decoding model to obtain a reconstructed image corresponding to the first processed image; the decoding model is a decoder with a convolutional structure; and perform feature comparison processing on the reconstructed image and the first processed image to obtain a feature difference between the reconstructed image and the first processed image as a reconstruction loss between the reconstructed image and the first processed image.
9. The device according to claim 7, characterized in that The comparison unit is further configured to obtain a positive sample pair based on the first image feature and the second image feature; obtain multiple negative sample pairs based on the first image feature and the image features of the other sample images; each of the negative sample pairs includes the first image feature and the image features of any one of the other sample images; and obtain the difference between the feature distance of the positive sample pair and the sum of the feature distances of all positive and negative sample pairs as the comparison loss.
10. The device according to claim 9, characterized in that The device also includes a storage unit configured to store the second image feature of the second processed image and the image features of the other sample images in a dynamic attenuation queue in sequence according to the time sequence of obtaining the image features; the weights of the image features in the dynamic attenuation queue decrease in order from the latest to the earliest storage time; The comparison unit is further configured to obtain position information of the second image feature and the image features of the other sample images in the dynamic attenuation queue; and determine weights of the second image feature and the image features of the other sample images based on the position information; Obtain the feature distances between the second image feature and the image features of the other sample images and the first image feature, and the weighted sum of the weights corresponding to the second image feature and the image features of the other sample images, as the sum of the feature distances of all positive and negative sample pairs.
11. The device according to claim 7, characterized in that The training unit is further configured to perform a summation process on the contrast loss and the reconstruction loss to obtain a loss sum; when the loss sum does not meet a preset convergence condition, adjust the model parameters of the first image feature extraction model; use a new target sample image to re-train the first image feature extraction model after the parameter adjustment until the obtained loss sum meets the convergence condition, and obtain the image feature extraction model based on the model parameters at the time of convergence.
12. An image feature extraction device, characterized in that: The device comprises: An image acquisition unit, configured to acquire an image to be processed; A feature determination unit, configured to input the image to be processed into an image feature extraction model to obtain initial image features of the image to be processed; the image feature extraction model is trained by the method according to any one of claims 1 to 5; The transformation unit is configured to perform nonlinear transformation processing on the initial image features to obtain target image features as image features of the image to be processed.
13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the image feature extraction model training method as described in any one of claims 1 to 5 or the image feature extraction method as described in claim 6.
14. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the image feature extraction model training method as described in any one of claims 1 to 5 or the image feature extraction method as described in claim 6.
15. A computer program product, comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the image feature extraction model training method as described in any one of claims 1 to 5 or the image feature extraction method as described in claim 6.
Citation Information
Patent Citations
Image super-resolution reconstruction method
CN108550115A
Image super-resolution reconstruction method and device, storage medium and electronic equipment
CN112488923A
Feature extraction model training method and device and computer equipment
CN113177566A
Fine-grained ship identification method based on comparative learning
CN113255793A
Training method and device of image processing model
CN113435545A
