Image acquisition model training and image acquisition method and device

Abstract

The application provides an image acquisition model training and image acquisition method and device. The image acquisition model training method comprises: acquiring a plurality of image samples; inputting the plurality of image samples into an original image acquisition model, traversing the windows of the image samples by the original image acquisition model, obtaining a plurality of first images corresponding to each image sample, and performing mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample; randomly sampling the second image to obtain at least one third image corresponding to each image sample, and performing mask processing on the at least one third image according to a second mask rate to output a target image sample corresponding to each image sample; and updating model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample to obtain a trained image acquisition model. The image acquisition model trained by the method can effectively acquire a target image with high accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image acquisition model training method and an image acquisition method and device. BACKGROUND

[0002] With the rapid development of mobile Internet and smart terminal technology, electronic devices with camera devices have increasingly entered people's lives. Users can use these electronic devices to obtain a large amount of image data. Since the image data may be unclear, the electronic device needs to process these unclear image data to obtain a target image with better image quality.

[0003] The existing image acquisition method often uses a Masked Auto Encoder (MAE) model or a Local Masked Reconstruction (LoMaR) model to process the acquired image. The former will produce high computational cost and large time complexity in the image processing process, resulting in an inaccurate target image. The latter performs single mask processing within a local window, making the entire image processing process relatively simple, which also results in an inaccurate target image.

[0004] In summary, the existing image acquisition method has certain limitations, resulting in an inaccurate target image obtained by the electronic device using the existing model, i.e., the target image has poor image quality. SUMMARY

[0005] The present application provides an image acquisition model training method and an image acquisition method and device. The image acquisition model trained by the image acquisition model training method can effectively obtain a target image with high accuracy, and the target image has better image quality.

[0006] The present application provides an image acquisition model training method, comprising:

[0007] obtaining a plurality of image samples;

[0008] The plurality of image samples are input into an original image acquisition model, the image samples are traversed by the original image acquisition model, a plurality of first images corresponding to each image sample are obtained, and the plurality of first images are masked according to a first mask rate, to obtain a second image corresponding to each image sample, and any two first images do not overlap; the second image is randomly sampled to obtain at least one third image corresponding to each image sample, and the at least one third image is masked according to a second mask rate, so as to output a target image sample corresponding to each image sample through the original image acquisition model; wherein the size of the third image is greater than the size of the first image.

[0009] According to the plurality of image samples and the target image samples corresponding to each image sample, the model parameters corresponding to the original image acquisition model are updated to obtain a trained image acquisition model.

[0010] According to the image acquisition model training method provided by the application, the at least one third image is masked according to the second mask rate to output the target image sample corresponding to each image sample through the original image acquisition model, which comprises: masking the at least one third image according to the second mask rate to obtain a fourth image corresponding to the third image; and performing encoding processing and decoding processing on the fourth image to obtain the target image sample corresponding to each image sample.

[0011] According to the image acquisition model training method provided by the application, the plurality of first images corresponding to each image sample are obtained by traversing the window of the image sample through the original image acquisition model, which comprises: obtaining the size of the image sample through the original image acquisition model; in the case that the size of the image sample does not match a preset size threshold, adjusting the size of the image sample to obtain an adjusted image sample; in the case that the size of the adjusted image sample matches the preset size threshold, traversing the window of the adjusted image sample to obtain the plurality of first images corresponding to each image sample.

[0012] According to the image acquisition model training method provided by the application, the encoding processing and decoding processing are performed on the fourth image to obtain the target image sample corresponding to each image sample, which comprises: using an encoder in the original image acquisition model to perform encoding processing on the fourth image to obtain implicit features corresponding to the fourth image; and using a decoder in the original image acquisition model to perform decoding processing on the implicit features to obtain the target image sample corresponding to each image sample.

[0013] According to the image acquisition model training method provided by the application, the model parameters corresponding to the original image acquisition model are updated according to the plurality of image samples and the target image samples corresponding to the image samples, so that a trained image acquisition model is obtained, which comprises: normalizing the pixel values in the image samples to obtain normalized images; determining the mean square error between the pixel values in the target image samples corresponding to the image samples and the pixel values in the normalized images; and updating the model parameters corresponding to the original image acquisition model according to the mean square error, so as to obtain the trained image acquisition model.

[0014] The application provides an image acquisition method, comprising:

[0015] acquiring a to-be-processed image;

[0016] inputting the to-be-processed image into an image acquisition model to obtain a target image output by the image acquisition model, wherein the image acquisition model is trained by the image acquisition model training method according to any one of the preceding embodiments.

[0017] The application further provides an image acquisition model training device, comprising:

[0018] an acquisition module configured to acquire a plurality of image samples;

[0019] a processing module configured to input the plurality of image samples into an original image acquisition model, perform window traversal on the image samples by the original image acquisition model to obtain a plurality of first images corresponding to each image sample, perform mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, and ensure that any two first images do not overlap; perform random sampling on the second image to obtain at least one third image corresponding to each image sample, and perform mask processing on the at least one third image according to a second mask rate, so that a target image sample corresponding to each image sample is output by the original image acquisition model; wherein the size of the third image is greater than the size of the first image; and the model parameters corresponding to the original image acquisition model are updated according to the plurality of image samples and the target image samples corresponding to the image samples, so that a trained image acquisition model is obtained.

[0020] The application further provides an image acquisition device, comprising:

[0021] an acquisition module configured to acquire a to-be-processed image;

[0022] a processing module configured to input the to-be-processed image into an image acquisition model to obtain a target image output by the image acquisition model, wherein the image acquisition model is trained by the image acquisition model training method according to any one of the preceding embodiments.

[0023] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the image acquisition model training method or the image acquisition method.

[0024] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the image acquisition model training method or the image acquisition method.

[0025] The application further provides a computer program product, which includes a computer program, and the computer program is executable on a processor to implement the image acquisition model training method or the image acquisition method.

[0026] The application provides an image acquisition model training method and an image acquisition method and device, the image acquisition model training method includes the following steps: acquiring a plurality of image samples; inputting the plurality of image samples into an original image acquisition model; traversing windows on the image samples by the original image acquisition model to obtain a plurality of first images corresponding to each image sample; performing mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, and any two first images do not overlap; performing random sampling on the second image to obtain at least one third image corresponding to each image sample; and performing mask processing on the at least one third image according to a second mask rate to output a target image sample corresponding to each image sample by the original image acquisition model; wherein the size of the third image is greater than the size of the first image; and updating model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample to obtain a trained image acquisition model. The plurality of image samples are subjected to twice local mask processing by the method to obtain more accurate image details, and the image acquisition model finally trained can effectively acquire a target image with high accuracy and high image quality. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0028] Figure 1 is a flowchart of the image acquisition model training method provided by the present application;

[0029] Figure 2 is a scene schematic diagram of the image acquisition model training method provided by the present application;

[0030] Figure 3 is a flow schematic diagram of the image acquisition method provided by the present application;

[0031] Figure 4a is a learning rate change curve of the image acquisition model in the pre-training stage on the ImageNet1K dataset provided by the present application;

[0032] Figure 4b is a loss change curve of the image acquisition model in the pre-training stage on the ImageNet1K dataset provided by the present application;

[0033] Figure 4c is a loss change curve of the image acquisition model in the fine tune stage on the test dataset provided by the present application;

[0034] Figure 4d is a change curve of the image acquisition model in the fine tune stage on the ImageNet1K dataset provided by the present application, in which the precision ranking on the test dataset is first;

[0035] Figure 4e is a change curve of the image acquisition model in the fine tune stage on the ImageNet1K dataset provided by the present application, in which the precision ranking on the test dataset is fifth;

[0036] Figure 5 is a structure schematic diagram of the image acquisition model training device provided by the present application;

[0037] Figure 6 is a structure schematic diagram of the image acquisition device provided by the present application;

[0038] Figure 7 is a structure schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0040] It should be noted that the execution subject involved in the embodiments of the present invention can be an image acquisition model training device or an electronic device. Optionally, the electronic device may include: a computer, a mobile terminal, and a wearable device, etc.

[0041] The embodiments of the present invention will be further described below using an electronic device as an example.

[0042] like Figure 1 The diagram shown is a flowchart illustrating the image acquisition model training method provided by this invention, which may include:

[0043] 101. Obtain multiple image samples.

[0044] Image samples refer to the image data required by electronic devices when training the original image acquisition model.

[0045] Optionally, the electronic device may acquire multiple image samples, which may include: the electronic device acquiring multiple image samples from a preset image dataset.

[0046] Optionally, the preset image dataset can be the ImageNet 1K (IN1K) dataset, where 1K indicates that the images in the ImageNet dataset have more than 1,000 categories.

[0047] 102. Input multiple image samples into the original image acquisition model. The original image acquisition model iterates through the image samples and draws windows to obtain multiple first images corresponding to each image sample. Then, the multiple first images are masked according to the first mask rate to obtain the second image corresponding to each image sample. Then, the second images are randomly sampled to obtain at least one third image corresponding to each image sample. The at least one third image is masked according to the second mask rate so that the target image sample corresponding to each image sample is output through the original image acquisition model.

[0048] In this system, no two first images overlap, and the size of the third image is larger than the size of the first image.

[0049] The original image acquisition model can be called a pre-trained model.

[0050] Optionally, the first image can be referred to as the local receptive field, and the first preset size corresponding to the first image is m*m, where m≥2, m is an integer, and the unit of m is a patch, that is, the first image corresponds to m*m patches.

[0051] The relationship between the image size and the local receptive field can be expressed by the formula N×W. S ≤H represents W. S Indicates the size of the image; RF≤W Swherein H denotes the size of the input image after padding, RF denotes the size of the local receptive field of the first mask operation, and N denotes the number of windows for the second mask sampling.

[0052] Optionally, the first preset size can be pre-set by the model or customized by the user, which is not limited here.

[0053] Optionally, the second image has the same size as the image sample.

[0054] Optionally, the second preset size corresponding to the third image is n*n, n>m, n is an integer, that is, the second preset size is larger than the first preset size.

[0055] Optionally, the second preset size is one of 5*5, 7*7, 9*9, 11*11 and 14*14.

[0056] Optionally, in the case where the number of third images is multiple, any two third images in the multiple third images can be overlapped or not overlapped, which is not limited here.

[0057] In step 102, the mask processing means that the electronic device sets the pixel value of the image sample to 0, and discards the image information of this part.

[0058] It should be noted that the mask processing of the plurality of first images by the electronic device according to the first mask rate can be understood as the first local mask processing of the plurality of first images by the electronic device, and the first local mask processing can retain image detail hints in the local receptive field.

[0059] The mask processing of the at least one third image by the electronic device according to the second mask rate can be understood as the second local mask processing of the at least one third image by the electronic device, and the second local mask processing can make the number and position of patches corresponding to each third image random, thereby taking into account the semantic features of multi-receptive field, increasing the difficulty of pixel recovery, and helping the original image acquisition model to learn more robust feature representation in the pre-training stage.

[0060] Optionally, the first mask rate and the second mask rate can be the same or different, and the first mask rate and the second mask rate can be pre-set by the model or customized by the user, which is not limited here.

[0061] In the pre-training stage, the electronic device can first acquire a plurality of image samples; then the electronic device inputs the plurality of image samples into the original image acquisition model, and performs non-overlapping division on the image samples through the original image acquisition model to obtain a plurality of first images corresponding to each image sample, so that any two first images in the plurality of first images corresponding to each image sample are non-overlapping.

[0062] Among them, the size of the image sample = the number of first images * the first preset size, that is, the plurality of first images obtained by the electronic device can constitute a complete image sample.

[0063] For example, after the electronic device acquires the image sample, it can first determine that the size corresponding to the image sample is 8*8; then the electronic device performs non-overlapping division on the image sample to obtain 16 first images with a first preset size of 2*2.

[0064] Then, the electronic device performs mask processing on the plurality of first images one by one according to the first mask rate in the original image acquisition model, that is, as many first images as mask processing operations, so that the second image corresponding to the image sample can be obtained.

[0065] For example, assuming that the first mask rate is 25%, the electronic device has acquired 16 first images A of 2*2, that is, each first image corresponds to 4 patches. The electronic device performs mask processing on the first image A1 based on the first mask rate in the original image acquisition model, that is, randomly selects one patch for masking among the 4 patches corresponding to the first image A1 to obtain a mask image B1; then the electronic device performs mask processing on the first image A2 based on the original image acquisition model according to the first mask rate to obtain a mask image B2, and so on until the first image A 16 corresponding mask image B 16 is acquired; then the electronic device determines the second image corresponding to the image sample according to the mask image B1, the mask image B2, …, and the mask image B 16 .

[0066] Among them, the position of the first image A1 in the image sample corresponds to the position of the mask image B1 in the second image, …, the position of the first image A 16 in the image sample corresponds to the position of the mask image B 16 in the second image, that is, the mask image B1, the mask image B2, …, and the mask image B 16 can constitute a complete second image.

[0067] Then, the electronic device randomly samples the second image to obtain at least one third image; if the number of the third images is one, the electronic device can directly perform mask processing on the third image according to the second mask rate to obtain the target image sample corresponding to each image sample; if the number of the third images is multiple, the electronic device can perform mask processing on the multiple third images one by one according to the second mask rate, that is, the number of times of mask processing operation is equal to the number of third images, and then the target image sample corresponding to each image sample is obtained.

[0068] For example, assuming that the second mask rate is 60%, the electronic device has obtained three third images C1 of 5*5, that is, each third image corresponds to 25 patches. The electronic device performs mask processing on the third image C1 based on the second mask rate in the original image acquisition model, that is, randomly selects 15 patches from the 25 patches corresponding to the third image C1 to perform mask processing to obtain a fourth image D1; then, the electronic device performs mask processing on the third image C2 and the third image C3 based on the original image acquisition model and the second mask rate to obtain a fourth image D2 and a fourth image D3, respectively, and then obtains the target image sample corresponding to each image sample.

[0069] Any two of the fourth image D1, the fourth image D2, and the fourth image D3 can be overlapped or not overlapped, which is not limited here.

[0070] Optionally, the electronic device randomly samples the second image to obtain at least one third image can include that the electronic device randomly samples the second image by using a sliding window of a second preset size to obtain at least one third image.

[0071] The sliding window is used to limit the size of the third image.

[0072] After obtaining the second image, the electronic device can first obtain a sliding window of a second preset size; then, the electronic device randomly samples the second image by using the sliding window to obtain at least one third image.

[0073] In some embodiments, the electronic device traverses the window on the image sample by using the original image acquisition model to obtain multiple first images corresponding to each image sample can include that the electronic device obtains the size of the image sample by using the original image acquisition model; the electronic device adjusts the size of the image sample to obtain an adjusted image sample in a case where the size of the image sample does not match a preset size threshold; and the electronic device traverses the window on the adjusted image sample to obtain multiple first images corresponding to each image sample in a case where the size of the adjusted image sample matches the preset size threshold.

[0074] Optionally, the preset size threshold can be pre-set by the model or customized by the user, which is not limited here.

[0075] After obtaining the image sample, the electronic device can first determine the size corresponding to the image sample; then, the electronic device compares the size corresponding to the image sample with the preset size threshold: if the size corresponding to the image sample matches the preset size threshold, the electronic device can directly traverse the window on the image sample to obtain a plurality of first images; if the size corresponding to the image sample does not match the preset size threshold, the electronic device needs to adjust the size corresponding to the image sample until the size of the adjusted image sample matches the preset size threshold, at which time the electronic device can traverse the window on the adjusted image sample to obtain a plurality of first images corresponding to each image sample.

[0076] For example, assuming that the preset size threshold is 8*8. The electronic device determines that the size corresponding to the image sample is 8*7 and does not match the preset size threshold 8*8, at which time the electronic device adjusts the size of the image sample to obtain an adjusted image sample; then, the electronic device traverses the window on the adjusted image sample when the size of the adjusted image sample is 8*8 and matches the preset size threshold 8*8 to obtain a plurality of first images corresponding to each image sample.

[0077] Optionally, when the size corresponding to the image sample does not match the preset size threshold, it means that the patch division and the image cannot be divided evenly, that is, H and W cannot be divided by patch_size, at which time the electronic device can perform an adaptive padding operation on the image sample, which can be optional padding up, down, left, or right, and the padding value is 0.

[0078] In some embodiments, the electronic device can mask at least one third image according to a second masking rate to obtain target image samples corresponding to each image sample through the original image acquisition model, which can include: the electronic device masks at least one third image according to a second masking rate to obtain fourth images corresponding to the third images; and the electronic device encodes and decodes the fourth images to obtain target image samples corresponding to each image sample.

[0079] The encoding process can also be referred to as the compression process, which refers to obtaining encoded features of image content by using a neural network method.

[0080] The decoding process can also be referred to as the decompression process, which refers to restoring the encoded features to the image content represented by the encoded features by using a preset neural network method.

[0081] The electronic device can first mask the at least one third image according to the second mask rate to obtain a fourth image corresponding to the third image, that is, the number of the fourth image is also at least one; then the electronic device inputs the at least one fourth image into the original image acquisition model; if the number of the fourth image is one, the electronic device can use the original image acquisition model to perform encoding processing and decoding processing on the fourth image to obtain a target image sample; if the number of the fourth image is multiple, the electronic device can use the original image acquisition model to perform encoding processing and decoding processing on the multiple fourth images one by one, that is, as many fourth images as there are, as many encoding processing operations and decoding processing operations will be performed, and then target image samples corresponding to each image sample are obtained.

[0082] In some embodiments, the electronic device performing encoding processing and decoding processing on the fourth image to obtain target image samples corresponding to each image sample can include: the electronic device using an encoder in the original image acquisition model to perform encoding processing on the fourth image to obtain implicit features corresponding to the fourth image; and the electronic device using a decoder in the original image acquisition model to perform decoding processing on the implicit features to obtain target image samples corresponding to each image sample.

[0083] The implicit features refer to semantic vectors corresponding to the fourth image, and the semantic vectors represent feature dimensions corresponding to the second image.

[0084] Optionally, the encoder can be an encoder with a vision transformer (ViT), and the decoder can be a decoder with a multi-layer perceptron (MLP).

[0085] After obtaining the fourth image, the electronic device can arrange all patches corresponding to the fourth image in a raster order to obtain a patch sequence, wherein the all patches include patches that have been processed by the second local masking and patches that have not been processed by the second local masking; then the electronic device inputs the patch sequence into an encoder in the original image acquisition model to obtain a semantic vector corresponding to the fourth image output by the encoder; then the electronic device inputs the semantic vector into a decoder in the original image acquisition model to obtain decoding features output by the decoder, and the electronic device determines target image samples corresponding to each image sample according to the decoding features.

[0086] Optionally, the raster order can be pre-set by the model or customized by the user, which is not limited here.

[0087] 103. updating the model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample, to obtain the trained image acquisition model.

[0088] After the electronic device obtains the target image sample corresponding to each image sample, since the size of the target image sample is different from the size of the image sample, the electronic device cannot directly update the model parameters corresponding to the original image acquisition model according to the target image sample and the image sample. Therefore, the electronic device first restores the target image sample to the corresponding position of the second image according to the position of the target image sample to obtain a restored image, that is, the restored image can include the image after the first mask processing and the image after the second mask processing, and the size of the restored image is the same as the size of the image sample. Then, the electronic device iteratively updates the model parameters corresponding to the original image acquisition model according to the restored image and the image sample, to obtain the trained image acquisition model, and the trained image acquisition model has a better image acquisition accuracy.

[0089] In some embodiments, the electronic device can update the model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample, to obtain the trained image acquisition model, which can include: the electronic device normalizes the pixel values in the image sample to obtain a normalized image; the electronic device determines the mean square error between the pixel values in the target image sample corresponding to each image sample and the pixel values in the normalized image; and the electronic device updates the model parameters corresponding to the original image acquisition model according to the mean square error, to obtain the trained image acquisition model.

[0090] Optionally, the preset interval can be pre-set by the model or customized by the user, which is not limited here.

[0091] The electronic device can first normalize the pixel values in the image sample to obtain a normalized image, and restore the target image sample to the corresponding position of the second image to obtain a restored image, and the size of the normalized image is the same as the size of the restored image. Then, the electronic device determines the mean square error between the pixel values in the restored image and the pixel values in the normalized image, that is, the mean square error between the pixel values at each position, which can be used as the loss function of the original image acquisition model. Then, the electronic device iteratively updates the model parameters corresponding to the original image acquisition model according to the mean square error, to obtain the trained image acquisition model.

[0092] In summary, for example, Figure 2As shown, it is a scene schematic diagram of the image acquisition model training method provided by the application, which can include: an electronic device traverses a window on an obtained 8*8 image sample through an original image acquisition model to obtain 16 2*2 first images; then, the electronic device performs first mask processing on each first image according to a first mask rate of 25%, to obtain a 8*8 second image corresponding to the image sample, and performs random sampling on the second image to obtain a 5*5 third image; then, the electronic device performs second mask processing on the third image according to a second mask rate of 60% to obtain a 5*5 fourth image; after that, the electronic device arranges all patches in the fourth image according to raster order to obtain a patch sequence; then, the electronic device inputs the patch sequence into an encoder in the original image acquisition model to obtain implicit features output by the encoder corresponding to the fourth image, and inputs the implicit features into a decoder in the original image acquisition model to obtain decoding features output by the decoder; then, the electronic device determines a target image sample corresponding to the image sample according to the decoding features; finally, the electronic device updates model parameters corresponding to the original image acquisition model according to the target image sample and the image sample to obtain a trained image acquisition model.

[0093] In the embodiment of the application, a plurality of image samples are obtained; the plurality of image samples are input into an original image acquisition model, the image samples are traversed by windows through the original image acquisition model to obtain a plurality of first images corresponding to each image sample, and the plurality of first images are subjected to mask processing according to a first mask rate to obtain second images corresponding to each image sample; the second images are subjected to random sampling to obtain at least one third image corresponding to each image sample, and the at least one third image is subjected to mask processing according to a second mask rate to output target image samples corresponding to each image sample through the original image acquisition model; model parameters corresponding to the original image acquisition model are updated according to the plurality of image samples and the target image samples corresponding to each image sample to obtain a trained image acquisition model. Through this method, the plurality of obtained image samples can be subjected to twice local mask processing to obtain more accurate image details, and then through encoding processing and mask processing, the image can be better restored, and thus the image acquisition model finally trained can effectively obtain target images with higher accuracy, and the image quality of the target images is better.

[0094] It should be noted that the execution subject involved in the embodiment of the application can be an image acquisition model training device or an electronic device.

[0095] The embodiments of the application will be further described below with the electronic device as an example.

[0096] As Figure 3 shown is a flowchart of an image acquisition method provided by the present application, which can include:

[0097] 301. Obtain a to-be-processed image.

[0098] Optionally, the to-be-processed image can include biological (for example, cats, dogs, trees, etc.) and / or non-biological (for example, stones, high-rise buildings, etc.) and the like.

[0099] Optionally, the electronic device can obtain the to-be-processed image, which can include that the electronic device can obtain the to-be-processed image by using a camera.

[0100] 302. Input the to-be-processed image into an image acquisition model to obtain a target image output by the image acquisition model.

[0101] The image acquisition model is obtained by training an image acquisition model training method as Figure 1 shown.

[0102] After obtaining the to-be-processed image, the electronic device can directly input the to-be-processed image into the trained image acquisition model to obtain a target image output by the trained image acquisition model, and the image quality of the target image is better than that of the to-be-processed image.

[0103] Optionally, after step 302, the method can further include that the electronic device performs fine-tuning processing on the target image to obtain a classification result.

[0104] The fine-tuning processing can correspond to a fine-tuning model, and the fine-tuning model is an encoder part + a linear classification layer part of the pretrain model.

[0105] The fine-tuning model structure can include an encoder and a logic layer of the pretrained model. The encoder is generally the same as the encoder corresponding to the pretrained model, and the logic layer can generally include MLP or a single-layer Linear.

[0106] In the fine-tuning stage, part of the weights (encoder) of the pretrained model can be used as the initial training weights of the fine-tuning model for training.

[0107] Optionally, after the electronic device pre-trains the model on the ImageNet1K dataset, the electronic device can perform fine tuning on the ImageNet1K dataset to perform a target classification task. During the entire fine tuning process, the electronic device can provide all patches of images in the ImageNet1K dataset to the fine tuning model, and use the normalized feature mean of the images as the final representation before classification.

[0108] As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR).

[0109] Table 1:

[0110]

[0111] As can be seen from Table 1, the classification results corresponding to the image acquisition model involved in the embodiments of the present application are higher. In addition, compared with the MAE model, the time complexity is reduced, and the reconstruction efficiency and accuracy are balanced; compared with the LoMaR model, the semantic features of multiple receptive fields are taken into account, although the difficulty of pixel recovery is increased, but the representation learning is effectively promoted.

[0112] As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR). Figure 4a

[0113] As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR).

[0114] As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR). Figure 4b As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR).

[0115] Figure 4c As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR).

[0116] As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR). Figure 4d As shown in Table 1, the classification results of different image acquisition models provided by the present application on the ImageNet1K dataset are shown. The different image acquisition models can include: an original image acquisition model without pre-training (NoPretraining), a pre-trained image acquisition model (Ours), an MAE model (MAE B / 16), and a LoMaR model (LoMaR).

[0117] ​​Wherein, the accuracy ranking first can be expressed by Top-1 accuracy.

[0118] As shown in Figure 4e Fig. 5 is a variation curve of the accuracy ranking fifth on the test data set in the fine tune stage of the image acquisition model provided by the application on the ImageNet1K data set.

[0119] Wherein, the accuracy ranking fifth can be expressed by Top-5 accuracy.

[0120] From Figures 4a to 4e It can be seen that the image acquisition model training method involved in the embodiment of the application still has the ability to continue learning when the pre-training iteration number (epoch) is 390. In addition, in the case of appropriate increase in the pre-training iteration number, the fine tune stage will also obtain better classification performance.

[0121] In the embodiment of the application, the image to be processed is obtained; the image to be processed is input into the image acquisition model to obtain the target image output by the image acquisition model. Through this method, the obtained image to be processed can be input into the trained image acquisition model, so that the trained image acquisition model can accurately determine the target image, improve the accuracy of the target image, that is, the image quality of the target image is better.

[0122] The image acquisition model training device provided by the application is described below, and the image acquisition model training device described below can be correspondingly referred to the image acquisition model training method described above.

[0123] As shown in Figure 5 Fig. 5 is a structure diagram of the image acquisition model training device provided by the application, which can include:

[0124] The acquisition module 501 is configured to acquire a plurality of image samples.

[0125] The processing module 502 is configured to input the plurality of image samples into an original image acquisition model, traverse the image samples by the original image acquisition model to obtain a plurality of first images corresponding to each image sample, perform mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, and the first images are not overlapped; randomly sampling the second image to obtain at least one third image corresponding to each image sample, and performing mask processing on the at least one third image according to a second mask rate to output a target image sample corresponding to each image sample by the original image acquisition model; wherein the size of the third image is greater than the size of the first image; and updating the model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample to obtain a trained image acquisition model.

[0126] Optionally, the processing module 502 is specifically configured to perform mask processing on the at least one third image according to the second mask rate to obtain a fourth image corresponding to the third image; and perform encoding processing and decoding processing on the fourth image to obtain the target image sample corresponding to each image sample.

[0127] Optionally, the acquisition module 501 is specifically configured to acquire the size of the image sample by the original image acquisition model.

[0128] The processing module 502 is specifically configured to adjust the size of the image sample to obtain an adjusted image sample in a case where the size of the image sample does not match a preset size threshold; and traverse the adjusted image sample to obtain a plurality of first images corresponding to each image sample in a case where the size of the adjusted image sample matches the preset size threshold.

[0129] Optionally, the processing module 502 is specifically configured to perform encoding processing on the fourth image by using an encoder in the original image acquisition model to obtain implicit features corresponding to the fourth image; and perform decoding processing on the implicit features by using a decoder in the original image acquisition model to obtain the target image sample corresponding to each image sample.

[0130] Optionally, the processing module 502 is specifically configured to perform normalization processing on pixel values in the image sample to obtain a normalized image; determine a mean square error between the pixel values in the target image sample corresponding to each image sample and the pixel values in the normalized image; and update the model parameters corresponding to the original image acquisition model according to the mean square error to obtain the trained image acquisition model.

[0131] The image acquisition device provided by the application is described below. The image acquisition device described below can be referred to in conjunction with the image acquisition method described above.

[0132] As Figure 6 shown, it is a structural schematic diagram of an image acquisition device provided by the application, which can include:

[0133] The acquisition module 601 is configured to acquire a to-be-processed image.

[0134] The processing module 602 is configured to input the to-be-processed image into an image acquisition model to obtain a target image output by the image acquisition model, wherein the image acquisition model is obtained by training an image acquisition model training method as Figure 1 shown.

[0135] As Figure 7 shown, it is a structural schematic diagram of an electronic device provided by the application, which can include a processor 710, a communications interface 720, a memory 730 and a communications bus 740, wherein the processor 710, the communications interface 720 and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can invoke a logical instruction in the memory 730 to execute an image acquisition model training method, which includes: acquiring a plurality of image samples; inputting the plurality of image samples into an original image acquisition model, traversing a window for the image samples through the original image acquisition model to obtain a plurality of first images corresponding to each image sample, and performing mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, wherein any two first images do not overlap; then randomly sampling the second image to obtain at least one third image corresponding to each image sample, and performing mask processing on the at least one third image according to a second mask rate to output a target image sample corresponding to each image sample through the original image acquisition model; wherein the size of the third image is greater than the size of the first image; and updating model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample to obtain a trained image acquisition model. Or, the method is executed to perform an image acquisition method, which includes: acquiring a to-be-processed image; inputting the to-be-processed image into an image acquisition model to obtain a target image output by the image acquisition model, wherein the image acquisition model is obtained by training the image acquisition model training method as described above.

[0136] Further, the logic instructions in the memory 730 described above can be implemented in the form of software functional units and sold or used as standalone products, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partially contribute to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0137] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the image acquisition model training method provided by the above-mentioned methods. The method comprises: acquiring a plurality of image samples; inputting the plurality of image samples into an original image acquisition model, traversing the window for the image samples by the original image acquisition model, obtaining a plurality of first images corresponding to each image sample, and performing mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, any two first images do not overlap; then randomly sampling the second image to obtain at least one third image corresponding to each image sample, and performing mask processing on the at least one third image according to a second mask rate, so as to output a target image sample corresponding to each image sample by the original image acquisition model; wherein the size of the third image is greater than the size of the first image; updating the model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image samples corresponding to each image sample, so as to obtain a trained image acquisition model. Or, to execute an image acquisition method, the method comprises: acquiring a to-be-processed image; inputting the to-be-processed image into an image acquisition model to obtain a target image output by the image acquisition model, the image acquisition model being trained by the image acquisition model training method described above.

[0138] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the image acquisition model training method provided by any of the above methods, the method comprising: obtaining a plurality of image samples; inputting the plurality of image samples into an original image acquisition model, traversing the image samples by the original image acquisition model to obtain a plurality of first images corresponding to each image sample, and performing mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, any two first images being non-overlapping; randomly sampling the second image to obtain at least one third image corresponding to each image sample, and performing mask processing on the at least one third image according to a second mask rate to output a target image sample corresponding to each image sample by the original image acquisition model; wherein the size of the third image is greater than the size of the first image; and updating model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample to obtain a trained image acquisition model. Or, to execute an image acquisition method, the method comprising: obtaining an image to be processed; inputting the image to be processed into an image acquisition model to obtain a target image output by the image acquisition model, the image acquisition model being trained by the image acquisition model training method as described above.

[0139] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0140] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary universal hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0141] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an image acquisition model, characterized in that, The method comprises the following steps: obtaining a plurality of image samples; inputting the plurality of image samples into an original image acquisition model, traversing a window on the image samples by the original image acquisition model to obtain a plurality of first images corresponding to each image sample, and performing mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, wherein any two first images do not overlap; then performing random sampling on the second image to obtain at least one third image corresponding to each image sample, and performing mask processing on the at least one third image according to a second mask rate to output a target image sample corresponding to each image sample by the original image acquisition model; wherein the size of the third image is larger than the size of the first image; updating model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image samples corresponding to each image sample to obtain a trained image acquisition model; the mask processing on the at least one third image according to the second mask rate to output the target image sample corresponding to each image sample by the original image acquisition model comprises: mask processing on the at least one third image according to the second mask rate to obtain a fourth image corresponding to the third image; encoding and decoding processing on the fourth image to obtain the target image sample corresponding to each image sample.

2. The method of claim 1, wherein, the traversing a window on the image samples by the original image acquisition model to obtain a plurality of first images corresponding to each image sample comprises: obtaining the size of the image sample by the original image acquisition model; in the case that the size of the image sample does not match a preset size threshold, adjusting the size of the image sample to obtain an adjusted image sample; in the case that the size of the adjusted image sample matches the preset size threshold, traversing a window on the adjusted image sample to obtain a plurality of first images corresponding to each image sample.

3. The method of claim 1, wherein, the encoding and decoding processing on the fourth image to obtain the target image sample corresponding to each image sample comprises: encoding processing on the fourth image by an encoder in the original image acquisition model to obtain implicit features corresponding to the fourth image; decoding processing on the implicit features by a decoder in the original image acquisition model to obtain the target image sample corresponding to each image sample.

4. The method according to any one of claims 1 to 3, characterized in that, the updating of the model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image samples corresponding to each image sample to obtain a trained image acquisition model comprises: normalizing pixel values in the image samples to obtain a normalized image; determining a mean square error between pixel values in the target image samples corresponding to each image sample and pixel values in the normalized image; updating the model parameters corresponding to the original image acquisition model according to the mean square error to obtain a trained image acquisition model.

5. An image acquisition method characterized by, The method comprises the following steps: obtaining a to-be-processed image; inputting the image to be processed into an image acquisition model to obtain a target image output by the image acquisition model, wherein the image acquisition model is trained by the image acquisition model training method in any one of claims 1-4.

6. An image acquisition model training apparatus, characterized by comprising: The method comprises: an acquisition module configured to acquire a plurality of image samples; a processing module configured to input the plurality of image samples into an original image acquisition model, traverse a window for the image samples by the original image acquisition model to obtain a plurality of first images corresponding to each image sample, perform mask processing on the plurality of first images according to a first mask rate to obtain a second image corresponding to each image sample, and randomly sample the second image to obtain at least one third image corresponding to each image sample, and perform mask processing on the at least one third image according to a second mask rate to output a target image sample corresponding to each image sample by the original image acquisition model; wherein the size of the third image is greater than the size of the first image; and updating model parameters corresponding to the original image acquisition model according to the plurality of image samples and the target image sample corresponding to each image sample to obtain a trained image acquisition model. The mask processing on the at least one third image according to the second mask rate to output the target image sample corresponding to each image sample by the original image acquisition model comprises: performing mask processing on the at least one third image according to the second mask rate to obtain a fourth image corresponding to the third image; performing encoding processing and decoding processing on the fourth image to obtain the target image sample corresponding to each image sample.

7. An image acquisition device, characterized in that The method comprises: an acquisition module configured to acquire an image to be processed; a processing module configured to input the image to be processed into an image acquisition model to obtain a target image output by the image acquisition model, wherein the image acquisition model is trained by the image acquisition model training method in any one of claims 1-4.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the image acquisition model training method in any one of claims 1-4 or the image acquisition method in claim 5 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the image acquisition model training method in any one of claims 1-4 or the image acquisition method in claim 5 when executed by the processor.

Images