An image processing method, device, server and storage medium

Abstract

The present disclosure relates to an image processing method, apparatus, server, and storage medium. The method comprises: obtaining a first block image and a mask image corresponding to a second block image, where the first block image is obtained by cropping an original image, and the second block image is the image of the cropped image excluding the first block image; inputting the first block image and the mask image into an image prediction model, and outputting a first predicted image of the first block image and a second predicted image of the second block image through the image prediction model. Through the embodiments of the present disclosure, it is possible to accurately restore missing portions of the original image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to an image processing method and device, a server, and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence technology, pre-training models in the image field have emerged. A pre-training model includes a model irrelevant to a specific task obtained from large-scale data using a machine learning training method, and the pre-training model is fine-tuned in combination with a specific task in the later stage, so as to obtain a model with multiple task functions. Therefore, pre-training can reduce the difficulty of model training and improve the accuracy of model training.

[0003] In related technologies, one method uses a supervised training method to obtain a pre-training model in the image field. The supervised training relies on artificially annotated data, and the cost is high. Another method constructs a pre-training model in a self-supervised manner, inputs two image blocks, and outputs whether the two image blocks are the same image. The pre-training model is trained according to the corresponding relationship between the input and the output. The expression of the image features obtained by this method is not comprehensive enough. Therefore, the accuracy of other task models obtained by fine-tuning using the pre-training model is not high. SUMMARY

[0004] The present disclosure provides an image processing method, device, server, and storage medium to at least solve the problem of insufficient comprehensive expression of image features of a self-supervised pre-training model in related technologies. The technical solutions of the present disclosure are as follows.

[0005] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, including:

[0006] obtaining a mask image corresponding to a first sub-block image and a second sub-block image, the first sub-block image being obtained by cropping an original image, and the second sub-block image being an image other than the first sub-block image obtained by cropping;

[0007] inputting the first sub-block image and the mask image into an image prediction model, and outputting a first prediction image of the first sub-block image and a second prediction image of the second sub-block image through the image prediction model.

[0008] In a possible implementation manner, the image prediction model is obtained in the following manner:

[0009] obtaining a sample set, the sample set including a first sub-block image sample, a second sub-block image sample, and a mask image sample corresponding to the second sub-block image sample;

[0010] constructing an image prediction model, the image prediction model being provided with training parameters;

[0011] inputting the first sub-block image sample and the mask image sample into the image prediction model to generate a prediction result;

[0012] iteratively adjusting the training parameters based on a difference between the prediction result and the first sub-block image sample and the second sub-block image sample until the difference meets a preset requirement, to obtain the image prediction model.

[0013] In a possible implementation, the image prediction model comprises a self-attention mechanism network, and the inputting the first sub-block image sample and the mask image sample into the image prediction model to generate a prediction result comprises:

[0014] extracting features of the first sub-block image sample and features of the mask image sample respectively to obtain first input features and second input features;

[0015] inputting the first input features and the second input features into the self-attention mechanism network, and outputting first output features and second output features through the self-attention mechanism network;

[0016] determining first predicted images and second predicted images respectively according to the first output features and the second output features.

[0017] In a possible implementation, the extracting features of the first sub-block image sample and features of the mask image sample respectively to obtain first input features and second input features comprises:

[0018] extracting features of the first sub-block image sample to obtain first input features;

[0019] setting features of the mask image sample as preset values to obtain second input features.

[0020] In a possible implementation, the extracting features of the first sub-block image sample and features of the mask image sample respectively to obtain first input features and second input features comprises:

[0021] extracting features of the first sub-block image sample and features of the mask image sample respectively by using a convolutional neural network to obtain first input features and second input features.

[0022] In a possible implementation, the determining first predicted images and second predicted images respectively according to the first output features and the second output features comprises:

[0023] inputting the first output features and the second output features into a generation network, and outputting first predicted images and second predicted images through the generation network.

[0024] In a possible implementation, the extracting the feature of the first patch image sample and the feature of the mask image sample respectively to obtain the first input feature and the second input feature comprises:

[0025] The first patch image sample and the mask image sample are linearly mapped by using a mapping matrix to obtain the first input feature and the second input feature.

[0026] In a possible implementation, the determining the first prediction image and the second prediction image according to the first output feature and the second output feature respectively comprises:

[0027] The first output feature and the second output feature are linearly inversely mapped by using the mapping matrix to obtain the first prediction image and the second prediction image.

[0028] In a possible implementation, the inputting the first input feature and the second input feature into the self-attention mechanism network and outputting the first output feature and the second output feature through the self-attention mechanism network comprises:

[0029] A first position vector of the first patch image sample and a second position vector of the second patch image sample are obtained.

[0030] The first input feature and the first position vector and the second input feature and the second position vector are spliced respectively to obtain a first spliced feature and a second spliced feature.

[0031] The first spliced feature and the second spliced feature are input into the self-attention mechanism network, and the first output feature and the second output feature are output through the self-attention mechanism network.

[0032] In a possible implementation, the iteratively adjusting the training parameter based on the difference between the prediction result and the first patch image sample and the second patch image sample until the difference meets a preset requirement to obtain the image prediction model comprises:

[0033] The training parameter is iteratively adjusted based on the difference between the first prediction image and the first patch image sample and / or based on the difference between the second prediction image and the second patch image sample until the difference meets a preset requirement to obtain the image prediction model.

[0034] According to a second aspect of the embodiments of the present disclosure, an image processing apparatus is provided, comprising:

[0035] an acquisition module configured to acquire a mask image corresponding to a first patch image and a second patch image, the first patch image being obtained by cropping an original image, and the second patch image being an image other than the first patch image obtained by cropping the original image;

[0036] a prediction module configured to input the first patch image and the mask image into an image prediction model, and output a first prediction image of the first patch image and a second prediction image of the second patch image through the image prediction model.

[0037] According to a third aspect of the embodiments of the present disclosure, a task model is provided, which is trained by using the correspondence between the first output feature and / or the second output feature and the task type according to any one of the embodiments of the present disclosure.

[0038] According to a fourth aspect of the embodiments of the present disclosure, a server is provided, comprising:

[0039] a processor;

[0040] a memory for storing instructions executable by the processor;

[0041] The processor is configured to execute the instructions to implement the image processing method according to any one of the embodiments of the present disclosure.

[0042] According to a fifth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, when the instructions in the computer readable storage medium are executed by the processor of the server, the server can execute the image processing method according to any one of the embodiments of the present disclosure.

[0043] According to a sixth aspect of the embodiments of the present disclosure, a computer program product is provided, the computer program product comprising instructions, wherein the instructions are executed by the processor of the server, so that the server can execute the image processing method according to any one of the embodiments of the present disclosure.

[0044] In the embodiments of the present disclosure, the image prediction model is trained by using the cropped first patch image sample and the mask image sample of the second patch image of the original image, and in the training process, the context information (i.e. the first patch image sample information) of the second patch image can be used to generate the second prediction image of the second patch image, so that the prediction result, i.e. the second prediction image, is closer to the second patch image. Through the embodiments of the present disclosure, the missing part of the original image can be accurately recovered.

[0045] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the disclosure without imposing on the disclosure limitations that are inconsistent with the principles of the disclosure.

[0047] Figure 1 is a flowchart of an image prediction model training method according to an example embodiment.

[0048] Figure 2 is a flowchart of an image processing method according to an example embodiment.

[0049] Figure 3 is a block diagram of an image processing apparatus according to an example embodiment.

[0050] Figure 4 is a block diagram of a server according to an example embodiment. DETAILED DESCRIPTION

[0051] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0053] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, analyzed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties.

[0054] Figure 1 is a flowchart of an image prediction model training method according to an example embodiment, with reference to Figure 1The original image 101 is cropped to obtain patch images of the same size. Some patch images are randomly selected from the patch images for mask processing to obtain mask images. The remaining images that are not processed by the mask are referred to as first patch images, and the patch images processed by the mask are referred to as second patch images. Feature extraction is performed on the first patch images and the mask images respectively to obtain input features. The input features are spliced with position vectors and input into a self-attention mechanism network 103. Output features are obtained through the self-attention mechanism network. The output features are input into a generation network, and predicted images are obtained through the generation network. Based on the difference between the predicted images and the patch images, the parameters in the self-attention mechanism network are iteratively adjusted until the difference meets the preset requirements.

[0055] Figure 2 is a flowchart of an image processing method according to an example embodiment. Referring to Figure 2 An image processing method is shown in FIG. 1. The image processing method includes:

[0056] In step S201, a first patch image and a mask image corresponding to a second patch image are obtained. The first patch image is obtained by cropping an original image, and the second patch image is an image obtained by cropping the original image except the first patch image.

[0057] In the embodiments of the present disclosure, the mask image of the second patch image includes an image obtained by performing mask processing on the second patch image, for example, setting the value of the mask image to the same pixel value, so that it appears black or white. In another example, an image with all the same pixel values and the same size as the second patch image can also be used as the mask image of the second patch image.

[0058] In the embodiments of the present disclosure, the first patch image and the second patch image are derived from the same original image. The original image is cropped to obtain a plurality of patch images, and a part of the patch images is randomly selected as the second patch image, and the remaining patch images are used as the first patch image.

[0059] In step S203, the first patch image and the mask image are input into an image prediction model, and a first predicted image of the first patch image and a second predicted image of the second patch image are output through the image prediction model.

[0060] In the embodiments of the present disclosure, the image prediction model is set to be obtained according to the correspondence between the first prediction image and the first patch image sample, and the correspondence between the second prediction image and the mask image sample. The image prediction model can include various artificial neural network models, such as an LSTM network (long short-term memory artificial neural network), a Transformer network (self-attention mechanism network), and an improved network based on the LSTM network and the Transformer network. The image prediction model disclosed in the embodiments of the present disclosure can be applied in various application scenarios, for example, when part of the image in the original image is missing or blurred, the image prediction model disclosed in the present disclosure can be used to predict the missing part or the blurred part. Further, if the size of the first patch image and the second patch image is set to be small enough, for example, 1 pixel, the pixel value of the original image can be predicted by the image prediction model disclosed in the present disclosure.

[0061] In the embodiments of the present disclosure, the image prediction model is trained by using the cropped first patch image sample and the mask image sample of the second patch image of the original image. During training, the context information (i.e., the first patch image sample information) of the second patch image can be used to generate the second prediction image of the second patch image, so that the prediction result, the second prediction image, is closer to the second patch image. Through the embodiments of the present disclosure, accurate recovery of the missing part of the original image can be achieved.

[0062] In a possible implementation, the image prediction model is obtained in the following manner:

[0063] A sample set is obtained, which includes a first patch image sample, a second patch image sample, and a mask image sample corresponding to the second patch image sample;

[0064] An image prediction model is constructed, and the image prediction model is provided with training parameters;

[0065] The first patch image sample and the mask image sample are input into the image prediction model to generate a prediction result;

[0066] The training parameters are iteratively adjusted based on the difference between the prediction result and the second patch image sample until the difference meets a preset requirement, and the image prediction model is obtained.

[0067] In the embodiments of the present disclosure, any picture sample can be cropped to obtain a patch image sample. Some patch image samples are randomly selected from the patch image sample as second patch image samples, and the remaining patch image samples are used as first patch image samples. The second patch image samples are subjected to mask processing to obtain mask image samples of the second patch image samples.

[0068] In the embodiments of the present disclosure, the image prediction model can include an LSTM network model, a Transformer network, etc. In one example, the mask image sample corresponding to the first sub-block image sample and the second sub-block image sample in the same image sample is sequentially input into the image prediction model to generate a prediction result. In one example, the first difference between the prediction image corresponding to the mask image sample and the second sub-block image sample can be used as a loss function to iteratively adjust the training parameters. The first difference can also be combined with the second difference between the prediction image corresponding to the first sub-block image and the first sub-block image to be used as a loss function to iteratively adjust the training parameters. The first difference and the second difference can also be set with a weight ratio to be used as a loss function to iteratively adjust the training parameters.

[0069] In the embodiments of the present disclosure, the image prediction model can randomly select any picture during the training process without manual labeling of the picture, and belongs to a self-supervised training model. The training method of the present disclosure improves the convenience of model training and reduces the cost of model training.

[0070] In one possible implementation, the image prediction model includes a self-attention mechanism network, and the input of the first sub-block image sample and the mask image sample into the image prediction model to generate a prediction result includes:

[0071] The features of the first sub-block image sample and the features of the mask image sample are extracted respectively to obtain first input features and second input features;

[0072] The first input features and the second input features are input into the self-attention mechanism network, and the first output features and the second output features are output through the self-attention mechanism network;

[0073] The first prediction image and the second prediction image are determined according to the first output features and the second output features respectively.

[0074] In one example, the features of the first sub-block image sample and the features of the mask image sample can be extracted using a convolutional neural network (CNN) in the embodiments of the present disclosure. In another example, the feature vectors corresponding to the first sub-block image sample and the second sub-block image sample can be obtained in a linear mapping manner, and the corresponding feature vectors are used as the first input features and the second input features.

[0075] In the embodiments of the present disclosure, the self-attention mechanism network (Transformer network) is a variant model of a recurrent neural network, which is originally used for processing machine translation tasks. The self-attention mechanism network is composed of an encoder and a decoder, and the encoder and the decoder are stacked by a plurality of encoding modules and decoding modules. The encoding modules and the decoding modules are connected by different ways through basic layers such as direct connection, multi-head attention (Multi-Head Attention) and full connection. The multi-head attention includes a self-attention layer (Self-Attention), which can automatically use the feature information of the previous and subsequent block images when encoding the current block image, so that the output feature can better express the current block image.

[0076] In the embodiments of the present disclosure, the first input feature corresponds to the first output feature, and the second input feature corresponds to the second output feature. The first block image corresponds to the first predicted image, and the second block image corresponds to the second predicted image.

[0077] In the embodiments of the present disclosure, the self-attention mechanism network (Transformer network) can automatically obtain the context information of the input features, including the first input feature and the second input feature, so as to automatically predict the second output feature in the case that the second input feature has no information amount. After multiple iterations of training, the second output feature can accurately express the content of the second block image sample.

[0078] In a possible implementation, the extracting the features of the first block image sample and the features of the mask image sample respectively to obtain the first input feature and the second input feature comprises:

[0079] The features of the first block image sample are extracted to obtain the first input feature.

[0080] The features of the mask image sample are set to a preset value to obtain the second input feature.

[0081] In the embodiments of the present disclosure, the features of the first block image sample can be extracted by using a convolutional neural network (CNN). In another example, the feature vector of the first block image sample can be obtained in a linear mapping manner as the first input feature. The features of the mask image sample are set to a preset value to obtain the second input feature.

[0082] In the embodiments of the present disclosure, the mask image sample without information amount is set to a preset value, which can reduce the calculation amount of feature extraction of the mask image sample and improve the training efficiency of the model.

[0083] The extracting the features of the first sub-block image sample and the features of the mask image sample respectively to obtain first input features and second input features comprises:

[0084] The features of the first sub-block image sample and the features of the mask image sample are extracted respectively by using a convolutional neural network to obtain first input features and second input features.

[0085] In the embodiment of the disclosure, the convolutional neural network can include a CNN network, a LeNet network, an AlexNet network, a VggNet network, a ResNet network, etc. The convolutional neural network can extract some preliminary features in an image, similar to a filter. After a large amount of training, the model can automatically adjust the value of the convolution kernel, and then perform convolution with the image to extract the features in the image.

[0086] In the embodiment of the disclosure, the first input features are obtained by extracting the features of the first sub-block image sample by using the convolutional neural network, and the second input features are obtained by extracting the features of the second sub-block image sample, which can accurately and efficiently obtain the input features of the image sample.

[0087] In a possible implementation, the determining the first predicted image and the second predicted image according to the first output features and the second output features respectively comprises:

[0088] The first output features and the second output features are input into a generative network, and the first predicted image and the second predicted image are output through the generative network.

[0089] In the embodiment of the disclosure, after the features of the first sub-block image sample and the mask image sample are extracted by using the convolutional neural network, the output features can be converted into a predicted image through a generative network (Generative Adversarial Networks, GAN). In the embodiment of the disclosure, the generative network can also include a variant network model thereof, such as a DCGAN network, a W-GAN network, a LS-GAN network, an EB-GAN network, a DC-GAN network, a VAE-GAN network, etc. The generative network includes two sub-network models, one is a generative network model, which is used to generate a predicted image as consistent as possible with the sub-block image, and the other is a discriminative network model, which is used to make a correct judgment between the generated predicted image and the sub-block image.

[0090] Through the embodiment of the disclosure, the first predicted image and the second predicted image can be accurately obtained.

[0091] In a possible implementation, the extracting the features of the first sub-block image sample and the features of the mask image sample respectively to obtain first input features and second input features comprises:

[0092] linearly mapping the first patch image sample and the mask image sample by using a mapping matrix to obtain a first input feature and a second input feature.

[0093] In the process of linearly mapping the first patch image sample by using a mapping matrix to obtain the first input feature in the embodiments of the present disclosure, the following steps are included: for the first patch image with a size of, for example, N x N, the first patch image is first flattened into a vector of 1 x N, and then dimension reduction is performed by using a preset mapping matrix with a size of, for example, N x M, to obtain a feature vector of 1 x M, and the first input feature is obtained. 2 2 In the process of linearly mapping the first patch image sample by using a mapping matrix to obtain the first input feature in the embodiments of the present disclosure, the following steps are included: for the first patch image with a size of, for example, N x N, the first patch image is first flattened into a vector of 1 x N, and then dimension reduction is performed by using a preset mapping matrix with a size of, for example, N x M, to obtain a feature vector of 1 x M, and the first input feature is obtained.

[0094] In the embodiments of the present disclosure, the first input feature and the second input feature can be quickly and conveniently obtained.

[0095] In a possible implementation manner, the determining the first predicted image and the second predicted image according to the first output feature and the second output feature respectively includes:

[0096] linearly mapping the first output feature and the second output feature by using the mapping matrix to obtain the first predicted image and the second predicted image.

[0097] In the embodiments of the present disclosure, in the case of determining the first input feature and the second input feature by using the mapping matrix, the first output feature and the second output feature can be quickly and conveniently linearly mapped by using the mapping matrix to obtain the first predicted image and the second predicted image.

[0098] In a possible implementation manner, the inputting the first input feature and the second input feature into the self-attention mechanism network and outputting the first output feature and the second output feature through the self-attention mechanism network includes:

[0099] obtaining a first position vector of the first patch image sample and a second position vector of the second patch image sample;

[0100] splicing the first input feature and the first position vector and the second input feature and the second position vector respectively to obtain a first spliced feature and a second spliced feature;

[0101] inputting the first spliced feature and the second spliced feature into the self-attention mechanism network and outputting the first output feature and the second output feature through the self-attention mechanism network.

[0102] ​In the embodiments of the present disclosure, the first position vector of the first sub-block image sample is obtained, and the specific process includes: the following formula (1) and formula (2) are used for the sine-cosine position encoding of the horizontal coordinate (X coordinate) of the first sub-block image.

[0103]

[0104]

[0105] The following formula (3) and formula (4) are used for the sine-cosine position encoding of the vertical coordinate (Y coordinate) of the first sub-block image.

[0106]

[0107]

[0108] wherein PE_x and PE_y are two-dimensional matrices, the matrix size is consistent with the dimension of the first input feature of the first sub-block image sample, x represents the position encoding of the first sub-block image sample on the horizontal coordinate axis in the original image, y represents the position encoding of the first sub-block image sample on the vertical coordinate axis in the original image, d model represents the dimension of the first input feature. 2i and 2i+1 represent the positions on the corresponding feature dimension. In the embodiments of the present disclosure, the process of obtaining the position vector of the mask image sample is the same as the process of obtaining the position vector of the first sub-block image sample, which will not be described here.

[0109] In the embodiments of the present disclosure, the first input feature and the first position vector, and the second input feature and the second position vector are spliced respectively to obtain the first spliced feature and the second spliced feature; the first spliced feature and the second spliced feature are input into the self-attention mechanism network, and the first output feature and the second output feature are output through the self-attention mechanism network. By inputting the position vector of the first sub-block image sample and the position vector of the mask image sample, the image prediction model can obtain more position information, so that the output feature can more comprehensively express the corresponding sub-block image.

[0110] In a possible implementation, the training parameters are iteratively adjusted based on the difference between the prediction result and the first sub-block image sample and the second sub-block image sample until the difference meets a preset requirement, and the image prediction model is obtained, including:

[0111] The training parameters are iteratively adjusted based on the difference between the first prediction image and the first sub-block image sample and / or based on the difference between the second prediction image and the second sub-block image sample until the difference meets a preset requirement, and the image prediction model is obtained.

[0112] In the embodiments of the present disclosure, the difference between the first predicted image and the first sub-block image sample can be used as a loss function to iteratively adjust the training parameters; the difference between the second predicted image and the second sub-block image sample can also be used to iteratively adjust the training parameters; the total difference between the first predicted image and the first sub-block image sample and the second predicted image and the second sub-block image sample can also be used as a loss function, and different weight ratios can also be set between the first predicted image and the first sub-block image sample and between the second predicted image and the second sub-block image sample based on the total difference.

[0113] In the embodiments of the present disclosure, the loss of the predicted image and the sub-block image can include multiple ways, which can facilitate fast obtaining of the training model.

[0114] In a possible implementation, a task model is provided, which is trained by using the correspondence between the first output feature and / or the second output feature and the task type according to any one of the embodiments of the present disclosure.

[0115] In the embodiments of the present disclosure, the task type can include image target detection, image segmentation, image classification, image target tracking, image face recognition, etc. The training method of the task model can include supervised training, unsupervised training method, self-supervised training method, reinforcement learning and other training methods. In the embodiments of the present disclosure, the output feature (including the first output feature and the second output feature) can reshape the first sub-block image sample and the second sub-block image sample, so that the features of the image can be fully expressed, and thus the task model obtained by using the output feature for transfer learning has higher accuracy.

[0116] It should be understood that, although each step in the flowchart is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the figure can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0117] It can be understood that the same / similar parts between each embodiment of the above method in the specification can be mutually referred to, and each embodiment mainly explains the difference from other embodiments, and the related part can be referred to the description of other method embodiments.

[0118] Figure 3is a block diagram of an image processing apparatus according to an example embodiment. Referring to Figure 3 The apparatus includes an obtaining module 301 and a prediction model 302.

[0119] The obtaining module 301 is configured to obtain a first patch image and a mask image corresponding to a second patch image, the first patch image being obtained by cropping an original image, and the second patch image being an image other than the first patch image in the cropped image.

[0120] The prediction module 302 is configured to input the first patch image and the mask image into an image prediction model, and output a first prediction image of the first patch image and a second prediction image of the second patch image through the image prediction model.

[0121] As to the apparatus in the above embodiments, the specific manners in which the respective modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0122] Figure 4 is a block diagram of a server 400 according to an example embodiment. For example, the server 400 can be a server. Referring to Figure 4 The server 400 includes a processing component 420, which further includes one or more processors, and a memory resource represented by a memory 422, for storing instructions, such as application programs, executable by the processing component 420. The application programs stored in the memory 422 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 420 is configured to execute the instructions to perform the above method.

[0123] The server 400 can further include a power supply component 424 configured to perform power management of the server 400, a wired or wireless network interface 426 configured to connect the server 400 to a network, and an input / output (I / O) interface 428. The server 400 can operate based on an operating system stored in the memory 422, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0124] In an example embodiment, a computer readable storage medium including instructions, such as the memory 422 including instructions, is also provided, and the above method can be performed by the processor of the server 400 executing the instructions. The storage medium can be a computer readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0125] In an example embodiment, there is also provided a computer program product comprising instructions executable by a processor of the server 400 to perform the above method.

[0126] It should be noted that the above apparatus, server, computer readable storage medium, computer program product and the like according to the method embodiments can also include other implementation manners, and the specific implementation manners can be referred to the description of the related method embodiments, which will not be repeated here.

[0127] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the concepts disclosed herein. The disclosure is intended to cover any variations, uses or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure that come within known

[0128] It should be understood that the present disclosure is not limited to the precise structures herein described and illustrated in the drawings, and that various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. An image processing method, characterized in that: include: Obtaining mask images corresponding to a first block image and a second block image, wherein the first block image is obtained by cropping the original image, and the second block image is an image of the cropped image excluding the first block image; The first block image and the second block image are obtained by: cropping the original image to obtain block images of the same size; randomly selecting some block images from the block images for masking to obtain mask images, the remaining images not subjected to the masking are referred to as the first block images, and the block images subjected to the masking are referred to as the second block images; The first block image and the mask image are input into an image prediction model, and the image prediction model outputs a first predicted image of the first block image and a second predicted image of the second block image; the image prediction model is obtained in the following manner: obtaining a sample set, the sample set including first block image samples, second block image samples and mask image samples corresponding to the second block image samples; constructing an image prediction model, wherein training parameters are set in the image prediction model; inputting the first block image samples and the mask image samples into the image prediction model to generate a prediction result; based on the difference between the prediction result and the first block image samples and the second block image samples, iteratively adjusting the training parameters until the difference meets the preset requirements to obtain the image prediction model; inputting the first block image samples and the mask image samples into the image prediction model to generate a prediction result includes: extracting the features of the first block image samples and the features of the mask image samples respectively, and obtaining to the first input feature and the second input feature; input the first input feature and the second input feature into the self-attention mechanism network, and output the first output feature and the second output feature through the self-attention mechanism network; determine the first predicted image and the second predicted image respectively according to the first output feature and the second output feature; the image prediction model includes a self-attention mechanism network, and the self-attention mechanism network is a transformer network; the features of the first block image sample and the features of the mask image sample are extracted respectively to obtain the first input feature and the second input feature, including: using a convolutional neural network to extract the features of the first block image sample and the features of the mask image sample respectively to obtain the first input feature and the second input feature; the first predicted image and the second predicted image are determined respectively according to the first output feature and the second output feature, including: inputting the first output feature and the second output feature into the generation network, and outputting the first predicted image and the second predicted image through the generation network.

2. The method according to claim 1, characterized in that The extracting the features of the first block image sample and the features of the mask image sample respectively to obtain the first input feature and the second input feature includes: Extracting features of the first block image samples to obtain first input features; The feature of the mask image sample is set to a preset value to obtain a second input feature.

3. The method according to claim 1, characterized in that The extracting the features of the first block image sample and the features of the mask image sample respectively to obtain the first input feature and the second input feature includes: A mapping matrix is ​​used to perform a linear mapping transformation on the first block image samples and the mask image samples to obtain a first input feature and a second input feature.

4. The method according to claim 3, characterized in that The determining the first predicted image and the second predicted image according to the first output feature and the second output feature respectively includes: The first output feature and the second output feature are respectively subjected to linear mapping inverse transformation using the mapping matrix to obtain a first predicted image and a second predicted image.

5. The method according to claim 1, wherein Inputting the first input feature and the second input feature into the self-attention mechanism network, and outputting a first output feature and a second output feature through the self-attention mechanism network, including: Obtaining a first position vector of a first block image sample and a second position vector of a second block image sample; respectively concatenating the first input feature and the first position vector, and the second input feature and the second position vector to obtain a first concatenated feature and a second concatenated feature; The first splicing feature and the second splicing feature are input into the self-attention mechanism network, and the self-attention mechanism network outputs a first output feature and a second output feature.

6. The method according to claim 1, characterized in that The iterative adjustment of the training parameters based on the difference between the prediction result and the first block image and the second block image samples until the difference meets the preset requirements to obtain the image prediction model includes: Based on the difference between the first predicted image and the first block image sample and / or based on the difference between the second predicted image and the second block image sample, the training parameters are iteratively adjusted until the difference meets the preset requirements, thereby obtaining the image prediction model.

7. An image processing device, characterized in that include: an acquisition module, configured to acquire a mask image corresponding to a first block image and a second block image, wherein the first block image is obtained by cropping the original image, and the second block image is an image of the cropped image excluding the first block image; The first block image and the second block image are obtained by: cropping the original image to obtain block images of the same size; randomly selecting some block images from the block images for masking to obtain mask images, the remaining images not subjected to the masking are referred to as the first block images, and the block images subjected to the masking are referred to as the second block images; A prediction module, used to input the first block image and the mask image into an image prediction model, and output a first predicted image of the first block image and a second predicted image of the second block image through the image prediction model; a method for obtaining the image prediction model includes: obtaining a sample set, the sample set including first block image samples, second block image samples and mask image samples corresponding to the second block image samples; constructing an image prediction model, wherein training parameters are set in the image prediction model; inputting the first block image samples and the mask image samples into the image prediction model to generate a prediction result; based on the difference between the prediction result and the first block image samples and the second block image samples, iteratively adjusting the training parameters until the difference meets the preset requirements to obtain the image prediction model; inputting the first block image samples and the mask image samples into the image prediction model to generate a prediction result includes: extracting the features of the first block image samples and the features of the mask image samples respectively , obtaining a first input feature and a second input feature; inputting the first input feature and the second input feature into a self-attention mechanism network, and outputting a first output feature and a second output feature through the self-attention mechanism network; determining a first predicted image and a second predicted image respectively according to the first output feature and the second output feature; the image prediction model includes a self-attention mechanism network, and the self-attention mechanism network is a transformer network; extracting the features of the first block image sample and the features of the mask image sample respectively to obtain the first input feature and the second input feature, including: using a convolutional neural network to extract the features of the first block image sample and the features of the mask image sample respectively to obtain the first input feature and the second input feature; determining the first predicted image and the second predicted image respectively according to the first output feature and the second output feature, including: inputting the first output feature and the second output feature into a generation network, and outputting the first predicted image and the second predicted image through the generation network.

8. A task model, characterized in that: The task model is obtained by training using the correspondence between the first output feature and / or the second output feature and the task type as described in any one of claims 1 to 6.

9. A server, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of a server, the server is enabled to perform the image processing method according to any one of claims 1 to 6.

11. A computer program product comprising instructions, characterized in that: When the instruction is executed by a processor of the server, the server is enabled to execute the image processing method according to any one of claims 1 to 6.

Images