Image inpainting method, device, readable medium, and electronic device

Abstract

The present disclosure relates to an image inpainting method, device, readable medium and electronic equipment, the method comprising: obtaining a first image to be inpainted, a second image and text information, the second image being obtained after preprocessing the first image, and the text information being used to describe the first image; processing the first image, the second image and the text information according to a trained image inpainting model to obtain an inpainted image, wherein the processing comprises: respectively encoding the first image, the second image and the text information to obtain a first image feature sequence, a second image feature sequence and a semantic feature sequence; and decoding the first image feature sequence, the second image feature sequence and the semantic feature sequence to obtain the inpainted image, which can improve the effect of image inpainting.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to an image repairing method and device, readable medium and electronic equipment. BACKGROUND

[0002] Image repairing refers to the process of reconstructing the missing or damaged part of an image, which is a technology between image editing and image generation. In the related art, the pixel values of the missing part are directly predicted by a model, and the image repairing effect is poor, which can easily lead to distortion such as ripples and distortion in the repairing area. SUMMARY

[0003] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] In a first aspect, the present disclosure provides an image repairing method, comprising:

[0005] obtaining a first image to be repaired, a second image and text information, the second image being obtained after preprocessing the first image, and the text information being used to describe the first image;

[0006] processing the first image, the second image and the text information according to a trained image repairing model to obtain a repaired image, wherein the processing comprises:

[0007] encoding the first image, the second image and the text information respectively to obtain a first image feature sequence, a second image feature sequence and a semantic feature sequence;

[0008] decoding the first image feature sequence, the second image feature sequence and the semantic feature sequence to obtain the repaired image.

[0009] In a second aspect, the present disclosure provides an image repairing device, comprising:

[0010] an obtaining module configured to obtain a first image to be repaired, a second image and text information, the second image being obtained after preprocessing the first image, and the text information being used to describe the first image;

[0011] a processing module configured to process the first image, the second image and the text information according to a trained image repairing model to obtain a repaired image, wherein the processing module is further configured to:

[0012] The first image, the second image, and the text information are respectively encoded to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence;

[0013] The first image feature sequence, the second image feature sequence, and the semantic feature sequence are decoded to obtain the repaired image.

[0014] In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program, which, when executed by a processing device, implements the steps of the method of the first aspect.

[0015] In a fourth aspect, the present disclosure provides an electronic device comprising:

[0016] a storage device having stored thereon a computer program;

[0017] a processing device configured to execute the computer program in the storage device to implement the steps of the method of the first aspect.

[0018] According to the above technical solution, since the first image can reflect the global information in the image, the second image can reflect the contour information of the target in the first image, and the text information can reflect the semantic information rich in the first image, therefore, by processing the first image, the second image, and the text information through the image repairing model, the model can consider the global information, the contour information of the target, and the semantic information of the image to be repaired at the same time to realize the repair of the first image. Since the model inputs various types of information, the effect of the repaired image obtained through the various types of information is better, which can effectively reduce the distortion repair and obtain a good repair effect for the image with a large area of missing.

[0019] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0021] Figure 1 is a flowchart of an image repairing method according to an exemplary embodiment of the present disclosure.

[0022] Figure 2 is a schematic diagram of a first image according to an exemplary embodiment of the present disclosure.

[0023] Figure 3is a structural schematic diagram of an image inpainting model shown by an example embodiment of the present disclosure.

[0024] Figure 4 is a training flowchart of an image inpainting model shown by an example embodiment of the present disclosure.

[0025] Figure 5 is a block diagram of an image inpainting apparatus shown by an example embodiment of the present disclosure.

[0026] Figure 6 is a structural schematic diagram of an electronic device shown by an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings. While certain embodiments of the present disclosure will be illustrated and described, it is well understood that various changes in the form and details can be made without departing from the spirit and scope of the disclosure. It is to be understood that the drawings and the embodiments are only for the purpose of illustration and no limitations are to be implied therefrom.

[0028] It should be understood that each step recited in the method embodiments of the present disclosure can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0029] The term “comprising” and variations thereof as used herein are used inclusively, i.e., “comprising but not limited to.” The term “based on” means “based, at least in part, on.” The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments.” Related terms are defined in the description that follows.

[0030] It should be noted that the terms “first”, “second”, and the like in the present disclosure are merely used to distinguish different devices, modules or units, and do not imply the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the terms “one”, “multiple” in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that “one or more” should be understood unless otherwise explicitly indicated in the context.

[0032] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are merely for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0033] Figure 1 is a flowchart of an image inpainting method according to an example embodiment of the present disclosure, which can be applied to a terminal device. As shown in Figure 1 the flowchart includes the following steps:

[0034] In step 110, a first image to be inpainted, a second image, and text information are obtained, the second image being obtained by preprocessing the first image, and the text information being used to describe the first image.

[0035] In some embodiments, the first image can be an image with missing parts, and since the first image is the whole image to be restored, it can reflect global information in the image, such as object distribution information. The second image can be an image obtained by preprocessing the first image, and the preprocessing can include grayscale processing and / or edge detection.

[0036] Through grayscale processing, the RGB value of each pixel point in the first image can be unified to the same value, and then the RGB three-channel data of the first image can be converted into single-channel data. The single-channel data is simple to process, thereby simplifying the complexity of the channel data of the first image. Through edge detection, the edge region of the target in the first image can be identified, and thus the contour information of the target in the first image can be presented through the second image.

[0037] In some embodiments, the text information can be used to describe the first image. For example, as shown in Figure 2 the first image, the text information can be "a small blue airplane is parked on a field". In some embodiments, the text information can be obtained by processing the first image through an image description model, or obtained by describing the first image manually.

[0038] The image description model can be used to output a corresponding natural language description for an input image. The image description model can adopt a model trained according to any network structure. For example, the network structure can include any of the following: a convolutional neural network and a recurrent neural network, a residual network and a long short-term memory network, and an attention model. The text information can be obtained by processing the first image through the image description model, and the text information can reflect the semantic information rich in the first image.

[0039] In step 120, the first image, the second image, and the text information are processed according to the trained image inpainting model to obtain an inpainted image, wherein the processing includes: encoding the first image, the second image, and the text information respectively to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence; and decoding the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the inpainted image.

[0040] The image inpainting model can be a pre-trained machine learning model. For details of the training process of the image inpainting model, please refer to Figure 4 and the related description, which will not be repeated here.

[0041] Since the first image can reflect the global information in the image, the second image can reflect the contour information of the target in the first image, and the text information can reflect the semantic information rich in the first image, therefore, by processing the first image, the second image and the text information through the image inpainting model, the model can consider the global information, the contour information of the target and the semantic information of the image to be inpainted at the same time to realize the inpainting of the first image. Since the model inputs various types of information, the inpainting image obtained through the various types of information has better effect, which can effectively reduce the distortion inpainting and obtain good inpainting effect for images with large area missing.

[0042] As shown in Figure 3 In some embodiments, the image inpainting model 300 can include a first image encoding model 310, a second image encoding model 320 and a text encoding model 330, which respectively encode the first image, the second image and the text information to obtain a first image feature sequence, a second image feature sequence and a semantic feature sequence, including: encoding the first image according to the first image encoding model to obtain the first image feature sequence; encoding the second image according to the second image encoding model to obtain the second image feature sequence; and encoding the text information according to the text encoding model to obtain the semantic feature sequence.

[0043] In some embodiments, the first image encoding model and the second image encoding model each include a convolutional neural network and / or a first multi-layer encoder, the first image feature sequence is output by the convolutional neural network or the first multi-layer encoder in the first image encoding model, and the second image feature sequence is output by the convolutional neural network or the first multi-layer encoder in the second image encoding model.

[0044] In some embodiments, when the first image encoding model includes a convolutional neural network, the first image feature sequence is output by the convolutional neural network in the first image encoding model; when the first image encoding model includes a first multi-layer encoder, the first image feature sequence is output by the first multi-layer encoder in the first image encoding model; and when the first image encoding model includes a convolutional neural network and a first multi-layer encoder, the first image feature sequence is output by the first multi-layer encoder in the first image encoding model.

[0045] In some embodiments, when the second image encoding model comprises a convolutional neural network, the second image feature sequence is output by the convolutional neural network in the second image encoding model; when the second image encoding model comprises the first multi-layer encoder, the second image feature sequence is output by the first multi-layer encoder in the second image encoding model; when the second image encoding model comprises a convolutional neural network and the first multi-layer encoder, the second image feature sequence is output by the first multi-layer encoder in the second image encoding model.

[0046] In some embodiments, the first multi-layer encoder can employ a model comprising a plurality of encoding layers, such as a BP (Back Propagation) neural network or a multi-layer autoencoder, etc. In some embodiments, the first multi-layer encoder can also be a pre-constructed model, and the first multi-layer encoder can comprise a first feedforward neural network, an attention network, a convolutional network, and a second feedforward neural network connected in sequence.

[0047] The convolutional neural network can extract features from the input image, extract deep learning features in the image, and output two-dimensional features. For example, the convolutional neural network can convert the image to a size of 256*256, and then convert the image features to a sequence feature of 2048 dimensions through a 6-layer convolutional layer, a pooling layer, and a fully connected layer of a channel attention mechanism. The sequence feature of 2048 dimensions can be multiple.

[0048] The first multi-layer encoder can extract target key features in the image, and the features output by the first multi-layer encoder are one-dimensional features. In some embodiments, when the first image encoding model or the second image encoding model comprises a convolutional neural network and a first multi-layer encoder, the first multi-layer encoder is connected to the output end of the convolutional neural network. At this time, the features input to the first multi-layer encoder are two-dimensional features output by the aforementioned convolutional neural network, for example, a plurality of sequence features of 2048 dimensions. The first multi-layer encoder can extract the association between the two-dimensional features to extract target key features in the image, and convert the two-dimensional features to one-dimensional features.

[0049] In some embodiments, the target key features can be texture structures in the image. For example, as shown in FIG. 1, an airplane image is input to the image repair model, and the image repair model can extract the key features of the airplane, such as the wings, the body, and the tail, and repair the missing tail of the airplane. Figure 2 For example, as shown in FIG. 1, an airplane image is input to the image repair model, and the image repair model can extract the key features of the airplane, such as the wings, the body, and the tail, and repair the missing tail of the airplane. Figure 2 For example, as shown in FIG. 1, an airplane image is input to the image repair model, and the image repair model can extract the key features of the airplane, such as the wings, the body, and the tail, and repair the missing tail of the airplane.

[0050] In some embodiments, the text encoding model can comprise a trained second multi-layer encoder, and the text information is encoded according to the text encoding model to obtain a semantic feature sequence, comprising: processing the word vector of the text information according to the trained second multi-layer encoder to obtain the semantic feature sequence, wherein the word vector is obtained after the text information is processed by word segmentation.

[0051] In some embodiments, the word vector can be obtained after the text information is processed by a word segmentation model, and the word segmentation model can or can not be included in the text encoding model. In some embodiments, the word segmentation model can be a BPE (Byte Pair Encoding) word segmentation model, which can not segment high-frequency words in the text information corresponding to the preset corpus, and can segment low-frequency words in the text information corresponding to the preset corpus, while ensuring the granularity of word segmentation, and also considering the integrity and accuracy of word segmentation as much as possible.

[0052] In possible implementations, the second multi-layer encoder has the same structure as the first multi-layer encoder. For the structure of the second multi-layer encoder, please refer to the foregoing description of the first multi-layer encoder, which will not be repeated here.

[0053] In some embodiments, the second multi-layer encoder can be a pre-trained machine learning model. In some embodiments, the second multi-layer encoder can be trained based on the following manner: obtaining a plurality of first sample texts labeled with a semantic feature sequence; performing word segmentation on each first sample text to obtain a plurality of initial sample word vectors; performing mask processing on each initial sample word vector to obtain a plurality of target sample word vectors; training the initial second multi-layer encoder according to the plurality of target sample word vectors and the labeled semantic feature sequence to obtain the trained second multi-layer encoder.

[0054] The first sample text can be a text in multiple languages. By training the second multi-layer encoder with texts in multiple languages as samples, the trained second multi-layer encoder can encode text information in different languages.

[0055] In some embodiments, the labeled semantic feature sequence can be a label of the first sample text, which can be manually labeled. In some embodiments, each first sample text can be processed by a word segmentation model. For specific details of the word segmentation model, please refer to the foregoing description, which will not be repeated here.

[0056] The mask processing on each initial sample word vector can be a random mask, and a large number of target sample word vectors can be obtained through the random mask. Thus, a large amount of training data for training the second multi-layer encoder can be obtained through the random mask, and the coding effect of the trained second multi-layer encoder is improved. Meanwhile, the second multi-layer encoder is trained according to the target sample word vectors processed by the mask, so that the trained second multi-layer encoder can accurately predict the semantic feature sequence for the text information with missing parts in the model prediction stage.

[0057] In some embodiments, the training of the initial second multi-layer encoder according to the plurality of target sample word vectors and the labeled semantic feature sequence to obtain the trained second multi-layer encoder can include: iteratively updating parameters of the initial second multi-layer encoder according to the plurality of target sample word vectors corresponding to the plurality of first sample texts to reduce the first loss function value of each first sample text, to obtain the trained second multi-layer encoder.

[0058] In some embodiments, in the training process of the second multi-layer encoder, the parameters of the initial second multi-layer encoder can be iteratively updated based on the plurality of target sample word vectors corresponding to the plurality of first sample texts. Specifically, the parameters of the initial second multi-layer encoder can be continuously adjusted to reduce the first loss function value of the first sample text corresponding to each target sample word vector, so that the first loss function value meets a preset condition. For example, the first loss function value converges, or the first loss function value is less than a preset value. When the first loss function meets the preset condition, the model training is completed, and the trained second multi-layer encoder is obtained. The trained second multi-layer encoder can perform coding processing on the text information to obtain the semantic feature sequence.

[0059] In some embodiments, the first loss function value of the first sample text corresponding to each target sample word vector can be determined by: processing the target sample word vector corresponding to the first sample text through the second multi-layer encoder to obtain a first predicted semantic feature sequence of the first sample text; and determining the first loss function value based at least on the difference between the first predicted semantic feature sequence and the labeled semantic feature sequence.

[0060] As Figure 3As shown, in some embodiments, the image inpainting model 300 comprises a cross-attention network 340, a third multi-layer encoder 350, and a fully connected network 360, the fully connected network 360 comprises at least one fully connected layer, and the fully connected network 360 decodes the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the inpainting image, including: fusing the first image feature sequence, the second image feature sequence, and the semantic feature sequence according to the cross-attention network to obtain an initial fusion feature sequence; processing the initial fusion feature sequence according to the third multi-layer encoder to obtain a target fusion feature sequence; and processing the target fusion feature sequence according to the fully connected network to obtain the inpainting image.

[0061] The cross-attention network can capture the dependency information between the first image feature sequence, the second image feature sequence, and the semantic feature sequence, and the cross-attention network can realize the interaction and complementarity of image information and text information, and output an initial fusion feature sequence with a dimension of 2018. For example, the dependency information between the semantic feature sequences can refer to the probability of the occurrence of words in the text information, and the dependency information between the image feature sequences (e.g., the first image feature sequence or the second image feature sequence) can refer to the relationship between different regions in the image.

[0062] In possible implementations, the third multi-layer encoder can have the same structure as the first multi-layer encoder. For the structure of the third multi-layer encoder, please refer to the foregoing description of the first multi-layer encoder, which will not be repeated here.

[0063] In some embodiments, the third multi-layer encoder can comprise a plurality of convolutional layers, the convolutional layers adopt multi-channel convolutional layers with a convolution window shape of 1*1 and a convolution kernel size of 1*128, and output a target fusion feature sequence with a dimension of 1024. The third multi-layer encoder can further capture dependency information.

[0064] In possible implementations, the fully connected network can comprise two fully connected layers, and the fully connected network can map the target fusion feature sequence to the dimension of the image. For example, the fully connected network can map the 1024-dimensional features output by the target fusion feature sequence to 10240 dimensions, and then to 256*256*3 dimensions, to obtain the inpainting image.

[0065] The cross-attention network, the third multi-layer encoder, and the fully connected network are used to fuse the spatial layout information (e.g., the first image feature sequence and the second image feature sequence) and the text semantic information of the picture to restore the target in the missing part of the first image. Since the object distribution in the image and the text semantic are basically aligned, and there is little jump, the image decoding network composed of the cross-attention network, the third multi-layer encoder, and the fully connected network can be used to directly obtain the restored image based on the target fusion feature sequence. Thus, the complexity of the image restoration process is simplified, and the efficiency of the image restoration is improved.

[0066] Figure 4 is a training flowchart of an image restoration model according to an example embodiment of the present disclosure, as shown in Figure 4 The image restoration model can be trained based on the following process, as shown in

[0067] In step 410, a sample image and a second sample text are obtained, and the second sample text is used to describe the sample image.

[0068] In step 420, the sample image and the second sample text are respectively subjected to random mask processing to obtain a plurality of first sample images and a plurality of target second sample texts.

[0069] In step 430, each first sample image is preprocessed to obtain a plurality of second sample images.

[0070] In step 440, training data are constructed according to the plurality of first sample images, the plurality of second sample images, the plurality of target second sample texts, and the sample image, and an initial image restoration model is trained according to the training data to obtain a trained image restoration model.

[0071] The sample image can be a complete image, and the sample image can include one or more. By performing mask processing on the sample image, an image with missing parts can be obtained. In some embodiments, a sample image can be randomly masked by a mask with a preset shape. The preset shape can be specifically set according to actual needs, such as a square or a circle. The mask proportion of the sample image is not more than a first preset proportion of the area of the sample image. The first preset proportion is specifically set according to actual needs, for example, the first preset proportion can be 50%. In some embodiments, the pixels of the randomly masked part of the sample image can be uniformly set to 0 or 255.

[0072] The preprocessing of the first sample image is the same as the preprocessing of the first image, and details can be referred to step 110 and related descriptions thereof. The second sample text can be used to describe the sample image, and the second sample text is obtained in the same manner as the text information in the model application stage, and details can be referred to step 110 and related descriptions thereof. In some embodiments, a preset symbol can be used to randomly replace words in the second sample text to perform random mask processing on the second sample text. For example, the preset symbol can be “#” or “*”, etc. The mask proportion of the second sample text does not exceed a second preset proportion of the total number of words in the second sample text, and the second preset proportion can be set according to actual needs, for example, the second preset proportion can be 30%.

[0073] By performing random mask processing on the sample image and the second sample text respectively, a large number of first sample images and target second sample texts can be obtained, so that a large number of training data can be constructed subsequently. The image restoration model is trained by a large amount of training data, the image restoration model can be fully trained, the restoration effect of the image restoration model is improved, and the image restoration model obtains a more realistic restoration effect in the actual restoration process.

[0074] In some embodiments, constructing training data according to the plurality of first sample images, the plurality of second sample images, the plurality of target second sample texts, and the sample image can be: grouping the plurality of first sample images, the plurality of second sample images, and the plurality of target second sample texts according to the same sample image to obtain a plurality of groups of training samples, each group of training samples including a first sample image, a second sample image, and a target second sample text of the same sample image; and constructing training data according to the plurality of groups of training samples and the sample image.

[0075] In some embodiments, training the initial image restoration model according to the training data to obtain the trained image restoration model includes: iteratively updating parameters of the initial image restoration model according to the plurality of groups of training samples to reduce a second loss function value corresponding to each training sample, to obtain the trained image restoration model.

[0076] In some embodiments, the second loss function value corresponding to each training sample can be determined by: processing the training sample by the image restoration model to obtain a sample restoration image; and determining the second loss function value based at least on a difference between the sample restoration image and a sample image corresponding to the training sample.

[0077] In some embodiments, during the training process of the image inpainting model, the parameters of the initial image inpainting model can be iteratively updated based on a plurality of training samples. Specifically, the parameters of the initial image inpainting model can be continuously adjusted to reduce the second loss function value corresponding to each training sample, so that the second loss function value satisfies a preset condition. For example, the second loss function value converges, or the second loss function value is less than a preset value. When the second loss function satisfies the preset condition, the model training is completed, and a trained image inpainting model is obtained. The trained image inpainting model can process the first image to be inpainted, the second image, and the text information to obtain an inpainted image.

[0078] Figure 5 is a block diagram of an image inpainting device according to an example embodiment of the present disclosure, as shown in Figure 5 The image inpainting device 500 can include:

[0079] The acquisition module 510 is configured to acquire a first image to be inpainted, a second image, and text information, wherein the second image is obtained by preprocessing the first image, and the text information is used to describe the first image.

[0080] The processing module 520 is configured to process the first image, the second image, and the text information according to the trained image inpainting model to obtain an inpainted image, wherein the processing module 520 is further configured to:

[0081] perform encoding processing on the first image, the second image, and the text information respectively to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence;

[0082] perform decoding processing according to the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the inpainted image.

[0083] Optionally, the image inpainting model includes a first image encoding model, a second image encoding model, and a text encoding model, and the processing module 520 is further configured to:

[0084] perform the encoding processing on the first image according to the first image encoding model to obtain the first image feature sequence;

[0085] perform the encoding processing on the second image according to the second image encoding model to obtain the second image feature sequence; and

[0086] perform the encoding processing on the text information according to the text encoding model to obtain the semantic feature sequence;

[0087] The first image encoding model and the second image encoding model each include a convolutional neural network and / or a first multi-layer encoder, the first image feature sequence is output by the convolutional neural network or the first multi-layer encoder in the first image encoding model, and the second image feature sequence is output by the convolutional neural network or the first multi-layer encoder in the second image encoding model.

[0088] Optionally, the text encoding model includes a trained second multi-layer encoder, and the processing module 520 is further configured to:

[0089] The word vector of the text information is processed according to the trained second multi-layer encoder to obtain the semantic feature sequence, and the word vector is obtained after the text information is processed by word segmentation.

[0090] Optionally, the second multi-layer encoder is trained based on the following manner:

[0091] A plurality of first sample texts labeled with a semantic feature sequence are obtained.

[0092] Each of the first sample texts is processed by word segmentation to obtain a plurality of initial sample word vectors.

[0093] Each of the initial sample word vectors is processed by a mask to obtain a plurality of target sample word vectors.

[0094] An initial second multi-layer encoder is trained according to a plurality of the target sample word vectors and the labeled semantic feature sequence to obtain the trained second multi-layer encoder.

[0095] Optionally, the image restoration model includes a cross-attention network, a third multi-layer encoder, and a fully connected network, the fully connected network includes at least one fully connected layer, and the processing module 520 is further configured to:

[0096] The first image feature sequence, the second image feature sequence, and the semantic feature sequence are fused by the cross-attention network to obtain an initial fusion feature sequence.

[0097] The initial fusion feature sequence is processed by the third multi-layer encoder to obtain a target fusion feature sequence.

[0098] The target fusion feature sequence is processed by the fully connected network to obtain the restored image.

[0099] Optionally, the text information is obtained by processing the first image by an image description model, or is obtained by manually describing the first image.

[0100] Optionally, the image inpainting model is trained based on the following manner:

[0101] Obtain a sample image and a second sample text, the second sample text being used to describe the sample image;

[0102] Randomly mask the sample image and the second sample text respectively to obtain a plurality of first sample images and a plurality of target second sample texts;

[0103] Preprocess each of the first sample images to obtain a plurality of second sample images;

[0104] Construct training data according to the plurality of first sample images, the plurality of second sample images, the plurality of target second sample texts, and the sample image, and train an initial image inpainting model according to the training data to obtain a trained image inpainting model.

[0105] Reference is made below to Figure 6 , which shows a structural schematic diagram of an electronic device (e.g., a terminal device in Figure 1 ) suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Personal Computers), PMPs (Portable Multimedia Players), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0106] As shown in Figure 6 , the electronic device 600 can include a processing device (e.g., a central processor, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 602 or loaded into a random access memory (RAM) 603 from a storage device 608. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0107] ​In general, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, and the like; output devices 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 608 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it is understood that all of the illustrated devices are not required to implement or be present. More or less devices can alternatively be implemented or present.

[0108] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processing devices 601, the above-described functions defined in the methods of embodiments of the present disclosure are performed.

[0109] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, optical fiber, RF, etc., or any suitable combination of the above.

[0110] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0111] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and can not be assembled into the electronic device.

[0112] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: acquire a first image to be repaired, a second image obtained after pre-processing the first image, and text information used to describe the first image; and process the first image, the second image, and the text information according to a trained image repair model to obtain a repaired image, wherein the processing includes: performing encoding processing on the first image, the second image, and the text information respectively to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence; and performing decoding processing according to the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the repaired image.

[0113] Computer program code for carrying out operations of the present disclosure can be written in any of one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0114] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It is also noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0115] The modules described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0116] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0117] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage media can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media can include one or more lines of electrical wire, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] According to one or more embodiments of the present disclosure, example 1 provides an image inpainting method, comprising:

[0119] obtaining a first image to be inpainted, a second image, and text information, the second image being obtained after pre-processing the first image, and the text information being used to describe the first image;

[0120] processing the first image, the second image, and the text information according to the trained image inpainting model to obtain an inpainted image, wherein the processing comprises:

[0121] respectively encoding the first image, the second image, and the text information to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence;

[0122] decoding the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the inpainted image.

[0123] According to one or more embodiments of the present disclosure, example 2 provides the method of example 1, the image restoration model comprises a first image encoding model, a second image encoding model and a text encoding model, the encoding processing of the first image, the second image and the text information respectively obtains a first image feature sequence, a second image feature sequence and a semantic feature sequence, comprising:

[0124] the encoding processing of the first image according to the first image encoding model obtains the first image feature sequence;

[0125] the encoding processing of the second image according to the second image encoding model obtains the second image feature sequence; and

[0126] the encoding processing of the text information according to the text encoding model obtains the semantic feature sequence;

[0127] wherein, the first image encoding model and the second image encoding model each comprise a convolutional neural network and / or a first multi-layer encoder, the first image feature sequence is output by the convolutional neural network or the first multi-layer encoder in the first image encoding model, and the second image feature sequence is output by the convolutional neural network or the first multi-layer encoder in the second image encoding model.

[0128] According to one or more embodiments of the present disclosure, example 3 provides the method of example 2, the text encoding model comprises a trained second multi-layer encoder, and the encoding processing of the text information according to the text encoding model to obtain the semantic feature sequence comprises:

[0129] processing the word vector of the text information according to the trained second multi-layer encoder to obtain the semantic feature sequence, the word vector is obtained after the text information is processed by word segmentation.

[0130] According to one or more embodiments of the present disclosure, example 4 provides the method of example 3, the second multi-layer encoder is trained based on the following way:

[0131] obtaining a plurality of first sample texts labeled with semantic feature sequences;

[0132] performing word segmentation processing on each of the first sample texts to obtain a plurality of initial sample word vectors;

[0133] mask processing each of the initial sample word vectors to obtain a plurality of target sample word vectors;

[0134] The initial second multi-layer encoder is trained according to the plurality of target sample word vectors and the labeled semantic feature sequence, to obtain the trained second multi-layer encoder.

[0135] According to one or more embodiments of the present disclosure, example 5 provides the method of example 1, the image inpainting model comprises a cross-attention network, a third multi-layer encoder, and a fully connected network, the fully connected network comprises at least one fully connected layer, and the decoding processing according to the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the inpainting image comprises:

[0136] The first image feature sequence, the second image feature sequence, and the semantic feature sequence are fused according to the cross-attention network to obtain an initial fusion feature sequence;

[0137] The initial fusion feature sequence is processed according to the third multi-layer encoder to obtain a target fusion feature sequence;

[0138] The target fusion feature sequence is processed according to the fully connected network to obtain the inpainting image.

[0139] According to one or more embodiments of the present disclosure, example 6 provides the method of example 1, the text information is obtained by processing the first image through an image description model, or is obtained by describing the first image manually.

[0140] According to one or more embodiments of the present disclosure, example 7 provides the method of any one of examples 1-6, the image inpainting model is obtained by training based on the following manner:

[0141] Obtaining a sample image and a second sample text, the second sample text is used to describe the sample image;

[0142] Randomly masking the sample image and the second sample text respectively to obtain a plurality of first sample images and a plurality of target second sample texts;

[0143] Preprocessing each of the first sample images to obtain a plurality of second sample images;

[0144] Constructing training data according to a plurality of the first sample images, a plurality of the second sample images, a plurality of the target second sample texts, and the sample image, and training an initial image inpainting model according to the training data to obtain a trained image inpainting model.

[0145] According to one or more embodiments of the present disclosure, example 8 provides the method of example 1,

[0146] According to one or more embodiments of the present disclosure, example 8 provides an image inpainting apparatus, comprising:

[0147] an obtaining module configured to obtain a first image to be inpainted, a second image obtained after pre-processing the first image, and text information used to describe the first image;

[0148] a processing module configured to process the first image, the second image, and the text information according to a trained image inpainting model to obtain an inpainted image, wherein the processing module is further configured to:

[0149] perform encoding processing on the first image, the second image, and the text information respectively to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence;

[0150] perform decoding processing according to the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the inpainted image.

[0151] According to one or more embodiments of the present disclosure, example 9 provides a computer readable medium having a computer program stored thereon, which, when executed by a processing apparatus, implements the steps of the method of any one of examples 1-7.

[0152] According to one or more embodiments of the present disclosure, example 10 provides an electronic device, comprising:

[0153] a storage device having a computer program stored thereon;

[0154] a processing apparatus configured to execute the computer program in the storage device to implement the steps of the method of any one of examples 1-7.

[0155] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology applied. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the disclosed concept. For example, the above features are replaced with the technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.

[0156] Moreover, while operations have been depicted in a particular order, this should not be understood as requiring such an order nor limiting it to only those operations shown and described. One of ordinary skill in the art will recognize that many of the operations can be performed in a differing order, or be performed concurrently, that some operations can be performed in any order or omitted, and that some operations can be performed in parallel. Similarly, while several specific implementation details have been discussed in the context of the above discussion, these should not be construed as limitations on the scope of the present disclosure. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0157] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With respect to the devices in the above-described embodiments, in which various modules perform operations, the specific manner in which the various modules perform the operations has been described in detail in the embodiments relating to the method. Here, no detailed explanation will be given.

Claims

1. An image restoration method, characterized in that, include: A first image to be repaired, a second image, and text information are obtained. The second image is obtained after preprocessing the first image. The second image is used to reflect the contour information of the target in the first image. The text information is used to describe the first image. The first image, the second image, and the text information are processed according to the trained image restoration model to obtain a restored image, wherein the processing includes: The first image, the second image, and the text information are encoded respectively to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence; The repaired image is obtained by decoding the first image feature sequence, the second image feature sequence, and the semantic feature sequence. The image restoration model includes a cross-attention network, a third multi-layer encoder, and a fully connected network. The fully connected network includes at least one fully connected layer. The step of decoding the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the restored image includes: fusing the first image feature sequence, the second image feature sequence, and the semantic feature sequence using the cross-attention network to obtain an initial fused feature sequence; processing the initial fused feature sequence using the third multi-layer encoder to obtain a target fused feature sequence; and processing the target fused feature sequence using the fully connected network to obtain the restored image.

2. The image restoration method according to claim 1, characterized in that, The image restoration model includes a first image encoding model, a second image encoding model, and a text encoding model. The encoding process performed on the first image, the second image, and the text information to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence includes: The first image is encoded according to the first image encoding model to obtain the first image feature sequence; The second image is encoded according to the second image encoding model to obtain the feature sequence of the second image; and The text information is encoded according to the text encoding model to obtain the semantic feature sequence; The first image coding model and the second image coding model each include a convolutional neural network and / or a first multilayer encoder. The first image feature sequence is output by the convolutional neural network or the first multilayer encoder in the first image coding model, and the second image feature sequence is output by the convolutional neural network or the first multilayer encoder in the second image coding model.

3. The image restoration method according to claim 2, characterized in that, The text encoding model includes a trained second multilayer encoder. The step of encoding the text information according to the text encoding model to obtain the semantic feature sequence includes: The semantic feature sequence is obtained by processing the word vectors of the text information according to the trained second multi-layer encoder. The word vectors are obtained after the text information is segmented.

4. The image restoration method according to claim 3, characterized in that, The second multilayer encoder was trained in the following manner: Obtain multiple first sample texts labeled with semantic feature sequences; Each of the first sample texts is segmented into words to obtain multiple initial sample word vectors; Each initial sample word vector is masked to obtain multiple target sample word vectors; The initial second multilayer encoder is trained based on the word vectors of multiple target samples and the labeled semantic feature sequence to obtain the trained second multilayer encoder.

5. The image restoration method according to claim 1, characterized in that, The image restoration model includes a cross-attention network, a third multi-layer encoder, and a fully connected network. The fully connected network includes at least one fully connected layer. The step of decoding based on the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the restored image includes: The first image feature sequence, the second image feature sequence, and the semantic feature sequence are fused according to the cross-attention network to obtain an initial fused feature sequence; The initial fused feature sequence is processed according to the third multi-layer encoder to obtain the target fused feature sequence; The target fusion feature sequence is processed using the fully connected network to obtain the repaired image.

6. The image restoration method according to claim 1, characterized in that, The text information is obtained by processing the first image through an image description model, or by manually describing the first image.

7. The image restoration method according to any one of claims 1-6, characterized in that, The image restoration model was trained in the following manner: Acquire a sample image and a second sample text, wherein the second sample text is used to describe the sample image; Random masking is performed on the sample images and the second sample text respectively to obtain multiple first sample images and multiple target second sample texts; Each of the first sample images is preprocessed to obtain multiple second sample images; Training data is constructed based on multiple first sample images, multiple second sample images, multiple target second sample texts, and the sample images. The initial image restoration model is then trained based on the training data to obtain a trained image restoration model.

8. An image restoration device, characterized in that, include: The acquisition module is configured to acquire a first image to be repaired, a second image, and text information. The second image is obtained after preprocessing the first image and is used to reflect the contour information of the target in the first image. The text information is used to describe the first image. The processing module is configured to process the first image, the second image, and the text information according to a trained image restoration model to obtain a restored image, wherein the processing module is further configured to: The first image, the second image, and the text information are encoded respectively to obtain a first image feature sequence, a second image feature sequence, and a semantic feature sequence; The repaired image is obtained by decoding the first image feature sequence, the second image feature sequence, and the semantic feature sequence. The image restoration model includes a cross-attention network, a third multi-layer encoder, and a fully connected network. The fully connected network includes at least one fully connected layer. The processing module is further configured to implement the above-mentioned step of decoding the first image feature sequence, the second image feature sequence, and the semantic feature sequence to obtain the restored image in the following manner: fusing the first image feature sequence, the second image feature sequence, and the semantic feature sequence according to the cross-attention network to obtain an initial fused feature sequence; processing the initial fused feature sequence according to the third multi-layer encoder to obtain a target fused feature sequence; and processing the target fused feature sequence according to the fully connected network to obtain the restored image.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processing device, the program implements the steps of the method described in any one of claims 1-7.

10. An electronic device, characterized in that, include: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the method according to any one of claims 1-7.

Images