Image processing method, device and computer readable storage medium

Abstract

The embodiment of the present invention discloses an image processing method, the method comprising: obtaining a first image to be trained, and generating a text area generator based on the first image to be trained; wherein the first image to be trained is an image with text information; using the text area generator to process the first image to be trained to generate a label image to be processed; wherein the label image to be processed is used to indicate the area covered by the text in the first image to be trained; generating a text-missing image based on the label image to be processed and the first image to be trained, and generating an image repair generator based on the text-missing image; obtaining an image to be processed with text information; using the text area generator and the image repair generator to repair the image to be processed to obtain a target image. The embodiment of the present invention also discloses an image processing device and a computer-readable storage medium.

Description

Technical Field

[0001] The present invention relates to image processing technology in the field of communications, and in particular to an image processing method, device and computer-readable storage medium. Background Art

[0002] With the rapid development of globalization and the Internet, many e-commerce companies have chosen to take their products global so that they can reach consumers in various countries quickly and easily. This requires launching their own product main sites in various countries. Naturally, the product details, comments, product introductions, etc. in the main sites need to be converted from Chinese to the text of the corresponding country; among them, the most important thing is to convert the product details images into images in the corresponding language. This requires erasing the text in the original image and repairing the erased area, and then writing the corresponding language text into the original text position.

[0003] At present, image inpainting technology is generally used to solve the above problems; and most of the related technologies use fast-marching algorithms, patch matching algorithms or deep learning algorithms to apply to image inpainting technology to achieve the erasure of text in product detail images and the restoration of erased areas. However, such image inpainting solutions in the related technologies have the problem that the texture restoration of the text area is not clear and complete, resulting in poor image restoration effect. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present invention hope to provide an image processing method, device and computer-readable storage medium, which solve the problem that the texture restoration of the text area in the image restoration scheme in the related art is not clear and complete, and improve the effect of image restoration.

[0005] The technical solution of the present invention is achieved in this way:

[0006] An image processing method, the method comprising:

[0007] Acquire a first image to be trained, and generate a text region generator based on the first image to be trained; wherein the first image to be trained is an image with text information;

[0008] The first image to be trained is processed by the text region generator to generate a label image to be processed; wherein the label image to be processed is used to indicate the area covered by the text in the first image to be trained;

[0009] Generate a text-missing image based on the to-be-processed label image and the first to-be-trained image, and perform training based on the text-missing image to generate an image restoration generator;

[0010] Obtaining an image to be processed with text information;

[0011] The text region generator and the image restoration generator are used to restore the image to be processed to obtain a target image.

[0012] In the above solution, the step of obtaining the first image to be trained includes:

[0013] Acquire a second image to be trained; wherein the second image to be trained does not contain any text information;

[0014] Text information is added to the second image to be trained to obtain the first image to be trained.

[0015] In the above solution, the step of generating a text area generator based on the first image to be trained includes:

[0016] Based on the first image to be trained, generating a label image to be trained; wherein the label image to be trained is used to indicate an area covered by text in the first image to be trained;

[0017] The text area generator is generated by performing training based on the first image to be trained and the label image to be trained.

[0018] In the above solution, the step of generating a label image to be trained based on the first image to be trained includes:

[0019] The color of the region where the characters are located in the first image to be trained is set to a first color, and the color of other regions except the characters in the first image to be trained is set to a second color, to obtain the label image to be trained.

[0020] In the above solution, the training based on the first to-be-trained image and the to-be-trained label image to generate the text area generator includes:

[0021] The initial text generator is trained based on the first image to be trained, the label image to be trained and a first loss function to generate the text area generator; wherein the first loss function is a loss function corresponding to the initial text generator.

[0022] In the above solution, the initial text generator is trained based on the first image to be trained, the label image to be trained and the first loss function to generate the text area generator, including:

[0023] Training a first initial discriminator based on the first image to be trained, the label image to be trained, and the second loss function to generate a first discriminator; wherein the second loss function is a loss function corresponding to the initial discriminator;

[0024] Based on the first image to be trained, the label image to be trained, the first loss function and the first discriminator, the initial text generator is trained to generate the text area generator.

[0025] In the above solution, the training of the first initial discriminator based on the first image to be trained, the label image to be trained and the second loss function to generate the first discriminator includes:

[0026] Using the initial text generator to process the first image to be trained to generate a reference label image;

[0027] The first initial discriminator is trained based on the reference label image, the label image to be trained and the second loss function to generate the first discriminator.

[0028] In the above scheme, the architecture of the text region generator includes a UNet architecture; and the first discriminator includes a Markov discriminator.

[0029] In the above solution, the step of generating a text-missing image based on the label image to be processed and the first image to be trained includes:

[0030] Determine a first pixel matrix corresponding to the to-be-processed label image and a second pixel matrix corresponding to the first to-be-trained image;

[0031] The text-missing image is generated based on the first pixel matrix and the second pixel matrix.

[0032] In the above solution, generating the text-missing image based on the first pixel matrix and the second pixel matrix includes:

[0033] Generate a third pixel matrix based on the first pixel matrix and the second pixel matrix;

[0034] An image corresponding to the third pixel matrix is ​​determined to obtain the text-missing image.

[0035] In the above solution, the training based on the text-missing image to generate an image restoration generator includes:

[0036] Based on the text-missing image, the second image to be trained and the third loss function, the initial image restoration generator is trained to generate the image restoration generator; wherein the third loss function is the loss function corresponding to the image restoration generator.

[0037] In the above solution, the initial image restoration generator is trained based on the text-missing image, the second image to be trained and the third loss function to generate the image restoration generator, including:

[0038] Training the second initial discriminator based on the text-missing image, the second image to be trained and a fourth loss function to generate a second discriminator; wherein the fourth loss function is a loss function corresponding to the second initial discriminator;

[0039] Based on the text-missing image, the second image to be trained, the third loss function and the second discriminator, the initial image restoration generator is trained to generate the image restoration generator.

[0040] In the above solution, the training of the second initial discriminator based on the text-missing image, the second image to be trained and the fourth loss function to generate the second discriminator includes:

[0041] Using the initial image restoration generator to restore the text-missing image to generate a reference image;

[0042] The second initial discriminator is trained based on the reference image, the second image to be trained and the fourth loss function to generate the second discriminator.

[0043] In the above scheme, the image restoration generator and the text region generator are used to restore the image to be processed to obtain a target image, including:

[0044] Using the text region generator to process the image to be processed to generate an intermediate label image;

[0045] Based on the intermediate label image and the image to be processed, generating an intermediate text-missing image;

[0046] The image restoration generator is used to restore the intermediate text-missing image to obtain the target image.

[0047] An image processing device, the device comprising: a processor, a memory and a communication bus;

[0048] The communication bus is used to realize the communication connection between the processor and the memory;

[0049] The processor is used to execute the image processing program stored in the memory to implement the following steps:

[0050] Acquire a first image to be trained, and generate a text region generator based on the first image to be trained; wherein the first image to be trained is an image with text information;

[0051] The first image to be trained is processed by the text region generator to generate a label image to be processed; wherein the label image to be processed is used to indicate the area covered by the text in the first image to be trained;

[0052] Generate a text-missing image based on the to-be-processed label image and the first to-be-trained image, and perform training based on the text-missing image to generate an image restoration generator;

[0053] Obtaining an image to be processed with text information;

[0054] The image to be processed is repaired using the image repair generator and the text area generator to obtain a target image.

[0055] A computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any of the above-mentioned image processing methods.

[0056] Because a first image to be trained with text information can be obtained and a text area generator can be generated based on the first image to be trained, and the text area generator is used to process the first image to be trained to generate a label image to be processed for indicating the area covered by the text in the image to be processed, and then a text-missing image is generated based on the label image to be processed and the first image to be trained, and an image repair generator is generated based on the text-missing image, and finally, the image to be processed with text information is obtained, and the text area generator and the image repair generator are used to repair the image to be processed to obtain the target image. In this way, a text area generator and an image repair generator can be obtained based on training, and the image to be processed can be directly repaired. The text area generator can extract the area covered by the text in the image to be processed, instead of extracting a rectangular area including the text in the image to be processed as in the relative technology, and makes full use of the information of other areas in the image to be processed except the text area, solves the problem that the texture repair of the text area in the image repair scheme in the related technology is not clear and complete, and improves the effect of image repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A schematic diagram of a flow chart of an image processing method provided by an embodiment of the present invention;

[0058] Figure 2 A schematic flow chart of another image processing method provided by an embodiment of the present invention;

[0059] Figure 3 A schematic flow chart of another image processing method provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of a text area generator in an image processing method provided by an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of generating a text-missing image in an image processing method provided by an embodiment of the present invention;

[0062] Figure 6 A schematic diagram of generating an image restoration generator in an image processing method provided by an embodiment of the present invention;

[0063] Figure 7 A schematic diagram of generating a target image in an image processing method provided by an embodiment of the present invention;

[0064] Figure 8 A schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0066] An embodiment of the present invention provides an image processing method, which can be applied to an image processing device. Figure 1 As shown, the method comprises the following steps:

[0067] Step 101: Obtain a first image to be trained, and generate a text region generator through training based on the first image to be trained.

[0068] The first image to be trained is an image with text information.

[0069] In an embodiment of the present invention, the first image to be trained may be a pre-acquired image with text information, and multiple images used to train the text region generator; wherein the text information in the first image to be trained may be located in the background region of the image, or in the object region of the image; that is, the text information in the first image to be trained may be located on the background of the image, or on the object of the image. It should be noted that the first image to be trained may be directly acquired, or may be generated based on an image without text information.

[0070] Step 102: Use a text region generator to process the first image to be trained to generate a label image to be processed.

[0071] The label image to be processed is used to indicate the area covered by the characters in the first image to be trained.

[0072] In an embodiment of the present invention, the size of the label image to be processed is the same as the size of the first image to be trained; in a feasible implementation method, one color is used in the area of ​​the label image to be processed corresponding to the area covered by the text information of the first image to be trained, and another color is used in the area of ​​the label image to be processed corresponding to the other area of ​​the first image to be trained except the area covered by the text information.

[0073] Step 103: Generate a text-missing image based on the label image to be processed and the first image to be trained, and perform training based on the text-missing image to generate an image restoration generator.

[0074] Among them, the size of the text-missing image is the same as the size of the first image to be trained, and the area in the text-missing image corresponding to the area covered by the text information of the first image to be trained is blank, and the information in the text-missing image except the blank area can be the same as the information of the first image to be trained.

[0075] Step 104: Acquire the image to be processed with text information.

[0076] Among them, the image to be processed can be an image that currently needs to delete the text in the image; it should be noted that the text information in the image to be processed can be located in the area where the background of the image is located, or it can be located in the area where the object of the image is located; that is to say, the text information in the image to be processed can be located on the background of the image, or it can be located on the object of the image.

[0077] Step 105: Use the text region generator and the image restoration generator to restore the image to be processed to obtain a target image.

[0078] In an embodiment of the present invention, the size of the target image is the same as that of the image to be processed, and the area in the target image corresponding to the area covered by the text information of the image to be processed has been filled with corresponding information, and the information of other areas of the target image except for the area corresponding to the area covered by the text information of the image to be processed is the same as that of the image to be processed; it should be noted that the information filled in the target image matches the information of the area around the area filled with the information in the target image.

[0079] The image processing method provided by the embodiment of the present invention obtains a first image to be trained with text information and generates a text area generator based on the first image to be trained, and uses the text area generator to process the first image to be trained to generate a label image to be processed for indicating the area covered by the text in the image to be processed, then generates a text-missing image based on the label image to be processed and the first image to be trained, and generates an image repair generator based on the text-missing image, and finally obtains the image to be processed with text information, and uses the text area generator and the image repair generator to repair the image to be processed to obtain a target image. In this way, the text area generator and the image repair generator can be obtained based on the training, and the image to be processed can be directly repaired. The text area generator can extract the area covered by the text in the image to be processed, instead of extracting a rectangular area including the text in the image to be processed as in the relative technology, and makes full use of the information of other areas in the image to be processed except the text area, solves the problem that the texture repair of the text area in the image repair scheme in the related technology is not clear and complete, and improves the effect of image repair.

[0080] Based on the above embodiments, an embodiment of the present invention provides an image processing method, referring to Figure 2 As shown, the method comprises the following steps:

[0081] Step 201: An image processing device obtains a second image to be trained.

[0082] The second image to be trained does not contain any text information.

[0083] In the embodiment of the present invention, the second image to be trained may be a plurality of images without text information; and the first image to be trained may be obtained by processing the second image to be trained.

[0084] Step 202: The image processing device adds text information to the second image to be trained to obtain the first image to be trained.

[0085] The image processing device may implement adding text information to the second image to be trained through code; in a feasible implementation, the Python language may be used to write code to implement adding text information to the second image to be trained.

[0086] Step 203: The image processing device generates a label image to be trained based on the first image to be trained.

[0087] The label image to be trained is used to indicate the area covered by the text in the first image to be trained.

[0088] In an embodiment of the present invention, the color of the area where the text information in the first image to be trained is located can be set to the first color, and the color of other areas in the first image to be trained except the area where the text information is located can be set to the second color, so as to obtain a label image to be trained; in a feasible implementation method, the first color can be black, and the second color can be white; it should be noted that the first image to be trained can be binarized to obtain the label image to be trained. During the binarization process, the area covered by the text information in the first image to be trained can be set to 0, and other areas except the area covered by the text information can be set to 1.

[0089] Step 204: The image processing device performs training based on the first image to be trained and the label image to be trained to generate a text area generator.

[0090] In the embodiment of the present invention, the initial text region generator may be trained using the first image to be trained and the label image to be trained to obtain the text region generator.

[0091] Wherein, step 204 can be implemented in the following manner:

[0092] The image processing device trains the initial text generator based on the first image to be trained, the label image to be trained and the first loss function to generate a text area generator.

[0093] Among them, the first loss function is the loss function corresponding to the initial text generator.

[0094] In an embodiment of the present invention, the image processing device may input the first image to be trained into the initial text region generator to generate a reference label image, and then use the first loss function to compare the reference label image and the label image to be trained, and continuously adjust the parameters in the initial text region generator based on the comparison result and the Adam optimizer, and finally train the text region generator. In a feasible implementation, the learning rate lr in the process of training the generated text generator can be 2e-4, β1 in the Adam optimizer can be 0.5, and β2 can be 0.99. It should be noted that the architecture of the initial text generator includes the UNet architecture.

[0095] Step 205: The image processing device processes the first image to be trained using a text region generator to generate a label image to be processed.

[0096] The label image to be processed is used to indicate the area covered by the characters in the first image to be trained.

[0097] In the embodiment of the present invention, the label image to be processed may be generated after the first image to be trained is input into the text region generator.

[0098] Step 206: The image processing device determines a first pixel matrix corresponding to the label image to be processed and a second pixel matrix corresponding to the first image to be trained.

[0099] The first pixel matrix can be obtained based on each pixel value of the label image to be processed; the pixel value of each pixel of the label image to be processed can be obtained, and then the first pixel matrix is ​​generated based on the pixel value of each pixel. Similarly, the second pixel matrix can be obtained based on each pixel value of the first image to be trained; the pixel value of each pixel of the first image to be trained can be obtained, and then the second pixel matrix is ​​generated based on the pixel value of each pixel.

[0100] Step 207: The image processing device generates a text-missing image based on the first pixel matrix and the second pixel matrix.

[0101] The text-missing image may be obtained by obtaining a pixel matrix based on the first pixel matrix and the second pixel matrix, and then acquiring an image corresponding to the obtained pixel matrix to obtain the text-missing image.

[0102] Step 208: The image processing device trains the initial image restoration generator based on the text-missing image, the second image to be trained, and the third loss function to generate an image restoration generator.

[0103] Among them, the third loss function is the loss function corresponding to the image restoration generator.

[0104] In an embodiment of the present invention, the image processing device may input the text-missing image into the initial image restoration generator to generate a reference image, then use a third loss function to compare the reference image and the second image to be trained, and continuously adjust the parameters in the initial image restoration generator based on the comparison result, and finally train the image restoration generator. It should be noted that the third loss function may be L1 loss and GAN loss.

[0105] Step 209: The image processing device obtains the image to be processed with text information.

[0106] Step 210: The image processing device uses a text region generator and an image restoration generator to restore the image to be processed to obtain a target image.

[0107] The target image may be obtained by first inputting the image to be processed into a text area generator to generate an intermediate label image, and then inputting an image based on the intermediate label image and the image to be processed into an image restoration generator.

[0108] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0109] The image processing method provided by the embodiment of the present invention can obtain a text area generator and an image restoration generator based on training, and directly restore the image to be processed. The text area generator can extract the area covered by the text in the image to be processed, instead of extracting a rectangular area including the text in the image to be processed as in the related technology. It makes full use of the information of other areas in the image to be processed except the text area, solves the problem that the texture restoration of the text area in the image restoration scheme in the related technology is not clear and complete, and improves the effect of image restoration.

[0110] Based on the above embodiments, an embodiment of the present invention provides an image processing method, referring to Figure 3 As shown, the method comprises the following steps:

[0111] Step 301: An image processing device obtains a second image to be trained.

[0112] The second image to be trained does not contain any text information.

[0113] Step 302: The image processing device adds text information to the second image to be trained to obtain the first image to be trained.

[0114] Step 303: The image processing device sets the color of the region where the text is located in the first image to be trained to the first color, and sets the color of other regions except the text in the first image to be trained to the second color, to obtain a label image to be trained.

[0115] The label image to be trained is used to indicate the area covered by the text in the first image to be trained.

[0116] In the embodiment of the present invention, the first color is different from the second color. The first color may be white, and the second color may be black.

[0117] Step 304: The image processing device trains the first initial discriminator based on the first image to be trained, the label image to be trained, and the second loss function to generate a first discriminator.

[0118] Among them, the second loss function is the loss function corresponding to the initial discriminator.

[0119] It should be noted that step 304 can be timed in the following ways:

[0120] Step 304a: The image processing device uses an initial text generator to process the first image to be trained to generate a reference label image.

[0121] The reference label image may be obtained by directly inputting the first image to be trained into the initial text generator. It should be noted that the size of the reference label image is the same as that of the first image to be trained; one color is used for the area in the reference label image corresponding to the area covered by the text information of the first image to be trained, and another color is used for the area in the reference label image corresponding to the other areas of the first image to be trained except the area covered by the text information. In a feasible implementation, the color of the area in the reference label image corresponding to the area covered by the text information of the first image to be trained may be white, and the color of the area in the reference label image corresponding to the other areas of the first image to be trained except the area covered by the text information may be black.

[0122] Step 304b: The image processing device trains the first initial discriminator based on the reference label image, the label image to be trained, and the second loss function to generate a first discriminator.

[0123] The second loss function is used to compare the reference label image and the label image to be trained, and then the parameters of the first initial discriminator are continuously updated according to the comparison result, and finally the first discriminator is obtained. Figure 4 As shown, the first image A to be trained can be input into the initial text generator to generate a reference label image B, and then the reference label image can be input into the first initial discriminator for training to obtain the first discriminator.

[0124] Step 305: The image processing device trains the initial text generator based on the first image to be trained, the label image to be trained, the first loss function and the first discriminator to generate a text area generator.

[0125] The architecture of the text generator includes a UNet architecture; and the first discriminator includes a Markov discriminator. It should be noted that the block in the discriminator of the embodiment of the present invention can adopt a size of 60×60.

[0126] In an embodiment of the present invention, a first image to be trained can be input into an initial text generator to generate a reference label image, and then a first loss function is used to compare the reference label image and the label image to be trained, and the reference label image is corrected according to the comparison result. Then, the repaired reference label image and the label image to be trained are input into a first discriminator for comparison processing, and the parameters of the initial text generator are continuously updated according to the comparison result, and finally a text area generator is obtained.

[0127] Step 306: The image processing device processes the first image to be trained using a text region generator to generate a label image to be processed.

[0128] The label image to be processed is used to indicate the area covered by the characters in the first image to be trained.

[0129] Step 307: The image processing device determines a first pixel matrix corresponding to the label image to be processed and a second pixel matrix corresponding to the first image to be trained.

[0130] Step 308: The image processing device generates a third pixel matrix based on the first pixel matrix and the second pixel matrix.

[0131] Among them, the third pixel matrix can be obtained by generating a pixel matrix based on the first pixel matrix and the second pixel, and assigning values ​​to specific elements in the generated pixel matrix; in a feasible implementation method, the first pixel matrix can be multiplied by the second pixel matrix, and then the values ​​of the elements in the pixel matrix obtained after the multiplication that are not 0 are set to 255 to obtain the third pixel matrix.

[0132] Step 309: The image processing device determines the image corresponding to the third pixel matrix to obtain a text-missing image.

[0133] In the embodiment of the present invention, Figure 5 As shown, the first image A to be trained is input into the text region generator to generate a label image C to be processed, and then a text-missing image D is generated based on the label image C to be processed and the first image A to be trained.

[0134] Step 310: The image processing device trains a second initial discriminator based on the text-missing image, the second image to be trained, and the fourth loss function to generate a second discriminator.

[0135] Among them, the fourth loss function is the loss function corresponding to the second initial discriminator.

[0136] It should be noted that step 310 trains the second initial discriminator based on the text-missing image, the second image to be trained and the fourth loss function, and the generation of the second discriminator can be achieved in the following manner:

[0137] Step 310a: The image processing device uses an initial image restoration generator to restore the text-missing image to generate a reference image.

[0138] The reference image may be an image obtained by directly inputting the text-missing image into the initial image restoration generator.

[0139] Step 310b: The image processing device trains the second initial discriminator based on the reference image, the second image to be trained and the fourth loss function to generate a second discriminator.

[0140] Among them, the fourth loss function is used to compare the reference image and the first image to be trained, and then the parameters of the second initial discriminator are continuously updated according to the comparison result, and finally the second discriminator is obtained.

[0141] Step 311: The image processing device trains the initial image restoration generator based on the text-missing image, the second image to be trained, the third loss function and the second discriminator to generate an image restoration generator.

[0142] The architecture of the image restoration generator may include a UNet structure; the second discriminator uses a spectral normalized Markovian discriminator (SN-patchGAN); the size of the block in the present invention may be 60×60. In a feasible implementation, the present invention uses a text-missing image, a second image to be trained, and a third loss function to train the initial image restoration generator to generate an image restoration generator. The structural diagram corresponding to the process can be as follows: Figure 6 As shown, the text-missing image D and the label image to be processed C are used for training to obtain a reference image E, and then the second discriminator is used to discriminate the reference image to finally generate an image restoration generator.

[0143] In an embodiment of the present invention, the shallow layer of the encoding part of the image restoration generator adopts a self-attention structure to encode the relationship information between any two pixels in the text-missing image; in this way, when there is a structural and texture relationship between the pixels farther away from the text-missing area and other parts of the image, an area similar to or related to the text-missing area can be generated, ensuring the consistency of the information of the repaired area and the surrounding area to be repaired, greatly improving the repair effect. The shallow layer of the second discriminator adopts self-attention to obtain more information, which is convenient for fast training. Moreover, the convolution in the second discriminator and the image restoration generator adopts gated convolution. Among them, the gated convolution can adopt the function Gating y,x To indicate that W g and W f It represents the weight of the pixel, I represents the pixel of the text missing image and the second image to be trained, and the feature function can be used y,x To express; Gating y,x =∑∑W g *I, Feature y,x =∑∑W f *I; When performing convolution operation, formula O can be used y，x =φ(Feature y,x )*σ(Gating y,x ) is used; it should be noted that when performing convolution operations, convolution operations can be performed layer by layer.

[0144] Step 312: The image processing device obtains the image to be processed with text information.

[0145] Step 313: The image processing device uses a text region generator to process the image to be processed to generate an intermediate label image.

[0146] Step 314: The image processing device generates an intermediate text-missing image based on the intermediate label image and the image to be processed.

[0147] It should be noted that the specific implementation process of generating the intermediate text-missing image based on the intermediate label image and the image to be processed in this embodiment is the same as the implementation process of generating the text-missing image based on the label image to be processed and the first image to be trained in the above embodiment. Please refer to the description in the above embodiment and will not be repeated here.

[0148] Step 315: The image processing device uses an image restoration generator to restore the image with missing middle text to obtain a target image.

[0149] Among them, Figure 7 As shown, the image F to be processed is input into the text area generator to generate an intermediate label image G, and then an intermediate text missing image H is generated based on the intermediate label image G and the image to be processed F. Finally, the intermediate text missing image H is input into the image restoration generator to obtain the target image K.

[0150] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0151] The image processing method provided by the embodiment of the present invention can obtain a text area generator and an image restoration generator based on training, and directly restore the image to be processed. The text area generator can extract the area covered by the text in the image to be processed, instead of extracting a rectangular area including the text in the image to be processed as in the related technology. It makes full use of the information of other areas in the image to be processed except the text area, solves the problem that the texture restoration of the text area in the image restoration scheme in the related technology is not clear and complete, and improves the effect of image restoration.

[0152] Based on the above embodiments, an embodiment of the present invention provides an image processing device, which can be applied to Figures 1 to 3 In an image processing method provided in a corresponding embodiment, referring to Figure 8 As shown, the device may include: a processor 41, a memory 42 and a communication bus 43, wherein:

[0153] The communication bus 43 is used to realize the communication connection between the processor 41 and the memory 42;

[0154] The processor 41 is used to execute the image processing program stored in the memory 42 to implement the following steps:

[0155] Acquire a first image to be trained, and generate a text area generator by training based on the first image to be trained;

[0156] Wherein, the first image to be trained is an image with text information;

[0157] Using a text region generator to process the first image to be trained to generate a label image to be processed;

[0158] The label image to be processed is used to indicate the area covered by the text in the first image to be trained;

[0159] Generate a text-missing image based on the label image to be processed and the first image to be trained, and perform training based on the text-missing image to generate an image restoration generator;

[0160] Obtaining an image to be processed with text information;

[0161] The text region generator and the image restoration generator are used to restore the image to be processed and obtain the target image.

[0162] In other embodiments of the present invention, the processor 41 is used to execute the acquisition of the first to-be-trained image stored in the memory 42 to implement the following steps:

[0163] Acquire a second image to be trained;

[0164] Wherein, the second image to be trained does not contain text information;

[0165] Text information is added to the second image to be trained to obtain the first image to be trained.

[0166] In other embodiments of the present invention, the processor 41 is used to execute the text area generator stored in the memory 42 for training based on the first image to be trained, so as to implement the following steps:

[0167] Based on the first image to be trained, generating a label image to be trained;

[0168] The label image to be trained is used to indicate the area covered by the text in the first image to be trained;

[0169] Training is performed based on the first image to be trained and the label image to be trained to generate a text area generator.

[0170] In other embodiments of the present invention, the processor 41 is used to execute the steps stored in the memory 42 to generate a label image to be trained based on the first image to be trained, so as to implement the following steps:

[0171] The color of the region where the text is located in the first image to be trained is set to the first color, and the color of other regions except the text in the first image to be trained is set to the second color, to obtain a label image to be trained.

[0172] In other embodiments of the present invention, the processor 41 is used to execute the training based on the first image to be trained and the label image to be trained stored in the memory 42 to generate a text area generator to implement the following steps:

[0173] Training the initial text generator based on the first image to be trained, the label image to be trained and the first loss function to generate a text area generator;

[0174] Among them, the first loss function is the loss function corresponding to the initial text generator.

[0175] In other embodiments of the present invention, the processor 41 is used to execute the training of the initial text generator based on the first image to be trained, the label image to be trained and the first loss function stored in the memory 42 to generate a text area generator, so as to implement the following steps:

[0176] Training a first initial discriminator based on a first image to be trained, a label image to be trained, and a second loss function to generate a first discriminator;

[0177] Among them, the second loss function is the loss function corresponding to the initial discriminator;

[0178] Based on the first image to be trained, the label image to be trained, the first loss function and the first discriminator, the initial text generator is trained to generate a text area generator.

[0179] In other embodiments of the present invention, the processor 41 is used to execute the training of the first initial discriminator based on the first image to be trained, the label image to be trained and the second loss function stored in the memory 42 to generate the first discriminator, so as to implement the following steps:

[0180] Using an initial text generator to process the first image to be trained to generate a reference label image;

[0181] The first initial discriminator is trained based on the reference label image, the label image to be trained, and the second loss function to generate a first discriminator.

[0182] In other embodiments of the present invention, the architecture of the text region generator includes a UNet architecture; and the first discriminator includes a Markov discriminator.

[0183] In other embodiments of the present invention, the processor 41 is used to execute the steps stored in the memory 42 to generate a text-missing image based on the label image to be processed and the first image to be trained, and to generate a first discriminator, so as to implement the following steps:

[0184] Determine a first pixel matrix corresponding to the label image to be processed and a second pixel matrix corresponding to the first image to be trained;

[0185] Generate a text-missing image based on the first pixel matrix and the second pixel matrix;

[0186] Training is performed based on the text-missing image and the first image to be trained to generate an image restoration generator.

[0187] In other embodiments of the present invention, the processor 41 is used to execute the steps stored in the memory 42 to generate a text-missing image based on the first pixel matrix and the second pixel matrix, so as to implement the following steps:

[0188] Based on the first pixel matrix and the second pixel matrix, generating a third pixel matrix;

[0189] An image corresponding to the third pixel matrix is ​​determined to obtain a text-missing image.

[0190] In other embodiments of the present invention, the processor 41 is used to execute the training based on the text-missing image stored in the memory 42 to generate an image restoration generator to implement the following steps:

[0191] Based on the text-missing image, the second image to be trained and the third loss function, the initial image restoration generator is trained to generate an image restoration generator;

[0192] Among them, the third loss function is the loss function corresponding to the image restoration generator.

[0193] In other embodiments of the present invention, the processor 41 is used to execute the training of the initial image restoration generator based on the text-missing image, the second image to be trained and the third loss function stored in the memory 42 to generate the image restoration generator, so as to implement the following steps:

[0194] Training the second initial discriminator based on the text-missing image, the second image to be trained, and the fourth loss function to generate a second discriminator;

[0195] Among them, the fourth loss function is the loss function corresponding to the second initial discriminator;

[0196] Based on the text-missing image, the second image to be trained, the third loss function and the second discriminator, the initial image restoration generator is trained to generate an image restoration generator.

[0197] In other embodiments of the present invention, the processor 41 is used to execute the training of the second initial discriminator based on the text-missing image, the second image to be trained and the fourth loss function stored in the memory 42 to generate the second discriminator, so as to implement the following steps:

[0198] The initial image restoration generator is used to restore the text-missing image and generate a reference image.

[0199] The second initial discriminator is trained based on the reference image, the second image to be trained and the fourth loss function to generate a second discriminator.

[0200] In other embodiments of the present invention, the processor 41 is used to execute the image restoration generator and the text area generator stored in the memory 42 to restore the image to be processed to obtain a target image, so as to implement the following steps:

[0201] The text region generator is used to process the image to be processed to generate an intermediate label image;

[0202] Based on the intermediate label image and the image to be processed, an intermediate text missing image is generated;

[0203] The image inpainting generator is used to inpaint the image with missing text in the middle to obtain the target image.

[0204] It should be noted that the specific implementation process of the steps executed by the processor in this embodiment can refer to Figures 1 to 3 The implementation process of the image processing method provided in the corresponding embodiment will not be repeated here.

[0205] The image processing device provided by the embodiment of the present invention can obtain a text area generator and an image restoration generator based on training, and directly restore the image to be processed. The text area generator can extract the area covered by the text in the image to be processed, instead of extracting a rectangular area including the text in the image to be processed as in the related technology. It makes full use of the information of other areas in the image to be processed except the text area, solves the problem of unclear and incomplete texture restoration of the text area in the image restoration scheme in the related technology, and improves the effect of image restoration.

[0206] Based on the foregoing embodiments, an embodiment of the present invention provides a computer-readable storage medium, the computer-readable storage medium stores one or more programs, the one or more programs can be executed by one or more processors to implement Figures 1 to 3 The corresponding embodiments provide steps in the image processing method.

[0207] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0208] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0211] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a first image to be trained, and generate a text region generator based on the first image to be trained; wherein the first image to be trained is an image with text information; The first image to be trained is processed by the text region generator to generate a label image to be processed; wherein the label image to be processed is used to indicate the area covered by the text in the first image to be trained; Generate a text-missing image based on the to-be-processed label image and the first to-be-trained image, and perform training based on the text-missing image to generate an image restoration generator; Obtaining an image to be processed with text information; The text region generator and the image restoration generator are used to restore the image to be processed to obtain a target image.

2. The method according to claim 1, characterized in that The step of obtaining a first image to be trained comprises: Acquire a second image to be trained; wherein the second image to be trained does not contain any text information; Text information is added to the second image to be trained to obtain the first image to be trained.

3. The method according to claim 1 or 2, characterized in that: The step of generating a text area generator through training based on the first image to be trained includes: Based on the first image to be trained, generating a label image to be trained; wherein the label image to be trained is used to indicate an area covered by text in the first image to be trained; The text area generator is generated by performing training based on the first image to be trained and the label image to be trained.

4. The method according to claim 3, characterized in that The step of generating a label image to be trained based on the first image to be trained comprises: The color of the region where the characters are located in the first image to be trained is set to a first color, and the color of other regions except the characters in the first image to be trained is set to a second color, to obtain the label image to be trained.

5. The method according to claim 3, characterized in that: The step of training based on the first image to be trained and the label image to be trained to generate the text area generator includes: The initial text generator is trained based on the first image to be trained, the label image to be trained and a first loss function to generate the text area generator; wherein the first loss function is a loss function corresponding to the initial text generator.

6. The method according to claim 5, characterized in that The step of training the initial text generator based on the first image to be trained, the label image to be trained and the first loss function to generate the text area generator includes: Training a first initial discriminator based on the first image to be trained, the label image to be trained, and the second loss function to generate a first discriminator; wherein the second loss function is a loss function corresponding to the initial discriminator; Based on the first image to be trained, the label image to be trained, the first loss function and the first discriminator, the initial text generator is trained to generate the text area generator.

7. The method according to claim 6, characterized in that The step of training the first initial discriminator based on the first image to be trained, the label image to be trained, and the second loss function to generate the first discriminator includes: Using the initial text generator to process the first image to be trained to generate a reference label image; The first initial discriminator is trained based on the reference label image, the label image to be trained and the second loss function to generate the first discriminator.

8. The method according to claim 6, characterized in that The architecture of the text region generator includes a UNet architecture; the first discriminator includes a Markov discriminator.

9. The method according to claim 1, characterized in that: The step of generating a text-missing image based on the to-be-processed label image and the first to-be-trained image includes: Determine a first pixel matrix corresponding to the to-be-processed label image and a second pixel matrix corresponding to the first to-be-trained image; The text-missing image is generated based on the first pixel matrix and the second pixel matrix.

10. The method according to claim 9, characterized in that The step of generating the text-missing image based on the first pixel matrix and the second pixel matrix includes: Generate a third pixel matrix based on the first pixel matrix and the second pixel matrix; An image corresponding to the third pixel matrix is ​​determined to obtain the text-missing image.

11. The method according to claim 2, characterized in that The step of training based on the text-missing image to generate an image restoration generator comprises: Based on the text-missing image, the second image to be trained and the third loss function, the initial image restoration generator is trained to generate the image restoration generator; wherein the third loss function is the loss function corresponding to the image restoration generator.

12. The method according to claim 11, characterized in that The step of training the initial image restoration generator based on the text-missing image, the second image to be trained and the third loss function to generate the image restoration generator comprises: Training the second initial discriminator based on the text-missing image, the second image to be trained and a fourth loss function to generate a second discriminator; wherein the fourth loss function is a loss function corresponding to the second initial discriminator; Based on the text-missing image, the second image to be trained, the third loss function and the second discriminator, the initial image restoration generator is trained to generate the image restoration generator.

13. The method according to claim 12, characterized in that The step of training the second initial discriminator based on the text-missing image, the second image to be trained and the fourth loss function to generate the second discriminator includes: Using the initial image restoration generator to restore the text-missing image to generate a reference image; The second initial discriminator is trained based on the reference image, the second image to be trained and the fourth loss function to generate the second discriminator.

14. The method according to claim 1, characterized in that The method of using the image restoration generator and the text region generator to restore the image to be processed to obtain a target image includes: Using the text region generator to process the image to be processed to generate an intermediate label image; Based on the intermediate label image and the image to be processed, generating an intermediate text-missing image; The image restoration generator is used to restore the intermediate text-missing image to obtain the target image.

15. An image processing device, characterized in that: The device comprises: a processor, a memory and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute the image processing program stored in the memory to implement the following steps: Acquire a first image to be trained, and generate a text area generator based on the first image to be trained; wherein the first image to be trained is an image with text information; use the text area generator to process the first image to be trained to generate a label image to be processed; wherein the label image to be processed is used to indicate the area covered by the text in the first image to be trained; generate a text-missing image based on the label image to be processed and the first image to be trained, and generate an image restoration generator based on the text-missing image; Obtaining an image to be processed with text information; The image to be processed is repaired using the image repair generator and the text area generator to obtain a target image.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the image processing method according to any one of claims 1 to 14.

Images