Image processing method, apparatus, device, storage medium and program product

Abstract

The application provides an image processing method, device and equipment, a storage medium and a program product, belongs to the field of image text processing of artificial intelligence, can reduce the model burden, and comprises the following steps: performing an iteration step, which comprises the following steps: splicing an original image and an erasure image of a previous iteration to obtain a spliced image of a current iteration; performing text erasure on the spliced image of the current iteration by using a text erasing model to output a predicted image of the current iteration; superimposing the predicted image of the current iteration and the original image according to position information of text to be erased in the original image to obtain an erasure image of the current iteration; when the iteration is performed to the last time, taking the erasure image of the last iteration as a target image; in the erasure image of the current iteration, a first region corresponding to the position information is derived from a region corresponding to the position information in the predicted image of the current iteration, and a second region other than the first region is derived from the original image.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, storage medium and computer program product, and also to a text erasure model processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology and artificial intelligence, image text erasure technology has gained increasing applications, playing an important role in privacy protection, visual information translation, and image content editing. Image text erasure technology removes text from an image and restores the original background after the text is removed. This task not only erases the text from the original image but also considers restoring the original background of the text area while erasing the text, which is the technical difficulty of this task.

[0003] Currently, image text erasure technology is mostly implemented based on end-to-end models. The end-to-end model identifies the location of the text area in the image and erases the text from the entire image. On the one hand, in addition to the ability to erase text, the end-to-end model also needs to implicitly have the ability to recognize text areas, which puts a heavy burden on the model. In addition, since the text is directly erased from the entire image, non-text areas are usually also affected. The non-text areas cannot remain consistent with the original image, which affects the erasure effect. Summary of the Invention

[0004] Based on this, it is necessary to provide an image processing method, device, computer equipment, storage medium and computer program product that can reduce the model burden and improve the erasure effect in response to the above technical problems, and also provide a text erasure model processing method, device, computer equipment, storage medium and computer program product.

[0005] The present application provides an image processing method, the method comprising:

[0006] Acquire an original image and position information of text to be erased in the original image;

[0007] Executing an iterative step; the iterative step includes: splicing the original image with the erased image of the previous iteration to obtain a spliced ​​image of the current iteration; erasing the current text on the spliced ​​image of the current iteration using a trained text erasure model, and outputting a predicted image of the current iteration; and superimposing the predicted image of the current iteration with the original image according to the position information to obtain an erased image of the current iteration;

[0008] When the iteration stop condition is met, an erased image of the last iteration is obtained, and the erased image of the last iteration is used as a target image obtained by performing multiple text erasures on the original image;

[0009] In which, the stitched image of the first iteration is obtained by stitching the original image with the original image; in the erased image of the current iteration obtained after superposition, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

[0010] The present application provides an image processing device, comprising:

[0011] A text position acquisition module, used to acquire the original image and the position information of the text to be erased in the original image;

[0012] An iterative erasure module is configured to execute an iterative step; the iterative step comprising: splicing the original image with the erased image of the previous iteration to obtain a spliced ​​image of the current iteration; erasing the text of the current iteration on the spliced ​​image of the current iteration using a trained text erasure model, and outputting a predicted image of the current iteration; and superimposing the predicted image of the current iteration with the original image according to the position information to obtain an erased image of the current iteration;

[0013] a target image acquisition module, configured to obtain an erased image of the last iteration when an iteration stop condition is satisfied, and use the erased image of the last iteration as a target image obtained by performing multiple text erasures on the original image;

[0014] In which, the stitched image of the first iteration is obtained by stitching the original image with the original image; in the erased image of the current iteration obtained after superposition, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

[0015] In one embodiment, the text position acquisition module is used to acquire an original image; perform text detection on the original image to determine a text area in the original image; and use the coordinates of the text area as position information of the text to be erased in the original image.

[0016] In one embodiment, the iterative erasing module is used to obtain an erased image of a previous iteration; and to splice the original image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration.

[0017] In one embodiment, an iterative erasure module is used to input the spliced ​​image of the current iteration into a trained text erasure model; perform feature extraction on the spliced ​​image of the current iteration through the encoding network in the text erasure model to obtain image features of the spliced ​​image of the current iteration; perform image reconstruction based on the image features through the transposed convolutional network in the text erasure model, and output a predicted image of the current iteration.

[0018] In one embodiment, the apparatus further includes a mask image generating module, configured to generate a mask image corresponding to the original image according to the position information, wherein the mask image is used to distinguish between a text area and a non-text area in the original image;

[0019] The iterative erasing module is used to superimpose the predicted image of the current iteration with the original image according to the mask image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

[0020] In one embodiment, the mask image generation module is further used to determine the text area and non-text area in the original image according to the position information; set the pixel value in the text area in the original image to 1, and set the pixel value in the non-text area in the original image to 0, to obtain a binary mask image.

[0021] In one embodiment, the erased image of the current iteration is obtained by the following formula:

[0022] dst=mask*output+(1-mask)*src;

[0023] Wherein, mask represents the mask image, output represents the predicted image of the current iteration, src represents the original image, and dst represents the erased image of the current iteration.

[0024] In one embodiment, the apparatus further comprises an image acquisition module for receiving an original image including the text to be erased sent by the online collaborative document application;

[0025] The device further includes a target image feedback module for returning the target image obtained by the multiple erasures to the online collaborative document application, so that the document can be edited through the online collaborative document using the restored target image as an editing background.

[0026] In one embodiment, the apparatus includes an image acquisition module configured to acquire an original image including the text to be translated;

[0027] The device further includes a translation processing module for obtaining a translation corresponding to the text to be translated; and pasting the translation into the target image according to the position of the text to be translated in the original image to obtain a translated image corresponding to the original image.

[0028] The present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above-mentioned image processing method.

[0029] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is used by a processor to execute the above-mentioned image processing method.

[0030] The present application provides a computer program product, including a computer program, which implements the above-mentioned image processing method when executed by a processor.

[0031] In the above-mentioned image processing method, apparatus, computer equipment, storage medium and computer program product, when the text erasure model has a light burden, that is, it only has the ability to erase text, by inputting the entire original image into the text erasure model and performing multiple iterative erasures, a target image with a better erasure effect can also be obtained. Specifically, in each iteration, the original image and the erased image of the previous iteration are spliced ​​to obtain a spliced ​​image. The spliced ​​image carries the original image information and the image information after the previous erasure. The spliced ​​image is input into the text erasure model, and the output predicted image can not only better erase the text in the text area, but also retain the boundary information between the text area and the non-text area, so that the background repair effect in the text area is better. Further, according to the position information of the text area where the text to be erased is located in the original image, the predicted image is superimposed on the original image to obtain the erased image of the current iteration, so that the first area corresponding to the position information in the erased image is derived from the predicted image, and the second area other than the first area is derived from the original image. In this way, the image content outside the text area can be completely retained, and the erasure result can be fully adapted to the position information of the text to be erased in the original image, thereby improving the erasure effect.

[0032] This application provides a method for processing a text erasure model, the method comprising:

[0033] Acquire a training sample, where the training sample includes a sample image, position information of text in the sample image, and a label image corresponding to the sample image that does not include the text;

[0034] For the training sample, an iterative step is performed until an iteration stop condition is satisfied; the iterative step includes: splicing the sample image with the erased image of the previous iteration to obtain a spliced ​​image of the current iteration, inputting the spliced ​​image of the current iteration into a predetermined neural network model, performing text erasure on the spliced ​​image of the current iteration, outputting a predicted image of the current iteration, and superimposing the predicted image of the current iteration with the sample image according to the position information to obtain an erased image of the current iteration;

[0035] Based on the erased image, predicted image and label image of each iteration, the loss of each iteration of the sample image is calculated, and the erasure loss corresponding to the training sample is obtained according to the sum of the losses of multiple iterations obtained when the iteration stops. After updating the network parameters of the predetermined neural network according to the erasure loss, the method returns to the step of obtaining the training sample and continues training until the training stop condition is met, thereby obtaining a trained text erasure model.

[0036] The present application provides a device for processing a text erasure model, the device comprising:

[0037] A training sample acquisition module is used to acquire a training sample, wherein the training sample includes a sample image, position information of text in the sample image, and a label image corresponding to the sample image that does not include the text;

[0038] an iterative erasing module, configured to execute an iterative step for the training sample until an iteration stop condition is satisfied; the iterative step comprising: splicing the sample image with the erased image of the previous iteration to obtain a spliced ​​image of the current iteration, inputting the spliced ​​image of the current iteration into a predetermined neural network model, performing text erasure on the spliced ​​image of the current iteration, outputting a predicted image of the current iteration, and superimposing the predicted image of the current iteration with the sample image according to the position information to obtain an erased image of the current iteration;

[0039] The loss parameter adjustment module is used to calculate the loss of each iteration of the sample image based on the erased image, the predicted image and the label image of each iteration, obtain the erasure loss corresponding to the training sample according to the sum of the losses of multiple iterations obtained when the iteration stops, update the network parameters of the predetermined neural network according to the erasure loss, return to the step of obtaining the training sample to continue training until the training stop condition is met, and obtain a trained text erasure model.

[0040] In one embodiment, the training sample acquisition module is used to use an image that does not include text as a label image; add text to the label image to obtain a sample image; perform text detection on the sample image to obtain position information of the text in the sample image.

[0041] In one embodiment, the iterative erasing module is used to obtain an erased image of a previous iteration; and to splice the sample image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration.

[0042] In one embodiment, the iterative erasing module is used to input the spliced ​​image of the current iteration into a predetermined neural network model; perform feature extraction on the spliced ​​image of the current iteration through the encoding network in the predetermined neural network model to obtain image features of the spliced ​​image of the current iteration; perform image reconstruction based on the image features through the transposed convolutional network in the predetermined neural network model, and output a predicted image of the current iteration.

[0043] In one embodiment, the apparatus further comprises a mask image generating module, configured to generate a mask image corresponding to the sample image according to the position information, wherein the mask image is used to distinguish between a text area and a non-text area in the sample image;

[0044] The iterative erasing module is further used to superimpose the predicted image of the current iteration with the sample image according to the mask image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the sample image.

[0045] In one embodiment, the mask image generation module is further used to determine the text area and non-text area in the sample image according to the position information; set the pixel value in the text area in the sample image to 1, and set the pixel value in the non-text area in the sample image to 0, to obtain a binary mask image.

[0046] In one embodiment, the loss parameter adjustment module is further used to determine the text areas in the predicted image and the label image of each iteration according to the position information of the text in the sample image; calculate the distance between the predicted image of each iteration and the text area in the label image as the text area erasure loss of each iteration; calculate the image difference between the erased image of each iteration and the label image as the image difference loss of each iteration; calculate the feature difference between the erased image of each iteration and the label image as the feature difference loss of each iteration; based on the text area erasure loss, image difference loss and feature difference loss of each iteration, obtain the loss of each iteration of the sample image.

[0047] In one embodiment, the loss parameter adjustment module is further used to input the erased image of each iteration and the label image into the trained feature extraction model respectively to obtain the image features corresponding to the erased image of each iteration and the label image respectively; calculate the distance between the image features corresponding to the erased image of each iteration and the label image respectively to obtain the perceptual loss between the erased image of each iteration and the label image; calculate the corresponding Gram matrix according to the image features corresponding to the erased image of each iteration, calculate the corresponding Gram matrix according to the image features corresponding to the label image, calculate the difference between the Gram matrices of the erased image of each iteration and the label image respectively, and obtain the style loss between the erased image of each iteration and the label image; obtain the feature difference loss based on the perceptual loss and the style loss.

[0048] The present application provides a computer device including a memory and a processor, wherein the memory stores a computer program and the processor executes the above-mentioned text erasure model processing method.

[0049] The present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the above-mentioned text erasure model processing method.

[0050] The present application provides a computer program product, including a computer program, which implements the above-mentioned text erasure model processing method when executed by a processor.

[0051] In the processing method device, computer equipment, storage medium and computer program product of the above-mentioned text erasure model, since the trained model only needs to have text erasure capability but does not need text detection capability to reduce the model burden while having a good erasing effect, during the training process, the sample image and the erased image of the previous iteration are iteratively spliced ​​to obtain a spliced ​​image, which carries the sample image information and the information of the image after the previous erasure. The spliced ​​image is input into the model and the predicted image is output, so that the model learns to pay attention to the text area that needs to be erased in the sample image, and then the text is detected according to the text area where the text to be erased in the sample image is located. The position information of this area is used to superimpose the predicted image and the sample image to obtain the erased image of the current iteration, so that the first area corresponding to the position information in the erased image is derived from the predicted image, and the second area other than the first area is derived from the sample image. Then, based on the erased image, predicted image and label image of each iteration, the loss of each iteration of the sample image is calculated. The text erasure model obtained by optimization based on the loss of multiple iterations can output an output that can not only better erase the text in the text area, but also retain the boundary information between the text area and the non-text area, so as to achieve the repair effect on the background in the text area. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A diagram illustrating an application environment of an image processing method or a text erasure model processing method in one embodiment;

[0053] Figure 2 1 is a flow chart of an image processing method according to an embodiment;

[0054] Figure 3 is a schematic diagram of an original image in one embodiment;

[0055] Figure 4 is a schematic diagram of a text area in an original image where text to be erased is located in one embodiment;

[0056] Figure 5 is a schematic diagram of a target image in one embodiment;

[0057] Figure 6 is a schematic diagram of an original image including text to be translated in one embodiment;

[0058] Figure 7 is a schematic diagram of a text area where text to be translated is located in one embodiment;

[0059] Figure 8A is a schematic diagram of a target image in another embodiment;

[0060] Figure 8B is a schematic diagram of a target image with a translation pasted thereon in one embodiment;

[0061] Figure 9 A schematic diagram of the structure of a text erasure model in one embodiment;

[0062] Figure 10 is a flowchart of an image processing method in another embodiment;

[0063] Figure 11 Schematic diagram of a process flow of a text erasure model processing method in one embodiment;

[0064] Figure 12 Schematic diagram of a flow chart of a method for processing a text erasure model in another embodiment;

[0065] Figure 13 is a structural block diagram of an image processing device in one embodiment;

[0066] Figure 14 is a structural block diagram of a processing device for a text erasure model in one embodiment;

[0067] Figure 15 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0069] Mentioning "embodiment" in this application means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is understood explicitly and implicitly by those skilled in the art that the embodiments described in this application can be combined with other embodiments. It should be noted that the "first", "second", etc. involved in the introduction of this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0070] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0071] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0072] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0073] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0074] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0075] The solution provided in the embodiments of the present application relates to an artificial intelligence image text processing technology, which may include an image text erasure technology. The image text erasure technology involves text detection and text erasure; wherein, text detection is to locate the area where the text is located in the image, and text erasure is to remove the text in the image and restore the background obscured by the text.

[0076] Traditional end-to-end image text erasure techniques do not require the location of text regions; the end-to-end model automatically locates and removes text regions within an image. Training of the end-to-end model requires paired, intact images before and after erasure. This approach requires the end-to-end model to possess text detection capabilities, placing a significant burden on the model. Furthermore, the resulting image after text erasure does not accurately match the text detection results from the original image.

[0077] Through improvements in the training process and inference process, the text erasure model of this application does not need to have the ability of text detection, is completely separated from the text detection end, and focuses on text erasure and background repair of text areas. At the same scale, the text erasure model of this application has better text erasure effect, and will not modify the image content outside the text area, and is compatible with the text detection results of the image.

[0078] Figure 1 1 is an application environment diagram of an image processing method or a text erasure model processing method in an embodiment. The terminal 102 communicates with the server 104 via a communication network. The terminal 102 can interact with the server 104 via a communication network. The terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster or cloud server composed of multiple servers. The server 104 can store sample images, location information of text in the sample images, and label images corresponding to the sample images, etc. through a data storage system; the data storage system can be integrated in the server 104 or can be set separately from the server 104.

[0079] The image processing method provided by the present application can be executed by a terminal alone, such as an online document application on the terminal. The method can be executed by a server alone. The method can also be executed collaboratively by a terminal and a server. For example, the terminal can send the original image to the server. After receiving the original image, the server obtains the position information of the text to be erased in the original image; performs an iterative step; the iterative step includes: splicing the original image with the erased image of the previous iteration to obtain a spliced ​​image of the current iteration; using a trained text erasure model, performing text erasure on the spliced ​​image of the current iteration, and outputting a predicted image of the current iteration; according to the position information, superimposing the predicted image of the current iteration with the original image to obtain the erased image of the current iteration; when the iteration stop condition is met, the erased image of the last iteration is obtained, and the erased image of the last iteration is used as the target image obtained by performing multiple text erasures on the original image, and the target image is fed back to the terminal. The first iteration of the stitched image is obtained by stitching the original image with the original image; in the erased image of the current iteration obtained after superposition, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image. Of course, the terminal can also obtain the original image and the position information of the text to be erased in the original image, and send them to the server together. The server can then use the image processing method provided in the embodiment of the application to achieve text erasure.

[0080] Figure 2 A flowchart of an image processing method in one embodiment is shown, in which a computer device ( Figure 1 The method is described as an example of executing the terminal or server in FIG. 1 ; the method includes the following steps:

[0081] Step S202: Acquire the original image and the position information of the text to be erased in the original image.

[0082] The original image may be an image including text, and text erasing on the original image is the process of erasing the text in the original image. The text in the original image may be referred to as text to be erased. The area where the text to be erased is located is the text area, and the area other than the text area in the original image may be referred to as the non-text area. Obviously, the text area includes not only text but also the background of the image. Text erasing not only requires erasing the text to be erased from the original image, but also requires restoring the background of the text area where the text to be erased is located as much as possible, and restoring the background obscured by the text to be erased.

[0083] The position information of the text to be erased is used to represent the position of the text to be erased in the original image. The position information can be the coordinates of the text area where the text to be erased is located. In one embodiment, the text area where the text to be erased is located can be the minimum rectangular box surrounding the text to be erased, and the corresponding position information is the position information of the minimum rectangular box. In an original image, there can be multiple texts to be erased, and the text areas where the multiple texts to be erased are located can be multiple non-connected areas corresponding to the multiple texts in the original image.

[0084] In one embodiment, the original image may be an image sent by an online collaborative document application, the image including text to be erased, and the image is an image that cannot be directly edited, such as a PowerPoint image including non-editable text. The terminal obtains the original image based on the online collaborative document application and sends the original image to the server. After the server erases the text on the original image to obtain the target image, it can also return the target image obtained by multiple erasures to the online collaborative document application, so that the document can be edited using the restored target image as the editing background through the online collaborative document. The online collaborative document application is a document application that can be edited online by multiple objects synchronously or asynchronously.

[0085] Figure 3 is a schematic diagram of an original image in an embodiment, referring to Figure 3 The text in the original image, such as "2. Early stage of valley development," is all to be erased. In an image text editing scenario, the text to be erased in the original image may be non-editable text. The terminal can send the image to a server via an online collaborative document application. The server then uses the image processing method provided in an embodiment of the present application to erase the text from the image, and then performs text editing based on the target image obtained after erasure.

[0086] Figure 4 A schematic diagram of a text area where text to be erased is located in an original image in one embodiment, referring to Figure 4 , the server can perform text detection on the original image and determine the location information of the text to be erased in the original image. Figure 4 As shown, the position information of each box surrounding the text is the position information of each text to be erased.

[0087] Figure 5 is a schematic diagram of a target image in one embodiment, referring to Figure 5 , using the image processing method provided in the embodiment of the present application, Figure 3 The original image shown is erased to obtain Figure 5The target image shown in the figure shows that not only has the text in the original image been erased, but the background obscured by the text has also been restored. This background seamlessly blends with the background of the non-text areas, demonstrating a significant restoration effect. Furthermore, the server can feed this target image back to an online collaborative document application, allowing document editors to use the restored target image as the editing background for document editing.

[0088] In the above embodiment, the text erasure model is applied to the image text editing scenario, and the text erasure model is used to erase the text in the area where the text to be erased corresponding to the document editing is located and to repair the background of the area. The non-text area can use the original image, and the image content outside the text area will not be modified, so that the target image is as close to the original image as possible.

[0089] In one embodiment, the original image may include text to be translated. The computer device may obtain the original image including the text to be translated; after performing multiple text erasures on the original image to obtain a target image, the computer device may also obtain a translation corresponding to the text to be translated; and the translation may be pasted onto the target image based on the position of the text to be translated in the original image to obtain a translated image corresponding to the original image.

[0090] In one embodiment, the original image containing the text to be translated can be captured in real time by a terminal camera. For example, when a user is reading an English book, they can use a camera to capture the contents of the book as the original image. After applying the image processing method of the present application to obtain a target image with the English text erased, the translated text can be further pasted into the target image to directly display the corresponding Chinese translation image. Of course, the original image containing the text to be translated can also be obtained locally or from other devices.

[0091] Figure 6 is a schematic diagram of an original image including text to be translated in one embodiment, Figure 7 Shown Figure 6 The text area where the text to be translated is located in the original image shown, Figure 8A Shows the Figure 6 The target image is obtained after the text is erased from the original image shown. It can be seen that the text in the original image has been erased. Figure 8B Shows the Figure 6 The server can obtain a translated image by translating the original image shown, that is, after obtaining the target image, further obtain a translation corresponding to the text to be translated, paste the translation into the target image according to the position of the text to be translated in the original image, so that the position of the translation in the target image corresponds to the position of the text to be translated in the original image, and obtain a translated image corresponding to the original image to feed back to the terminal.

[0092] In the above embodiment, the text erasure model is applied to the image-text translation scenario. The text erasure model is used to erase the text in the area where the text to be translated is located and to repair the background of the area. The non-text area can use the original image, and the image content outside the text area will not be modified, so that the translated image is as close to the original image as possible.

[0093] In one embodiment, the position information of the original image can be determined by text detection. Specifically, the computer device can obtain the original image, perform text detection on the original image, determine the text area in the original image, and use the coordinates of the text area as the position information of the text to be erased in the original image.

[0094] Among them, text detection refers to identifying the position of text in the image, that is, determining the text area in the image. In this embodiment, the text area is the area in the original image where the text to be erased is located. It should be noted that text detection can be performed by other units in the server besides the text erasure model, such as the text detection model in the server, which can be implemented based on OCR (optical character recognition) technology; of course, text detection can also be performed by the terminal, which performs text detection on the original image, obtains the position information of the text to be erased in the original image, and sends the original image and position information to the server, which then performs text erasure. The text detection process is the process of identifying the pixels in the image that belong to the text. The area formed by these pixels is the text area. Generally, the text area is represented by the minimum rectangular area surrounding these pixels. The position information of the text area, that is, the position information of the text to be erased, can be represented by the coordinates of the minimum rectangular area. Alternatively, the coordinates of the upper left corner pixel of the minimum rectangular area, the height and width of the minimum rectangular area can be used as the position information of the text to be erased.

[0095] In the above embodiment, text detection is removed from the text erasure model, and the text erasure model can focus on text erasure, thereby achieving complete decoupling of the text erasure model and the text detection function, and reducing the burden on the text erasure model.

[0096] Step S204, executing the iteration step; the iteration step includes: splicing the original image with the erased image of the previous iteration to obtain the spliced ​​image of the current iteration; erasing the text of the spliced ​​image of the current iteration through the trained text erasure model, and outputting the predicted image of the current iteration; superimposing the predicted image of the current iteration with the original image according to the position information to obtain the erased image of the current iteration.

[0097] The text erasure model is a model used for text erasure. It has the ability to erase text, but does not require text detection. The text erasure model can iterate multiple times during the text erasure process. During each iteration, the input image to the text erasure model is obtained by concatenating the original image with the erased image from the previous iteration. The predicted image output by the text erasure model is further superimposed with the original image to obtain the erased image for that iteration.

[0098] In one embodiment, the original image and the erased image from the previous iteration are spliced ​​together by image channel. For example, if the original image is a three-channel image and the erased image from each iteration is also a three-channel image, the number of image channels in the resulting spliced ​​image will be six. Furthermore, during the first iteration, the original image and the erased image may be spliced ​​together such that the input dimensions of the first iteration are aligned with the input dimensions of subsequent iterations.

[0099] Step S206, when the iteration stop condition is met, the erased image of the last iteration is obtained, and the erased image of the last iteration is used as the target image obtained by performing multiple text erasing on the original image; wherein, the spliced ​​image of the first iteration is obtained by splicing the original image with the original image; in the erased image of the current iteration obtained after superposition, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

[0100] The iteration stopping condition can be the number of iterations, which can be determined based on the actual situation. Take the example of the iteration stopping condition of three iterations as an example:

[0101] During the first iteration, the computer device splices the original image with the original image itself to obtain a spliced ​​image that is input into the text erasure model in the first iteration; the text erasure model performs text erasure corresponding to the first iteration based on the spliced ​​image of the first iteration and outputs a predicted image of the first iteration; then, the computer device superimposes the predicted image of the first iteration and the original image according to the position information of the text to be erased in the original image to obtain an erased image of the first iteration. In the erased image of the first iteration, the first area corresponding to the position information of the text to be erased is derived from the area of ​​the predicted image of the first iteration, and the second area of ​​the erased image of the first iteration, excluding the first area, is derived from the original image; specifically, the pixel values ​​in the first area of ​​the erased image of the first iteration adopt the pixel values ​​of the first area of ​​the predicted image of the first iteration, and the pixel values ​​in the second area of ​​the erased image of the first iteration adopt the pixel values ​​of the second area of ​​the original image.

[0102] During the second iteration, the computer device splices the erased image of the first iteration with the original image to obtain a spliced ​​image that is input into the text erasure model in the second iteration; the text erasure model performs text erasure corresponding to the second iteration based on the spliced ​​image of the second iteration and outputs a predicted image of the second iteration; then, the computer device superimposes the predicted image of the second iteration with the original image according to the position information of the text to be erased in the original image to obtain an erased image of the second iteration. In the erased image of the second iteration, the first area corresponding to the position information of the text to be erased is derived from the area of ​​the predicted image of the second iteration, and the second area of ​​the erased image of the second iteration, excluding the first area, is derived from the original image; specifically, the pixel values ​​in the first area of ​​the erased image of the second iteration adopt the pixel values ​​of the predicted image of the second iteration, and the pixel values ​​in the second area of ​​the erased image of the second iteration adopt the pixel values ​​of the second area of ​​the original image.

[0103] During the third iteration (also known as the last iteration), the computer device splices the erased image of the second iteration with the original image to obtain a spliced ​​image that is input into the text erasure model in the third iteration; the text erasure model performs text erasure corresponding to the third iteration based on the spliced ​​image of the third iteration, and outputs a predicted image of the third iteration; then, the computer device superimposes the predicted image of the third iteration with the original image according to the position information of the text to be erased in the original image, to obtain an erased image of the third iteration. In the erased image of the third iteration, the first area corresponding to the position information of the text to be erased is derived from the area of ​​the predicted image of the third iteration, and the second area of ​​the erased image of the third iteration, excluding the first area, is derived from the original image; specifically, the pixel values ​​in the first area of ​​the erased image of the third iteration adopt the pixel values ​​of the predicted image of the third iteration, and the pixel values ​​in the second area of ​​the erased image of the third iteration adopt the pixel values ​​of the second area of ​​the original image.

[0104] After the third iteration is completed, the erased image of the third iteration is used as the target image obtained by performing multiple text erasing on the original image.

[0105] In the above-mentioned image processing method, when the burden of the text erasure model is light, that is, when it only has the ability to erase text, by inputting the entire original image into the text erasure model and performing multiple iterative erasing, a target image with better erasure effect can also be obtained. Specifically, in each iteration, the original image and the erased image of the previous iteration are spliced ​​to obtain a spliced ​​image. The spliced ​​image carries the original image information and the image information after the previous erasure. The spliced ​​image is input into the text erasure model. The output predicted image can not only effectively erase the text in the text area, but also preserve the boundary information between the text area and the non-text area, so that the background in the text area is better restored. Further, according to the position information of the text area where the text to be erased is located in the original image, the predicted image is superimposed with the original image to obtain the erased image of the current iteration. The first area corresponding to the position information in the erased image is derived from the predicted image, and the second area other than the first area is derived from the original image. In this way, the image content outside the text area can be completely preserved, and the erasure result can be fully adapted to the position information of the text to be erased in the original image, thereby improving the erasure effect.

[0106] In one embodiment, the original image is spliced ​​with the erased image of the previous iteration, including: obtaining the erased image of the previous iteration; splicing the original image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration.

[0107] The number of image channels of the erased image in each iteration may be consistent with the number of image channels of the original image. For example, the number of image channels of the erased image in each iteration and the number of image channels of the original image may be 3 image channels, and the 3-channel image may be, for example, RGB image channels.

[0108] Take the current iteration as the second iteration, the previous iteration as the first iteration, and the image channel as RGB as an example:

[0109] The computer device can obtain the erased image of the first iteration, which includes three image channels of RGB; then, the erased image of the first iteration and the original image can be spliced ​​on the RGB image channel to obtain a spliced ​​image of six image channels, and the spliced ​​image of the six image channels is used as the spliced ​​image input to the text erasure model in the second iteration.

[0110] In the above embodiment, the spliced ​​image of each iteration includes the original image, which allows the text erasure model to perform text erasure with the participation of the original image, thereby preventing the predicted image output by the text erasure model from gradually deviating from the original image during multiple iterations.

[0111] In one embodiment, the text of the spliced ​​image of the current iteration is erased by a trained text erasure model, and a predicted image of the current iteration is output, including: inputting the spliced ​​image of the current iteration into the trained text erasure model; extracting features of the spliced ​​image of the current iteration through the encoding network in the text erasure model to obtain image features of the spliced ​​image of the current iteration; reconstructing the image based on the image features through the transposed convolutional network in the text erasure model, and outputting the predicted image of the current iteration.

[0112] The network structure of the text erasure model includes an encoding network and a decoding network. In one embodiment, the computer device forms the network structure of the text erasure model by replacing part of the encoding network of the u-net with a residual network (resnet) and replacing part of the decoding network of the u-net with an upsampling network with a transposed convolutional network.

[0113] Figure 9 A schematic diagram of the structure of a text erasure model in one embodiment, referring to Figure 9 ,The text erasure model includes an encoding network and a decoding network. The encoding network includes a convolutional network, a residual network and a downsampling network, and the decoding network includes a transposed convolutional network, a convolutional network and a splicing network.

[0114] After obtaining the spliced ​​image of the current iteration, the computer device can input the spliced ​​image of the current iteration into the text erasure model, perform feature extraction on the spliced ​​image of the current iteration through the encoding network of the text erasure model, and obtain the image features of the spliced ​​image of the current iteration; then, through the transposed convolutional network of the text erasure model, perform image reconstruction based on the image features of the spliced ​​image of the current iteration, and output the predicted image of the current iteration.

[0115] In the above embodiment, the encoding network of the text erasure model is used to extract features of the spliced ​​image of the current iteration, and the transposed convolutional network of the text erasure model is used to reconstruct the image, so that the reconstructed predicted image is closer to the original image, thereby improving the restoration degree of the predicted image.

[0116] In one embodiment, the above method also includes: generating a mask image corresponding to the original image according to the position information; superimposing the predicted image of the current iteration with the original image according to the position information to obtain the erased image of the current iteration, including: superimposing the predicted image of the current iteration with the original image according to the mask image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

[0117] Among them, the mask image is used to distinguish the text area and non-text area in the original image, that is, the mask image can distinguish which areas correspond to the areas where the text to be erased is located in the original image, and which areas correspond to the areas where the text to be erased is not located in the original image.

[0118] After the computer device obtains the position information of the text to be erased in the original image, it can generate a mask image corresponding to the original image according to the position information, and according to the mask image, superimpose the predicted image of the current iteration and the original image to obtain the erased image of the current iteration.

[0119] Here, by performing the superposition according to the mask image, the first region corresponding to the position information of the text to be erased in the resulting erased image of the current iteration can be derived from the region corresponding to the position information in the predicted image of the current iteration, and the second region other than the first region can be derived from the original image. Specifically, by performing the superposition according to the mask image, the pixel values ​​of the first region corresponding to the position information of the text to be erased in the resulting erased image of the current iteration can be the pixel values ​​of the text region in the predicted image of the current iteration, and the pixel values ​​of the second region other than the first region can be the pixel values ​​of the non-text region in the original image.

[0120] In the above embodiment, the predicted image of the current iteration and the original image are superimposed using a mask image generated according to position information. The text area in the erased image of the current iteration obtained by superposition is derived from the predicted image of the current iteration and the non-text area is derived from the original image, so that the text erasure model can focus on erasing the text area without the need for text detection.

[0121] In one embodiment, a mask image corresponding to an original image is generated according to position information, including: determining text areas and non-text areas in the original image according to the position information; setting pixel values ​​in the text areas in the original image to 1 and setting pixel values ​​in the non-text areas in the original image to 0 to obtain a binary mask image.

[0122] After obtaining the position information of the text to be erased in the original image, the computer device can determine the text area and non-text area in the original image, set the pixel value of the text area to 1, and set the pixel value of the non-text area to 0 to obtain a binary mask image.

[0123] In one embodiment, the erased image of the current iteration is obtained by the following formula: dst = mask*output+(1-mask)*src; where mask represents the mask image, output represents the predicted image of the current iteration, src represents the original image, and dst represents the erased image of the current iteration.

[0124] Take the pixel value in the first row and first column of the erased image of the current iteration (denoted as dst[1,1]) as an example: dst[1,1]=mask[1,1]*output[1,1]+(1-mask[1,1])*src[1,1]

[0125] Where mask[1,1] represents the pixel value in the first row and first column of the binary mask image, output[1,1] represents the pixel value in the first row and first column of the predicted image of the current iteration, and src[1,1] represents the pixel value in the first row and first column of the original image. When mask[1,1]=0, dst[1,1]=(1-mask[1,1])*src[1,1]; when mask[1,1]=1, dst[1,1]=mask[1,1]*output[1,1].

[0126] Similarly, the pixel values ​​of the erased image of the current iteration in other rows and columns can be obtained in the above manner to form the erased image of the current iteration.

[0127] In the above embodiment, the above formula is used to extract the pixel values ​​of the text area of ​​the predicted image of the current iteration through the binarized mask image and use them as the pixel values ​​of the text area of ​​the erased image of the current iteration, and extract the pixel values ​​of the non-text area of ​​the original image and use them as the pixel values ​​of the non-text area of ​​the erased image of the current iteration.

[0128] Figure 10 This is a schematic diagram of an image processing method in one embodiment. The method can be applied to an image text editing scenario or an image text translation scenario. The method is described by taking the execution of the method by a computer device as an example. Figure 10 , the method may include the following steps:

[0129] The computer device obtains an original image, performs text detection on the original image, determines the text area in the original image, and uses the coordinates of the text area as the position information of the text to be erased in the original image; according to the position information of the text to be erased in the original image, determines the text area and non-text area in the original image, sets the pixel value in the text area in the original image to 1, and sets the pixel value in the non-text area in the original image to 0, thereby obtaining a binary mask image.

[0130] During the first iteration, the computer device can splice the original image and the original image itself on the channel to obtain a spliced ​​image of the first iteration; the text erasure model performs text erasure corresponding to the first iteration based on the spliced ​​image of the first iteration, and outputs a predicted image of the first iteration; then, according to the mask image, the predicted image of the first iteration and the original image are superimposed to obtain an erased image of the first iteration.

[0131] During the second iteration, the erased image of the first iteration and the original image can be spliced ​​to obtain a spliced ​​image of the second iteration; the text erasure model performs text erasure corresponding to the second iteration based on the spliced ​​image of the second iteration, and outputs a predicted image of the second iteration; then, according to the mask image, the predicted image of the second iteration and the original image are superimposed to obtain an erased image of the second iteration.

[0132] Continue to iterate in the same manner as the second iteration. When the iteration reaches the end, the erased image of the second-to-last iteration and the original image can be spliced ​​to obtain the spliced ​​image of the last iteration; the text erasure model erases the text corresponding to the last iteration based on the spliced ​​image of the last iteration, and outputs the predicted image of the last iteration; then, according to the mask image, the predicted image of the last iteration and the original image are superimposed to obtain the erased image of the last iteration, and output as the target image.

[0133] The erased image of each iteration is obtained by the following formula:

[0134] dst=mask*output+(1-mask)*src; where mask represents the mask image, output represents the predicted image of this iteration, src represents the original image, and dst represents the erased image of this iteration.

[0135] In the above embodiment, text detection and text erasure are completely decoupled, and the image obtained by text erasure can be fully compatible with the detection result of text detection, which brings great convenience to engineering applications; the original image can be input into the text erasure model as a whole, so that the text erasure model can erase the text on the original image from the perspective of the entire image, without being limited to a partial area. Compared with the solution of cropping the text area and processing it separately, the text erasure model can extract the background features of the entire image, improve the quality of background restoration, avoid the problem of color difference, and at the same time, the time consumption of text erasure will not increase with the increase of text area, thereby improving the efficiency of text erasure; and, since the text erasure model can erase text based on the entire image, the text erasure model does not need to have text detection capability; in addition, During the process of multiple iterations of text erasure by the text erasure model, the image input to the text erasure model in each iteration includes the erased image of the previous iteration, which is obtained by superimposing the predicted image of the previous iteration and the original image. The first area corresponding to the position information of the erased image of the previous iteration is derived from the area corresponding to the position information in the predicted image of the previous iteration, and the second area other than the erased image of the previous iteration is derived from the original image. Therefore, the text erasure model can focus on erasing the area where the erased text is located and repairing the background of the area. At the same scale, the text erasure model has better text erasure effect than the end-to-end model. The non-text area can use the original image and will not modify the image content outside the text area.

[0136] This application also provides a method for processing a text erasure model. Figure 11 A flowchart of a method for processing a text erasure model in one embodiment, wherein the method is performed by a computer device ( Figure 1 The method is described as an example of executing the terminal or server in FIG. 1 ; the method includes the following steps:

[0137] Step S1102 : obtaining a training sample, where the training sample includes a sample image, location information of text in the sample image, and a label image corresponding to the sample image but not including text.

[0138] Each training sample consists of three parts: a sample image with text, the location information of the text in the sample image, and a label image without text corresponding to the sample image.

[0139] In one embodiment, obtaining a training sample includes: using an image without text as a label image; adding text to the label image to obtain a sample image; and performing text detection on the sample image to obtain position information of the text in the sample image.

[0140] Taking into account the low efficiency of image acquisition after erasing text, this embodiment obtains training samples through data synthesis. Specifically, the computer device can obtain an image that does not include text and use it as a label image. By randomly adding text to the label image, a sample image is obtained. Then, text detection is performed on the sample image to obtain the position information of the text in the sample image, or the position information used when randomly adding text is used as the position information of the text in the sample image.

[0141] In one embodiment, obtaining a training sample includes: obtaining a text-editable image; parsing the text-editable image, deleting the editable text in the text-editable image to obtain an image without text and use it as a sample image; and converting the text-editable image into a non-text-editable image and use it as a label image.

[0142] In the above embodiment, training samples are obtained by adding texts as a data synthesis method, thereby improving the efficiency of obtaining training samples.

[0143] In addition, in order to make the text erasure model have better generalization ability in the photo-taking scenario, the computer device can add noise to the sample image according to the image data amplification method corresponding to various photo-taking scenarios to obtain multiple sample images to form multiple training samples.

[0144] Step S1104, for the training sample, executing the iterative step until the iteration stop condition is met; the iterative step includes: splicing the sample image with the erased image of the previous iteration to obtain the spliced ​​image of the current iteration, inputting the spliced ​​image of the current iteration into a predetermined neural network model, erasing the text of the spliced ​​image of the current iteration, outputting the predicted image of the current iteration, and superimposing the predicted image of the current iteration with the sample image according to the position information to obtain the erased image of the current iteration.

[0145] Among them, the network structure of the predetermined neural network model can refer to Figure 9 ,After the training of the predetermined neural network model is completed, the ,obtained predetermined neural network model can be used as a text erasure ,model.

[0146] In this step, text erasure is performed multiple times on the sample image of the training sample. Each iteration involves image splicing and superposition. The process of multiple text erasures can be referred to the introduction of step S204 and will not be repeated here.

[0147] Step S1106: Based on the erased image, predicted image and label image of each iteration, the loss of each iteration of the sample image is calculated. The erasure loss corresponding to the training sample is obtained according to the sum of the losses of multiple iterations obtained when the iteration stops. After updating the network parameters of the predetermined neural network according to the erasure loss, return to the step of obtaining the training sample to continue training until the training stop condition is met, and a trained text erasure model is obtained.

[0148] When the network parameters of the predetermined neural network are updated, multiple iterations are performed on each training sample, and each iteration will generate corresponding losses, and the losses of each iteration are determined based on the erased image, predicted image and label image of the iteration; then, the sum of the losses of multiple iterations of the training sample is used as the erasure loss of the training sample, and the network parameters of the predetermined neural network model are updated according to the erasure loss of the training sample; then, the next training sample is obtained, and the above processing is performed on the next training sample to update the network parameters of the predetermined neural network model again until the training stop condition is met, and the predetermined neural network model at this time is used as the text erasure model.

[0149] In the above embodiment, since the trained model only needs to have text erasure capability and does not need text detection capability to reduce the model burden while achieving good erasure effect, during the training process, the sample image and the erased image of the previous iteration are iteratively spliced ​​to obtain a spliced ​​image. The spliced ​​image carries the sample image information and the information of the image after the previous erasure. The spliced ​​image is input into the model and a predicted image is output, so that the model learns to focus on the text area that needs to be erased in the sample image. Then, according to the position information of the text area where the text to be erased is located in the sample image, the predicted image and the sample image are superimposed to obtain the erased image of the current iteration, so that the first area corresponding to the position information in the erased image is derived from the predicted image, and the second area other than the first area is derived from the sample image. Then, based on the erased image, predicted image and label image of each iteration, the loss of each iteration of the sample image is calculated. The text erasure model obtained by optimization based on the loss of multiple iterations can output a model that can not only effectively erase the text in the text area, but also preserve the boundary information between the text area and the non-text area, so as to achieve the background repair effect in the text area.

[0150] In one embodiment, after splicing the sample image with the erased image of the previous iteration, a spliced ​​image of the current iteration is obtained, including: obtaining the erased image of the previous iteration; splicing the sample image with the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration.

[0151] The number of image channels of the erased image in each iteration may be consistent with the number of image channels of the sample image. For example, the number of image channels of the erased image in each iteration and the number of image channels of the sample image may be 3, and the 3 image channels may be, for example, RGB channels.

[0152] Take the current iteration as the second iteration, the previous iteration as the first iteration, and the image channel as RGB as an example:

[0153] The computer device can obtain the erased image of the first iteration, which includes three image channels of RGB; then, the erased image of the first iteration and the sample image can be spliced ​​on the RGB image channel to obtain a spliced ​​image of six image channels, and the spliced ​​image of the six image channels is used as the spliced ​​image that needs to be input into the predetermined neural network model in the second iteration.

[0154] In the above embodiment, each iterative stitched image includes a sample image, which allows the predetermined neural network model to perform text erasure with the participation of the sample image, so that the output prediction image is as close to the sample image as possible.

[0155] In one embodiment, the spliced ​​image of the current iteration is input into a predetermined neural network model, the text of the spliced ​​image of the current iteration is erased, and the predicted image of the current iteration is output, including: inputting the spliced ​​image of the current iteration into a predetermined neural network model; performing feature extraction on the spliced ​​image of the current iteration through the encoding network in the predetermined neural network model to obtain image features of the spliced ​​image of the current iteration; performing image reconstruction based on the image features through the transposed convolutional network in the predetermined neural network model, and outputting the predicted image of the current iteration.

[0156] The network structure of the predetermined neural network model includes an encoding network and a decoding network. In one embodiment, the network structure of the predetermined neural network model can be formed by replacing part of the network in the encoding network of u-net (a neural network-based encoding and decoding structure) with a residual network, and replacing part of the network in the decoding network of u-net from an upsampling network to a transposed convolutional network.

[0157] After obtaining the spliced ​​image of the current iteration, the computer device can input the spliced ​​image of the current iteration into a predetermined neural network model, perform feature extraction on the spliced ​​image of the current iteration through the encoding network of the predetermined neural network model, and obtain the image features of the spliced ​​image of the current iteration; then, through the transposed convolution network of the predetermined neural network model, perform image reconstruction based on the image features of the spliced ​​image of the current iteration, and output the predicted image of the current iteration.

[0158] In the above embodiment, the encoding network of the predetermined neural network model is used to extract features of the spliced ​​image of the current iteration, and the transposed convolutional network of the predetermined neural network model is used to reconstruct the image, so that the reconstructed predicted image is closer to the original image, thereby improving the restoration degree of the predicted image.

[0159] In one embodiment, the processing method of the text erasure model also includes: the computer device can generate a mask image corresponding to the sample image according to the position information, and the mask image is used to distinguish the text area and the non-text area in the sample image; according to the mask image, the predicted image of the current iteration and the sample image are superimposed to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the sample image.

[0160] After the computer device obtains the position information of the text in the sample image, it can generate a mask image corresponding to the sample image according to the position information, and superimpose the predicted image of the current iteration and the sample image according to the mask image to obtain the erased image of the current iteration.

[0161] Here, by superimposing the mask images, the first region corresponding to the position information of the to-be-erased text in the resulting erased image of the current iteration can be derived from the region corresponding to the position information in the predicted image of the current iteration, and the second region other than the first region can be derived from the sample image. Specifically, by superimposing the mask images, the pixel values ​​of the first region corresponding to the position information of the text in the resulting erased image of the current iteration can be the pixel values ​​of the text region in the predicted image of the current iteration, and the pixel values ​​of the second region other than the first region can be the pixel values ​​of the non-text region of the sample image.

[0162] In the above embodiment, the predicted image and sample image of the current iteration are superimposed using a mask image generated according to position information, and the text area in the erased image of the current iteration obtained by superposition is derived from the predicted image of the current iteration and the non-text area is derived from the sample image, so that the predetermined neural network model can focus on erasing the text area without the need for text detection.

[0163] In one embodiment, the computer device can determine the text area and non-text area in the sample image according to the position information; set the pixel value in the text area in the sample image to 1, and set the pixel value in the non-text area in the sample image to 0, to obtain a binary mask image.

[0164] After obtaining the position information of the text in the sample image, the computer device can determine the text area and non-text area in the sample image, set the pixel value of the text area to 1, and set the pixel value of the non-text area to 0 to obtain a binary mask image.

[0165] In one embodiment, a computer device can determine the text areas in the predicted image and the label image of each iteration according to the position information of the text in the sample image; calculate the distance between the predicted image and the text area in the label image of each iteration as the text area erasure loss of each iteration; calculate the image difference between the erased image and the label image of each iteration as the image difference loss of each iteration; calculate the feature difference between the erased image and the label image of each iteration as the feature difference loss of each iteration; and obtain the loss of each iteration of the sample image based on the text area erasure loss, image difference loss and feature difference loss of each iteration.

[0166] The loss of each iteration includes text area erasure loss, image difference loss and feature difference loss, which can be represented by the following formula: L = L l1 +L ssim +L vgg , where L represents the loss of each iteration, L l1 Characterize the text region erasure loss, L ssim Characterize the image difference loss, L vgg Characterizes feature difference loss; the text area erasure loss of each iteration characterizes the distance difference of pixel values ​​in the text area between the predicted image and the label image of that iteration; the image difference loss of each iteration characterizes the structural similarity loss between the erased image and the label image of that iteration in the entire image area; the feature difference loss of each iteration characterizes the perceptual loss and style loss between the erased image and the label image of that iteration in the entire image area.

[0167] Regarding the text area erasure loss, the computer device can determine the text area in the predicted image and the label image of each iteration according to the position information of the text in the sample image, calculate the distance between the pixel values ​​of the predicted image and the label image on the text area of ​​each iteration and use it as the text area erasure loss of each iteration.

[0168] Specifically, the text area erasure loss of each iteration can be calculated by the following formula: l1 =||(img out -img gt )*mask||1; where img out Represents the predicted image of this iteration, img gt Represents the label image, and mask represents the mask image obtained according to the position information of the text in the sample image.

[0169] Regarding image difference loss, image difference loss is a loss that characterizes the structural similarity loss between the predicted image of this iteration and the label image in the entire image area. Since the non-text area of ​​the erased image of this iteration is derived from the label image, and the text area is derived from the predicted image of this iteration, the erased image of this iteration and the label image are different in the text area. Therefore, the difference in the text area makes the natural connection between the text area and the non-text area different. Therefore, in order to make the text erasure model have better connection ability and allow the background corresponding to the text area to be better connected to the background corresponding to the non-text area, the embodiment of the present application introduces image difference loss for model training.

[0170] The computer device can calculate the image difference between the erased image and the label image of each iteration as the image difference loss of each iteration according to the following formula; the formula is L ssim =-SSIM(I out , I gt ); where I out Characterize the erased image of each iteration, I gt Characterize the label image; and I out and I gt Calculate as x and y of SSIM(x,y) respectively.

[0171] The calculation formula for SSIM(x, y) is: Among them, μ x is the average value of the pixel values ​​of image x, μy is the average value of the pixel values ​​of image y, is the variance of the pixel values ​​of image x, is the variance of the pixel values ​​of image y, σ xy is the covariance of images x and y, c1=(k1L) 2 and c2=(k2L) 2 It is a constant used to maintain stability, L is the dynamic range of pixel values, k1=0.01k2=0.03.

[0172] By using the mean as an estimate of brightness, the variance as an estimate of contrast, and the covariance as a measure of structural similarity, the image difference loss can measure the difference between two erased images and the labeled image from three dimensions: brightness, contrast, and structure.

[0173] Regarding feature difference loss, the feature difference between the erased image and the labeled image in each iteration is mainly reflected in: the perceptual loss and style loss of the erased image and the labeled image in each iteration over the entire image area. Therefore, the computer device can calculate the perceptual loss and style loss of the erased image and the labeled image in each iteration over the entire image area, and use them as the feature difference loss between the two over the entire image area.

[0174] In the above embodiment, based on the distance between the predicted image and the label image in each iteration on the text area, the text area erasure loss of each iteration is obtained, which can make the network parameters of the predetermined neural network model more targeted at the erasure of the text area and the reconstruction of the background of the text area, so that the text erasure model focuses on the erasure of the local area and the reconstruction of the background of the text area; in addition, based on the image difference loss between the erased image and the label image in each iteration on the entire image area, the network parameters of the predetermined neural network model can be made more targeted at the natural connection between the background of the text area and the background of the non-text area, so that the background of the text area reconstructed by the text erasure model can be more naturally connected to the background of the non-text area; and, based on the feature difference loss between the erased image and the label image in each iteration on the entire image area, the network parameters of the predetermined neural network model can be made more targeted at the stylistic consistency between the background of the text area and the background of the non-text area, so that the style of the background of the text area reconstructed by the text erasure model is close to the background of the non-text area.

[0175] In one embodiment, regarding the feature difference loss of each iteration, the computer device can input the erased image and the label image of each iteration into a trained feature extraction model respectively to obtain the image features corresponding to the erased image and the label image of each iteration; calculate the distance between the image features corresponding to the erased image and the label image of each iteration to obtain the perceptual loss between the erased image and the label image of each iteration; calculate the corresponding Gram matrix based on the image features corresponding to the erased image of each iteration, calculate the corresponding Gram matrix based on the image features corresponding to the label image, calculate the difference between the Gram matrices of the erased image and the label image of each iteration, and obtain the style loss between the erased image and the label image of each iteration; based on the perceptual loss and the style loss, obtain the feature difference loss.

[0176] The trained feature extraction model may be a VGG (Visual Geometry Group) feature extraction model.

[0177] The feature difference loss of each iteration includes perceptual loss and style loss. When calculating the feature difference loss of each iteration, the computer device can input the erased image and the label image of that iteration into the trained feature extraction model to obtain the image features corresponding to the erased image and the label image of that iteration; then, the computer device can calculate the L1 distance between the image features corresponding to the erased image and the label image of that iteration to obtain the perceptual loss between the erased image and the label image of that iteration; then, the computer device calculates the corresponding Gram matrix according to the image features corresponding to the erased image of that iteration, calculates the corresponding Gram matrix according to the image features corresponding to the label image, calculates the difference between the Gram matrices of the erased image and the label image of that iteration, and obtains the style loss between the erased image and the label image of that iteration; based on the sum of the perceptual loss and the style loss, the feature difference loss of that iteration is obtained.

[0178] In the above embodiment, the feature difference loss of each iteration includes perceptual loss and style loss, which can make the network parameters of the predetermined neural network model more targeted at the stylistic consistency between the background of the text area and the background of the non-text area, so that the style of the background of the text area reconstructed by the text erasure model is close to the background of the non-text area.

[0179] Figure 12 FIG. 1 is a schematic diagram of a method for processing a text erasure model in one embodiment. The method is described by taking a computer device as an example. The method may include the following steps:

[0180] The computer device can use an image that does not include text as a label image; after adding text to the label image, a sample image is obtained; text detection is performed on the sample image to obtain the position information of the text in the sample image and form a mask image; and a training sample is formed based on the sample image, the position information of the text in the sample image, and the label image that does not include text corresponding to the sample image.

[0181] For the training sample, the iterative step is executed until the iteration stop condition is met; the iterative step includes: splicing the sample image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration, inputting the spliced ​​image of the current iteration into a predetermined neural network model, erasing the text of the spliced ​​image of the current iteration, outputting the predicted image of the current iteration, and superimposing the predicted image of the current iteration with the sample image according to the mask image to obtain the erased image of the current iteration.

[0182] The iterative process during training is similar to that of the text erasure model. For details, please refer to the introduction to the iterative process of the text erasure model.

[0183] The computer device performs multiple iterations on each training sample, and each iteration will generate corresponding losses, and the loss of each iteration is determined based on the erased image, predicted image and label image of that iteration; then, the sum of the losses of multiple iterations of the training sample is used as the erasure loss of the training sample, and the network parameters of the predetermined neural network model are updated once according to the erasure loss of the training sample; then the next training sample is obtained, and the above-mentioned processing is performed on the next training sample to perform the next update on the network parameters of the predetermined neural network model until the training stop condition is met, and the predetermined neural network model at this time is used as the text erasure model.

[0184] Regarding the loss of a single iteration, the specific calculation includes the following steps:

[0185] The computer device determines the text areas in the predicted image and the label image of the iteration according to the position information of the text in the sample image; and calculates the distance between the predicted image and the text area in the label image of the iteration as the text area erasure loss of the iteration.

[0186] The text area erasure loss of this iteration can be calculated by the following formula: l1 =||(img out -img gt )*mask||1; where img out Represents the predicted image of this iteration, img gt Represents the label image, and mask represents the mask image obtained according to the position information of the text in the sample image.

[0187] The computer device calculates the image difference between the erased image and the label image of the iteration as the image difference loss of the iteration;

[0188] Specifically, according to the formula L ssim =-SSIM(I out , I gt ), calculate the image difference between the erased image and the label image of this iteration and use it as the image difference loss of this iteration; where, I out Characterize the erased image of this iteration, I gt Characterize the label image; and I out and I gt Calculate as x and y of SSIM(x,y) respectively.

[0189] The computer device calculates the feature difference between the erased image and the label image of the iteration as the feature difference loss of the iteration.

[0190] Specifically, the computer device inputs the erased image and the label image of this iteration into the trained feature extraction model respectively to obtain the image features corresponding to the erased image and the label image of this iteration; calculates the distance between the image features corresponding to the erased image and the label image of this iteration to obtain the perceptual loss between the erased image and the label image of this iteration; calculates the corresponding Gram matrix based on the image features corresponding to the erased image of this iteration, calculates the corresponding Gram matrix based on the image features corresponding to the label image, calculates the difference between the Gram matrices of the erased image and the label image of this iteration, and obtains the style loss between the erased image and the label image of this iteration; based on the perceptual loss and the style loss, obtains the feature difference loss of this iteration.

[0191] The sum of the text area erasure loss, image difference loss and feature difference loss of this iteration is taken as the loss of this iteration.

[0192] In the above embodiment, during training, the overall forward process is consistent with the inference process of the text erasure model, and the network parameters of the predetermined neural network model are updated by comprehensively considering the sum of the losses of each training sample in multiple iterations, so that the text erasure model has the ability to erase text; and the spliced ​​image of each iteration includes the sample image, which allows the predetermined neural network model to perform text erasure with the participation of the sample image, so that the output prediction image is as close to the sample image as possible; in addition, based on the distance between the prediction image of each iteration and the label image on the text area, the text area erasure loss of each iteration is obtained, which can make the network parameters of the predetermined neural network model more targeted at the erasure of the text area and the reconstruction of the background of the text area, so that the text erasure model Focus on erasing this area and reconstructing the background of the text area; in addition, based on the image difference loss between the erased image and the label image in each iteration in the entire image area, the network parameters of the predetermined neural network model can be made more targeted at the natural connection between the background of the text area and the background of the non-text area, so that the background of the text area reconstructed by the text erasure model can be more naturally connected to the background of the non-text area; and, the feature difference loss between the erased image and the label image in each iteration in the entire image area can make the network parameters of the predetermined neural network model more targeted at the stylistic consistency between the background of the text area and the background of the non-text area, so that the style of the background of the text area reconstructed by the text erasure model is close to the background of the non-text area.

[0193] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0194] Based on the same inventive concept, embodiments of the present application also provide an image processing device for implementing the aforementioned image processing method. The solution to the problem provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations and technical effects of one or more of the following image processing device embodiments can be found in the limitations and technical effects of the image processing method above, and will not be further elaborated here.

[0195] Figure 13 FIG. 1 is a structural block diagram of an image processing device in one embodiment. Figure 13 , the device comprises:

[0196] The text position acquisition module 1302 is used to acquire the original image and the position information of the text to be erased in the original image;

[0197] The iterative erasure module 1304 is configured to perform an iterative step, wherein the iterative step includes: splicing the original image with the erased image of the previous iteration to obtain a spliced ​​image of the current iteration; erasing the text of the spliced ​​image of the current iteration using the trained text erasure model, and outputting a predicted image of the current iteration; and superimposing the predicted image of the current iteration with the original image based on position information to obtain an erased image of the current iteration;

[0198] The target image acquisition module 1306 is configured to obtain an erased image of the last iteration when an iteration stop condition is met, and use the erased image of the last iteration as a target image obtained by performing multiple text erasures on the original image;

[0199] Among them, the stitched image of the first iteration is obtained by stitching the original image with the original image; in the erased image of the current iteration obtained after superposition, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

[0200] In one embodiment, the text position acquisition module 1302 is used to acquire an original image; perform text detection on the original image to determine a text area in the original image; and use the coordinates of the text area as position information of the text to be erased in the original image.

[0201] In one embodiment, the iterative erasing module 1304 is configured to obtain an erased image of a previous iteration; and to splice the original image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration.

[0202] In one embodiment, the iterative erasure module 1304 is used to input the spliced ​​image of the current iteration into a trained text erasure model; perform feature extraction on the spliced ​​image of the current iteration through the encoding network in the text erasure model to obtain image features of the spliced ​​image of the current iteration; perform image reconstruction based on the image features through the transposed convolutional network in the text erasure model, and output a predicted image of the current iteration.

[0203] In one embodiment, the apparatus further includes a mask image generation module for generating a mask image corresponding to the original image according to the position information, wherein the mask image is used to distinguish between text areas and non-text areas in the original image;

[0204] The iterative erasing module 1304 is used to superimpose the predicted image of the current iteration with the original image according to the mask image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

[0205] In one embodiment, the mask image generation module is further used to determine the text area and non-text area in the original image according to the position information; set the pixel value in the text area in the original image to 1, and set the pixel value in the non-text area in the original image to 0, to obtain a binary mask image.

[0206] In one embodiment, the erased image of the iteration is obtained by the following formula:

[0207] dst=mask*output+(1-mask)*src;

[0208] Among them, mask represents the mask image, output represents the predicted image of the current iteration, src represents the original image, and dst represents the erased image of the current iteration.

[0209] In one embodiment, the apparatus further comprises an image acquisition module for receiving an original image including the text to be erased sent by the online collaborative document application;

[0210] The device also includes a target image feedback module for returning the target image obtained by multiple erasures to the online collaborative document application, so that the document can be edited through the online collaborative document with the restored target image as the editing background.

[0211] In one embodiment, the apparatus includes an image acquisition module for acquiring an original image including the text to be translated;

[0212] The device also includes a translation processing module for obtaining a translation corresponding to the text to be translated; and pasting the translation to the target image according to the position of the text to be translated in the original image to obtain a translated image corresponding to the original image.

[0213] In the above-mentioned image processing device, when the burden of the text erasure model is light, that is, when it only has the ability to erase text, by inputting the entire original image into the text erasure model and performing multiple iterative erasures, a target image with better erasure effect can also be obtained. Specifically, in each iteration, the original image and the erased image of the previous iteration are spliced ​​to obtain a spliced ​​image. The spliced ​​image carries the original image information and the image information after the previous erasure. The spliced ​​image is input into the text erasure model. The output predicted image can not only effectively erase the text in the text area, but also preserve the boundary information between the text area and the non-text area, so that the background in the text area is better restored. Further, according to the position information of the text area where the text to be erased is located in the original image, the predicted image is superimposed with the original image to obtain the erased image of the current iteration. The first area corresponding to the position information in the erased image is derived from the predicted image, and the second area other than the first area is derived from the original image. In this way, the image content outside the text area can be completely preserved, and the erasure result can be fully adapted to the position information of the text to be erased in the original image, thereby improving the erasure effect.

[0214] Each module in the above-mentioned image processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0215] Based on the same inventive concept, the present application also provides a device for processing a text erasure model for implementing the aforementioned method for processing a text erasure model. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations and technical effects of the one or more embodiments of the device for processing a text erasure model provided below can be found in the limitations and technical effects of the method for processing a text erasure model described above, and will not be repeated here.

[0216] Figure 14 FIG. 1 is a structural block diagram of a processing device for a text erasure model in one embodiment. Figure 14 , the device comprises:

[0217] A training sample acquisition module 1402 is configured to acquire a training sample, where the training sample includes a sample image, location information of text in the sample image, and a label image corresponding to the sample image that does not include text.

[0218] The iterative erasing module 1404 is configured to execute iterative steps for the training sample until an iteration stop condition is satisfied. The iterative steps include: splicing the sample image with the erased image of the previous iteration to obtain a spliced ​​image of the current iteration, inputting the spliced ​​image of the current iteration into a predetermined neural network model, performing text erasure on the spliced ​​image of the current iteration, outputting a predicted image of the current iteration, and superimposing the predicted image of the current iteration with the sample image according to position information to obtain an erased image of the current iteration.

[0219] The loss parameter adjustment module 1406 is used to calculate the loss of each iteration of the sample image based on the erased image, predicted image and label image of each iteration, obtain the erasure loss corresponding to the training sample according to the sum of the losses of multiple iterations obtained when the iteration stops, update the network parameters of the predetermined neural network according to the erasure loss, and return to the step of obtaining the training sample to continue training until the training stop condition is met, thereby obtaining a trained text erasure model.

[0220] In one embodiment, the training sample acquisition module 1402 is configured to use an image without text as a label image; add text to the label image to obtain a sample image; and perform text detection on the sample image to obtain location information of the text in the sample image.

[0221] In one embodiment, the iterative erasing module 1404 is configured to obtain an erased image of a previous iteration; and to splice the sample image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration.

[0222] In one embodiment, the iterative erasing module 1404 is used to input the spliced ​​image of the current iteration into a predetermined neural network model; perform feature extraction on the spliced ​​image of the current iteration through the encoding network in the predetermined neural network model to obtain image features of the spliced ​​image of the current iteration; perform image reconstruction based on the image features through the transposed convolutional network in the predetermined neural network model, and output a predicted image of the current iteration.

[0223] In one embodiment, the apparatus further includes a mask image generating module for generating a mask image corresponding to the sample image according to the position information, wherein the mask image is used to distinguish between a text area and a non-text area in the sample image;

[0224] The iterative erasing module 1404 is further used to superimpose the predicted image of the current iteration and the sample image according to the mask image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the sample image.

[0225] In one embodiment, the mask image generation module is further used to determine the text area and non-text area in the sample image according to the position information; set the pixel value in the text area in the sample image to 1, and set the pixel value in the non-text area in the sample image to 0, to obtain a binary mask image.

[0226] In one embodiment, the loss parameter adjustment module 1406 is further used to determine the text areas in the predicted image and the label image of each iteration according to the position information of the text in the sample image; calculate the distance between the predicted image of each iteration and the text area in the label image as the text area erasure loss of each iteration; calculate the image difference between the erased image and the label image of each iteration as the image difference loss of each iteration; calculate the feature difference between the erased image and the label image of each iteration as the feature difference loss of each iteration; and obtain the loss of the sample image for each iteration based on the text area erasure loss, image difference loss and feature difference loss of each iteration.

[0227] In one embodiment, the loss parameter adjustment module 1406 is further used to input the erased image and the label image of each iteration into the trained feature extraction model respectively to obtain the image features corresponding to the erased image and the label image of each iteration; calculate the distance between the image features corresponding to the erased image and the label image of each iteration to obtain the perceptual loss between the erased image and the label image of each iteration; calculate the corresponding Gram matrix according to the image features corresponding to the erased image of each iteration, calculate the corresponding Gram matrix according to the image features corresponding to the label image, calculate the difference between the Gram matrices of the erased image and the label image of each iteration, and obtain the style loss between the erased image and the label image of each iteration; based on the perceptual loss and the style loss, obtain the feature difference loss.

[0228] The processing device of the above-mentioned text erasure model only requires the trained model to have text erasure capability and does not require text detection capability, thereby reducing the model burden while achieving good erasure effect. During the training process, the sample image and the erased image of the previous iteration are iteratively spliced ​​to obtain a spliced ​​image. The spliced ​​image carries the sample image information and the information of the image after the previous erasure. The spliced ​​image is input into the model and a predicted image is output, so that the model learns to focus on the text area that needs to be erased in the sample image. Then, according to the position information of the text area where the text to be erased is located in the sample image, the predicted image and the sample image are superimposed to obtain the erased image of the current iteration, so that the first area corresponding to the position information in the erased image is derived from the predicted image, and the second area other than the first area is derived from the sample image. Then, based on the erased image, predicted image and label image of each iteration, the loss of each iteration of the sample image is calculated. The text erasure model obtained by optimizing based on the loss of multiple iterations can output a model that can not only effectively erase the text in the text area, but also preserve the boundary information between the text area and the non-text area, so as to achieve the background repair effect of the text area.

[0229] Each module in the processing device of the text erasure model can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0230] In one embodiment, a computer device is provided. The computer device may be a terminal or a server, and its internal structure diagram may be as follows: Figure 15 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O interface) and a communication interface connected through a system bus. The processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image processing data or processing data of a text erasure model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method and / or a text erasure model processing method is implemented.

[0231] Those skilled in the art will understand that Figure 15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0232] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0233] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0234] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0235] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0236] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0237] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0238] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Performing text detection on the original image using a text detection model to obtain location information of the text to be erased in the original image; Executing an iterative step; the iterative step includes: splicing the original image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration; erasing the current text on the spliced ​​image of the current iteration using a trained text erasure model, and outputting a predicted image of the current iteration; and superimposing the predicted image of the current iteration with the original image according to the position information to obtain an erased image of the current iteration, wherein the text erasure model does not have a text detection capability, but only has a text erasure capability; When the iteration stop condition is met, an erased image of the last iteration is obtained, and the erased image of the last iteration is used as a target image obtained by performing multiple text erasures on the original image; In which, the multi-channel stitched image of the first iteration is obtained by stitching the original image with the original image on a preset image channel; in the erased image of the current iteration obtained after superposition, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

2. The method according to claim 1, characterized in that The step of performing text detection on the original image using a text detection model to obtain position information of the text to be erased in the original image includes: Get the original image; The text detection model is used to perform text detection on the original image to determine a text area in the original image, where the coordinates of the text area are position information of the text to be erased in the original image.

3. The method according to claim 1, characterized in that The method of performing text erasure on the spliced ​​image of the current iteration by using the trained text erasure model and outputting the predicted image of the current iteration includes: Inputting the spliced ​​image of the current iteration into the trained text erasure model; Performing feature extraction on the spliced ​​image of the current iteration through the encoding network in the text erasure model to obtain image features of the spliced ​​image of the current iteration; The image is reconstructed based on the image features through the transposed convolutional network in the text erasure model, and the predicted image of the current iteration is output.

4. The method according to claim 1, wherein The method further comprises: generating a mask image corresponding to the original image according to the position information, wherein the mask image is used to distinguish a text area from a non-text area in the original image; The step of superimposing the predicted image of the current iteration with the original image according to the position information to obtain the erased image of the current iteration includes: According to the mask image, the predicted image of the current iteration is superimposed with the original image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

5. The method according to claim 4, characterized in that Generating a mask image corresponding to the original image according to the position information includes: determining a text area and a non-text area in the original image according to the position information; The pixel values ​​in the text area of ​​the original image are set to 1, and the pixel values ​​in the non-text area of ​​the original image are set to 0, to obtain a binary mask image.

6. The method according to claim 5, characterized in that The erased image of the current iteration is obtained by the following formula: dst=mask*output+(1-mask)*src; Wherein, mask represents the mask image, output represents the predicted image of the current iteration, src represents the original image, and dst represents the erased image of the current iteration.

7. The method according to claim 1, characterized in that The method further comprises: receiving an original image including text to be erased sent by an online collaborative document application; After performing multiple text erasing operations on the original image to obtain the target image, the method further includes: The target image obtained by erasing multiple times is returned to the online collaborative document application, so that the document is edited through the online collaborative document using the restored target image as the editing background.

8. The method according to claim 1, characterized in that The method further comprises: Obtaining an original image including text to be translated; After performing multiple text erasing operations on the original image to obtain the target image, the method further includes: Obtaining a translation corresponding to the text to be translated; According to the position of the text to be translated in the original image, the translation is pasted into the target image to obtain a translation image corresponding to the original image.

9. A method for processing a text erasure model, characterized in that: The method comprises: Obtaining a training sample, the training sample including a sample image, position information of text in the sample image, and a label image corresponding to the sample image that does not include the text, wherein the position information of the text in the sample image is obtained by performing text detection on the sample image using a text detection model; For the training sample, an iterative step is performed until an iteration stop condition is satisfied; the iterative step includes: splicing the sample image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration, inputting the spliced ​​image of the current iteration into a predetermined neural network model, performing the current text erasure on the spliced ​​image of the current iteration, outputting a predicted image of the current iteration, and superimposing the predicted image of the current iteration with the sample image according to the position information to obtain an erased image of the current iteration; Based on the erased image, predicted image and label image of each iteration, the loss of each iteration of the sample image is calculated, and the erasure loss corresponding to the training sample is obtained according to the sum of the losses of multiple iterations obtained when the iteration stops. After updating the network parameters of the predetermined neural network according to the erasure loss, return to the step of obtaining the training sample to continue training until the training stop condition is met, and obtain a trained text erasure model. The trained text erasure model does not have text detection capability, but only has text erasure capability.

10. The method according to claim 9, characterized in that The obtaining of training samples includes: Use the image without text as the label image; After adding text to the label image, a sample image is obtained; Perform text detection on the sample image to obtain position information of the text in the sample image.

11. The method according to claim 9, characterized in that The step of inputting the spliced ​​image of the current iteration into a predetermined neural network model, erasing the current text on the spliced ​​image of the current iteration, and outputting the predicted image of the current iteration comprises: Inputting the spliced ​​image of the current iteration into a predetermined neural network model; Performing feature extraction on the spliced ​​image of the current iteration through the encoding network in the predetermined neural network model to obtain image features of the spliced ​​image of the current iteration; The image is reconstructed based on the image features through the transposed convolutional network in the predetermined neural network model, and the predicted image of the current iteration is output.

12. The method according to claim 9, characterized in that The method further comprises: generating a mask image corresponding to the sample image according to the position information, wherein the mask image is used to distinguish a text area from a non-text area in the sample image; The step of superimposing the predicted image of the current iteration with the sample image according to the position information to obtain the erased image of the current iteration includes: According to the mask image, the predicted image of the current iteration is superimposed with the sample image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information originates from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area originates from the sample image.

13. The method according to claim 9, characterized in that The calculating the loss of each iteration of the sample image based on the erased image, the predicted image and the label image of each iteration includes: Determining the text areas in the predicted image and the label image for each iteration respectively according to the position information of the text in the sample image; Calculate the distance between the predicted image of each iteration and the text region in the label image as the text region erasure loss of each iteration; Calculating the image difference between the erased image and the label image at each iteration as the image difference loss at each iteration; Calculate the feature difference between the erased image and the label image at each iteration as the feature difference loss at each iteration; Based on the text region erasure loss, image difference loss and feature difference loss of each iteration, the loss of each iteration of the sample image is obtained.

14. The method according to claim 13, characterized in that Calculating the feature difference between the erased image and the label image in each iteration as the feature difference loss in each iteration includes: Inputting the erased image and the label image of each iteration into a trained feature extraction model respectively to obtain image features corresponding to the erased image and the label image of each iteration; Calculating the distance between the image features corresponding to the erased image of each iteration and the label image to obtain the perceptual loss between the erased image of each iteration and the label image; Calculating a corresponding Gram matrix based on the image features corresponding to the erased image of each iteration, calculating a corresponding Gram matrix based on the image features corresponding to the label image, calculating the difference between the Gram matrices of the erased image of each iteration and the label image, and obtaining a style loss between the erased image of each iteration and the label image; The feature difference loss is obtained based on the perceptual loss and the style loss.

15. An image processing device, characterized in that: The device comprises: A text location acquisition module is used to perform text detection on the original image using a text detection model to obtain location information of the text to be erased in the original image; An iterative erasure module is configured to execute an iterative step; the iterative step includes: splicing the original image and the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration; performing text erasure on the spliced ​​image of the current iteration using a trained text erasure model, and outputting a predicted image of the current iteration; and superimposing the predicted image of the current iteration with the original image according to the position information to obtain an erased image of the current iteration. The text erasure model does not have text detection capability, but only text erasure capability. a target image acquisition module, configured to obtain an erased image of the last iteration when an iteration stop condition is satisfied, and use the erased image of the last iteration as a target image obtained by performing multiple text erasures on the original image; In which, the multi-channel stitched image of the first iteration is obtained by stitching the original image with the original image on a preset image channel; in the erased image of the current iteration obtained after superposition, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

16. The device according to claim 15, characterized in that The text position acquisition module is also used to acquire the original image; perform text detection on the original image through a text detection model to determine the text area in the original image, and the coordinates of the text area are the position information of the text to be erased in the original image.

17. The device according to claim 15, characterized in that The iterative erasure module is used to input the spliced ​​image of the current iteration into a trained text erasure model; perform feature extraction on the spliced ​​image of the current iteration through the encoding network in the text erasure model to obtain image features of the spliced ​​image of the current iteration; perform image reconstruction based on the image features through the transposed convolutional network in the text erasure model, and output a predicted image of the current iteration.

18. The device according to claim 15, characterized in that The device further comprises: a mask image generating module, configured to generate a mask image corresponding to the original image according to the position information, wherein the mask image is used to distinguish between a text area and a non-text area in the original image; The iterative erasing module is used to superimpose the predicted image of the current iteration with the original image according to the mask image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the original image.

19. The device according to claim 18, characterized in that The mask image generation module is further used to determine the text area and non-text area in the original image according to the position information; set the pixel value in the text area in the original image to 1, and set the pixel value in the non-text area in the original image to 0, to obtain a binary mask image.

20. The device according to claim 19, characterized in that The erased image of the current iteration is obtained by the following formula: dst=mask*output+(1-mask)*src; Wherein, mask represents the mask image, output represents the predicted image of the current iteration, src represents the original image, and dst represents the erased image of the current iteration.

21. The device according to claim 15, characterized in that The device further comprises: An image acquisition module, configured to receive an original image including text to be erased, sent by an online collaborative document application; The device further comprises: The target image feedback module is used to return the target image obtained by the multiple erasures to the online collaborative document application, so that the document can be edited through the online collaborative document using the restored target image as the editing background.

22. The device according to claim 15, characterized in that The device further comprises: An image acquisition module, used to acquire an original image including the text to be translated; The device further comprises: The image acquisition module is used to obtain a translation corresponding to the text to be translated after performing multiple text erasures on the original image to obtain the target image; and paste the translation into the target image according to the position of the text to be translated in the original image to obtain a translated image corresponding to the original image.

23. A processing device for a text erasure model, characterized in that: The device comprises: A training sample acquisition module is used to acquire a training sample, wherein the training sample includes a sample image, position information of text in the sample image, and a label image corresponding to the sample image that does not include the text, wherein the position information of the text in the sample image is obtained by performing text detection on the sample image using a text detection model; An iterative erasing module is configured to execute an iterative step for the training sample until an iteration stop condition is satisfied; the iterative step comprises: splicing the sample image with the erased image of the previous iteration on a preset image channel to obtain a multi-channel spliced ​​image of the current iteration, inputting the spliced ​​image of the current iteration into a predetermined neural network model, performing text erasure on the spliced ​​image of the current iteration, outputting a predicted image of the current iteration, and superimposing the predicted image of the current iteration with the sample image according to the position information to obtain an erased image of the current iteration; The loss parameter adjustment module is used to calculate the loss of each iteration of the sample image based on the erased image, the predicted image and the label image of each iteration, obtain the erasure loss corresponding to the training sample according to the sum of the losses of multiple iterations obtained when the iteration stops, update the network parameters of the predetermined neural network according to the erasure loss, return to the step of obtaining the training sample to continue training until the training stop condition is met, and obtain a trained text erasure model. The trained text erasure model does not have text detection capability, but only has text erasure capability.

24. The device according to claim 23, characterized in that The training sample acquisition module is used to use an image that does not include text as a label image; add text to the label image to obtain a sample image; perform text detection on the sample image to obtain position information of the text in the sample image.

25. The device according to claim 23, characterized in that The iterative erasing module is used to input the spliced ​​image of the current iteration into a predetermined neural network model; perform feature extraction on the spliced ​​image of the current iteration through the encoding network in the predetermined neural network model to obtain image features of the spliced ​​image of the current iteration; perform image reconstruction based on the image features through the transposed convolutional network in the predetermined neural network model, and output a predicted image of the current iteration.

26. The device according to claim 23, characterized in that The device further comprises: a mask image generating module, configured to generate a mask image corresponding to the sample image according to the position information, wherein the mask image is used to distinguish a text area from a non-text area in the sample image; The iterative erasing module is further used to superimpose the predicted image of the current iteration with the sample image according to the mask image to obtain the erased image of the current iteration, so that in the erased image of the current iteration, the first area corresponding to the position information is derived from the area corresponding to the position information in the predicted image of the current iteration, and the second area other than the first area is derived from the sample image.

27. The device according to claim 23, characterized in that The loss parameter adjustment module is further used to determine the text areas in the predicted image and the label image of each iteration according to the position information of the text in the sample image; calculate the distance between the predicted image and the text area in the label image of each iteration as the text area erasure loss of each iteration; Calculating the image difference between the erased image and the label image at each iteration as the image difference loss at each iteration; Calculate the feature difference between the erased image and the label image at each iteration as the feature difference loss at each iteration; Based on the text region erasure loss, image difference loss and feature difference loss of each iteration, the loss of each iteration of the sample image is obtained.

28. The device according to claim 27, characterized in that The loss parameter adjustment module is further used to input the erased image and the label image of each iteration into the trained feature extraction model respectively to obtain image features corresponding to the erased image and the label image of each iteration; Calculating the distance between the image features corresponding to the erased image of each iteration and the label image to obtain the perceptual loss between the erased image of each iteration and the label image; Calculating a corresponding Gram matrix based on the image features corresponding to the erased image of each iteration, calculating a corresponding Gram matrix based on the image features corresponding to the label image, calculating the difference between the Gram matrices of the erased image of each iteration and the label image, and obtaining a style loss between the erased image of each iteration and the label image; The feature difference loss is obtained based on the perceptual loss and the style loss.

29. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 14 is implemented.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

31. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images