Image processing method, image processing device and program

Abstract

To provide a technique that makes it possible to selectively erase only a specific character string from an image.SOLUTION: An image processing method includes: extracting, by a background extraction part, a background image having all character strings erased from an input image including the character strings; using, by a character extraction part, the input image and background image to extract the character strings; erasing, by a character string erasure part, a specific character string from an image of the extracted character string; and integrating, by a reconstitution part, the background image and the image of the character strings which have the specific character string erased.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to techniques for an image processing method, an image processing device, and a program. [Background technology]

[0002] 2. Description of the Related Art Techniques for removing text regions from images containing text (Scene Text Removal) are being developed (see, for example, Non-Patent Documents 1 and 2). Scene text removal (STR) or scene text eraser is the task of removing text regions from a scene image. As a result of STR, text regions in a scene image are filled in as if they were part of the surrounding area. This results in the disappearance of text information from the image. Note that traditional STR methods attempt to remove all text regions in a scene image. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Nakamura, Zhu, Yanai, Uchida, "Scene text eraser,",Proc. International Conference on Document Analysis and Recognition, (ICDAR), 2017. [Non-Patent Document 2] Tursun, Zeng, Denman, Sivapalan, Sridharan, Fookes, "MTR-Net: A generic scene text eraser," Proc. International Conference on Document Analysis and Recognition (ICDAR), 2019. Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional techniques were designed to remove all scene text, so it was not possible to select the text to be removed. Because characters contain a lot of useful information, removing all scene text would result in a large loss of information.

[0005] In view of the above circumstances, the present invention provides a technique that enables selective erasure of only a specific character string from an image. [Means for solving the problem]

[0006] One aspect of the present invention is an image processing method in which a background extraction unit extracts a background image from an input image containing character strings, where all of the character strings have been erased, a character string extraction unit extracts the character string using the input image and the background image, a character string erasure unit erases a specified character string from the extracted image of the character string, and a reconstruction unit integrates the background image with the image of the character string from which the specified character string has been erased.

[0007] One aspect of the present invention is an image processing device comprising: a background extraction unit that extracts a background image from an input image containing character strings, where all of the character strings have been erased; a string extraction unit that extracts the character string using the input image and the background image; a string erasure unit that erases a specified character string from the extracted image of the character string; and a reconstruction unit that integrates the background image and an image of the character string from which the specified character string has been erased.

[0008] One aspect of the present invention is a program for causing a computer to function as the image processing device described above. Effect of the Invention

[0009] The present invention makes it possible to selectively erase only specific character strings from an image. [Brief description of the drawings]

[0010] [Figure 1] FIG. 13 is a diagram illustrating an example of removing a character string. [Diagram 2] FIG. 1 illustrates an example of a configuration of an image processing apparatus according to an embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an overview of a configuration and processing of an embodiment. [Figure 4] 1 is a flowchart showing an outline of a processing procedure performed in an embodiment. [Diagram 5] 11 is a flowchart of a first processing procedure in pre-learning. [Figure 6] 13 is a flowchart of a second processing procedure in pre-learning. [Figure 7] 13 is a flowchart of a third processing procedure in pre-learning. [Figure 8] 13 is a flowchart of a fourth processing procedure in pre-learning. [Figure 9] 13 is a flowchart of a synchronous learning process procedure. [Figure 10] 13 is a flowchart of a selective character string erasure process. [Figure 11] FIG. 13 is a diagram showing an example of an image used in learning. [Figure 12] FIG. 13 is a diagram showing an example of an evaluation result. [Figure 13] FIG. 13 is a diagram showing an example of an evaluation result. [Figure 14] FIG. 13 is a diagram showing an example of a quantitative evaluation result. [Figure 15] FIG. 13 is a diagram showing an example of an evaluation result using a character font image dataset. [Figure 16] FIG. 13 is a diagram showing an example of a generated image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described in detail with reference to the drawings.

[0012] FIG. 1 is a diagram showing an example of character string removal. The image with code g1 is the original image. The image with code g2 is an image of the case where a text region is removed using conventional technology. The image with code g3 is an image of the case where characters are selectively removed using the method of this embodiment. In this embodiment, target words are automatically found and removed end-to-end.

[0013] For example, assume that an input image I has a set of K word candidates that can be deleted, Ω={“drug,” “store,” “coke”}, and “drug” is to be deleted. In this case, the image processing device 1 (FIG. 2) deletes ω(∈Ω)=“drug” and ― Do not eliminate the words “store” and “coke” from ∪(Ω\ω).

[0014] 2 is a diagram showing an example of the configuration of the image processing device of this embodiment. The image processing device 1 includes, for example, an acquisition unit 11, a learning unit 12, an extraction unit 13, a character string erasure unit 14, a reconstruction unit 15, a loss calculation unit 16, an update unit 17, an output unit 18, and a storage unit 19. The learning unit 12 includes, for example, a learning data preparation unit 121. The extraction unit 13 includes, for example, a background extraction unit 131 and a character string extraction unit 132. The loss calculation unit 16 includes, for example, a background extraction loss calculation unit 161, a string extraction loss calculation unit 162, a string deletion loss calculation unit 163, and a reconstruction loss calculation unit 164. The update unit 17 includes, for example, a background extraction unit coefficient update unit 171, a character string extraction unit coefficient update unit 172, a character string deletion unit coefficient update unit 173, and a reconstruction unit coefficient update unit 174.

[0015] The acquisition unit 11 acquires an image.

[0016] The learning data preparation unit 121 performs a learning data preparation process, which is a process of preparing learning data that is a collection of a large number of sets of a plurality of images.

[0017] The background extraction unit 131 performs background extraction processing, which is processing for erasing all character strings from an image that includes character strings and extracting a background image. That is, the background image is an image in which all character strings have been erased from the input image.

[0018] The character string extraction unit 132 performs a character string extraction process, which is a process of erasing the background from an image including a character string and extracting only the character string.

[0019] The character string erasing unit 14 performs a character string erasing process, which is a process for erasing a specified character string from an image in which only the character string is extracted.

[0020] The reconstruction unit 15 performs reconstruction processing, which is processing for synthesizing the image from which the specified character string has been erased with the background image.

[0021] The background extraction loss calculation unit 161 performs a background extraction loss calculation process, which is a process for evaluating whether the result of the background extraction process is desirable.

[0022] The character string extraction loss calculation unit 162 performs a character string extraction loss calculation process, which is a process for evaluating whether the result of the character string extraction process is desirable.

[0023] The character string erasure loss calculation unit 163 performs a character string erasure loss calculation process, which is a process for evaluating whether the result of the character string erasure process is desirable or not.

[0024] The reconstruction loss calculation unit 164 performs a reconstruction loss calculation process, which is a process for evaluating whether the result of the reconstruction process is desirable.

[0025] The background extraction unit coefficient update unit 171 performs a background extraction unit coefficient update process, which is a process of updating the background extraction unit coefficients based on the calculated loss.

[0026] The character string extraction unit coefficient update unit 172 performs a character string extraction unit coefficient update process, which is a process of updating the character string extraction unit coefficients based on the calculated loss.

[0027] The string erasure unit coefficient update unit 173 performs a string erasure unit coefficient update process, which is a process of updating the string erasure unit coefficients based on the calculated loss.

[0028] The reconstruction unit coefficient update unit 174 performs a reconstruction unit coefficient update process, which is a process of updating the reconstruction unit coefficients based on the calculated loss.

[0029] The output unit 18 outputs the estimated image obtained by combining the image from which the specified character string has been erased and the background image by the reconstruction unit 15 to an external device. The external device is, for example, an image display device, a personal computer, a mobile terminal, or the like.

[0030] The memory unit 19 stores, for example, parameters used by each processing unit, programs used by the voice processing device at the beginning of the learning process, programs used at the later stages of learning, and programs used when selecting and erasing character strings using a learned module described later.

[0031] The image processing device 1 is configured using a processor such as a CPU (Central Processing Unit) and a memory. The image processing device 1 functions as, for example, an acquisition unit 11, a learning unit 12, an extraction unit 13, a character string erasure unit 14, a reconstruction unit 15, a loss calculation unit 16, an update unit 17, and an output unit 18 by the processor executing a program. All or part of the functions of the image processing device 1 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a semiconductor storage device (for example, an SSD: Solid State Drive), or a storage device such as a hard disk or a semiconductor storage device built into a computer system. The above program may be transmitted via an electric communication line.

[0032] Fig. 3 is a diagram showing an overview of the configuration and processing of this embodiment. As shown in Fig. 3, the image processing device 1 can be expressed by a plurality of modules. Each module is a mathematical model and a network. An overview of the processing in the case of selective character string deletion will be explained using each trained module. The input and output of each module during training will be described later.

[0033] The image g11 is input to a Background Extraction Module (reference symbol g12) and a Text Extraction Module (reference symbol g14). As described above, the Background Extraction Module (reference symbol g12) corresponds to background extraction processing. The Text Extraction Module (reference symbol g14) corresponds to character string extraction processing.

[0034] A background extraction module (reference symbol g12) extracts an estimated background image g13 by erasing all character strings from an image that includes character strings, and outputs the extracted estimated background image g13. The image g11 and the estimated background image g13 are input to the Text Extraction Module (g14). The Text Extraction Module (g14) erases the background from the image including the character string, extracts only the character string, and outputs the extracted estimated character string image g15.

[0035] The Selective Word Removal Module (reference symbol g17) receives the estimated character string image g15 and the input label g16. The Selective Word Removal Module (reference symbol g17) corresponds to a character string erasure process. The Selective Word Removal Module (reference symbol g17) extracts an estimated erased character string image g18 and outputs the extracted estimated erased character string image g18.

[0036] The estimated background image g13 and the estimated erased character string image g18 are input to the reconstruction module (g19). The reconstruction module (g19) corresponds to a reconstruction process. The reconstruction module (g19) synthesizes the image from which the specified character string has been erased with the background image, and outputs the synthesized estimated image g20.

[0037] The model is trained in an end-to-end manner after pre-training. Specifically, each module is first pre-trained individually, and then all modules are fine-tuned end-to-end. For example, the Selective Word Removal Module (symbol g17) is trained with text images with no background changes, and each module can be pre-trained with a sufficient amount of synthetic images prepared for that module. Therefore, according to this embodiment, the multi-module structure allows the model training cost to be dramatically reduced.

[0038] [Explanation of each process] Next, each process used in the learning process and image processing will be described.

[0039] (Learning sample preparation process) In the training data preparation process, there is no input, and the output is a set of training samples. The training samples are the training background images ∞ (x i ) - , training input image x i , training string image z i , Learning string image with deletion 〇 (z i ) - , learning target image y i Five images and training input labels c i Group of 〇 ((x i ) - ,x i ,z i ,(z i ) - ,y i ,c i )

[0040] Background image for learning (x i ) - is an image that does not contain any strings. i is an image containing only text on a plain background. A learning input image xi is an image in which a learning text image is superimposed on a learning background image. A learning erased text image (z i ) - is an image obtained by erasing a specific character string from the learning character string image. i is an image in which a training erased character string image is superimposed on a training background image. In other words, it is an image in which a specific character string has been erased from a training input image. Also, the training input label c i is the identification information (ID) of the string to be deleted.

[0041] In the learning sample preparation process, a method in which a character string is selected from a predetermined set of character strings is generally used, but multiple character strings may be selected, or a character string may be directly input without preparing a set of character strings in advance. A learning sample does not need to have all six of the components mentioned above; at a minimum, it will be sufficient if it has three items: a learning input image, a learning target image, and a learning input label. In this embodiment, a large number of training samples is called a training sample set.

[0042] (Background extraction processing) In the background extraction process, the inputs during learning are the learning input image and the background extraction unit coefficients, and the inputs during use are the input image and the background extraction unit coefficients (the same applies to the following similar processes). The background extraction unit coefficients are model parameters of the background extraction unit 131 that executes this process as an image. Here, the background extraction unit 131 is constructed with the aim of erasing all character strings from the input image. In this embodiment, the output image of the background extraction unit 131 is called an estimated background image. The output during learning is a learning estimated background image, and the output during use is an estimated background image (the same applies to the following similar processes). During learning, the background extraction unit 131 learns, for example, with an MSE (Mean Squared Error) loss function between the learning estimated background image and the background image that is the teacher data.

[0043] That is, during pre-learning and simultaneous learning, the background extraction unit 131 receives an input image for learning and outputs an estimated background image for learning. During selective character string erasure, the background extraction unit 131 receives an input image and outputs an estimated background image. Both the input image and output image of the background extraction unit 131 are three-channel RGB images. For the background extraction process, we use conventional STR technologies, such as those described in Non-Patent Documents 1 and 2, which are prior art.

[0044] In this embodiment, a neural network structure called U-Net (see, for example, Reference 1) is used as a method for removing the background and extracting only the character string. The U-Net used in the background extraction process of this embodiment has, for example, three convolution layers, three deconvolution layers, and skip connections. In addition, four residual layers are used in the bottleneck in the U-Net. The U-Net lowers the spatial resolution using multiple convolution layers, and then increases the spatial resolution using the same number of deconvolution layers to output an image with the same spatial resolution as the input image. In addition, the U-Net creates a shortcut between the convolution layer output and the deconvolution layer output, which have the same spatial resolution, and reflects information with high spatial resolution in the output.

[0045] Reference 1;Olaf Ronneberger, Philipp Fischer, et al., “U-Net: Convolutional networks for biomedical image segmentation”, Proc. International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), 2015

[0046] (String extraction process) In the early stages of learning, it is assumed that the character string extraction unit 132 is pre-trained independently. Therefore, the inputs in the early stages of learning are the learning input image, the learning background image included in the learning sample, and the character string extraction unit coefficients. In the later stages of learning, it is assumed that the whole including the character string extraction unit 132 is trained simultaneously. Therefore, the inputs in the later stages of learning are the learning input image, the learning estimated background image which is the output of the background extraction process, and the character string extraction unit coefficients. The inputs during use are the input image, the estimated background image, and the character string extraction unit coefficients. The character string extraction unit coefficients are model parameters of the character string extraction unit 132 which executes this process as an image. The output is an estimated character string image for learning. The character string extraction unit 132 is constructed with the aim of erasing the background from the input image and extracting only the character string. In this embodiment, the output image of the character string extraction unit 132 is called an estimated character string image.

[0047] That is, the character string extraction unit 132 receives a learning input image and a learning background image during pre-learning, and receives a learning input image and a learning estimated background image during the simultaneous learning process, and outputs an estimated character string image for learning. During selective character string erasure, the character string extraction unit 132 receives an input image and an estimated background image, and outputs an estimated character string image.

[0048] In this embodiment, a neural network structure called U-Net (see, for example, Reference 1) is used as a method for removing the background and extracting only the character string. The U-Net of the character string extraction unit 132 in this embodiment has, for example, four convolution layers, four deconvolution layers, and skip connections.

[0049] The input image of the character string extraction unit 132 is a six-channel image obtained by superimposing two three-channel RGB images, the input image and the background image. The output of the character string removal unit 14 is an RGBA image (RGB plus an alpha channel) showing the extracted text. Non-text areas of the output image are treated as transparent (α=0), while text areas are treated as non-transparent (α=1), and have RGB colors. In this embodiment, the estimated character string image output by the character string extraction unit 132 holds an alpha channel in addition to the channels of the input image.

[0050] (String deletion process) The inputs in the early stages of learning are a learning string image, a learning input label, and a string erasure unit coefficient. The inputs in the later stages of learning are an estimated string image for learning, a learning input label, and a string erasure unit coefficient. The inputs during use are an estimated string image, an input label, and a string erasure coefficient. The string erasure unit coefficient is a model parameter of the string erasure unit 14 that executes this process as an image. The output during learning is an estimated erased character string image for learning. The output during use is an estimated erased character string image. As in the character string extraction process, the output of the character string erasure unit 14, which aims to erase the character string specified by the input label, is set as an estimated erased character string image.

[0051] That is, string erasure unit 14 receives a learning string image and a learning input label during pre-learning, and receives an estimated learning string image and a learning input label during simultaneous learning, and outputs an estimated erased string image for learning. When selectively erasing a string, string erasure unit 14 receives an estimated string image and an input label, and outputs an estimated erased string image.

[0052] In this embodiment, a neural network structure called Conditional U-Net (see, for example, Reference 2) is used as a method for erasing a specified character string. Conditional U-Net is a structure that can add a class condition to the above-mentioned U-Net.

[0053] Specifically, the condition is expressed as a K-dimensional one-hot vector, and the target word ω is specified. That is, among the K elements, the element corresponding to ω is set to 1, and the others are set to 0. In this embodiment, the Conditional U-Net includes a FiLM (Feature-wise Linear Modulation) layer to supply the condition vector to the bottleneck part of the standard U-Net. Specifically, the output of the FiLM layer modifies the feature map of the bottleneck part of the U-Net. Note that FiLM is a formalization of the conditional affine transformation.

[0054] Reference 2; Gabriel Meseguer-Brocal, Geoffroy Peeters, “Conditioned-U-Net: Introducing a control mechanism in the U-Net for multiple source separations”, Proc. International Society for Music Information Retrieval Conference (ISMIR), 2019

[0055] As shown in FIG. 3 and described above, the input to the character string erasure unit 14 is an RGBA image containing only text and a condition vector. The output of the character string erasure unit 14 is ― This is a text-only RGBA image showing only the word ∪(Ω\ω).

[0056] (Reconstruction process) The inputs in the early stages of learning are the erased character string image for learning, the background image for learning, and the reconstruction unit coefficients. The inputs in the later stages of learning are the estimated erased character string image for learning, the estimated background image for learning, and the reconstruction unit coefficients. The inputs during use are the estimated erased character string image, the estimated background image, and the reconstruction unit coefficients. The reconstruction unit coefficients are model parameters of the reconstruction unit 15 that executes this process as an image. The output during learning is an estimated image for learning. The output during use is an estimated image. As in the character string extraction process, the output of the reconstruction unit 15 is an estimated image.

[0057] That is, the reconstructing unit 15 receives the erased learning string image and the learning background image during pre-learning, receives the estimated erased learning string image and the estimated learning background image during the simultaneous learning process, and outputs an estimated learning image. The reconstructing unit 15 receives the estimated erased learning string image and the estimated background image during the selective string erasure process, and outputs an estimated image.

[0058] In this embodiment, as a method for synthesizing an image of only character strings with a background image, for example, U-Net is used as in the character string extraction process. By using this U-Net, in the reconstruction process, the erased character string image having an alpha channel is synthesized with the background image having no alpha channel, and an estimated image having no alpha channel is output. The RGB image (estimated background image) output from the background extraction unit 131 and the RGBA image output from the character string erasure unit 14 are layered as a 7-channel image and input to the reconstruction unit 15.

[0059] (Background extraction loss calculation process) The inputs are a training background image and a training estimated background image. The output is a training background extraction loss. Note that in this process, the degree to which the training estimated background image is close to the training background image is evaluated and output as the training background extraction loss. In this embodiment, the loss is calculated using, for example, the mean square error between the training background image and the training estimated background image.

[0060] (String extraction loss calculation process) The inputs are a training string image and a training estimated string image. The output is the training string extraction loss. In this embodiment, the loss is calculated using, for example, the mean square error between the learning character string image and the learning estimated character string image.

[0061] (String deletion loss calculation process) The inputs are a learning erased character string image and a learning estimated erased character string image. The output is the training string erasure loss. In this embodiment, the loss is calculated using, for example, the mean square error between the learning erased character string image and the learning estimated erased character string image.

[0062] (Reconstruction loss calculation process) The inputs are a training estimated image and a training target image. The output is the reconstruction loss for training. Note that in this process, the degree to which the estimated image for training is close to the target image for training is evaluated and output as the reconstruction loss for training. In this embodiment, the loss is calculated using, for example, the mean square error between the learning estimated image and the learning target image.

[0063] (Background extraction coefficient update process) In the early stage of learning, pre-learning is performed by the background extraction unit alone. Therefore, the inputs in the early stage of learning are the training background extraction loss and the current background extraction unit coefficients, and the model parameters of the background extraction unit 131 are updated based on the training background extraction loss. In the later stage of learning, the entire model is learned simultaneously. Therefore, the inputs in the later stage of learning are the training reconstruction loss and the current background extraction unit coefficients, and the model parameters are updated based on the training reconstruction loss. The output is the updated background extraction coefficients. Note that in this process, the model parameters of the background extraction unit 131 are updated based on the loss.

[0064] It is assumed that the background extraction process is represented by a neural network. Therefore, in this embodiment, by backpropagating the loss to the neural network, it is possible to update the background extraction unit coefficients corresponding to the model parameters of the background extraction unit 131 constituting the background extraction process.

[0065] (String extraction coefficient update process) The inputs for the early learning stage are the training string extraction loss and the current string extraction unit coefficients. The inputs for the later learning stage are the training reconstruction loss and the current string extraction unit coefficients. The output is the updated string extractor coefficients.

[0066] It is assumed that the character string extraction process is represented by a neural network. Therefore, in this embodiment, by backpropagating the loss to the neural network, it is possible to update the character string extraction unit coefficients corresponding to the model parameters of the character string extraction unit 132 constituting the character string extraction process.

[0067] (String deletion section coefficient update process) The inputs for the early learning stage are the training string erasure loss and the current string erasure coefficients. The inputs for the later learning stage are the training reconstruction loss and the current string erasure coefficients. The output is the updated string eraser coefficients.

[0068] The string erasure process is assumed to be represented by a neural network. Therefore, in this embodiment, the loss is back-propagated to the neural network, so that the string erasure unit coefficients corresponding to the model parameters of the string erasure unit 14 constituting the string erasure process can be updated.

[0069] (Reconstruction coefficient update process) The input is the training reconstruction loss and the current reconstruction coefficients. The output is the updated reconstruction unit coefficients.

[0070] The reconstruction process is assumed to be expressed by a neural network. Therefore, in this embodiment, the loss is back-propagated to the neural network, so that the reconstruction unit coefficients corresponding to the model parameters of the reconstruction unit 15 constituting the reconstruction process can be updated.

[0071] [Processing Procedure] First, an outline of the processing procedure performed in this embodiment will be described with reference to Fig. 4, which is a flowchart showing the outline of the processing procedure performed in this embodiment. (Step S1) The image processing device 1 performs pre-learning in order to individually optimize each process of background extraction, character string extraction, character string erasure, and reconstruction. (Step S2) The image processing device 1 performs synchronous learning in order to simultaneously optimize the processing of all processing units. (Step S3) The image processing device 1 performs selective character string erasure in order to implement selective character string erasure using the learned model.

[0072] (Pre-learning process) First, the pre-learning process will be described with reference to Figs. 5 to 8. The output of this process is a set of learning samples. Through the following process, the learning data preparation unit 121 prepares learning data that is a collection of a large number of sets of multiple images. As described above, since a learning sample does not need to have all six elements, processing is performed on the learning input image, learning target image, and learning input label, and processing on the other elements does not need to be performed.

[0073] FIG. 5 is a flowchart of a first processing procedure in the pre-learning. (Step S11) The acquisition unit 11 acquires an input image for learning. (Step S12) The background extraction unit 131 performs background extraction processing on the learning input image. (Step S13) The background extraction loss calculation unit 161 performs a background extraction loss calculation process. (Step S14) The background extraction unit coefficient update unit 171 performs a background extraction unit coefficient update process. After the process, the learning data preparation unit 121 returns to the process of step S11 and repeats the processes of steps S11 to S14.

[0074] FIG. 6 is a flowchart of a second processing procedure in the pre-learning. (Step S21) The acquisition unit 11 acquires a learning input image and a learning background image. (Step S22) The character string extraction unit 132 receives the learning input image and the learning background image, and extracts a character string. (Step S23) The character string extraction loss calculation unit 162 performs a character string extraction loss calculation process. (Step S24) The character string extraction unit coefficient update unit 172 performs a character string extraction unit coefficient update process. After the process, the learning data preparation unit 121 returns to the process of step S21 and repeats the processes of steps S21 to S24. Through these processes, the character string extraction unit coefficients are updated and the character string extraction unit 132 is trained.

[0075] FIG. 7 is a flowchart of a third processing procedure in the pre-learning. (Step S31) The acquisition unit 11 acquires a learning character string image and a learning input label. (Step S32) The character string erasure unit 14 receives the learning character string image and the learning input label, and performs a character string erasure process. (Step S33) The character string erasure loss calculation unit 163 performs a character string erasure loss calculation process. (Step S34) The character string erasure unit coefficient update unit 173 performs a character string erasure unit coefficient update process. After the process, the learning data preparation unit 121 returns to the process of step S31, and repeats the processes of steps S31 to S34. Through these processes, the character string erasure unit coefficients are updated and learning of the character string erasure unit 14 is performed.

[0076] As described above, the learning of the string erasure unit 14 is an image pair such as images g53 and g54 in FIG. 4. A sufficient number of image pairs are required to cover the style and color variations of the target word specified by ω. The background image has been removed by the preceding module (Text Extraction Module (reference symbol g14 in FIG. 3)), and therefore the string erasure unit 14 does not need to consider the background variations. In other words, if it is necessary to take background variations into consideration in the string erasure process, it is necessary to have a very large number of combinations of background and target word. In contrast, according to this embodiment, the background has been removed, and therefore the learning load can be reduced.

[0077] FIG. 8 is a flowchart of a fourth processing procedure in the pre-learning. (Step S41) The acquisition unit 11 acquires a learning deleted character string image and a learning background image. (Step S42) The reconstructing unit 15 receives the learning erased character string image and the learning background image and performs reconstruction processing. (Step S43) The reconstructing loss calculation unit 164 performs a reconstructing loss calculation process. (Step S44) The reconstruction unit coefficient update unit 174 performs a reconstruction unit coefficient update process. After the process, the learning data preparation unit 121 returns to the process of step S41 and repeats the processes of steps S41 to S44. Through these processes, the reconstruction unit coefficients are updated and the reconstruction unit 15 is trained.

[0078] (Synchronous learning process) FIG. 9 is a flowchart of a processing procedure for the synchronous learning process. (Step S51) The acquisition unit 11 acquires a learning input image and a learning input label. (Step S52) The background extraction unit 131 performs background extraction processing on the learning input image. (Step S53) The character string extraction unit 132 receives the estimated background image and the learning input image, and extracts character strings. (Step S54) The character string erasure unit 14 receives the estimated character string image and the learning input label, and performs a character string erasure process. (Step S55) The reconstructing unit 15 receives the estimated background image and the estimated erased character string image and performs reconstruction processing. (Step S56) The reconstructing loss calculation unit 164 performs a reconstructing loss calculation process. (Step S57) The reconstruction unit coefficient update unit 174 performs a reconstruction unit coefficient update process. (Step S58) The character string erasure unit coefficient update unit 173 performs a character string erasure unit coefficient update process. (Step S59) The character string extraction unit coefficient update unit 172 performs a character string extraction unit coefficient update process. (Step S60) The background extraction unit coefficient update unit 171 performs a background extraction unit coefficient update process. After the process, the learning data preparation unit 121 returns to the process of step S51 and repeats the processes of steps S51 to S60. Through these processes, the character string erasing section coefficients, character string extraction section coefficients, background extraction section coefficients and reconstruction section coefficients are updated, and learning is performed for background extraction section 131, character string extraction section 132, character string erasing section 14 and reconstruction section 15.

[0079] (Selective string deletion process; when in use) FIG. 10 is a flow chart of the selective character string deletion process. (Step S71) The acquisition unit 11 acquires an input image and an input label. (Step S72) The background extraction unit 131 performs background extraction processing on the learning input image. (Step S73) The character string extraction unit 132 receives the estimated background image and the input image and extracts a character string. (Step S74) The character string erasure unit 14 receives the estimated character string image and the input label and performs a character string erasure process. (Step S75) The reconstructing unit 15 receives the estimated background image and the estimated erased character string image and performs reconstruction processing. (Step S76) The output unit 18 outputs to an external device an estimated image obtained by combining the image from which the specified character string has been erased by the reconstruction unit 15 with the background image.

[0080] [evaluation] We constructed training samples using SynthText (see, for example, Reference 3), a method that can naturally embed text strings into natural images. The procedure for constructing the training sample is as follows. I. First, we prepared several natural images and used them as background images. II. Next, we prepared a set of strings that could be erased in the experiment and strings that would not be erased even in the evaluation, and constructed string images by randomly changing the combination of any number of strings to be erased, any number of strings not to be erased, and the fonts of each string. III. We also constructed an erased string image by removing one of the strings selected in II. This removed string becomes the input label. IV.Furthermore, the input image and the target image were constructed by synthesizing the background image and the character string image, and the background image and the erased character string image, respectively, using SynthText.

[0081] Reference 3; Ankush Gupta, Andrea Vedaldi, et al., “Synthetic data for text localization in natural images”, Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

[0082] FIG. 11 is a diagram showing an example of an image used in learning. Image g51 is an example of a background image. Image g51 is an example of an image including the string {"natural", "drug"} in the background image. Image g53 is an example of an image of the string {"natural", "drug"}. Image g54 is an example of a string image with the target string "drug" deleted. Image g55 is an example of a useful output image of the image processing device 1.

[0083] In order to make the evaluation as objective as possible, five country names were used as target word candidates Ω. In the evaluation, the strings "France", "China", "Germany", "Japan", and "India" were used as target words.

[0084] 12 and 13 are diagrams showing examples of evaluation results. In Figures 12 and 13, images g101, g111, g121, g131, and g141 are input images. Images g102, g112, g122, g132, and g142 are target images. Images g103, g113, g123, g133, and g143 are estimated images by the method of this embodiment. Images g104, g114, g124, g134, and g144 are result images of an attempt to selectively erase strings using only Conditional U-Net.

[0085] Also, in FIG. 12, for images g101 to g104, "France" was specified as the input label, and the aim was to erase this character string from the image. For images g111 to g114, "China" was specified as the input label, and the aim was to erase this character string from the image. For images g121 to g124, "Germany" was specified as the input label, and the aim was to erase this character string from the image. For images g131 to g134, "Japan" was specified as the input label, and the aim was to erase this character string from the image. For images g141 to g144, "India" was specified as the input label, and the aim was to erase this character string from the image.

[0086] 12 and 13, the method of the present embodiment can erase only the appropriately specified character string. In contrast, Conditional U-Net cannot erase the specified character string.

[0087] Fig. 14 is a diagram showing an example of a quantitative evaluation result. In Fig. 14, "Proposed" is a quantitative evaluation result of the method of this embodiment, and "CU-Net" is a quantitative evaluation result of the Conditional U-Net method. This evaluation evaluates to what extent the output result of selective character string erasure contains character strings to be erased, whether character strings other than those to be erased are contained, and the image quality of the output result.

[0088] The evaluation indices are recall (R), precision (P), mean square error (MSE), signal-to-noise ratio (PSNR), and structural similarity (SSIM). A low recall is required for target words, and a high recall is required for non-target words. For other indices, it is desirable to have a high precision, signal-to-noise ratio, and SSIM, and a low mean square error.

[0089] As shown in Fig. 14, the method of the present embodiment is able to erase a significant proportion of the character strings to be erased that were included in the input image, especially from the recall of the target word. On the other hand, the conventional Conditional U-Net method is unable to erase most of the character strings to be erased that were included in the input image. In addition, it can be seen from the recall of non-target words that most of the non-erased character strings remain intact. Furthermore, the results of MSE, SN, and SSIM show that the image does not deteriorate significantly even when the character string is erased. These facts demonstrate the effectiveness of the method of this embodiment.

[0090] FIG. 15 is a diagram showing an example of an evaluation result using a character font image dataset. The character font image dataset MyFonts (see Reference 4) was used. In this data, multiple impression word labels are assigned to each font, and these were used as labels. Also, the character type (A, B, C, ..., Z) was used as additional condition information. In FIG. 15, "Full model" indicates this embodiment.

[0091] Reference 4; Tianlang Chen, Zhaowen Wang, et al., “Large-scale tag-based font retrieval with generative feature learning”, Proc. International Conference on Computer Vision (ICCV), 2019

[0092] In the evaluation, the method was compared with the following four conventional techniques known as conditional image generation methods. ·Comparative example 1; C-GAN (see reference 5) ·Comparative example 2; AC-GAN (see reference 6) Comparative example 3; CP-GAN (see reference 7) Comparative example 4; Imp2Font (see reference 8)

[0093] Reference 5; Mehdi Mirza, Simon Osindero, “Conditional generative adversarial nets”, arXiv preprint 1411.1784, 2014 Reference 6; Augustus Odena, Christopher Olah, et al., “Conditional image synthesis with auxiliary classifier GANs”, Proc. International Conference on Machine Learning (ICML), 2017. Reference 7; Takuhiro Kaneko, Yoshitaka Ushiku, “Class-Distinct and Class-Mutual Image Generation with GANs”, BMVC 2019 (Spotlight), 2019, [online], [Internet search searched June 2, 2023],<URL: https: / / takuhirok.github.io / CP-GAN / > Reference 8;Matsuda, Kimura, Uchida, “Impression2font: Generating fonts by specifying impressions”, Proc. International Conference on Document Analysis and Recognition (ICDAR), 2021

[0094] In addition, in the evaluation, we also compared the results with the case where the label editing process used in the method of this embodiment was not used (w / o CMLE) and the case where the label compression process was not used (w / o ILSC).The following four types of evaluation indices were used. FID (see Reference 9) is an index used to evaluate the quality of the generated image. A smaller value indicates a better result. Intra-FID (see Reference 10): This is an index that averages the FID for each label type. A smaller value indicates a better result. mAP (mean Average Precision)-train, mAP-test These are indices that extend GAN-train and GAN-test, which are widely used as evaluation indices for generative adversarial networks, to multi-label conditions, and are indices that evaluate both the diversity and quality of generated images in an integrated manner. A larger value indicates a better result.

[0095] Reference 9; Martin Heusel, Hubert Ramsauer, et al., “GANs trained by a two time-scale update rule converge to a local Nash equilibrium”, Proc. Conference on Neural Information Processing Systems (NeurIPS), 2017 Reference 10;Takeru Miyato, Masanori Koyama, “cGANs with projection discriminator”, Proc. International Conference on Learning Representations (ICLR), 2018

[0096] As shown in FIG. 15, the method of this embodiment obtained results that were better than or comparable to the conventional method.

[0097] Fig. 16 is a diagram showing an example of a generated image. In Fig. 16, "Full model" indicates this embodiment. In Fig. 16, each row corresponds to an input label, and each column corresponds to an image generation method. In the evaluation, since the image generated varies depending on the input noise, four generated images are shown for each combination of input label and image generation method.

[0098] As shown in Fig. 16, the conventional Imp2Font method generates images with little variety when the input label is fixed. In contrast, the method of the present embodiment generates diverse and highly readable font images while retaining the impression of the input label.

[0099] The image processing device 1 configured in this way erases the background from the input image, erases the specified character string from the image containing only the extracted character string, and combines the background image from which all character strings have been erased with the image containing only the character string from which the specified character string has been erased. This makes it possible for the image processing device 1 configured in this way to generate an image from which only the specified character string has been erased from the input image.

[0100] (Modification) The above-mentioned image is not limited to a still image, but may be a moving image. In this case, for example, the character string image specified by the above-mentioned method is removed from the image for each frame.

[0101] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0102] The present invention provides a device and method for erasing specified text images from still images and moving images. [Explanation of symbols]

[0103] 1...image processing device, 11...acquisition unit, 12...learning unit, 13...extraction unit, 14...character string erasure unit, 15...reconstruction unit, 16...loss calculation unit, 17...update unit, 18...output unit, 19...storage unit, 121...learning data preparation unit, 131...background extraction unit, 132...character string extraction unit, 161...background extraction loss calculation unit, 162...character string extraction loss calculation unit, 163...character string erasure loss calculation unit, 164...reconstruction loss calculation unit, 171...background extraction unit coefficient update unit, 172...character string extraction unit coefficient update unit, 173...character string erasure unit coefficient update unit, 174...reconstruction unit coefficient update unit

Claims

1. A background extraction unit extracts a background image from an input image including a character string, the background image being obtained by erasing all of the character strings; a character string extraction unit extracts the character string by using the input image and the background image; a character string erasing unit erasing a specified character string from the extracted image of the character string; a reconstruction unit integrates the background image with an image of the character string from which the specified character string has been erased; Image processing methods.

2. The image processing device In order to individually optimize each process of background extraction, text extraction, text erasure, and reconstruction, we conduct pre-learning. In order to simultaneously optimize the processing of all processing units, synchronous learning is performed. using the learned background extraction unit, the character string extraction unit, the character string erasure unit, and the reconstruction unit, to integrate the background image and an image of a character string from which the specified character string has been erased from the input image, thereby performing selective character string erasure. The image processing method according to claim 1 .

3. the background extraction unit, during learning, extracts an estimated background image from a learning input image in which all of the character strings have been erased, and learns parameters of the background extraction unit using a loss function between the estimated background image and a background image that is training data; The image processing method according to claim 1 .

4. the character string extraction unit extracts the character string using a learning input image and a learning background image, and learns parameters of the character string extraction unit using a loss function between an estimated character string image of the extracted character string and a character string image serving as training data; The image processing method according to claim 1 .

5. the character string erasure unit learns parameters of the character string erasure unit by using a loss function between an erased character string image obtained by erasing a character string specified by the learning input label from a learning character string image and the learning erased character string image; The image processing method according to claim 1 .

6. the reconstruction unit synthesizes an image in which a specified character string is erased from an input image including the character string with the background image, and learns parameters of the reconstruction unit using a loss function between the synthesized image and a learning target image; The image processing method according to claim 1 .

7. The background extraction unit extracts a background image from the learning input image, the character string extraction unit extracts a character string image from the character string image by using the background image and the learning input image; the reconstruction unit reconstructs an image obtained by erasing a specified character string from an input image including the character string, by combining the image with the background image; A reconstruction loss calculation unit calculates a reconstruction loss between the synthesized image and a learning target image; a reconstruction unit coefficient update unit updates a reconstruction unit coefficient, which is a parameter of the reconstruction unit, based on the reconstruction loss; a string erasure unit coefficient update unit updates a string erasure unit coefficient, which is a parameter of the string erasure unit, based on a string erasure loss function between an erased string image obtained by erasing a string designated by the learning input label from the learning string image and the training erased string image; a character string extraction unit coefficient update unit updates a character string extraction unit coefficient, which is a parameter of the character string extraction unit, based on a character string extraction loss function between the extracted character string image and a learning character string image; a background extraction unit coefficient update unit updates a background extraction unit coefficient, which is a parameter of the background extraction unit, based on a background extraction loss between the background image and a learning background image; The image processing method according to claim 1 .

8. a background extraction unit that extracts a background image from an input image including a character string, the background image being obtained by erasing all of the character strings; a character string extraction unit that extracts the character string by using the input image and the background image; a character string erasing unit that erases a specified character string from the extracted image of the character string; a reconstruction unit that integrates the background image and an image of a character string from which the specified character string has been erased; An image processing device comprising:

9. Computer, A program for causing the image processing device according to claim 8 to function.

Images