Text mask extraction method, device, storage medium, and program product

Abstract

Embodiments of the present disclosure provide a text mask extraction method and device, a storage medium and a program product. The method comprises: obtaining an original image, the display content of the original image containing text characters; inputting the original image into a preset text mask extraction model, obtaining the features of each pixel in the original image through a UNET network in the text mask extraction model, and determining the text character mask corresponding to the original image according to the features of each pixel in the original image; wherein the preset text mask extraction model is obtained by first training and then jointly training with a pre-trained text recognition model. In the embodiments of the present disclosure, the text character mask is extracted from the original image containing text through the pre-trained text mask extraction model, the segmentation accuracy is improved by means of the UNET network in the text mask extraction model, the smooth text character mask can be extracted, and the quality of the text character mask is effectively improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer and network communication technology, and in particular to a method, apparatus, storage medium, and program product for extracting text masks. Background Technology

[0002] Image masking is used to control the area or process of image processing by occluding all or part of a selected image, graphic, or object. Image masking can also be used for images containing text characters to construct a mask of the text characters from the image, so that image processing is performed only on the text characters or the background.

[0003] In some image mask extraction methods, adaptive threshold segmentation algorithms are used to segment pixels and separate the mask area from the background area. However, the segmentation accuracy is low. When extracting text character masks from images containing text characters, the extracted text character masks have jagged edges and the quality of the text character masks is poor. Summary of the Invention

[0004] This disclosure provides a method, apparatus, storage medium, and program product for extracting text masks, thereby improving the quality of text character masks by extracting smooth text character masks from original images containing text.

[0005] In a first aspect, embodiments of this disclosure provide a method for extracting a text mask, comprising:

[0006] Acquire the original image, the display content of which includes text characters;

[0007] The original image is input into a preset text mask extraction model. The features of each pixel in the original image are obtained through the UNET network in the text mask extraction model. The text character mask corresponding to the original image is determined based on the features of each pixel in the original image.

[0008] The preset text mask extraction model is obtained by first conducting preliminary training and then jointly training it with a pre-trained text recognition model.

[0009] Secondly, embodiments of this disclosure provide a text mask extraction device, comprising:

[0010] An acquisition unit is used to acquire an original image, wherein the display content of the original image includes text characters;

[0011] The processing unit is used to input the original image into a preset text mask extraction model, obtain the features of each pixel in the original image through the UNET network in the text mask extraction model, and determine the text character mask corresponding to the original image based on the features of each pixel in the original image.

[0012] The preset text mask extraction model is obtained by first conducting preliminary training and then jointly training it with a pre-trained text recognition model.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory;

[0014] The memory stores computer-executed instructions;

[0015] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the text mask extraction method as described in the first aspect and various possible designs of the first aspect.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the text mask extraction method described in the first aspect and various possible designs of the first aspect.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including computer execution instructions, which, when executed by a processor, implement the text mask extraction method described in the first aspect and various possible designs of the first aspect.

[0018] The text mask extraction method, device, storage medium, and program product provided in this disclosure involve acquiring an original image whose display content includes text characters; inputting the original image into a preset text mask extraction model; obtaining the features of each pixel in the original image through the UNET network in the text mask extraction model; and determining the text character mask corresponding to the original image based on the features of each pixel in the original image. The preset text mask extraction model is obtained by initial training followed by joint training with a pre-trained text recognition model. In this disclosure, the pre-trained text mask extraction model extracts text character masks from original images containing text. By leveraging the jointly trained text mask extraction model and the UNET network within it, segmentation accuracy is improved, resulting in smooth text character masks and effectively improving the quality of the text character masks. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 An example diagram of a text mask extraction model provided in an embodiment of this disclosure;

[0021] Figure 2 A schematic flowchart of a text mask extraction method provided in an embodiment of this disclosure;

[0022] Figure 3 This is an example diagram of the UNET network in a text mask extraction model provided in an embodiment of this disclosure;

[0023] Figure 4 A schematic diagram of the model training process in a text mask extraction method provided in an embodiment of this disclosure;

[0024] Figure 5 A schematic diagram illustrating the combined model during model training, provided in an embodiment of this disclosure;

[0025] Figure 6 A structural block diagram of a text mask extraction device provided in an embodiment of this disclosure;

[0026] Figure 7 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] Image masking is used to control the area or process of image processing by occluding all or part of a selected image, graphic, or object. Image masking can also be used for images containing text characters to construct a mask of the text characters from the image, so that image processing is performed only on the text characters or the background.

[0029] In some image mask extraction methods, adaptive thresholding algorithms are used for pixel segmentation, such as the OTSU (Otsu's method or maximum inter-class variance) algorithm. This method involves converting the image to grayscale to obtain a corresponding grayscale image, and then using the OTSU algorithm to process the grayscale image to determine the segmentation threshold. Based on the segmentation threshold and the grayscale values ​​of each pixel, the mask region and the background region can be segmented. However, its segmentation accuracy is low. When extracting text character masks from images containing text characters, the extracted text character masks have jagged edges, resulting in poor quality.

[0030] To extract a smooth text character mask from an original image containing text and improve the quality of the text character mask, this disclosure provides a text mask extraction method. The method involves acquiring an original image whose display content includes text characters; inputting the original image into a preset text mask extraction model, as shown in the example below. Figure 1 As shown, the UNET network in the text mask extraction model is used to obtain the features of each pixel in the original image, and the text character mask corresponding to the original image is determined based on the features of each pixel in the original image.

[0031] The preset text mask extraction model is obtained by first conducting preliminary training and then jointly training it with a pre-trained text recognition model. In this embodiment, the pre-trained text mask extraction model extracts text character masks from the original image containing text. By leveraging the UNET network in the text mask extraction model to improve segmentation accuracy, a smooth text character mask can be extracted, effectively improving the quality of the text character mask.

[0032] The text mask extraction method provided in this disclosure will be described in detail below with reference to specific embodiments.

[0033] refer to Figure 2 , Figure 2 This is a schematic flowchart illustrating a text mask extraction method according to an embodiment of the present disclosure. The method of this embodiment can be applied to terminal devices or servers, and the text mask extraction method includes:

[0034] S201. Obtain the original image, wherein the display content of the original image includes text characters.

[0035] In this embodiment, when it is necessary to obtain a text character mask from an original image containing text, the original image to be processed is first obtained and used as an input image to be input into a preset text mask extraction model. The specific process of obtaining the original image is not limited in this embodiment.

[0036] S202. Input the original image into a preset text mask extraction model, obtain the features of each pixel in the original image through the UNET network in the text mask extraction model, and determine the text character mask corresponding to the original image based on the features of each pixel in the original image.

[0037] The preset text mask extraction model is obtained by first conducting preliminary training and then jointly training it with a pre-trained text recognition model.

[0038] In this embodiment, a text mask extraction model is pre-trained. This model includes a UNET network, a sigmoid activation function, and an output layer, such as... Figure 1 As shown, the UNET network is one of the most commonly used and simplest segmentation models, including a contracting path and an expanding path. It combines downsampling and upsampling, and low-level and high-level information, significantly improving segmentation accuracy. In this embodiment, the UNET network is used to obtain the features of each pixel in the original image. The UNET network in this embodiment is as follows: Figure 3 As shown, the features of each pixel are then mapped to a range of 0 to 1 using an activation function, generating a probability corresponding to each pixel. The output layer determines the text character mask corresponding to the original image based on the probability corresponding to each pixel and a preset probability threshold. Specifically, the output layer compares the probability corresponding to each pixel with the preset probability threshold and converts the probability corresponding to each pixel to 0 or 1, where 0 represents the background and 1 represents the text character. Thus, each pixel of the original image is represented by 0 and 1, resulting in the text character mask corresponding to the original image.

[0039] Optionally, the text mask extraction model is first trained separately and then jointly trained with a pre-trained text recognition model. During the joint training process, the model parameters of the text recognition model are kept fixed, and only the model parameters of the text mask extraction model are optimized. Through joint training, the accuracy of the text recognition model is improved, and the resulting text character mask is smoother and of higher quality.

[0040] Furthermore, based on the probability corresponding to each pixel and a preset probability threshold, the text character mask corresponding to the original image is determined. The specific process includes:

[0041] For any pixel, determine whether the probability corresponding to the pixel is greater than the preset probability threshold; if the probability corresponding to the pixel is greater than the preset probability threshold, then set the value of the pixel to 1; if the probability corresponding to the pixel is not greater than the preset probability threshold, then set the value of the pixel to 0; based on the value of each pixel, obtain the text character mask corresponding to the original image.

[0042] Optionally, after determining the text character mask corresponding to the original image, the style of the text characters and / or background can be modified according to the text character mask corresponding to the original image (e.g., modifying the color of the text characters and / or modifying the background color, etc.) to generate an image containing the same text characters as the original image. The generated image can be used as a training image for some other text processing models, and the text characters contained in the generated image are known (the same as the original image, only the text characters in the original image need to be labeled), without the need for re-labeling, thus reducing the labeling cost.

[0043] The text mask extraction method in this embodiment involves acquiring an original image whose content includes text characters; inputting the original image into a preset text mask extraction model; using the UNET network within the text mask extraction model to obtain the features of each pixel in the original image; and determining the corresponding text character mask based on the features of each pixel in the original image. The preset text mask extraction model is initially trained and then jointly trained with a pre-trained text recognition model. In this embodiment, the pre-trained text mask extraction model extracts text character masks from original images containing text. By leveraging the jointly trained text mask extraction model and the UNET network within it, segmentation accuracy is improved, resulting in smooth text character masks and effectively enhancing the quality of the text character masks.

[0044] Based on the above embodiments, this embodiment also provides a method for training a text mask extraction model. The executing entity can be the same as in the above embodiments, or it can be a different executing entity. The training process is as follows: Figure 4 As shown, the specific steps are as follows:

[0045] S301. Obtain multiple training images containing text, the text characters contained in each training image, and the text character mask corresponding to each training image.

[0046] In this embodiment, multiple training images containing text are first acquired. Optionally, multiple arbitrary images containing text can be acquired. The text characters in the images are located and recognized by an optical character recognition (OCR) tool (or other text recognition algorithms or tools). The images are then segmented according to the located text characters to obtain training images and the text characters contained in the training images.

[0047] A preset text character mask extraction algorithm is used to process any training image to obtain the text character mask corresponding to the training image.

[0048] Optionally, any training image can be converted to grayscale to obtain a corresponding grayscale image; a preset adaptive threshold segmentation algorithm is used to process the grayscale image to determine the segmentation threshold, such as the OTSU (Otsu method or maximum inter-class variance) algorithm, maximum entropy algorithm, etc.; further, the number of first-class pixels with grayscale values ​​greater than the segmentation threshold and the number of second-class pixels with grayscale values ​​not greater than the segmentation threshold are counted; considering that for images containing text, the number of background pixels is greater than the number of text character pixels, according to the statistical results, the pixel class with the greater number of first-class pixels and second-class pixels is determined as background pixels, and the other class is determined as text character pixels; finally, the value of the text character pixels is set to 1, and the value of the background pixels is set to 0, and the text character mask corresponding to the training image is obtained according to the value of each pixel.

[0049] Of course, other methods can also be used in this embodiment to obtain training images, text characters contained in each training image, and text character masks corresponding to each training image; no limitation is made here.

[0050] S302. Train the initial text mask extraction model based on the training image and the corresponding text character mask.

[0051] In this embodiment, any training image and its corresponding text character mask are used as a set of training data. The training image is input into the initial text mask extraction model to obtain the predicted text character mask output by the text mask extraction model. The loss is calculated by comparing the predicted text character mask with the text character mask corresponding to the training image. The text mask extraction model is then optimized based on the loss. Through the above process, preliminary semi-supervised learning of the text mask extraction model can be achieved.

[0052] S303. The trained text mask extraction model and the pre-trained text recognition model are combined to obtain a combined model. In the combined model, the output of the text mask extraction model is converted into image features and then input into the text recognition model.

[0053] In this embodiment, since the text character mask corresponding to the training image is obtained using a preset text character mask extraction algorithm, the trained text mask extraction model has certain errors. To correct these errors, this embodiment combines the trained text mask extraction model with a pre-trained text recognition model to obtain a combined model. In this combined model, the output of the text mask extraction model is converted into image features and then input into the text recognition model. Optionally, the pre-trained text recognition model can be any model capable of recognizing text from an image, such as a CRNN (Convolutional Recurrent Neural Network) text recognition model. In this embodiment, by jointly training the text mask extraction model and the text recognition model, the model parameters of the text recognition model are fixed, and the text mask extraction model is optimized, thus improving the accuracy of the text mask extraction model. Therefore, after the output of the text mask extraction model is converted into image features and input into the text recognition model, the accuracy of the text recognition result of the text recognition model can be effectively improved.

[0054] Furthermore, since the text mask extraction model requires optimization of the UNET network, this embodiment... Figure 5 The combined model is constructed as shown. Specifically, the combined model process is as follows: for any training image, the UNET network in the text mask extraction model is used to obtain the features of each pixel in the training image, and the features of each pixel are used to generate the probability corresponding to each pixel through the activation function; the probability corresponding to each pixel is multiplied by the value of the corresponding pixel in the training image, and the result of the multiplication is input into the text recognition model for text recognition.

[0055] S304. Based on the training image, the corresponding text character mask, and the corresponding text character, train the combined model. During the training process, keep the model parameters of the text recognition model fixed and optimize the model parameters of the text mask extraction model.

[0056] In this embodiment, any training image, its corresponding text character mask, and the corresponding text character are used as a set of training data to train the combined model. The training image is used as input. The UNET network in the text mask extraction model obtains the features of each pixel in the training image, and the features of each pixel are used to generate a probability corresponding to each pixel through an activation function. The probability corresponding to each pixel is multiplied by the value of the corresponding pixel in the training image, and the result is input into the text recognition model for text recognition. Finally, the text mask extraction model outputs a predicted text character. Furthermore, the text mask extraction model also outputs a predicted text character mask. Further, a first loss value is obtained between the predicted text character mask output by the text mask extraction model and the text character mask corresponding to the training image; optionally, the first loss value can be Dice Loss. A second loss value is obtained between the predicted text character output by the text recognition model and the text character corresponding to the training image; optionally, the second loss value can be CTC (Connectionist) loss. Temporal Classification (connected temporal classification) Loss: Since the text recognition model has been pre-trained and its accuracy is guaranteed, it can be assumed that the first and second loss values ​​are caused by the text mask extraction model. Therefore, the model parameters of the text recognition model are kept fixed, while the model parameters of the text mask extraction model are optimized.

[0057] In particular, since the text mask extraction model requires optimization of the UNET network, the model parameters of the UNET network can be optimized based on the first and second loss values. By minimizing the total loss value, the model parameters of the UNET network are adjusted, refining the UNET network segmentation results and improving segmentation accuracy.

[0058] During the testing phase, the test images only need to be processed by the finally trained text mask extraction model to obtain high-quality text character masks, achieving pixel-level automatic extraction of text characters from images.

[0059] Corresponding to the text mask extraction method in the above embodiment, Figure 6 This is a structural block diagram of a text mask extraction device provided for embodiments of this disclosure. For ease of explanation, only the parts relevant to embodiments of this disclosure are shown. (Refer to...) Figure 6 The text mask extraction device 600 includes: an acquisition unit 601 and a processing unit 602.

[0060] The acquisition unit 601 is used to acquire the original image, wherein the display content of the original image includes text characters;

[0061] The processing unit 602 is used to input the original image into a preset text mask extraction model, obtain the features of each pixel in the original image through the UNET network in the text mask extraction model, and determine the text character mask corresponding to the original image based on the features of each pixel in the original image; wherein, the preset text mask extraction model is obtained by first undergoing preliminary training and then jointly training with a pre-trained text recognition model.

[0062] In one or more embodiments of this disclosure, when the processing unit 602 determines the text character mask corresponding to the original image based on the features of each pixel in the original image, it is used to:

[0063] The features of each pixel are used to generate the probability corresponding to each pixel through an activation function;

[0064] The text character mask corresponding to the original image is determined based on the probability of each pixel and a preset probability threshold.

[0065] In one or more embodiments of this disclosure, when the processing unit 602 determines the text character mask corresponding to the original image based on the probability corresponding to each pixel and a preset probability threshold, it is used to:

[0066] For any pixel, determine whether the probability corresponding to the pixel is greater than the preset probability threshold;

[0067] If the probability corresponding to the pixel is greater than the preset probability threshold, then the value of the pixel is set to 1; if the probability corresponding to the pixel is not greater than the preset probability threshold, then the value of the pixel is set to 0.

[0068] Based on the value of each pixel, the text character mask corresponding to the original image is obtained.

[0069] In one or more embodiments of this disclosure, after determining the text character mask corresponding to the original image, the processing unit 602 is further configured to:

[0070] Based on the text character mask corresponding to the original image, modify the style of the text characters and / or background to generate an image containing the same text characters as the original image.

[0071] In one or more embodiments of this disclosure, the method further includes a training unit 603, configured to:

[0072] Obtain multiple training images containing text, the text characters contained in each training image, and the text character mask corresponding to each training image;

[0073] The initial text mask extraction model is trained based on the training images and the corresponding text character masks.

[0074] The trained text mask extraction model and the pre-trained text recognition model are combined to obtain a combined model. In the combined model, the output of the text mask extraction model is converted into image features and then input into the text recognition model.

[0075] The combined model is trained based on the training image, the corresponding text character mask, and the corresponding text character. During the training process, the model parameters of the text recognition model are kept fixed, while the model parameters of the text mask extraction model are optimized.

[0076] In one or more embodiments of this disclosure, when the training unit 603 converts the output of the text mask extraction model into image features and inputs them into the text recognition model in the combined model, it is used for:

[0077] For any training image, the UNET network in the text mask extraction model is used to obtain the features of each pixel in the training image, and the features of each pixel are used to generate the probability corresponding to each pixel through an activation function;

[0078] The probability corresponding to each pixel is multiplied by the value of the corresponding pixel in the training image, and the result of the multiplication is input into the text recognition model for text recognition.

[0079] In one or more embodiments of this disclosure, when the training unit 603 trains the combined model based on the training image, the corresponding text character mask, and the corresponding text character, it is used to:

[0080] Input any training image into the combined model for processing;

[0081] Obtain the first loss value between the predicted text character mask output by the text mask extraction model and the text character mask corresponding to the training image;

[0082] Obtain a second loss value between the predicted text characters output by the text recognition model and the text characters corresponding to the training image;

[0083] The model parameters of the text mask extraction model are optimized based on the first loss value and the second loss value.

[0084] In one or more embodiments of this disclosure, the training unit 603, when optimizing the model parameters of the text mask extraction model based on the first loss value and the second loss value, is configured to:

[0085] The model parameters of the UNET network in the text mask extraction model are optimized based on the first loss value and the second loss value.

[0086] In one or more embodiments of this disclosure, the first loss value is Dice Loss, and / or the second loss value is CTC Loss.

[0087] In one or more embodiments of this disclosure, the training unit 603, when acquiring multiple training images containing text, text characters contained in each training image, and text character masks corresponding to each training image, is used to:

[0088] Multiple arbitrary images containing text are acquired, and the text characters in the images are located and recognized using an optical character recognition (OCR) tool. The images are then segmented based on the located text characters to obtain training images and the text characters contained in the training images.

[0089] A preset text character mask extraction algorithm is used to process any training image to obtain the text character mask corresponding to the training image.

[0090] In one or more embodiments of this disclosure, when the training unit 603 processes any training image using a preset text character mask extraction algorithm to obtain the text character mask corresponding to the training image, it is used to:

[0091] Convert any training image to grayscale to obtain the corresponding grayscale image;

[0092] The grayscale image is processed using a preset adaptive threshold segmentation algorithm to determine the segmentation threshold;

[0093] Count the number of first-class pixels in the grayscale image whose grayscale value is greater than the segmentation threshold, and the number of second-class pixels whose grayscale value is not greater than the segmentation threshold;

[0094] The pixel class with a larger number of pixels between the first type and the second type is identified as background pixels, and the other type is identified as text character pixels;

[0095] The value of the text character pixel is set to 1, and the value of the background pixel is set to 0. The text character mask corresponding to the training image is obtained based on the value of each pixel.

[0096] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0097] refer to Figure 7The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing embodiments of the present disclosure. The electronic device 700 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0098] like Figure 7 As shown, the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0099] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0100] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of embodiments of this disclosure.

[0101] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0102] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0103] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0104] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0105] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0106] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0107] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0108] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0109] In a first aspect, according to one or more embodiments of this disclosure, a method for extracting a text mask is provided, comprising:

[0110] Acquire the original image, the display content of which includes text characters;

[0111] The original image is input into a preset text mask extraction model. The features of each pixel in the original image are obtained through the UNET network in the text mask extraction model. The text character mask corresponding to the original image is determined based on the features of each pixel in the original image.

[0112] The preset text mask extraction model is obtained by first conducting preliminary training and then jointly training it with a pre-trained text recognition model.

[0113] According to one or more embodiments of this disclosure, determining the text character mask corresponding to the original image based on the features of each pixel in the original image includes:

[0114] The features of each pixel are used to generate the probability corresponding to each pixel through an activation function;

[0115] The text character mask corresponding to the original image is determined based on the probability of each pixel and a preset probability threshold.

[0116] According to one or more embodiments of this disclosure, determining the text character mask corresponding to the original image based on the probability corresponding to each pixel and a preset probability threshold includes:

[0117] For any pixel, determine whether the probability corresponding to the pixel is greater than the preset probability threshold;

[0118] If the probability corresponding to the pixel is greater than the preset probability threshold, then the value of the pixel is set to 1; if the probability corresponding to the pixel is not greater than the preset probability threshold, then the value of the pixel is set to 0.

[0119] Based on the value of each pixel, the text character mask corresponding to the original image is obtained.

[0120] According to one or more embodiments of this disclosure, after determining the text character mask corresponding to the original image, the method further includes:

[0121] Based on the text character mask corresponding to the original image, modify the style of the text characters and / or background to generate an image containing the same text characters as the original image.

[0122] According to one or more embodiments of this disclosure, the method further includes:

[0123] Obtain multiple training images containing text, the text characters contained in each training image, and the text character mask corresponding to each training image;

[0124] The initial text mask extraction model is trained based on the training images and the corresponding text character masks.

[0125] The trained text mask extraction model and the pre-trained text recognition model are combined to obtain a combined model. In the combined model, the output of the text mask extraction model is converted into image features and then input into the text recognition model.

[0126] The combined model is trained based on the training image, the corresponding text character mask, and the corresponding text character. During the training process, the model parameters of the text recognition model are kept fixed, while the model parameters of the text mask extraction model are optimized.

[0127] According to one or more embodiments of this disclosure, the step of converting the output of the text mask extraction model into image features and then inputting them into the text recognition model in the combined model includes:

[0128] For any training image, the UNET network in the text mask extraction model is used to obtain the features of each pixel in the training image, and the features of each pixel are used to generate the probability corresponding to each pixel through an activation function;

[0129] The probability corresponding to each pixel is multiplied by the value of the corresponding pixel in the training image, and the result of the multiplication is input into the text recognition model for text recognition.

[0130] According to one or more embodiments of this disclosure, training the combined model based on the training image, the corresponding text character mask, and the corresponding text character includes:

[0131] Input any training image into the combined model for processing;

[0132] Obtain the first loss value between the predicted text character mask output by the text mask extraction model and the text character mask corresponding to the training image;

[0133] Obtain a second loss value between the predicted text characters output by the text recognition model and the text characters corresponding to the training image;

[0134] The model parameters of the text mask extraction model are optimized based on the first loss value and the second loss value.

[0135] According to one or more embodiments of this disclosure, optimizing the model parameters of the text mask extraction model based on the first loss value and the second loss value includes:

[0136] The model parameters of the UNET network in the text mask extraction model are optimized based on the first loss value and the second loss value.

[0137] According to one or more embodiments of this disclosure, the first loss value is Dice Loss, and / or the second loss value is CTC Loss.

[0138] According to one or more embodiments of this disclosure, obtaining a plurality of training images containing text, text characters contained in each training image, and a text character mask corresponding to each training image includes:

[0139] Multiple arbitrary images containing text are acquired, and the text characters in the images are located and recognized using an optical character recognition (OCR) tool. The images are then segmented based on the located text characters to obtain training images and the text characters contained in the training images.

[0140] A preset text character mask extraction algorithm is used to process any training image to obtain the text character mask corresponding to the training image.

[0141] According to one or more embodiments of this disclosure, the step of processing any training image using a preset text character mask extraction algorithm to obtain a text character mask corresponding to the training image includes:

[0142] Convert any training image to grayscale to obtain the corresponding grayscale image;

[0143] The grayscale image is processed using a preset adaptive threshold segmentation algorithm to determine the segmentation threshold;

[0144] Count the number of first-class pixels in the grayscale image whose grayscale value is greater than the segmentation threshold, and the number of second-class pixels whose grayscale value is not greater than the segmentation threshold;

[0145] The pixel class with a larger number of pixels between the first type and the second type is identified as background pixels, and the other type is identified as text character pixels;

[0146] The value of the text character pixel is set to 1, and the value of the background pixel is set to 0. The text character mask corresponding to the training image is obtained based on the value of each pixel.

[0147] Secondly, according to one or more embodiments of this disclosure, a text mask extraction device is provided, comprising:

[0148] An acquisition unit is used to acquire an original image, wherein the display content of the original image includes text characters;

[0149] The processing unit is used to input the original image into a preset text mask extraction model, obtain the features of each pixel in the original image through the UNET network in the text mask extraction model, and determine the text character mask corresponding to the original image based on the features of each pixel in the original image.

[0150] The preset text mask extraction model is obtained by first conducting preliminary training and then jointly training it with a pre-trained text recognition model.

[0151] According to one or more embodiments of this disclosure, when the processing unit determines the text character mask corresponding to the original image based on the features of each pixel in the original image, it is configured to:

[0152] The features of each pixel are used to generate the probability corresponding to each pixel through an activation function;

[0153] The text character mask corresponding to the original image is determined based on the probability of each pixel and a preset probability threshold.

[0154] According to one or more embodiments of this disclosure, when the processing unit determines the text character mask corresponding to the original image based on the probability corresponding to each pixel and a preset probability threshold, it is configured to:

[0155] For any pixel, determine whether the probability corresponding to the pixel is greater than the preset probability threshold;

[0156] If the probability corresponding to the pixel is greater than the preset probability threshold, then the value of the pixel is set to 1; if the probability corresponding to the pixel is not greater than the preset probability threshold, then the value of the pixel is set to 0.

[0157] Based on the value of each pixel, the text character mask corresponding to the original image is obtained.

[0158] According to one or more embodiments of this disclosure, after determining the text character mask corresponding to the original image, the processing unit is further configured to:

[0159] Based on the text character mask corresponding to the original image, modify the style of the text characters and / or background to generate an image containing the same text characters as the original image.

[0160] According to one or more embodiments of this disclosure, the method further includes a training unit for:

[0161] Obtain multiple training images containing text, the text characters contained in each training image, and the text character mask corresponding to each training image;

[0162] The initial text mask extraction model is trained based on the training images and the corresponding text character masks.

[0163] The trained text mask extraction model and the pre-trained text recognition model are combined to obtain a combined model. In the combined model, the output of the text mask extraction model is converted into image features and then input into the text recognition model.

[0164] The combined model is trained based on the training image, the corresponding text character mask, and the corresponding text character. During the training process, the model parameters of the text recognition model are kept fixed, while the model parameters of the text mask extraction model are optimized.

[0165] According to one or more embodiments of this disclosure, when the training unit converts the output of the text mask extraction model into image features and inputs them into the text recognition model in the combined model, it is used for:

[0166] For any training image, the UNET network in the text mask extraction model is used to obtain the features of each pixel in the training image, and the features of each pixel are used to generate the probability corresponding to each pixel through an activation function;

[0167] The probability corresponding to each pixel is multiplied by the value of the corresponding pixel in the training image, and the result of the multiplication is input into the text recognition model for text recognition.

[0168] According to one or more embodiments of this disclosure, when the training unit trains the combined model based on the training image, the corresponding text character mask, and the corresponding text character, it is used to:

[0169] Input any training image into the combined model for processing;

[0170] Obtain the first loss value between the predicted text character mask output by the text mask extraction model and the text character mask corresponding to the training image;

[0171] Obtain a second loss value between the predicted text characters output by the text recognition model and the text characters corresponding to the training image;

[0172] The model parameters of the text mask extraction model are optimized based on the first loss value and the second loss value.

[0173] According to one or more embodiments of this disclosure, when the training unit optimizes the model parameters of the text mask extraction model based on the first loss value and the second loss value, it is used to:

[0174] The model parameters of the UNET network in the text mask extraction model are optimized based on the first loss value and the second loss value.

[0175] According to one or more embodiments of this disclosure, the first loss value is Dice Loss, and / or the second loss value is CTC Loss.

[0176] According to one or more embodiments of this disclosure, when the training unit acquires multiple training images containing text, text characters contained in each training image, and text character masks corresponding to each training image, it is used to:

[0177] Multiple arbitrary images containing text are acquired, and the text characters in the images are located and recognized using an optical character recognition (OCR) tool. The images are then segmented based on the located text characters to obtain training images and the text characters contained in the training images.

[0178] A preset text character mask extraction algorithm is used to process any training image to obtain the text character mask corresponding to the training image.

[0179] According to one or more embodiments of this disclosure, when the training unit processes any training image using a preset text character mask extraction algorithm to obtain a text character mask corresponding to the training image, it is used to:

[0180] Convert any training image to grayscale to obtain the corresponding grayscale image;

[0181] The grayscale image is processed using a preset adaptive threshold segmentation algorithm to determine the segmentation threshold;

[0182] Count the number of first-class pixels in the grayscale image whose grayscale value is greater than the segmentation threshold, and the number of second-class pixels whose grayscale value is not greater than the segmentation threshold;

[0183] The pixel class with a larger number of pixels between the first type and the second type is identified as background pixels, and the other type is identified as text character pixels;

[0184] The value of the text character pixel is set to 1, and the value of the background pixel is set to 0. The text character mask corresponding to the training image is obtained based on the value of each pixel.

[0185] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0186] The memory stores computer-executed instructions;

[0187] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the text mask extraction method as described in the first aspect and various possible designs of the first aspect.

[0188] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the text mask extraction method described in the first aspect and various possible designs of the first aspect is implemented.

[0189] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including computer execution instructions that, when executed by a processor, implement the text mask extraction method described in the first aspect and various possible designs of the first aspect.

[0190] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0191] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0192] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for extracting text masks, characterized in that, include: Acquire the original image, the display content of which includes text characters; The original image is input into a preset text mask extraction model. The features of each pixel in the original image are obtained through the UNET network in the text mask extraction model. The text character mask corresponding to the original image is determined based on the features of each pixel in the original image. The preset text mask extraction model is obtained by first undergoing preliminary training and then jointly training it with a pre-trained text recognition model. The process of first undergoing initial training and then jointly training with a pre-trained text recognition model includes: Obtain multiple training images containing text, the text characters contained in each training image, and the text character mask corresponding to each training image; The initial text mask extraction model is trained based on the training images and the corresponding text character masks. The trained text mask extraction model and the pre-trained text recognition model are combined to obtain a combined model. In the combined model, the output of the text mask extraction model is converted into image features and then input into the text recognition model. The combined model is trained based on the training image, the corresponding text character mask, and the corresponding text character. During the training process, the model parameters of the text recognition model are kept fixed, while the model parameters of the text mask extraction model are optimized.

2. The method according to claim 1, characterized in that, The step of determining the text character mask corresponding to the original image based on the features of each pixel in the original image includes: The features of each pixel are used to generate the probability corresponding to each pixel through an activation function; The text character mask corresponding to the original image is determined based on the probability of each pixel and a preset probability threshold.

3. The method according to claim 2, characterized in that, The step of determining the text character mask corresponding to the original image based on the probability corresponding to each pixel and a preset probability threshold includes: For any pixel, determine whether the probability corresponding to the pixel is greater than the preset probability threshold; If the probability corresponding to the pixel is greater than the preset probability threshold, then the value of the pixel is set to 1; if the probability corresponding to the pixel is not greater than the preset probability threshold, then the value of the pixel is set to 0. Based on the value of each pixel, the text character mask corresponding to the original image is obtained.

4. The method according to any one of claims 1-3, characterized in that, After determining the text character mask corresponding to the original image, the method further includes: Based on the text character mask corresponding to the original image, modify the style of the text characters and / or background to generate an image containing the same text characters as the original image.

5. The method according to claim 1, characterized in that, The step of converting the output of the text mask extraction model into image features and then inputting them into the text recognition model in the combined model includes: For any training image, the UNET network in the text mask extraction model is used to obtain the features of each pixel in the training image, and the features of each pixel are used to generate the probability corresponding to each pixel through an activation function; The probability corresponding to each pixel is multiplied by the value of the corresponding pixel in the training image, and the result of the multiplication is input into the text recognition model for text recognition.

6. The method according to claim 1 or 5, characterized in that, The step of training the combined model based on the training image, the corresponding text character mask, and the corresponding text character includes: Input any training image into the combined model for processing; Obtain the first loss value between the predicted text character mask output by the text mask extraction model and the text character mask corresponding to the training image; Obtain a second loss value between the predicted text characters output by the text recognition model and the text characters corresponding to the training image; The model parameters of the text mask extraction model are optimized based on the first loss value and the second loss value.

7. The method according to claim 6, characterized in that, The optimization of the model parameters of the text mask extraction model based on the first loss value and the second loss value includes: The model parameters of the UNET network in the text mask extraction model are optimized based on the first loss value and the second loss value.

8. The method according to claim 1, characterized in that, The step of obtaining multiple training images containing text, the text characters contained in each training image, and the text character mask corresponding to each training image includes: Multiple arbitrary images containing text are acquired, and the text characters in the images are located and recognized using an optical character recognition (OCR) tool. The images are then segmented based on the located text characters to obtain training images and the text characters contained in the training images. A preset text character mask extraction algorithm is used to process any training image to obtain the text character mask corresponding to the training image.

9. The method according to claim 8, characterized in that, The step of processing any training image using a preset text character mask extraction algorithm to obtain the text character mask corresponding to the training image includes: Convert any training image to grayscale to obtain the corresponding grayscale image; The grayscale image is processed using a preset adaptive threshold segmentation algorithm to determine the segmentation threshold; Count the number of first-class pixels in the grayscale image whose grayscale value is greater than the segmentation threshold, and the number of second-class pixels whose grayscale value is not greater than the segmentation threshold; The pixel class with a larger number of pixels among the first type and the second type is identified as background pixels, and the other type is identified as text character pixels; The value of the text character pixel is set to 1, and the value of the background pixel is set to 0. The text character mask corresponding to the training image is obtained based on the value of each pixel.

10. A text mask extraction device, characterized in that, include: An acquisition unit is used to acquire an original image, wherein the display content of the original image includes text characters; The processing unit is used to input the original image into a preset text mask extraction model, obtain the features of each pixel in the original image through the UNET network in the text mask extraction model, and determine the text character mask corresponding to the original image based on the features of each pixel in the original image. The preset text mask extraction model is obtained by first undergoing preliminary training and then jointly training it with a pre-trained text recognition model. The device also includes a training unit for: Obtain multiple training images containing text, the text characters contained in each training image, and the text character mask corresponding to each training image; The initial text mask extraction model is trained based on the training images and the corresponding text character masks. The trained text mask extraction model and the pre-trained text recognition model are combined to obtain a combined model. In the combined model, the output of the text mask extraction model is converted into image features and then input into the text recognition model. The combined model is trained based on the training image, the corresponding text character mask, and the corresponding text character. During the training process, the model parameters of the text recognition model are kept fixed, while the model parameters of the text mask extraction model are optimized.

11. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-9.

13. A computer program product, characterized in that, Includes computer execution instructions, which, when executed by a processor, implement the method as described in any one of claims 1-9.

Images