Text-refining network

Abstract

Methods for text segmentation, comprehensive: Receiving (600) an image that includes a foreground text section and a background section; Classifying (605) each pixel of the image as foreground text or background using a neural network (130) that refines a segmentation prediction (520) using a key vector representing features of the foreground text segment, the key vector being based on the segmentation prediction (520); and Identifying (610) the foreground text section based on the classification, the procedure further comprising: Encoding (800) the image (300, 400, 500) to generate a feature map (510); Decoding (805) the feature map (510) to generate the segmentation prediction (520); and Identifying (810) the key vector based on the segmentation prediction (520), and including the identification of the key vector: Calculating (900) a cosine similarity of the segmentation prediction (520); Applying (905) a second bias based on cosine similarity; Applying (910) a second softmax to an output of the second bias; Combining (915) the second softmax with the feature card; and Applying (920) a pooling layer to generate the key vector.

Description

[0001] The following generally concerns computer-based vision and, in particular, the segmentation of text patterns using machine learning.

[0002] Computer-aided vision encompasses techniques for acquiring, processing, analyzing, and understanding digital images. Digital images can, for example, be fed into artificial systems capable of extracting information. Computer-aided vision includes tasks such as image processing, image segmentation, and object detection.

[0003] Segmentation is the process of dividing a digital image into multiple segments or image objects. In some cases, a segment or object is represented by a set of pixels and a corresponding class label (that is, a semantic description of the class to which the pixels belong). As such, the output of an image segmentation network can be in the form of an image mask corresponding to one or more classes. Image segmentation facilitates the simplification and modification of the digital image's representation into a processed image that users can further analyze and modify. Image segmentation has applications in areas such as image search and retrieval, image synthesis, image editing, and human-computer interaction.

[0004] A variety of tools can be used to perform image segmentation. However, conventional segmentation tools lack knowledge of the specific visual properties of text. Therefore, these tools are prone to errors when identifying text. For example, curves, flourishes, and embellishments surrounding the text may be incorrectly segmented as text, or only one section of the text may be correctly identified, while other sections with similar color and texture may be missed. Therefore, there is a need in the prior art for an improved image segmentation tool that can accurately classify and decompose foreground text information within an image.

[0005] The publications TANG, Youbao, WU, Xianggian, “Scene text detection and segmentation based on cascaded convolution neural networks”, IEEE transactions on image processing, Vol. 26, 2017, No. 3, pp. 1509–1520, and HE, Tong, et al., “Text-attentional convolutional neural network for scene text detection”, Preprint. [v2] Thu, 24 Mar 2016, 23:25:52 UTC. 2016-03-24. https: / / doi.org / 10.48550 / arXiv.1510.03283, each describe exemplary neural networks for recognizing text in scenes.

[0006] The Wikipedia entry "Residual neural network", Wikipedia https: / / en.wikipedia.org / w / index.php?Qtitle=Residual_neural_network&oldid=978101817, describes a residual neural network.

[0007] US 2007 / 0 286 499 A1 describes a method for classifying image data overlaid with text.

[0008] CN 108 536 678 A describes a method for extracting text key information using a neural network.

[0009] The present disclosure describes systems and methods for performing text segmentation with the features specified in the dependent claims. Embodiments of the disclosure perform an initial segmentation and then refine the segmentation by identifying features common to the text-identified sections of the image. In some examples, a key vector representing the common features is combined with image features using an attention component. According to certain embodiments, a machine learning model is trained using multiple training tasks based on the initial segmentation and the refined segmentation.

[0010] A method, a device, and a non-temporary, computer-readable medium for text segmentation are described. Embodiments of the method, the device, and the non-temporary, computer-readable medium are configured to receive an image containing a foreground text segment and a background segment, classify each pixel of the image as foreground text or background using a neural network that refines a segmentation prediction using a key vector representing features of the foreground text segment (the key vector being based on the segmentation prediction), and identify the foreground text segment based on the classification.

[0011] Described are a method, a setup, and a non-temporary computer-readable medium for training a neural network for text segmentation. Embodiments of the method, setup, and non-temporary computer-readable medium are configured to identify a training set comprising training images and ground-truth segmentation information for each of the training images, wherein the ground-truth segmentation information identifies a foreground text segment and a background segment; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each of the training images; and compute an initial loss function based on a comparison of the segmentation prediction and the ground-truth segmentation information.Identifying a key vector to represent features of the foreground text segment based on the segmentation prediction, classifying each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each of the training images, calculating a second loss function based on a comparison of the refined segmentation prediction and the ground-truth segmentation information, and updating parameters of the neural network based on the first loss function and the second loss function.

[0012] The text segmentation method and the method for training a neural network for text segmentation each further include: Encoding the image to generate a feature map; Decoding the feature map to generate the segmentation prediction; and identifying the key vector based on the segmentation prediction, wherein identifying the key vector includes: Calculating a cosine similarity of the segmentation prediction; Applying a second bias based on cosine similarity; Applying a second softmax to an output of the second bias; Combining the second softmax with the feature card; and Applying a pooling layer to generate the key vector.

[0013] A device and method for text segmentation are described. Embodiments of the device and method include an encoder configured to encode an image to generate a feature map, wherein the image contains a foreground text section and a background section; a first decoder configured to decode the feature map to generate a segmentation prediction; a key generator configured to identify a key vector to represent features of the foreground text section based on the segmentation prediction; an attention component configured to generate a combined feature map based on a key vector; and a second decoder configured to generate a refined segmentation prediction based on the combined feature map.The key generator includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer.

[0014] The drawings are briefly described below. Fig. Figure 1 shows an example of a system for text segmentation according to aspects of the present revelation. Fig. Figure 2 shows an example of a process for text segmentation according to aspects of the present revelation. Fig. 3 and Fig. Figure 4 shows examples of text segmentation according to aspects of the present revelation. Fig. Figure 5 shows an example of a text segmentation device according to aspects of the present disclosure. Fig. Figure 6 shows an example of a process for text segmentation according to aspects of the present revelation. Fig. Figure 7 shows an example of a process for generating a segmentation prediction according to aspects of the present disclosure. Fig. 8 and Fig. Figure 9 shows examples of a process for generating a key vector according to aspects of the present disclosure. Fig. Figure 10 shows an example of a process for generating a refined segmentation prediction according to aspects of the present disclosure. Fig. Figure 11 shows an example of a process for training a neural network to segment text according to aspects of the present revelation. Fig. Figure 12 shows an example of a process for expanding a training set according to aspects of the present disclosure.

[0015] This disclosure describes systems and methods for performing text segmentation. Embodiments of the disclosure perform an initial segmentation and then refine the segmentation by identifying features (e.g., textures) that are common to the preliminary text-identified sections of the image. In some examples, a key vector representing the text features is combined with the encoded image (i.e., features of the overall image) using an attention component. According to certain embodiments, a machine learning model is trained using multiple training tasks based on the initial segmentation and the refined segmentation.

[0016] Image segmentation is a crucial component of computer vision. Text segmentation is a subset of image segmentation that focuses on text recognition and has several important applications. For example, optical character recognition (OCR) tools locate text segments before recognizing the text content.

[0017] Some image segmentation tools generate bounding boxes that identify an area containing text. However, for downstream text-related applications such as style transfer, text removal, and image editing, knowing the approximate location of the text (i.e., using the bounding boxes) may not be sufficient. Such applications depend on more complex subsequent image processing steps. Therefore, there is a need for text segmentation tools that segment an image at the pixel level.

[0018] Furthermore, conventional segmentation models are not specifically designed for text segmentation and therefore lack knowledge of text's specific visual properties. This means that text has specific visual characteristics that differ from those of other objects (such as faces, body parts, buildings, or natural objects). For example, text may exhibit weaker spatial constraints compared to regular objects. Additionally, the semantic information associated with text foreground textures can vary from image to image. Therefore, the similarity of text features between different images may be lower than for other objects, while the similarity of text features within an image may be higher than for other objects.

[0019] As a result, conventional image segmentation tools are prone to errors when classifying or segmenting text. For example, curves, flourishes, and embellishments around the text may be mistakenly segmented as text, or only one section of the text may be correctly identified, while other sections with similar color and texture may be missing.

[0020] Embodiments of the present disclosure include an improved text segmentation tool that addresses the technical problems of conventional networks. For example, the text segmentation tool of the present disclosure employs the unconventional steps of identifying key features associated with a text segment of an image and then utilizing these features to generate refined text segmentation. By identifying features common to text segments of the image, an improved text segmentation tool can provide more accurate results at the pixel level. For example, sections of text that might be missed by conventional segmentation tools are correctly classified as text, while non-text segments of an image that are not text are not classified as text.

[0021] Embodiments of the present disclosure include a neural network comprising a deep learning backbone network for text segmentation and a network extension that utilizes the output of the backbone network and refines its output using a customized query key attention mechanism to increase classification accuracy. Embodiments of the present disclosure, in both regular and artistic configurations, process text patterns embedded in the image (that is, instead of exclusively processing regular objects in an everyday scene). The text patterns can have different types and textures depending on the application and the user.

[0022] In some embodiments, the neural network includes an interchangeable encoder network and a customized query key attention mechanism that compares foreground text information over an image. A neural network of the present disclosure overcomes problems in recognizing text images. For example, some text regions lack advanced prior spatial knowledge, and their textures have comparatively little semantic meaning. In some embodiments, the neural network incorporates the unconventional techniques of integrating re-thresholding with channel-wise given cosine similarity, foreground feature spooling, and query key attention. As a result, the improved neural network increases the accuracy of text segmentation. (Overview of the system)

[0023] Fig. Figure 1 shows an example of a text segmentation system according to aspects of the present disclosure. The example shown includes a user 100, a user device 105, a text segmentation device 110, a cloud 135, and a database 140. In one example, the text segmentation device 110 includes a processor unit 115, a memory unit 120, an image application 125, and a neural network 130. However, in some embodiments, the components and functions of the text segmentation device 110 are localized on the user device 105.

[0024] A user 100 can select an image containing foreground text, and the text segmentation device 110 can identify the foreground text (for example, using a machine learning model). The user 100 can then edit or otherwise use the text identified by the text segmentation device 110.

[0025] User 100 can communicate with the text segmentation device 110 via the user device 105 and the cloud 135. The user device 105 can be a PC, a laptop computer, a mainframe computer, a palmtop computer, a personal assistant, a mobile device, or any other suitable processing device.

[0026] A Processor Unit 115 is an intelligent hardware device (for example, a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the Processor Unit 115 is configured to operate a memory array using a memory controller. In other cases, a memory controller is integrated into the Processor Unit 115. In some cases, the Processor Unit 115 is configured to execute computer-readable instructions stored in memory to perform various functions.In some embodiments, the processor unit 115 includes special-purpose components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0027] Examples of the Memory Unit 120 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of a Memory Unit 120 include solid-state storage and a hard disk drive. In some examples, the Memory Unit 120 is used to store computer-readable and computer-executable software containing instructions that, when executed, cause a processor to perform various functions described here. In some cases, the Memory Unit 120 includes, among other things, a BIOS (Basic Input / Output System) that controls the basic operation of hardware or software, such as interaction with peripheral components or devices. In some cases, the memory controller operates memory cells. The memory controller may include, for example, a row decoder, a column decoder, or both.In some cases, memory cells within a storage unit store 120 pieces of information in the form of a logical state.

[0028] A Cloud 135 is a computer network configured to provide on-demand availability of computer system resources, such as data storage and computing power. In some examples, the Cloud 135 provides resources without active user intervention. The term "Cloud 135" is sometimes used to describe data centers accessible to many users via the internet. Some large cloud networks handle functions distributed across multiple locations from central servers. A server is referred to as an edge server if it has a direct or close connection to a user. In some cases, the Cloud 135 is limited to a single organization. In other examples, the Cloud 135 is available to many organizations.In one example, Cloud 135 comprises a multi-layered communications network including multiple edge routers and core routers. In another example, Cloud 135 relies on a local collection of switches at a single physical location.

[0029] A database is an organized collection of data. A database stores data, for example, in a specific format known as a schema. A database can be structured as a single database, a distributed database, multiple distributed databases, or a disaster recovery backup database. In some cases, a database controller manages the data storage and processing within a database. In some cases, the user interacts with the database controller. In other cases, the database controller can operate automatically without user interaction. In some examples, the database might contain a selection of images that include foreground text.

[0030] According to some embodiments, the image application 125 receives an image containing a foreground text section and a background section. The text segmentation device 110 identifies the foreground text section based on the classification by the neural network 130. In some examples, the image application 125 modifies the texture of the foreground text section to generate a modified image.

[0031] In some embodiments, the image application 125 identifies initial training images. In some examples, the image application 125 identifies the foreground text portion and the background portion for each of the initial training images. In some examples, the image application 125 modifies the initial training images based on the identified foreground text portion to generate additional training images, the training set including the initial training images and the additional training images. In some examples, the image application 125 replaces the background portion while retaining the foreground text portion.

[0032] In some examples, Image Application 125 includes a graphics editor (such as Microsoft® Paint or Adobe® Photoshop). The user can select a digital image and input it into the neural network for text segmentation using Image Application 125. The user can also edit the output image from the neural network (that is, a processed image in which the foreground text has been identified) using Image Application 125.

[0033] In some embodiments, the text segmentation device 110 incorporates an artificial neural network (ANN). An ANN is a hardware or software component that includes a number of connected nodes (i.e., artificial neurons) that correspond, in a sense, to the neurons in the human brain. Each connection or edge transmits a signal from one node to another (like the physical synapses in the brain). When a node receives a signal, it processes the signal and then transmits the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is calculated as a function of the sum of its inputs. Each node and edge is assigned one or more node weights that determine how the signal is processed and transmitted.

[0034] During the training process, these weights are adjusted to improve the accuracy of the result (by minimizing a loss function that, in a sense, represents the difference between the current result and the target result). The weight of an edge increases or decreases the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which a signal is not transmitted at all. In some cases, the nodes are grouped into layers. Different layers perform different transformations on their inputs. The initial layer is known as the input layer, and the final layer is known as the output layer. In some cases, signals traverse certain layers multiple times.

[0035] In some examples, the Text Segmentation Device 110 incorporates a convolutional neural network (CNN). For instance, an encoder used to identify image features may include convolutional layers. A CNN is a class of neural network commonly used in computer vision or image classification systems. In some cases, a CNN can enable the processing of digital images with minimal preprocessing. A CNN may be characterized by the use of convolutional (or cross-correlation) hidden layers. These layers apply a convolutional operation to the input before signaling the result to the next layer. Each convolutional node can process data for a limited input field (that is, the receiving field).During a forward pass of the CNN, filters in each layer can be convolved over the input volume, calculating the dot product between the filter and the input. During the training process, the filters can be modified to activate when they detect a specific feature within the input.

[0036] According to some embodiments, the neural network 130 classifies each pixel of the image as foreground text or background, which refines a segmentation prediction using a key vector representing features of the foreground text segment, the key vector being based on the segmentation prediction.

[0037] In some embodiments, neural network 130 classifies each pixel of the training images as foreground text or background to generate a segmentation prediction for each of the training images. In some examples, neural network 130 computes a first loss function based on a comparison of the segmentation prediction and the ground-truth segmentation information. Neural network 130 computes a second loss function based on a comparison of the refined segmentation prediction and the ground-truth segmentation information. Parameters of neural network 130 are updated based on the first and second loss functions. In some cases, the neural network is also referred to as a text-refining network.

[0038] Embodiments of the present disclosure provide a deep learning neural network. The neural network includes an interchangeable encoder network and a customized query key attention mechanism that spreads and compares text foreground information over an image. In one embodiment, the neural network includes re-thresholding with channel-wise given cosine similarity, foreground feature pooling, and query key attention. As a result, the neural network improves the accuracy of the classification and the results of the segmentation prediction.

[0039] According to one embodiment, the neural network comprises two components: a backbone network for segmentation prediction and a network extension that uses the output from the backbone network and refines its prediction using a customized query key attention mechanism. The backbone network is an interchangeable encoder network and provides a segmentation result on text regions. In some examples, the backbone network includes ResNet101, DeepLabV3+, or another suitable image processing network. However, embodiments of the present disclosure are not limited to the choice of backbone network. The neural network does not impose any restrictions on the choice of backbone network, and users can select one according to their specific tasks and the computing environment.

[0040] Fig. Figure 2 shows an example of a text segmentation process according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of codes to control or regulate functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to procedures and processes described according to aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0041] In step 200, a user selects an image that includes text in the foreground. The user can upload a digital image to the image application (for example, Adobe® Photoshop). In some examples, the user selects a digital image from the database or the cloud. The text in the foreground is embedded in the scene and includes text patterns, such as English letters. According to one example, the text in the foreground includes an artistic font or a curved / swirly font where the letters of this customized font are well-proportioned, smooth, and flowing. In some cases, the operations of this step involve a user as described by Fig. 1 has been described, or will be carried out by this.

[0042] In process 205, the system performs text segmentation on the image. The system incorporates a neural network capable of classifying and segmenting objects related to foreground text at the pixel level. The neural network can segment various types of text patterns, such as regular fonts, artistic fonts, and custom text styles embedded within the user-selected image. In some cases, the operations of this step may involve a neural network, as illustrated by... Fig. 1 has been described, or can be carried out by this.

[0043] In process 210, the system identifies the text foreground. The text foreground of the image is segmented, preserved, and output by the system. The neural network filters out non-textual information or objects that are not related to the text segmentation, such as environments and objects that are not labeled as text. Unlike an input image, where text is blended with its environment and not readily apparent, the user can easily see the text foreground in the system's output image. In some cases, the processes of this step involve an image application, as illustrated by Fig. 1 has been described, or can be carried out by this.

[0044] In Operation 215, the user edits the text foreground of the image. The user can continue editing the text foreground of the image using the image application (for example, Adobe® Photoshop). In some examples, the user changes the content of the text and / or text pattern (for example, by using a font with a different style) based on the identified text foreground portion of the image. In some cases, the operations of this step involve a user as described by Fig. 1 has been described, or can be carried out by this. (Text segmentation)

[0045] Fig. Figure 3 shows an example of text segmentation according to aspects of the present disclosure. The example shown includes an original image 300 and a text foreground 305. In some cases, the original image 300 includes a set of objects from a scene. Some of the objects include text. According to one example, the original image 300 includes a highway exit sign (drivers know that it is an exit sign because of the text “EXIT ONLY” written in capital letters on the sign). The exit sign shows the number of the exit assigned to a road junction (for example, 20A) and the destination of the exit if a driver takes that junction (for example, Highway XY).Beyond the text, the original image also includes a scene with surroundings, such as roofs of houses, leaves of trees, sky and the montage structure of the exit sign.

[0046] Text foreground 305 contains only text information within the image. Text foreground 305 does not include any surroundings in the scene that are not labeled as text. In the example described above, text foreground 305 includes a regular font, an artistic font, or a combination of both (for example, "EXIT 20 A," "NORTH," "Likelike Hwy," and "EXIT ONLY" are shown in the text foreground).

[0047] According to some embodiments, the original image 300 is used for training and is therefore fed into the neural network. The text foreground 305 is the ground truth annotation. A loss function is calculated between the segmentation prediction based on the original image and the ground truth annotation. In some cases, the original image 300 is fed into the neural network for derivation.

[0048] Fig. Figure 4 shows an example of text segmentation according to aspects of the present disclosure. The example shown includes an original image 400 and a text foreground 405. The original image 400 contains text or text patterns with a user-specified font. In some cases, the text patterns include user-created artistic embellishments. According to one example, the original image 400 includes a person's hand holding an artistic, non-regular text font. The text includes "vacation calories do not count." In some cases, the letters of the text font are not readily recognizable to a user because the text is mixed with other non-textual information in the scene and / or the letters have curved or ornate shapes that make the text less obvious to the user.There are non-textual elements in the original image, such as trees, umbrellas, and houses. The user is interested in knowing the exact content of the text (for example, the spelling of each word in the sentence) and can then proceed to edit the identified text from the original image.

[0049] Text foreground 405 contains only textual information. Text foreground 405 has no environmental information that is not labeled as text within the scene. In the example described above, text foreground 405 contains the text "vacation calories do not count". The text is set in an artistic font (for example, some of the English letters are in a curved, swirly, or adapted font so that the letters are well-proportioned, smooth, and flowing).

[0050] According to some embodiments, the original image 400 is used for training and is therefore fed into the neural network. The text foreground 405 is the ground truth annotation. A loss function is calculated between the segmentation prediction based on the original image and the ground truth annotation. In some cases, the original image 400 is fed into the neural network for derivation. (Architecture)

[0051] Fig. Figure 5 shows an example of a text segmentation device according to aspects of the present disclosure. The example shown includes an original image 500, an encoder 505, a feature map 510, a first decoder 515, a segmentation prediction 520, a key generator 525, an attention component 530, a combined feature map 535, a second decoder 540, a refined segmentation prediction 545, a text foreground 550, a first loss 555, and a second loss 560.

[0052] In some embodiments, the neural network (that is, the network model) includes a backbone network and a refinement extension. The backbone network is an interchangeable encoder network selected by users based on their tasks and the computing environment. The backbone network is also referred to as Encoder 505. The refinement extension is a query key attention mechanism that processes the (output) feature map 510 from the backbone network. In one embodiment, the neural network includes the Encoder 505, the first decoder 515, the key generator 525, the attention component 530, and the second decoder 540. In some cases, the neural network is also referred to as a text-refining network or model.

[0053] The original image 500 is an input image for the decoder 505. In some cases, the original image is referred to as input image x. The original image 500 contains text or text patterns in a specific font. In some cases, the text patterns include artistic embellishments by a user. For example, the original image 500 includes, as in Fig. Image 5 depicts a person's hand holding an artistic, non-regular font of text that reads "vacation calories do not count." In some cases, the font of the text is not immediately apparent to a user. There are non-textual environments in the original image, such as streets, umbrellas, and houses. A user is interested in knowing the exact content of the text (for example, the spelling of the text) and can then proceed to edit the identified text from the original image. The original image 500 is an example of the corresponding element, which is based on Fig. 3 and Fig. 4 has been described, or includes aspects thereof.

[0054] According to some embodiments, the encoder 105 encodes the input image x to generate a feature card 510. The feature card 510 is then used with x f designated.

[0055] The encoder 505 encodes an image from the training images to generate a feature map 510. One of the training images is, for example, the original image 500 (that is, the input image x). In some embodiments, the encoder 505 is configured to encode an image to generate a feature map 510, where the image includes a foreground text section and a background section. In some examples, the encoder 505 incorporates a ResNet architecture. In some other examples, the encoder 505 incorporates a DeepLabV3+ architecture. However, embodiments of the present disclosure are not limited to the choice of encoder network.

[0056] According to some embodiments, the first decoder 515 decodes the feature map 510 to generate the segmentation prediction 520. The segmentation prediction 520 is expressed as x semThis is referred to as... In some examples, the first decoder 515 includes a convolution layer, a first bias, and a first softmax. The first decoder 515 applies a convolution layer to the feature card 510. The first decoder 515 applies a first bias to an output of the convolution layer. The first decoder 515 applies a first softmax to an output of the first bias.

[0057] According to one embodiment, the model calculates the segmentation prediction x sem via an extra folding layer with a bias and a softmax layer. Then x sem using the ground truth annotation y with a cross-entropy loss L sem (that is, the first loss 555) is monitored. Since the model L sem directly using x sem and y is calculated, it requires close monitoring of the parameters of the encoder 505.

[0058] According to some embodiments, the key generator 525 identifies the key vector based on the segmentation prediction 520. In some examples, the key generator 525 calculates a cosine similarity of the segmentation prediction 520. The key generator 525 applies a second bias based on the cosine similarity. The key generator 525 applies a second softmax to an output of the second bias. The key generator 525 combines the second softmax with the feature map 510. The key generator 525 applies a pooling layer to generate the key vector.

[0059] According to some embodiments, the key generator 525 identifies a key vector for representing features of the foreground text segment based on the segmentation prediction 520. In some examples, the key generator 525 includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer. The key generator 525 computes a cosine similarity of the segmentation prediction 520. The key generator 525 applies a second bias based on the cosine similarity. The key generator 525 applies a second softmax to an output of the second bias. The key generator 525 combines the second softmax with the feature map 510. The key generator 525 applies a pooling layer to generate the key vector.

[0060] According to one embodiment, the model assumes a threshold re-thresholding x sembefore and includes a modified segmentation prediction that uses x̂ sem This is referred to as the threshold regeneration step, in which only the foreground prediction remains activated with high reliability. The model performs this threshold regeneration step by introducing a new bias at the convolutional layer, thereby increasing x sem is generated. This new bias is linearly proportional to the channel-wise given cosine similarity of x. sem . cossim(x,y)=xT⋅y‖x‖‖y‖

[0061] According to one embodiment, the channel-wise given cosine similarity of x sem inversely proportional to the prediction accuracy of this. x sem Therefore, if it is inaccurate, its channel-wise given cosine similarity increases, and the bias in the background channel also increases. This equivalently reduces the activation in the foreground region in x̂. semduring the passage of the softmax. The result is that only the foreground prediction remains with high reliability in x̂. sem activated.

[0062] The model then performs a pooling of a feature vector v. k within the highly reliable range of x̂ sem before, to represent the foreground texture of the text. Such pooling is achieved by the weighted sum between the feature map x f and x̂ sem carried out. vk=∑n=1…Nxfn⋅x^semn

[0063] Here, N denotes the number of pixels in the map, which is why only the channel dimension (C) is preserved. This means the model not only captures foreground text information in v k within an image, but also reduces the dependence on v k with backbone parameters (that is, encoder parameters) between different images.

[0064] In some embodiments, the attention component 530 classifies each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction 545 for each of the training images. In some examples, the attention component 530 combines the key vector and the feature map 510 to generate an attention map. In some examples, the attention component 530 combines the attention map and the feature map 510 to generate a combined feature map 535. In some examples, the combined feature map 535 includes color information for each pixel of the image. In some embodiments, the attention component 530 is configured to generate a combined feature map 535 based on the key vector.In some examples, the attention component 530 includes a second product function and a chaining function.

[0065] According to one embodiment, the model v uses k as key and x f as a request and calculates attention x att through the dot product. Since the dot product is performed at all pixel locations, the texture information is globally spread or distributed to all parts of the image.

[0066] In some embodiments, the second decoder 540 decodes the combined feature map 535 to generate a refined segmentation prediction 545, identifying the foreground text section based on the refined segmentation prediction 545. In some embodiments, the second decoder 540 is configured to generate a refined segmentation prediction 545 based on the combined feature map 535. In some examples, the second decoder 540 includes convolution layers, batch normalization layers, and ReLU layers. The refined segmentation prediction 545 is used with x rfn designated.

[0067] According to one embodiment, the model links the input image x and the feature map x. f and the newly calculated attention x attand passes this new matrix through two additional Conv-5×5 layers and one Conv-1×1 layer to obtain the final refined result x rfn to obtain. x rfn The ground truth annotation y is used with the cross-entropy loss L. rfn (that is, using the second loss of 560) monitored during the training period. x rfn This serves as the final output of the model during the derivation time (i.e., when using the model for prediction). The final loss L (that is, the second loss 560) is a weighted sum of L. sem and L rfn . Lsem=1|C|×|N|∑c=1..C∑n=1..Nyc,nlogxsemc,n Lrfn=1|C|×|N|∑c=1..C∑n=1..Nyc,nlogxrfnc,n L=wsemLsem+wrfnLrfn

[0068] According to one embodiment, the model can be trained end-to-end. During the training process, the hyperparameters are set, for example, as follows: The weights for both L sem as with L rfn The training length is set to 20,500 total iterations. The learning rate is 0.01 and decreases slowly from 0.01 to 0 due to the 0.9 poly rules. A warmup of 500 iterations is used at the beginning of the training. One embodiment of the present disclosure uses the pretrained ImageNet model as the initial model. No hyperparameters are required during the derivation time.

[0069] In some cases, the text foreground 550 is a ground-truth image (denoted by y). For example, text foreground 550 contains text information (or a query) that reads "vacation calories do not count". Text foreground 550 includes a regular font, an artistic font, or a combination of both. Unlike the original image, text foreground 550 does not contain any non-textual information (such as contextual information or surroundings). Text foreground 550 is an example of the corresponding element that is based on Fig. 3 and Fig. 4 has been described, or includes aspects thereof. (Derivation)

[0070] Fig. Figure 6 shows an example of a text segmentation process according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of codes to control or regulate functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to procedures and processes described according to aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0071] In operation 600, the system receives an image containing a foreground text section and a background section. In some cases, the operations of this step involve an image application, as described by Fig. The image application includes a graphics editor (for example, Microsoft® Paint or Adobe® Photoshop). The image application is described in section 1, or can be executed by it.

[0072] In some implementations, the received image contains text patterns in a regular font and / or artistic designs embedded within the image. In some cases, the image can be used to train the neural network during a training phase and is therefore input into the neural network. In other cases, the image is used for derivation, and the neural network generates text segmentation results based on the image.

[0073] In operation 605, the system classifies each pixel of the image as foreground text or background using a neural network that refines a segmentation prediction using a key vector representing features of the foreground text segment, the key vector being based on the segmentation prediction. In some cases, the operations of this step involve a neural network, as shown by Fig. 1 has been described, or can be performed by it.

[0074] According to one embodiment, the neural network includes a backbone network and a network extension that accesses the output of the backbone network. The backbone network includes a CNN model (for example, ResNet-101, DeepLabV3+). DeepLabV3+ is a CNN model used for semantic segmentation and includes an encoder-decoder network. ResNet-101 is a CNN model with a depth of 101 layers. In some examples, the ResNet-101 network is pre-trained on the ImageNet database. However, embodiments of the present disclosure are not limited to the choice of backbone network. The neural network does not impose any restrictions on the choice of backbone network, and users can select the backbone network according to their specific tasks and the computing environment.

[0075] In some cases, the backbone network is referred to as the encoder. The network extension refines the segmentation prediction from the backbone network using a query key attention mechanism. The neural network can decompose the text foreground from the image at the pixel level. Therefore, the accuracy of the text segmentation is increased.

[0076] In one embodiment, the neural network includes an encoder (i.e., a backbone network), a first decoder, a key generator, an attention component, and a second decoder. The encoder is configured to encode an image to generate a feature map, where the image contains a foreground text segment and a background segment. The first decoder is configured to decode the feature map to generate a segmentation prediction. The key generator is configured to identify a key vector to represent features of the foreground text segment based on the segmentation prediction. The attention component is configured to generate a combined feature map based on the key vector. The second decoder is configured to generate a refined segmentation prediction based on the combined feature map.

[0077] In operation 610, the system identifies the foreground text segment based on the classification. A user can then proceed to edit the foreground text segment using the graphics editor (for example, using Microsoft® Paint or Adobe® Photoshop). In some cases, the operations of this step involve an image application, as described by Fig. 1 has been described, or can be carried out by this. (Initial segmentation)

[0078] Fig. Figure 7 shows an example of a process for generating a segmentation prediction according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of codes to control or regulate functional elements of a facility. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to procedures and processes described according to aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0079] In process 700, the system applies one or more convolutional layers to the feature map. Convolutional layers in a convolutional neural network apply learned filters to input images to generate feature maps. In some cases, the operations of this step involve a first decoder, as described by Fig. 5, or can be performed by it. According to some embodiments, the encoder is configured to encode an image to generate a feature map, wherein the image includes a foreground text section and a background section. The first decoder is configured to decode the feature map to generate a segmentation prediction. In some cases, the encoder includes a ResNet architecture. The first decoder includes a convolution layer, a first bias, and a first softmax.

[0080] In operation 705, the system applies an initial bias to an output from the convolutional layer. The initial bias value or values ​​are used to shift left or right an activation function of the neural network used to learn the model. In some examples, changing the model's weights can alter the slope of the activation function. The overall curve of the activation function can be shifted left or right using the initial bias. In some cases, the operations of this step involve an initial decoder, as described by Fig. 5 has been described, or can be carried out by this.

[0081] In operation 710, the system applies a first softmax to an output of the first bias. The first softmax is also called the first softmax function. According to one embodiment, the first softmax function is used as the activation function of the neural network to normalize the network's output to a probability distribution over predicted output classes. After applying the first softmax function, each component of the feature map lies in the interval (0, 1), and the components sum to 1. These values ​​are interpreted as probabilities. In some cases, the operations of this step involve a first decoder, as described by Fig. 5 is described, or can be carried out by this. (Text feature identification)

[0082] Fig. Figure 8 shows an example of a process for generating a key vector according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of codes to control or regulate functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to procedures and processes described according to aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0083] In process 800, the system encodes the image to generate a feature map. The neural network includes a backbone network (also called an encoder). The encoder extracts the feature map from an input image. In some cases, the operations of this step involve an encoder, as illustrated by Fig. 5 has been described, or can be carried out by this.

[0084] In operation 805, the system decodes the feature map to generate the segmentation prediction. The encoder calculates the segmentation prediction using the first decoder. The first decoder includes a convolution layer, a first bias, and a first softmax. In some cases, the operations of this step involve a first decoder as defined by Fig. 5 has been described, or can be carried out by this.

[0085] In process 810, the system identifies the key vector based on the segmentation prediction. The key generator is configured to identify a key vector to represent features of the foreground text segment based on the segmentation prediction. According to one embodiment, the key generator includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer.

[0086] In one embodiment, the neural network performs a sophisticated threshold re-thresholding of the segmentation prediction and obtains a modified segmentation prediction. The neural network executes this threshold re-thresholding step by introducing a new bias at the convolution layer, thereby generating the segmentation prediction. This new bias is linearly proportional to the channelwise cosine similarity of the segmentation prediction. In another embodiment, the channelwise cosine similarity of the segmentation prediction is inversely proportional to its prediction accuracy. Therefore, if the segmentation prediction becomes inaccurate, its channelwise cosine similarity increases, and the bias in the background channel increases. This equivalently reduces the activation in the foreground region for the modified segmentation prediction as it passes through the softmax.Therefore, only the foreground prediction remains activated with high reliability in the modified segmentation prediction.

[0087] The neural network then pools a feature vector within the highly reliable domain of the modified segmentation prediction to represent the foreground texture of the text. This pooling is performed by a simple weighted sum between the feature map and the modified segmentation prediction. The neural network not only aggregates text foreground information into the feature vector within an image, but also reduces the feature vector's dependence on backbone parameters across different images. In some cases, the feature vector is also referred to as the key vector, as illustrated by... Fig. 5 has been described. In some cases, the processes of this step involve a key generator, as described in section 5. Fig. 5 has been described, or will be carried out by this.

[0088] Fig. Figure 9 shows an example of a process for generating a key vector according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of codes to control or regulate functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to procedures and processes described according to aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0089] In process 900, the system calculates a cosine similarity of the segmentation prediction. In some cases, the processes of this step involve a key generator, as described by Fig. 5 has been described, or can be carried out by it.

[0090] In one embodiment, the neural network performs a sophisticated threshold re-sholding of the segmentation prediction and obtains the modified segmentation prediction in which only the foreground prediction remains activated with high reliability. The neural network executes this threshold re-sholding by introducing a new bias at the convolution layer, thereby generating the segmentation prediction. This new bias is linearly proportional to the channelwise cosine similarity of the segmentation prediction.

[0091] According to one embodiment, the channel-wise cosine similarity of the segmentation prediction is inversely proportional to its prediction accuracy. Therefore, if the segmentation prediction becomes inaccurate, its channel-wise cosine similarity increases, and the bias in the background channel increases. This correspondingly reduces the activation in the foreground during the modified segmentation prediction process when the softmax function is applied. Consequently, only the foreground prediction remains activated with high reliability during the modified segmentation prediction process.

[0092] In process 905, the system applies a second bias based on cosine similarity. In some cases, the processes of this step involve a key generator, as described by Fig. 5 has been described, or can be carried out by this.

[0093] In operation 910, the system applies a second softmax to an output of the second bias. In some cases, the operations of this step involve a key generator, as described by Fig. 5 has been described, or can be carried out by this.

[0094] In process 915, the system combines the second softmax with the feature map. According to one embodiment, the modified segmentation prediction and the feature map are combined and fed into a pooling layer. In some cases, the processes of this step involve a key generator, as described by Fig. 5 has been described, or can be carried out by this.

[0095] In operation 920, the system applies a pooling layer to generate the key vector. A pooling layer is a new layer added after the convolutional layer. The pooling layer is used to order layers within a convolutional neural network, a process that can be repeated once or multiple times within a neural network model. The pooling layer operates on each feature map separately to create a new set of the same number of pooled feature maps. Pooling involves selecting a pooling operation. In some cases, the size of the pooling operation is smaller than the size of the feature map. For example, the size of the pooling operation might be 2×2 pixels with a stride of 2 pixels. Average pooling or maximum pooling operations can be used.In some cases, the processes of this step involve a key generator, as shown in . Fig. 5 has been described, or can be carried out by this.

[0096] According to one embodiment, the model assumes a pooling of a feature vector v. k (that is, a key vector) within the highly reliable domain of the modified segmentation prediction to represent the foreground texture of the text. Such pooling is performed by a simple weighted sum between the feature map and the modified segmentation prediction. The weighted sum is denoted as v k = vk=∑n=1..Nxfn⋅x^semn formulated where N denotes the number of pixels in the map, such that only the channel dimension (C) is preserved. The neural network not only summarizes text foreground information in v kwithin an image, but also reduces the dependence on v k with backbone parameters between different images. (Refined segmentation)

[0097] Fig. Figure 10 shows an example of a process for generating a refined segmentation prediction according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of code to control or regulate functional elements of a facility. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to procedures and processes described according to aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0098] In process 1000, the system combines the key vector and the feature map to generate an attention map. According to one embodiment, the neural network uses v k The function takes the text as a key vector and the feature map as a query and calculates the attention map using the dot product. In some cases, a dot product is performed at all pixel locations, applying the texture information globally to all parts of the image to highlight parts of the image with features similar to the text features. In some cases, the operations of this step involve an attention component, as determined by the text features. Fig. 5 has been described, or can be carried out by this.

[0099] In process 1005, the system combines the attention map and the feature map to generate a combined feature map. According to one embodiment, an attention component is configured to generate a combined feature map based on the key vector. The attention component includes a second product function and a concatenation function. According to one embodiment, the neural network concatenates the input image (that is, the original image), the feature map, and the newly calculated attention map. The resulting matrix is ​​a combined feature map. In some cases, the operations of this step involve an attention component as defined by Fig. 5 has been described, or can be carried out by this.

[0100] In process 1010, the system decodes the combined feature map to generate a refined segmentation prediction, identifying the foreground text section based on this prediction. According to one embodiment, a second decoder is configured to generate a refined segmentation prediction based on the combined feature map. This second decoder includes convolution layers, batch normalization layers, and ReLU layers.

[0101] In one embodiment, the neural network iterates through the combined feature map using two additional Conv-5×5 layers and one Conv-1×1 layer to obtain the refined final result. The refined final result is also referred to as the refined segmentation prediction. In some cases, the operations of this step involve a second decoder, as illustrated by Fig. 5 has been described, or can be carried out by this. (Training)

[0102] Fig. Figure 11 shows an example of a process for training a neural network for text segmentation according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of codes to control or regulate functional elements of a device. Additionally or alternatively, certain processes are performed using specialized hardware. Generally, these operations are performed according to procedures and processes described in accordance with aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0103] In operation 1100, the system identifies a training set comprising training images and ground-truth segmentation information for each of the training images, where the ground-truth segmentation information identifies a foreground text section and a background section. In some cases, the foreground text section consists solely of text information. For example, the foreground text section reads "Vacation calories do not count." The text uses a non-regular and artistic font. Some letters in the text have curved or swirly shapes. In some cases, the operations of this step involve an image application, such as those described by Fig. 1 has been described, or can be carried out by this.

[0104] In process 1105, the system classifies each pixel of the training images as foreground text or background to generate a segmentation prediction for each of the training images. In some cases, the processes of this step involve a neural network, as shown by Fig. 1 has been described, or can be carried out by this.

[0105] In one embodiment, the neural network extracts the feature map from the input image using an encoder and further calculates a segmentation prediction via an additional convolutional layer with a bias and a softmax layer. The output of the softmax layer, or softmax function, is the segmentation prediction, where each component (or element) of the segmentation prediction lies within the interval (0, 1). The values ​​(0, 1) corresponding to the respective component are interpreted as probabilities, for example, the probability that a pixel of a training image is considered foreground text or background.

[0106] In process 1110, the system calculates an initial loss function based on a comparison of the segmentation prediction and the ground-truth segmentation information. The segmentation prediction is combined with the ground-truth annotation using a cross-entropy loss L. sem (that is, the first loss function) is monitored. Since the neural network monitors the first loss function L sem Calculated directly using a segmentation prediction and ground-truth segmentation information, it enforces strong monitoring of all backbone parameters. In some cases, the processes of this step involve a neural network, as demonstrated by... Fig. 1 has been described, or can be carried out by this.

[0107] In operation 1115, the system identifies a key vector to represent features of the foreground text segment based on the segmentation prediction. According to one embodiment, a key generator is configured to identify a key vector to represent features of the foreground text segment based on the segmentation prediction. The key generator includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer. In some cases, the operations of this step involve a key generator, as shown by Fig. 5 has been described, or can be carried out by this.

[0108] In process 1120, the system classifies each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each of the training images. In some cases, the processes of this step involve an attention component, as determined by Fig. 5 has been described, or can be carried out by this.

[0109] In one embodiment, the neural network uses the key vector and the feature map as a query and computes the attention map by the dot product of the key vector and the feature map. In some cases, a dot product is performed at all pixel locations, and texture information is applied globally to all parts of the image to highlight parts of the image with features similar to the text features. The model concatenates the input image, the feature map, and the newly computed attention map and passes this new matrix through two additional 5×5 conversion layers and one additional 1×1 conversion layer to obtain the refined segmentation prediction.

[0110] In process 1125, the system calculates a second loss function based on a comparison of the refined segmentation prediction and the ground-truth segmentation information. According to one embodiment, the refined segmentation prediction is calculated using a cross-entropy loss L. rfn (that is, the second loss function) is monitored during the training period and serves as the final output of the neural network during the derivation period (that is, for prediction). The final loss L is a weighted sum of L sem and L rfn (that is, from the first loss function and the second loss function). In some cases, the processes of this step involve a neural network, as shown by Fig. 1 has been described, or can be carried out by this.

[0111] In process 1130, the system updates parameters of the neural network based on the first loss function and the second loss function. In some cases, the processes of this step involve a neural network as described by Fig. 1 has been described, or can be carried out by this.

[0112] According to one embodiment, the neural network can be trained end-to-end. During the training phase, the hyperparameters are set, for example, as follows: The weights for both L sem as with L rfnThe training length is set to 20,500 total iterations. The learning rate is 0.01 and decreases slowly from 0.01 to 0 due to the 0.9 poly rules. A warmup of 500 iterations is used at the beginning of training. In some cases, the neural network uses the pre-trained ImageNet model as the initial model for text segmentation. The neural network's encoder is a pre-trained backbone network (that is, weights or parameters from ImageNet are pre-trained for segmentation classification). According to one embodiment, the neural network's encoder is already trained to identify textures, edges, and the like. During end-to-end training of the neural network, the encoder's weights are updated (that is, the weights of the ImageNet backbone network are not fixed). No hyperparameters are needed during the derivation time.

[0113] Fig. Figure 12 shows an example of a process for extending a training set according to aspects of this disclosure. In some examples, these operations are performed by a system that includes a processor executing a set of code to control or regulate functional elements of a device. Additionally or alternatively, certain processes are performed using special-purpose hardware. Generally, these operations are performed according to procedures and processes described according to aspects of this disclosure. In some cases, the operations described here consist of several sub-steps or are performed in conjunction with other operations.

[0114] In operation 1200, the system identifies initial training images. In some cases, each of the initial training images contains one or more objects in the scene (for example, an everyday scene such as a highway exit sign). Each image includes, for example, textures, edges, and text patterns of various types. The text patterns may be a regular font, an artistic font, custom text styles, or a combination thereof. In some cases, the operations of this step involve an image application, as described by Fig. 1 has been described, or will be carried out by it.

[0115] In operation 1205, the system identifies the foreground text portion and the background portion for each of the initial training images. An image containing only the foreground text portion is referred to as a ground-truth image or ground-truth annotation. In some cases, the segmentation prediction from the neural network is monitored using the ground-truth image and a cross-entropy loss function. In some cases, the operations of this step involve an image application, such as those described by Fig. 1 has been described, or can be carried out by this.

[0116] In operation 1210, the system modifies the initial training images based on the identified foreground text segment to generate additional training images, with the training set including both the initial training images and the additional training images. In some examples, the additional training images do not have the same text patterns as the initial training images. For example, the additional training images have different textures and background colors. Random background swapping is another way the initial training images are modified. In some cases, multiscale training is used to create a large-sized training set. In some cases, the operations of this step involve an image application, as described by Fig. 1 has been described, or can be carried out by this. (Evaluation)

[0117] The results of the neural network (i.e., the text-refining network) are compared with the baseline model DeepLabV3+. The segmentation results of the neural network show a significant qualitative increase in prediction accuracy when compared with the results of DeepLabV3+. The evaluation metric includes fgloU (foreground intersection over union fgloU) to assess the segmentation models. The increase in quantitative results using the neural network is recorded. An ablation study, investigating the effectiveness of different parts of the neural network, is conducted. Further experimental results using two additional public text segmentation datasets (COCOTS and ICDAR17) are recorded.The basic model exhibits an fgloU score of 85.25% on a fitted text segmentation dataset (text segmentation dataset). The neural network of the present invention increases the fgloU score to 88.58%. In one example, the basic model exhibits an fgloU score of 71.13% on the COCOTS dataset. The neural network of the present disclosure increases the fgloU score to 73.89%. In another example, the basic model exhibits an fgloU score of 80.77% on the ICDAR17 dataset. The neural network of the present disclosure increases the fgloU score to 85.09%.

[0118] Accordingly, the present disclosure includes at least the following embodiments.

[0119] A method for text segmentation is described. Implementations of the method are configured to receive an image containing a foreground text section and a background section, classify each pixel of the image as foreground text or background using a neural network that refines a segmentation prediction using a key vector representing features of the foreground text section, the key vector being based on the segmentation prediction, and identify the foreground text section based on the classification.

[0120] A text segmentation device is described. The device includes a processor, a memory in electronic communication with the processor, and instructions stored in the memory. The instructions are executable to cause the processor to: receive an image containing a foreground text segment and a background segment; classify each pixel of the image as foreground text or background using a neural network that refines a segmentation prediction using a key vector representing features of the foreground text segment, where the key vector is based on the segmentation prediction; and identify the foreground text segment based on the classification.

[0121] This document describes a non-temporary, computer-readable medium that stores text segmentation code. In some examples, the code includes instructions executable by a processor to: receive an image containing a foreground text segment and a background segment; classify each pixel of the image as foreground text or background using a neural network that refines a segmentation prediction using a key vector representing features of the foreground text segment, the key vector being based on the segmentation prediction; and identify the foreground text segment based on the classification.

[0122] The method, setup and non-temporary computer-readable medium described above further include encoding the image to generate a feature map, decoding the feature map to generate the segmentation prediction, and identifying the key vector based on the segmentation prediction.

[0123] Some examples of the procedure, setup, and non-temporary computer-readable medium for decoding the feature map described above further include applying a convolutional layer to the feature map. Some examples further include applying an initial bias to an output of the convolutional layer. Some examples further include applying an initial softmax to an output of the initial bias.

[0124] The procedure, setup, and non-temporary computer-readable medium for identifying the key vector described above further include calculating a cosine similarity of the segmentation prediction, applying a second bias based on the cosine similarity, applying a second softmax to an output of the second bias, combining the second softmax with the feature map, and applying a pooling layer to generate the key vector.

[0125] Some examples of the procedure, setup, and non-temporary computer-readable medium described above further include combining the key vector and feature map to generate an attention map. Some examples further include combining the attention map and feature map to generate a combined feature map. Some examples further include decoding the combined feature map to generate a refined segmentation prediction, identifying the foreground text segment based on the refined segmentation prediction.

[0126] Some examples of the procedure, setup, and non-temporary computer-readable medium described above further include modifying a texture of the foreground text section to produce a modified image.

[0127] A method for training a neural network for text segmentation is described. Implementations of the method are configured to identify a training set comprising training images and ground-truth segmentation information for each of the training images, where the ground-truth segmentation information identifies a foreground text segment and a background segment; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each of the training images; compute an initial loss function based on a comparison of the segmentation prediction and the ground-truth segmentation information; and identify a key vector to represent features of the foreground text segment based on the segmentation prediction.Classifying each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each of the training images, calculating a second loss function based on a comparison of the refined segmentation prediction and the ground-truth segmentation information, and updating parameters of the neural network based on the first loss function and the second loss function.

[0128] This document describes a device for training a neural network for text segmentation. The device includes a processor, a memory in electronic communication with the processor, and instructions stored in the memory. The instructions are executable to cause the processor to: identify a training set comprising training images and ground-truth segmentation information for each of the training images, where the ground-truth segmentation information identifies a foreground text segment and a background segment; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each of the training images; and compute an initial loss function based on a comparison of the segmentation prediction and the ground-truth segmentation information.Identifying a key vector to represent features of the foreground text segment based on the segmentation prediction, classifying each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each of the training images, calculating a second loss function based on a comparison of the refined segmentation prediction and the ground-truth segmentation information, and updating parameters of the neural network based on the first loss function and the second loss function.

[0129] This document describes a non-temporary, computer-readable medium that stores code for training a neural network for text segmentation. In some examples, the code includes instructions executable by a processor to: identify a training set comprising training images and ground-truth segmentation information for each of the training images, where the ground-truth segmentation information identifies a foreground text segment and a background segment; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each of the training images; compute an initial loss function based on a comparison of the segmentation prediction and the ground-truth segmentation information; and identify a key vector to represent features of the foreground text segment based on the segmentation prediction.Classifying each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each of the training images, calculating a second loss function based on a comparison of the refined segmentation prediction and the ground-truth segmentation information, and updating parameters of the neural network based on the first loss function and the second loss function.

[0130] The procedure, setup and non-temporary computer-readable medium described above further include encoding an image of the training images to generate a feature map and decoding the feature map to generate the segmentation prediction.

[0131] Some examples of the procedure, setup, and non-temporary computer-readable medium for decoding the feature map described above further include applying a convolutional layer to the feature map. Some examples further include applying an initial bias to an output of the convolutional layer. Some examples further include applying an initial softmax to an output of the initial bias.

[0132] Furthermore, the procedure, setup, and non-temporary computer-readable medium for identifying the key vector described above include calculating a cosine similarity of the segmentation prediction, applying a second bias based on the cosine similarity, applying a second softmax to an output of the second bias, combining the second softmax with the feature map, and applying a pooling layer to generate the key vector.

[0133] Some examples of the procedure, setup, and non-temporary computer-readable medium described above further include combining the key vector and feature map to generate an attention map. Some examples further include combining the attention map and feature map to generate a combined feature map. Some examples further include decoding the combined feature map to generate the refined segmentation prediction, identifying the foreground text segment based on the refined segmentation prediction.

[0134] In some examples, the combined feature map includes color information for each pixel of the image.

[0135] Some examples of the procedure, setup, and non-temporary computer-readable medium described above further include identifying initial training images. Some examples further include identifying the foreground text segment and the background segment for each of the initial training images. Some examples further include modifying the initial training images based on the identified foreground text segment to generate additional training images, with the training set comprising the initial training images and the additional training images.

[0136] Some examples of the procedure, setup, and non-temporary computer-readable medium for modifying the initial training images described above further include replacing the background section while retaining the foreground text section. A text segmentation setup is described.Embodiments of the device include an encoder configured to encode an image to generate a feature map, wherein the image includes a foreground text section and a background section; a first decoder configured to decode the feature map to generate a segmentation prediction; a key generator configured to identify a key vector to represent features of the foreground text section based on the segmentation prediction; an attention component configured to generate a combined feature map based on the key vector; and a second decoder configured to generate a refined segmentation prediction based on the combined feature map.

[0137] This document describes a method for providing a text segmentation device. The method includes providing an encoder configured to encode an image to generate a feature map, where the image contains a foreground text section and a background section; a first decoder configured to decode the feature map to generate a segmentation prediction; a key generator configured to identify a key vector to represent features of the foreground text section based on the segmentation prediction; an attention component configured to generate a combined feature map based on a key vector; and a second decoder configured to generate a refined segmentation prediction based on the combined feature map.The key generator includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer.

[0138] In some examples, the encoder uses a ResNet architecture. In some examples, the first decoder includes a convolution layer, a first bias layer, and a first softmax layer. In some examples, the attention component includes a second product function and a concatenation function. In some examples, the second decoder includes convolution layers, batch normalization layers, and ReLU layers.

[0139] The description and the drawing presented here represent exemplary configurations, but do not represent all implementations within the scope of the claims. The processes and steps can, for example, be rearranged, combined, or modified in other ways. Furthermore, structures and devices can be represented in the form of block diagrams to illustrate the relationship between components and to avoid obscuring the described concepts. Similar components or features may have the same name but different reference symbols corresponding to other figures.

[0140] Some modifications to the disclosure are readily apparent to a person skilled in the art, and the principles defined here can also be applied to other variations without deviating from the scope of the patent claims. The disclosure is not limited to the examples and embodiments described here, but should be interpreted in the broadest sense as consistent with the principles and novel features disclosed herein.

[0141] The described methods can be implemented or performed by devices that include a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose computer can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. A processor can also be implemented as a combination of computing devices (for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working in conjunction with a DSP core, or any other such configuration).The functions described here can be implemented in hardware or software and can be executed by a processor, firmware, or any combination thereof. In software implementations executed by a processor, the functions can be stored as instructions or code on a computer-readable medium.

[0142] Computer-readable media include both non-temporary computer storage media and communication media, which are any medium that facilitates the transfer of code or data. A non-temporary storage medium can be any available medium that a computer can access. Examples of non-temporary computer-readable media include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), a compact disc (CD) or other optical disk storage, a magnetic disk storage, or any other non-temporary medium for carrying or storing data or code.

[0143] Furthermore, connection components can also be described as computer-readable media. If, for example, code or data is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave signals, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is included in the definition of "medium." Combinations of media are also included within the scope of computer-readable media. In the present disclosure and the following claims, the word "or" indicates an inclusive list such that, for example, the list consisting of X, Y, or Z means X, Y, Z, XY, XZ, YZ, or XYZ.Furthermore, the phrase "based on" is not used to represent a closed set of conditions. A step described, for example, "based on condition A" can be based on either condition A or condition B. In other words, the phrase "based on" should be interpreted as meaning "at least partially based on." Moreover, the words "a" or "an" indicate "at least one."

Claims

[1] Text segmentation methods, including: Receiving (600) an image that includes a foreground text section and a background section; Classifying (605) each pixel of the image as foreground text or background using a neural network (130) that refines a segmentation prediction (520) using a key vector representing features of the foreground text segment, the key vector being based on the segmentation prediction (520); and Identifying (610) the foreground text section based on the classification, the procedure further comprising: Encoding (800) the image (300, 400, 500) to generate a feature map (510); Decoding (805) the feature map (510) to generate the segmentation prediction (520); and Identifying (810) the key vector based on the segmentation prediction (520), and including the identification of the key vector: Calculating (900) a cosine similarity of the segmentation prediction (520); Applying (905) a second bias based on cosine similarity; Applying (910) a second softmax to an output of the second bias; Combining (915) the second softmax with the feature card; and Applying (920) a pooling layer to generate the key vector. [2] Method according to claim 1, wherein decoding the feature map comprises: Applying (700) a convolutional layer to the feature map (510); Applying (705) a first bias to an output of the folding layer; and Applying (710) a first softmax to an output of the first bias. [3] Method according to claim 1 or 2, further comprising: Combining (1000) the key vector and the feature map (510) to generate an attention map; Combining (1005) the attention map and the feature map (510) to generate a combined feature map (535); and Decoding (1010) the combined feature map (535) to generate a refined segmentation prediction (545), identifying the foreground text segment based on the refined segmentation prediction (545). [4] A method according to any of the preceding claims, further comprising: Modifying the texture of the foreground text section to create a modified image. [5] Method for training a neural network (130) for text segmentation, comprising: Identifying (1100) a training set comprising training images and ground truth segmentation information for each of the training images, wherein the ground truth segmentation information identifies a foreground text section and a background section; Classifying (1105) each pixel of the training images as foreground text or Background for generating a segmentation prediction (520) for each of the training images; Calculating (1110) a first loss function based on a comparison of the segmentation prediction (520) and the ground truth segmentation information; Identifying (1115) a key vector to represent features of the foreground text segment based on segmentation prediction (520); Classifying (1120) each pixel of the training images as foreground text or Background based on the key vector to generate a refined segmentation prediction (545) for each of the training images; Calculate (1125) a second loss function based on a comparison of the refined segmentation prediction (545) and the ground truth segmentation information; and Updating (1130) parameters of the neural network (130) based on the first loss function and the second loss function, the procedure further includes: Coding (800) an image (300, 400, 500) of the training images to generate a feature map (510); and Decoding (805) the feature map (510) to generate the segmentation prediction (520), and wherein Identifying (1115) the key vector includes: Calculating (900) a cosine similarity of the segmentation prediction (520); Applying (905) a second bias based on cosine similarity; Applying (910) a second softmax to an output of the second bias; Combining the second softmax with the feature card; and Applying (920) a pooling layer to generate the key vector. [6] Method according to claim 5, wherein the decoding (805) of the feature map (510) comprises: Applying (700) a folding layer to the feature map (510); Applying (705) a first bias to an output of the folding layer; and Applying (710) a first softmax to an output of the first bias. [7] Method according to claim 5 or 6, further comprising: Combining (1000) the key vector and the feature map (510) to generate an attention map; Combining (1005) the attention map and the feature map (510) to generate a combined feature map (535); and Decoding (1010) the combined feature map (535) to generate the refined segmentation prediction (545), identifying the foreground text segment based on the refined segmentation prediction (545). [8] Method according to claim 7, wherein: the combined feature map (535) includes color information for each pixel of the image. [9] Method according to any one of claims 5 to 8, further comprising: Identifying (1200) initial training images; Identify (1205) the foreground text section and the background section for each of the initial training images; and Modifying (1210) the initial training images based on the identified foreground text segment to generate additional training images, wherein the training set includes the initial training images and the additional training images. [10] Method according to claim 9, comprising modifying the initial training images: Replacing the background section while preserving the foreground text section. [11] Text segmentation device, comprising: an encoder (105, 505) configured to encode an image to generate a feature map (510), wherein the image includes a foreground text section and a background section;(510) a first decoder (515) configured to decode the feature map to generate a segmentation prediction (520); a key generator (525) configured to identify a key vector for representing features of the foreground text section based on the segmentation prediction (520); an attention component (530) configured to generate a combined feature map (535) based on the key vector; and a second decoder (540) configured to generate a refined segmentation prediction (545) based on the combined feature map (535), wherein: The key generator includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer. [12] Device according to claim 11, wherein: The encoder (105, 505) incorporates a ResNet architecture. [13] Device according to claim 11 or 12, wherein: the first decoder (515) includes a folding layer, a first bias and a first softmax. [14] Device according to any one of claims 11 to 13, wherein: the attention component (530) includes a second product function and a chaining function. [15] Device according to any one of claims 11 to 14, wherein: The second decoder includes convolution layers, batch normalization layers, and ReLU layers.

Images