Text refinement network

Abstract

Embodiments of this disclosure relate to text thinning networks. Systems and methods for text segmentation are described. Embodiments of the inventive concept are configured to: receive an image comprising a foreground text portion and a background portion; classify each pixel of the image as foreground text or background using a neural network, the neural network using a key vector representing features of the foreground text portion to thin the segmentation prediction, wherein the key vector is based on the segmentation prediction; and identify the foreground text portion based on the classification.

Description

Technical Field

[0001] This disclosure generally relates to computer vision, and more specifically to the use of machine learning to segment text patterns. Background Technology

[0002] Computer vision refers to the techniques for acquiring, processing, analyzing, and understanding digital images. For example, digital images can be input into artificial systems that can extract information. Computer vision includes tasks such as image processing, image segmentation, and object detection.

[0003] Image segmentation is the process of dividing a digital image into multiple segments or image objects. In some cases, segments or objects are represented by sets of pixels and corresponding category labels (i.e., semantic descriptions of the categories to which the pixels belong). Therefore, the output of an image segmentation network can be in the form of an image mask corresponding to one or more categories. Image segmentation helps simplify the representation of digital images and transforms them into processed images that users can further analyze and modify. Image segmentation has applications in fields such as image search and retrieval, image synthesis, image editing, and human-computer interaction.

[0004] Various tools can be used to perform image segmentation. However, traditional segmentation tools are unaware of the unique visual properties of text. Therefore, these tools are prone to errors when identifying text. For example, curves and decorations surrounding text may be incorrectly segmented as text, or only a portion of the text may be correctly identified, while other parts with similar colors and textures may be lost. Therefore, there is a need in the art for an improved image segmentation tool capable of accurately classifying and parsing foreground text information in an image. Summary of the Invention

[0005] This disclosure describes systems and methods for performing text segmentation. Embodiments of this disclosure perform initial segmentation, then refine the segmentation by identifying features common to portions of an image that are identified as text. In some examples, the key vector representing the common features is combined with image features using an attention component. According to some embodiments, a machine learning model is trained using multiple training tasks based on the initial segmentation and the refined segmentation.

[0006] A method, apparatus, and non-transient computer-readable medium for text segmentation are described. Embodiments of the method, apparatus, and non-transient computer-readable medium are configured to: receive an image comprising a foreground text portion and a background portion; classify each pixel of the image as foreground text or background using a neural network, the neural network refining the segmentation prediction using a key vector representing features of the foreground text portion, wherein the key vector is based on the segmentation prediction; and identify the foreground text portion based on the classification.

[0007] A method, apparatus, and non-transient computer-readable medium for training a neural network for text segmentation are described. Embodiments of the method, apparatus, and non-transient computer-readable medium are configured to: identify a training set comprising training images and ground-based segmentation information for each training image, wherein the ground-based segmentation information identifies foreground text portions and background portions; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each training image; compute a first loss function based on a comparison of the segmentation prediction and the ground-based segmentation information; identify a key vector representing features of the foreground text portions based on the segmentation prediction; classify each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each training image; compute a second loss function based on a comparison of the refined segmentation prediction and the ground-based segmentation information; and update the parameters of the neural network based on the first and second loss functions.

[0008] An apparatus and method for text segmentation are described. Embodiments of the apparatus and method include: an encoder configured to encode an image to generate a feature map, wherein the image includes a foreground text portion and a background portion; a first decoder configured to decode the feature map to generate a segmentation prediction; a key generator configured to identify a key vector representing features of the foreground text portion based on the segmentation prediction; an attention component configured to generate a combined feature map based on the key vector; and a second decoder configured to generate a refined segmentation prediction based on the combined feature map. Attached Figure Description

[0009] Figure 1 Examples of systems for text segmentation according to various aspects of this disclosure are shown.

[0010] Figure 2 Examples of processes for text segmentation according to various aspects of this disclosure are shown.

[0011] Figure 3 and Figure 4 Examples of text segmentation according to various aspects of this disclosure are shown.

[0012] Figure 5 Examples of text segmentation apparatuses according to various aspects of this disclosure are shown.

[0013] Figure 6 Examples of processes for text segmentation according to various aspects of this disclosure are shown.

[0014] Figure 7 Examples of processes for generating segmentation predictions according to various aspects of this disclosure are shown.

[0015] Figures 8 to 9 Examples of processes for generating key vectors according to various aspects of this disclosure are shown.

[0016] Figure 10 Examples of processes for generating refined segmentation predictions according to various aspects of this disclosure are shown.

[0017] Figure 11 An example of a process for training a neural network for text segmentation is shown, according to various aspects of this disclosure.

[0018] Figure 12 Examples of processes for expanding the training set according to various aspects of this disclosure are shown. Detailed Implementation

[0019] This disclosure describes systems and methods for performing text segmentation. Embodiments of this disclosure perform initial segmentation, then refine the segmentation by identifying features (e.g., texture) common to portions of an image temporarily identified as text. In some examples, key vectors representing text features are combined with an attention component to encode the image (i.e., features of the entire image). According to some embodiments, a machine learning model is trained using multiple training tasks based on the initial segmentation and refined segmentation.

[0020] Image segmentation is an important component of computer vision. Text segmentation is a subset of image segmentation focused on identifying text, and it has many important applications. For example, optical character recognition (OCR) tools locate text portions before identifying text content.

[0021] Some image segmentation tools generate bounding boxes to identify regions containing text. However, for downstream text-related applications such as style transfer, text removal, and image editing, simply knowing the approximate location of the text (i.e., using bounding boxes) may not be sufficient. Such applications depend on more complex subsequent image processing steps. Therefore, text segmentation tools that can segment images at the pixel level are needed.

[0022] Furthermore, traditional segmentation models are not specifically designed for text segmentation and therefore are unaware of the unique visual attributes of text. That is, text possesses unique visual attributes that differ from other objects (such as faces, body parts, buildings, or natural objects). For example, text may have weaker spatial connectivity compared to regular objects. Additionally, the semantic information associated with foreground textures of text may vary from image to image. Therefore, the similarity of text features between different images may be lower than that between other objects, while the similarity of text features within a single image may be higher than that between other objects.

[0023] Therefore, traditional image segmentation tools are prone to errors when classifying or segmenting text. For example, curves and decorations around text may be incorrectly segmented as text, or only a portion of the text may be correctly identified, while other parts with similar colors and textures may be lost.

[0024] Embodiments of this disclosure include improved text segmentation tools that address technical problems of conventional networks. For example, the text segmentation tool of this disclosure employs key features that identify the text portions of an image, and then uses these features to generate non-traditional steps for refined text segmentation. By identifying features common to the text portions of an image, the improved text segmentation tool can provide more accurate results at the pixel level. For example, text portions that would be missed by conventional segmentation tools are correctly classified as text, and non-text portions of the image that are not text are not classified as text.

[0025] Embodiments of this disclosure include: a neural network having a deep learning backbone network for text segmentation; and a network extension that uses the output from the backbone network and refines its output using a custom query-key attention mechanism to improve classification accuracy. Embodiments of this disclosure process regular fonts and artistically designed text patterns embedded in images (i.e., rather than specifically processing regular objects in everyday scenes). Depending on the application and the user, the text patterns can be of various types and textures.

[0026] In some embodiments, the neural network includes an interchangeable encoder network and a custom query-key attention mechanism that compares foreground text information on an image. The neural network of this disclosure overcomes the challenges of recognizing text images. For example, some text regions lack strong spatial prior knowledge, and their texture has relatively little semantic meaning. In some embodiments, the neural network includes unconventional techniques integrating channel direction cosine similarity re-thresholding, foreground feature pooling, and query-key attention. Therefore, the improved neural network enhances the accuracy of text segmentation.

[0027] System Overview

[0028] Figure 1 Examples of systems for text segmentation according to various aspects of this disclosure are shown. The illustrated examples include a user 100, a user device 105, a text segmentation apparatus 110, a cloud 135, and a database 140. In one embodiment, the text segmentation apparatus 110 includes a processor unit 115, a memory unit 120, an image application 125, and a neural network 130. However, in some embodiments, the components and functions of the text segmentation apparatus 110 reside on the user device 105.

[0029] User 100 can select an image that includes foreground text, and text segmentation device 110 can identify the foreground text (e.g., using a machine learning model). User 100 can then edit or otherwise use the text identified by text segmentation device 110.

[0030] User 100 can communicate with text segmentation device 110 via user equipment 105 and cloud 135. User equipment 105 can be a personal computer, laptop computer, mainframe computer, handheld computer, personal assistant, mobile device or any other suitable processing device.

[0031] Processor unit 115 is an intelligent hardware device (e.g., a general-purpose processing component, digital signal processor (DSP), central processing unit (CPU), graphics processing unit (GPU), microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), programmable logic device, discrete gate or transistor logic component, discrete hardware component, or any combination thereof). In some cases, processor unit 115 is configured to operate a memory array using a memory controller. In other cases, the memory controller is integrated into processor unit 115. In some cases, processor unit 115 is configured to execute computer-readable instructions stored in memory to perform various functions. In some embodiments, processor unit 115 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0032] Examples of memory cells 120 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of memory cells 120 include solid-state memory and hard disk drives. In some examples, memory cell 120 is used to store computer-readable, computer-executable software containing instructions that, when executed, cause a processor to perform the various functions described herein. In some cases, memory cell 120 includes a basic input / output system (BIOS) that controls the operation of underlying hardware or software, such as interaction with peripheral components or devices. In some cases, a memory controller operates the memory cells. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within memory cell 120 store information in the form of logical states.

[0033] A cloud 135 is a computer network configured to provide on-demand availability of computer system resources, such as data storage and computing power. In some examples, a cloud 135 provides resources without active management by the user. The term cloud 135 is sometimes used to describe a data center available to many users via the Internet. Some large cloud networks have the capability to be distributed across multiple locations from a central server. If a server has a direct or close connection to a user, it is specifically referred to as an edge server. In some cases, a cloud 135 is limited to a single organization. In other examples, a cloud 135 can be used by many organizations. In one example, a cloud 135 comprises a multi-tiered communication network that includes multiple edge routers and a core router. In another example, a cloud 135 is based on a local collection of switches in a single physical location.

[0034] Database 140 is an organized collection of data. For example, database 140 stores data in a specified format called a schema. Database 140 can be structured as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage the storage and processing of data in database 140. In some cases, user 100 interacts with the database 140 controller. In other cases, the database controller can operate automatically without user interaction. In some examples, database 140 includes the selection of images that include foreground text.

[0035] According to some embodiments, image application 125 receives an image including a foreground text portion and a background portion. Text segmentation device 110 identifies the foreground text portion based on classification from neural network 130. In some examples, image application 125 modifies the texture of the foreground text portion to produce a modified image.

[0036] According to some embodiments, image application 125 identifies initial training images. In some examples, image application 125 identifies a foreground text portion and a background portion for each initial training image in the initial training images. In some examples, image application 125 modifies the initial training images based on the identified foreground text portions to generate additional training images, wherein the training set includes the initial training images and the additional training images. In some examples, image application 125 replaces the background text portion while retaining the foreground text portion.

[0037] In some examples, the image application 125 includes a graphics editor (e.g. Paint or (Photoshop). Users can select a digital image and input it into a neural network for text segmentation using the image application 125. Users can also use the image application 125 to edit the output image from the neural network (i.e., the processed image with the foreground text already identified).

[0038] In some embodiments, the text segmentation apparatus 110 includes an artificial neural network (ANN). An ANN is a hardware or software component that includes a number of connected nodes (i.e., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge sends a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, it processes the signal and then sends the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is calculated as a function of the sum of its inputs. Each node and edge is associated with one or more node weights that determine how the signal is processed and sent.

[0039] During training, these weights are adjusted to improve the accuracy of the results (i.e., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). Edge weights increase or decrease the signal strength transmitted between nodes. In some cases, nodes have a threshold below which signals are not transmitted at all. In some examples, nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal traverses certain layers multiple times.

[0040] In some embodiments, the text segmentation apparatus 110 includes a convolutional neural network (CNN). For example, an encoder for identifying image features may include convolutional layers. CNNs are a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNNs can process digital images with minimal preprocessing. A characteristic of CNNs may be the use of convolutional (or cross-correlation) hidden layers. These layers apply a convolutional operation to the input before signaling the result to the next layer. Each convolutional node can process data within a finite input domain (i.e., receptive field). During the forward pass of the CNN, the filters of each layer may be convolved over the input volume, calculating the dot product between the filter and the input. During the training process, the filters can be modified so that they activate when specific features within the input are detected.

[0041] According to some embodiments, the neural network 130 classifies each pixel of an image as foreground text or background, and uses a key vector representing features of the foreground text portion to refine the segmentation prediction, wherein the key vector is based on the segmentation prediction.

[0042] According to some embodiments, the neural network 130 classifies each pixel of a training image as foreground text or background to generate a segmentation prediction for each training image. In some examples, the neural network 130 computes a first loss function based on a comparison between the segmentation prediction and ground truth segmentation information. The neural network 130 computes a second loss function based on a comparison between the thinning segmentation prediction and ground truth segmentation information. The parameters of the neural network 130 are updated based on the first and second loss functions. In some cases, the neural network is also referred to as a text thinning network.

[0043] Embodiments of this disclosure provide a deep learning neural network. The neural network includes an interchangeable encoder network and a custom query-key attention mechanism that propagates and compares textual foreground information across an image. In embodiments, the neural network includes integrated channel orientation cosine similarity re-thresholding, foreground feature pooling, and query-key attention. Therefore, the neural network improves the accuracy of classification and segmentation predictions.

[0044] According to embodiments, the neural network includes two components: a backbone network for segmentation prediction and a network extension that refines its predictions using the output from the backbone network and a custom query-key attention mechanism. The backbone network is an interchangeable encoder network and provides segmentation results for text regions. In some examples, the backbone network includes ResNet101, DeepLabV3+, or another suitable image processing network. However, embodiments of this disclosure are not limited to the choice of backbone network. The neural network has no restrictions on the choice of backbone network, and users can choose a backbone network based on their subtraction task and computing environment.

[0045] Figure 2 Examples of processes for text segmentation according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0046] In operation 200, the user selects an image that includes a text foreground. The user can upload the digital image to an image application (e.g.,...). (Photoshop). In some examples, the user selects a digital image from a database or the cloud. A text foreground is embedded in the scene and includes text patterns such as English letters. According to examples, the text foreground includes artistic or curvilinear fonts, where the letters of the custom font are symmetrical, smooth, and flowing. In some cases, this step involves referencing... Figure 1The user described, or the entity that can perform the action.

[0047] In operation 205, the system performs text segmentation on the image. This system includes a neural network that can classify and segment objects associated with the text foreground at the pixel level. The neural network is capable of segmenting various types of text patterns, such as regular fonts, artistic fonts, and custom text designs embedded in an image selected by the user. In some cases, this step involves referencing... Figure 1 The neural network described, or that can be executed by it.

[0048] In operation 210, the system identifies the text foreground. The text foreground of the image is segmented, saved, and output by the system. The neural network discards non-textual information or objects that do not fall into text segmentation, such as the surrounding environment and objects not labeled as text. Unlike input images where text and surrounding environment are mixed and not obvious, users can easily view the text foreground from the system's output image. In some cases, this step involves referencing... Figure 1 The image described is an application, or can be executed by it.

[0049] In operation 215, the user edits the text foreground of the image. The user can use image applications (e.g., Photoshop continues editing the text foreground of the image. In some examples, the user changes the text content and / or text pattern (e.g., using a different style font) based on the identified text foreground portion of the image. In some cases, this step involves referencing... Figure 1 The user described, or the entity that can perform the action.

[0050] Text Segmentation

[0051] Figure 3 Examples of text segmentation according to various aspects of this disclosure are shown. The examples shown include an original image 300 and a text foreground 305. In some cases, the original image 300 includes a collection of objects from a scene. Some objects include text. According to the example, the original image 300 includes a highway exit sign (drivers know this is an exit sign because of the capitalized text "EXIT ONLY" posted on the exit sign. The exit sign displays the exit number assigned to the fork in the road (e.g., 20A) and the destination if the driver exits at this fork (e.g., Likelike Hwy). In addition to the text, the original image also includes a scene about the surrounding environment, such as rooftops of houses, foliage, sky, and the structure where the exit sign is installed.

[0052] The text foreground 305 includes only the text information in the image. The text foreground 305 does not include the surrounding environment in the scene that is not labeled as text. In the example above, the text foreground 305 includes regular fonts, artistic fonts, or a combination of both (e.g., “EXIT 20A”, “NORTH”, “Likelike Hwy”, and “EXIT ONLY” are shown in the text foreground).

[0053] In some embodiments, the original image 300 is used for training and thus input into the neural network. The text foreground 305 is ground truth annotation. The loss function is calculated between the segmentation prediction based on the original image and the ground truth annotation. In some cases, the original image 300 is input into the neural network for inference.

[0054] Figure 4 Examples of text segmentation according to various aspects of this disclosure are shown. The examples shown include an original image 400 and a text foreground 405. The original image 400 includes text or text patterns with a font specified by the user. In some cases, the text patterns include artistic designs from the user. According to the example, the original image 400 includes text in an unconventional artistic font held by a person's hand. The text includes "Holiday calories not counted." In some cases, the text font is not easily recognizable as letters by the user because the text is mixed with other non-textual information in the scene and / or the letters have curved shapes, making the text less obvious to the user. Non-textual surroundings exist in the original image, such as streets, umbrellas, and houses. The user is interested in understanding the exact content of the text (e.g., the spelling of each word in a phrase) and can continue editing the identified text from the original image.

[0055] The text foreground 405 contains only text information. The text foreground 405 does not contain surrounding environment information that is not labeled as text within the scene. In the example above, the text foreground 405 includes the text stating "No calories counted during holidays." This text is in an artistic font (e.g., some English letters have curves or a custom font that makes the letters symmetrical, smooth, and flowing).

[0056] In some embodiments, the original image 400 is used for training and thus input into the neural network. The text foreground 405 is ground truth annotation. The loss function is calculated between the segmentation prediction based on the original image and the ground truth annotation. In some cases, the original image 400 is input into the neural network for inference.

[0057] Architecture

[0058] Figure 5Examples of text segmentation apparatuses according to various aspects of this disclosure are shown. The examples shown include an original image 500, an encoder 505, a feature map 510, a first decoder 515, a segmentation prediction 520, a key generator 525, an attention component 530, a combined feature map 535, a second decoder 540, a thinning segmentation prediction 545, a text foreground 550, a first loss 555, and a second loss 560.

[0059] According to some embodiments, the neural network (i.e., the network model) includes a backbone network and a thinning extension. The backbone network is an interchangeable encoder network selected by the user based on their task and computing environment. The backbone network is also referred to as encoder 505. The thinning extension is a query key attention mechanism that processes the query key from the (output) feature map 510 of the backbone network. According to embodiments, the neural network includes encoder 505, a first decoder 515, a key generator 525, an attention component 530, and a second decoder 540. In some cases, the neural network is also referred to as a text thinning network or model.

[0060] The original image 500 is the input image to the encoder 505. In some cases, the original image is referred to as input image x. The original image 500 includes text or text patterns in a specific font. In some cases, the text patterns include artistic designs from the user. For example, such as... Figure 5 The illustrated original image 500 includes a human hand holding text in an unconventional, artistic font that declares "Holiday Calories Not Counted." In some cases, the font of the text is not easily recognizable to the user. The original image contains non-textual elements such as streets, umbrellas, and houses. The user is interested in understanding the exact content of the text (e.g., the spelling) and can continue editing the identified text from the original image. Original image 500 is a reference. Figure 3 and Figure 4 Examples of the corresponding elements described, or including aspects thereof.

[0061] According to some embodiments, encoder 505 encodes the input image x to produce feature map 510. Feature map 510 is represented as x f Encoder 505 encodes images of the training images to produce feature maps 510. For example, one of the training images is the original image 500 (i.e., input image x). In some embodiments, encoder 505 is configured to encode images to produce feature maps 510, wherein the images include foreground text portions and background portions. In some examples, encoder 505 includes a ResNet architecture. In some other examples, encoder 505 includes a DeepLabV3+ architecture. However, embodiments of this disclosure are not limited to the choice of encoder network.

[0062] According to some embodiments, a first decoder 515 decodes the feature map 510 to produce a segmentation prediction 520. The segmentation prediction 520 is represented as x. sem In some examples, the first decoder 515 includes a convolutional layer, a first bias, and a first softmax. The first decoder 515 applies the convolutional layer to the feature map 510. The first decoder 515 applies the first bias to the output of the convolutional layer. The first decoder 515 applies the first softmax to the output of the first bias.

[0063] According to an embodiment, the model computes the segmentation prediction x via additional convolutional layers with bias and softmax layers. sem x sem Using cross-entropy loss L sem (i.e., the first loss is 555) is supervised by ground reality annotations y. Since the model directly uses x... sem Calculate L with y sem Therefore, it strongly supervises the parameters of encoder 505.

[0064] According to some embodiments, the key generator 525 identifies the key vector based on the segmentation prediction 520. In some examples, the key generator 525 calculates the cosine similarity of the segmentation prediction 520. The key generator 525 applies a second bias based on the cosine similarity. The key generator 525 applies a second softmax to the output of the second bias. The key generator 525 combines the second softmax with the feature map 510. The key generator 525 applies a pooling layer to produce the key vector.

[0065] According to some embodiments, key generator 525 identifies key vectors representing features of foreground text portions based on segmentation prediction 520. In some examples, key generator 525 includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer. Key generator 525 computes the cosine similarity of segmentation prediction 520. Key generator 525 applies the second bias based on the cosine similarity. Key generator 525 applies the second softmax to the output of the second bias. Key generator 525 combines the second softmax with feature map 510. Key generator 525 applies pooling layers to produce key vectors.

[0066] According to the embodiment, the model re-thresholds x sem And the prediction that only has a high confidence level remains initiated is represented as The modified segmentation prediction. This is achieved by analyzing the generated x... sem The convolutional layer adds a new bias, and the model performs this re-thresholding step. This new bias is related to x. sem The cosine similarity of the channel directions is linearly proportional.

[0067]

[0068] According to an embodiment, x sem The channel direction cosine similarity is inversely proportional to its prediction accuracy. Therefore, when x sem When it becomes inaccurate, its channel direction cosine similarity increases, and the bias on the background channel increases. This is equivalent to reducing the bias when passing through softmax. The foreground region is initiated. Therefore, only foreground predictions with high confidence are used. Keep it running.

[0069] Then, the model pools. Feature vector v within the high confidence region k This pooling is used to represent the foreground texture of the text. f and It is executed by weighted sum of the values.

[0070]

[0071] Here, N represents the number of pixels in the image, thus preserving only the channel dimension (C). By doing so, the model not only aggregates text foreground information into a single image (v...). k And it reduced v k Correlation with backbone (i.e., encoder) parameters between different images.

[0072] According to some embodiments, attention component 530 classifies each pixel of a training image as foreground text or background based on a key vector to generate a refined segmentation prediction 545 for each training image. In some examples, attention component 530 combines the key vector and feature map 510 to generate an attention map. In some examples, attention component 530 combines the attention map and feature map 510 to generate a combined feature map 535. In some examples, the combined feature map 535 includes color information for each pixel of the image. According to some embodiments, attention component 530 is configured to generate the combined feature map 535 based on the key vector. In some examples, attention component 530 includes a second product function and a cascade function.

[0073] According to the embodiment, the model uses v k Use x as a key f As a query, and by calculating the attention x through the dot product att Since the dot product is performed on all pixel locations, texture information is globally propagated to all parts of the image.

[0074] According to some embodiments, the second decoder 540 decodes the combined feature map 535 to produce a refined segmentation prediction 545, wherein the foreground text portion is identified based on the refined segmentation prediction 545. According to some embodiments, the second decoder 540 is configured to produce the refined segmentation prediction 545 based on the combined feature map 535. In some examples, the second decoder 540 includes convolutional layers, batch normalization layers, and ReLU layers. The refined segmentation prediction 545 is represented as x rfn .

[0075] According to an embodiment, the model cascades the input image x and the feature map x. f And the newly calculated attention x att The new matrix is ​​then passed through two additional conv5x5 layers and one conv1x1 layer to obtain the final refined result x. rfn x rfn During training time, cross-entropy loss L is used. rfn (i.e., the second loss of 560) is monitored by ground-based annotations y. rfn This serves as the model's final output during inference time (i.e., when making predictions using the model). The final loss L (i.e., the second loss 560) is L sem and L rfn The weighted sum.

[0076]

[0077]

[0078] L = w sem L sem +w rfn L rfn (5)

[0079] According to an embodiment, the model can be trained end-to-end. During the training process, the hyperparameters are set as follows, for example, L sem and L rfn The weights are set to 1. The training length is set to 20,500 total iterations. The learning rate is 0.01, and it decays slowly from 0.01 to 0 using a 0.9 polygon rule. A 500-iteration warm-up is used at the start of training. One embodiment of this disclosure uses an ImageNet pre-trained model as the initial model. No hyperparameters are required during inference time.

[0080] In some cases, the text foreground 550 is a ground-based image (represented as y). For example, the text foreground 550 includes textual information (or a query) stating "No calories counted during holidays." The text foreground 550 may use regular fonts, artistic fonts, or a combination of both. Unlike the original image, the text foreground 550 does not contain non-textual information (such as contextual information or surrounding environment). The text foreground 550 is a reference. Figure 3 and Figure 4 Examples of the corresponding elements described, or including aspects thereof.

[0081] inference

[0082] Figure 6 Examples of processes for text segmentation according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0083] In operation 600, the system receives an image including a foreground text portion and a background portion. In some cases, this step involves referencing... Figure 1 The image application described, or that can be executed by it. Image applications include graphics editors (e.g., Paint or Photoshop).

[0084] According to some embodiments, the received image includes regular fonts and / or artistic text patterns embedded in the image. In some cases, the image can be used to train the neural network during the training phase, thus being input into the neural network. In other cases, the image is used for inference, and the neural network generates text segmentation results based on the image.

[0085] In operation 605, the system uses a neural network to classify each pixel of the image as either foreground text or background. This neural network uses a key vector representing features of the foreground text portion to refine the segmentation prediction, where the key vector is based on the segmentation prediction. In some cases, this step involves referencing... Figure 1 The neural network described, or that can be executed by it.

[0086] According to embodiments, the neural network includes a backbone network and network extensions built on the outputs from the backbone network. The backbone network includes a CNN model (e.g., ResNet-101, DeepLabV3+). DeepLabV3+ is a CNN model for semantic segmentation, including an encoder-decoder network. ResNet-101 is a CNN model with 101 layers. In some examples, the ResNet-101 network is pre-trained on the ImageNet database. However, embodiments of this disclosure are not limited to the choice of backbone network. There are no restrictions on the choice of backbone network for the neural network, and users can choose the backbone network based on their task and computing environment.

[0087] In some cases, the backbone network is referred to as the encoder. Network extensions use a query-key attention mechanism to refine the segmentation predictions from the backbone network. The neural network can resolve the text foreground from an image at the pixel level. Therefore, the accuracy of text segmentation is improved.

[0088] According to an embodiment, the neural network includes an encoder (i.e., a backbone network), a first decoder, a key generator, an attention component, and a second decoder. The encoder is configured to encode an image to produce a feature map, wherein the image includes a foreground text portion and a background portion. The first decoder is configured to decode the feature map to produce a segmentation prediction. The key generator is configured to identify a key vector representing features of the foreground text portion based on the segmentation prediction. The attention component is configured to generate a combined feature map based on the key vector. The second decoder is configured to generate a refined segmentation prediction based on the combined feature map.

[0089] In operation 610, the system identifies foreground text portions based on classification. Users can use a graphical editor (e.g., Paint or Use Photoshop to continue editing the foreground text section. In some cases, this step involves referencing... Figure 1 The image described is an application, or can be executed by it.

[0090] Initial segmentation

[0091] Figure 7 Examples of processes for generating segmentation predictions according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0092] In operation 700, the system applies one or more convolutional layers to the feature map. The convolutional layers in a convolutional neural network apply learned filters to the input image to generate the feature map. In some cases, this step involves referencing... Figure 5 The first decoder is described, or may be executed by it. According to some embodiments, the encoder is configured to encode an image to produce a feature map, wherein the image includes a foreground text portion and a background portion. The first decoder is configured to decode the feature map to produce a segmentation prediction. In some cases, the encoder includes a ResNet architecture. The first decoder includes convolutional layers, a first bias, and a first softmax.

[0093] In operation 705, the system applies a first bias to the output of the convolutional layer. One or more first bias values ​​are used to shift the neural network's startup function to the left or right for model learning. In some examples, changing the model's weights can alter the steepness of the startup function. The entire curve of the startup function can be shifted to the left or right using the first bias. In some cases, this step involves referencing... Figure 5 The first decoder described, or that can be executed by it.

[0094] In operation 710, the system applies a first softmax to the output of the first bias. The first softmax is also referred to as the first softmax function. According to an embodiment, the first softmax function is used as a startup function for the neural network to normalize the network's output to a probability distribution over the predicted output class. After applying the first softmax function, each component of the feature map is in the interval (0,1), and the components sum to 1. These values ​​are interpreted as probabilities. In some cases, this step involves referencing... Figure 5 The first decoder described, or that can be executed by it.

[0095] Text feature identifier

[0096] Figure 8 Examples of processes for generating key vectors according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to methods and procedures described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0097] In operation 800, the system encodes the image to produce a feature map. The neural network includes a backbone network (i.e., also referred to as an encoder). The encoder extracts the feature map from the input image. In some cases, this step involves referencing... Figure 5 The encoder described, or the one that can be executed by it.

[0098] In operation 805, the system decodes the feature map to produce a segmentation prediction. The encoder computes the segmentation prediction using a first decoder. The first decoder includes a convolutional layer, a first bias, and a first softmax. In some cases, this step involves referencing... Figure 5 The first decoder described, or that can be executed by it.

[0099] In operation 810, the system identifies key vectors based on segmentation prediction. The key generator is configured to identify key vectors representing features of foreground text portions based on segmentation prediction. According to an embodiment, the key generator includes a cosine similarity function, a second bias, a second softmax function, a first product function, and a pooling layer.

[0100] According to an embodiment, the neural network performs a complex re-thresholding of the segmentation prediction and obtains a modified segmentation prediction. This re-thresholding step is performed by adding a new bias to the convolutional layers that generate the segmentation prediction. This new bias is linearly proportional to the channel orientation cosine similarity of the segmentation prediction. According to an embodiment, the channel orientation cosine similarity of the segmentation prediction is inversely proportional to its prediction accuracy. Therefore, when the segmentation prediction becomes inaccurate, its channel orientation cosine similarity increases, and the bias on the background channels increases. When applied via softmax, this effectively reduces the activation of foreground regions in the modified segmentation prediction. Therefore, only foreground predictions with high confidence remain activated in the modified segmentation prediction.

[0101] Then, the neural network pools the feature vectors within the high-confidence regions of the modified segmentation prediction to represent the foreground texture of the text. This pooling is performed through a simple weighted sum between the feature maps and the modified segmentation prediction. The neural network not only aggregates the text foreground information into a feature vector within a single image but also reduces the correlation between the feature vector and the backbone parameters across different images. In some cases, the feature vector is also represented as a reference... Figure 5 The described key vector. In some cases, this step involves referencing... Figure 5 The described key generator, or one that can be executed by it.

[0102] Figure 9Examples of processes for generating key vectors according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to methods and procedures described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0103] In operation 900, the system calculates the cosine similarity of the segmentation predictions. In some cases, this step involves referencing... Figure 5 The described key generator, or one that can be executed by it.

[0104] According to an embodiment, the neural network sophisticatedly re-thresholds the segmentation predictions and obtains modified segmentation predictions that only keep the foreground predictions with high confidence active. This re-thresholding is performed by adding a new bias to the convolutional layers that generate the segmentation predictions. This new bias is linearly proportional to the channel direction cosine similarity of the segmentation predictions.

[0105] According to the embodiment, the channel orientation cosine similarity of the segmentation prediction is inversely proportional to its prediction accuracy. Therefore, when the segmentation prediction becomes inaccurate, its channel orientation cosine similarity increases, and the bias on the background channel increases. When the softmax function is applied, this effectively reduces the activation of foreground regions in the modified segmentation prediction. Thus, only foreground predictions with high confidence remain activated in the modified segmentation prediction.

[0106] In operation 905, the system applies the second bias based on cosine similarity. In some cases, this step involves referencing... Figure 5 The described key generator, or one that can be executed by it.

[0107] In operation 910, the system applies a second softmax to the output of the second bias. In some cases, this step involves referencing... Figure 5 The described key generator, or one that can be executed by it.

[0108] In operation 915, the system combines the second softmax with the feature map. According to an embodiment, the modified segmentation prediction and feature map are combined and fed into a pooling layer. In some cases, this step involves referencing... Figure 5 The described key generator, or one that can be executed by it.

[0109] In operation 920, the system applies pooling layers to generate key vectors. Pooling layers are new layers added after convolutional layers. Pooling layers are used to order the layers within a convolutional neural network and can be repeated once or multiple times within the neural network model. Pooling layers operate individually on each feature map to create a new set of pooled feature maps of the same number. Pooling includes selection pooling operations. In some cases, the size of the pooling operation is smaller than the size of the feature map. For example, the size of the pooling operation is 2x2 pixels with a stride of 2 pixels. Average pooling or max pooling operations can be used. In some cases, this step involves referencing... Figure 5 The described key generator, or one that can be executed by it.

[0110] According to the embodiment, the feature vector v in the high-confidence region of the segmentation prediction after model pooling modification is... k (i.e., key vectors) to represent the foreground texture of the text. This pooling is performed by a simple weighted sum between the feature maps and the modified segmentation predictions. The weighted sum is formulated as follows: Here, N represents the number of pixels in the image, so it only preserves the channel size (C). Neural networks not only summarize text foreground information into a single image's v... k And it reduced v k Correlation of backbone parameters between different images.

[0111] Refine segmentation

[0112] Figure 10 Examples of processes for generating refined segmentation predictions according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0113] In operation 1000, the system combines the key vector and feature map to generate an attention map. According to an embodiment, the neural network uses v k The key vector is used, and the feature map is used as the query. An attention map is computed via dot products. In some cases, the dot products are performed on all pixel locations, and texture information is globally applied to all parts of the image to emphasize areas where image features resemble text features. In some cases, this step involves referencing... Figure 5 The attention component described, or that can be executed by it.

[0114] In operation 1005, the system combines the attention map and the feature map to produce a combined feature map. According to an embodiment, the attention component is configured to generate the combined feature map based on a key vector. The attention component includes a second product function and a cascade function. According to an embodiment, the neural network cascades the input image (i.e., the original image), the feature map, and the newly computed attention map. The resulting matrix is ​​the combined feature map. In some cases, this step involves referencing... Figure 5 The attention component described, or that can be executed by it.

[0115] In operation 1010, the system decodes the combined feature map to generate a refined segmentation prediction, wherein the foreground text portion is based on the refined segmentation prediction identifier. According to an embodiment, a second decoder is configured to generate the refined segmentation prediction based on the combined feature map. The second decoder includes convolutional layers, batch normalization layers, and ReLU layers.

[0116] According to an embodiment, the neural network passes the combined feature map through two additional conv5x5 layers and one conv1x1 layer to obtain the final thinning result. The final thinning result is also referred to as the thinned segmentation prediction. In some cases, this step involves referencing... Figure 5 The second decoder described, or which can be executed by it.

[0117] train

[0118] Figure 11 Examples of processes for training a neural network for text segmentation according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0119] In operation 1100, the system identifies a training set, which includes training images and ground truth segmentation information for each training image, wherein the ground truth segmentation information identifies the foreground text portion and the background portion. In some cases, the foreground text portion includes only text information. According to an example, the foreground text portion declares "Holiday calories not counted." The text has an unconventional artistic font. Some letters in the text have curved shapes. In some cases, this step involves referencing... Figure 1 The image described is an application, or can be executed by it.

[0120] In operation 1105, the system classifies each pixel of the training image as either foreground text or background to generate a segmentation prediction for each training image. In some cases, this step involves referencing... Figure 1 The neural network described, or that can be executed by it.

[0121] According to an embodiment, the neural network uses an encoder to extract feature maps from the input image and also computes segmentation predictions via additional convolutional layers and softmax layers with bias. The output of the softmax layer or softmax function is the segmentation prediction, where each component (or element) of the segmentation prediction is in an interval of (0,1). These (0,1) values ​​corresponding to the respective components are interpreted as probabilities, such as the probability of a pixel in the training image being considered as foreground text or background.

[0122] In operation 1110, the system calculates the first loss function based on a comparison between segmentation prediction and ground reality segmentation information. Segmentation prediction is performed using cross-entropy loss L. sem The ground-based annotations (i.e., the first loss function) are used for supervision. Since the neural network directly uses segmentation prediction and ground-based segmentation information to compute the first loss function L... sem Therefore, it provides strong supervision over all backbone parameters. In some cases, this step involves referencing... Figure 1 The neural network described, or that can be executed by it.

[0123] In operation 1115, the system identifies a key vector representing features of the foreground text portion based on segmentation prediction. According to an embodiment, a key generator is configured to identify a key vector representing features of the foreground text portion based on segmentation prediction. The key generator includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer. In some cases, this step involves referencing... Figure 5 The described key generator, or one that can be executed by it.

[0124] In operation 1120, the system classifies each pixel of the training image as foreground text or background based on the key vector to generate a refined segmentation prediction for each training image. In some cases, this step involves referencing... Figure 5 The attention component described, or that can be executed by it.

[0125] According to an embodiment, the neural network uses a key vector and a feature map as queries, and computes an attention map by the dot product of the key vector and the feature map. In some cases, the dot product is performed on all pixel locations, and texture information is globally applied to all parts of the image to emphasize parts of the image whose features are similar to text features. The model cascades the input image, feature map, and newly computed attention map, and passes this new matrix through two additional conv5x5 layers and one additional conv1x1 layer to obtain refined segmentation predictions.

[0126] In operation 1125, the system calculates a second loss function based on a comparison of refined segmentation prediction and ground reality segmentation information. According to the embodiment, the refined segmentation prediction uses a cross-entropy loss L during training time. rfn (That is, the second loss function) is used for supervision, and it acts as the first output of the neural network (i.e., for prediction) during inference time. The final loss L is L sem and L rfn (That is, a weighted sum of the first and second loss functions). In some cases, this step involves referencing... Figure 1 The neural network described, or that can be executed by it.

[0127] In operation 1130, the system updates the parameters of the neural network based on the first loss function and the second loss function. In some cases, this step involves referencing... Figure 1 The neural network described, or that can be executed by it.

[0128] According to an embodiment, the neural network can be trained end-to-end. During the training phase, the hyperparameters are set as follows, for example, L... sem and L rfn The weights are 1. The training length is set to 20,500 total iterations. The learning rate is 0.01, and it decays slowly from 0.01 to 0 using a 0.9 polygon rule. A 500-iteration warm-up is used at the start of training. In some cases, the neural network uses a pre-trained ImageNet model as the initial model for text segmentation. The encoder of the neural network is a pre-trained backbone network (i.e., the ImageNet weights and parameters are pre-trained for segmentation and classification). According to the embodiment, the encoder of the neural network has been trained to identify textures, edges, etc. During end-to-end training of the neural network, the encoder weights are updated (i.e., the ImageNet backbone weights are not fixed). No hyperparameters are needed during inference time.

[0129] Figure 12Examples of processes for expanding a training set according to various aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to various aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0130] In operation 1200, the system identifies the initial training images. In some cases, each image in the initial training images contains one or more objects in a scene (e.g., an everyday scene such as a highway exit). For example, each image includes various types of textures, edges, and text patterns. Text patterns can have regular fonts, artistic fonts, custom text designs, or combinations thereof. In some cases, this step involves referencing... Figure 1 The image described is an application, or can be executed by it.

[0131] In operation 1205, the system identifies the foreground text portion and the background portion for each initial training image in the initial training images. Images containing only the foreground text portion are referred to as ground truth images or ground truth annotations. In some cases, the segmentation prediction from the neural network is supervised using ground truth images with a cross-entropy loss function. In other cases, this step involves referencing... Figure 1 The image described is an application, or can be executed by it.

[0132] In operation 1210, the system modifies the initial training image based on the identified foreground text portion to generate additional training images, where the training set includes both the initial and additional training images. In some examples, the additional training images do not have the same text pattern as the initial training images. For example, the additional training images have different textures and background colors. Random background swapping is another way to modify the initial training images. In some cases, multi-scale training is applied to create a very large training set. In some cases, this step involves referencing... Figure 1 The image described is an application, or can be executed by it.

[0133] Evaluate

[0134] The results from the neural network (i.e., the text refinement network) were compared with the baseline model DeepLabV3+. When compared with results from DeepLabV3+, the segmentation results from the neural network showed a substantial qualitative improvement in prediction accuracy. Evaluation metrics included foreground cross over joint (fgIoU) to evaluate the segmentation model. The quantitative improvement using the neural network was documented. The effectiveness of different parts of the neural network was demonstrated in an ablation study. Further experimental results using two other public text segmentation datasets (COCOTS and ICDAR17) were documented. For example, the baseline model achieved an fgIoU score of 85.25% on the custom text segmentation dataset. The neural network of this disclosure improved the fgIoU score to 88.58%. In one example, the baseline model achieved an fgIoU score of 71.13% on the COCOTS dataset. The neural network of this disclosure improved the fgIoU score to 73.89%. In another example, the baseline model achieved an fgIoU score of 80.77% on the ICDAR17 dataset. The neural network disclosed herein improves the fgIoU score to 85.09%.

[0135] Example Implementation

[0136] Therefore, this disclosure includes at least the following embodiments.

[0137] A method for text segmentation is described. An embodiment of the method is configured to: receive an image comprising a foreground text portion and a background portion; classify each pixel of the image as foreground text or background using a neural network, the neural network refining the segmentation prediction using a key vector representing features of the foreground text portion, wherein the key vector is based on the segmentation prediction; and identify the foreground text portion based on the classification.

[0138] An apparatus for text segmentation is described. The apparatus includes a processor, a memory in electronic communication with the processor, and instructions stored in the memory. The instructions are operable to cause the processor to: receive an image comprising a foreground text portion and a background portion; classify each pixel of the image as foreground text or background using a neural network, the neural network refining the segmentation prediction using a key vector representing features of the foreground text portion, wherein the key vector is based on the segmentation prediction; and identify the foreground text portion based on the classification.

[0139] A non-transient computer-readable medium is described for storing code for text segmentation. In some examples, the code includes processor-executable instructions to: receive an image comprising a foreground text portion and a background portion; classify each pixel of the image as foreground text or background using a neural network, the neural network refining the segmentation prediction using a key vector representing features of the foreground text portion, wherein the key vector is based on the segmentation prediction; and identify the foreground text portion based on the classification.

[0140] Some examples of the methods, apparatuses, and non-transient computer-readable media described above also include: encoding an image to generate a feature map. Some examples also include: decoding the feature map to generate a segmentation prediction. Some examples also include: identifying a key vector based on the segmentation prediction.

[0141] Some examples of the methods, apparatuses, and non-transient computer-readable media for decoding feature maps include applying a convolutional layer to the feature map. Some examples also include applying a first bias to the output of the convolutional layer. Some examples further include applying a first softmax to the output of the first bias.

[0142] Some examples of the methods, apparatuses, and non-transient computer-readable media used for identifying key vectors include: computing cosine similarity of segmentation predictions; applying a second bias based on the cosine similarity; applying a second softmax to the output of the second bias; combining the second softmax with a feature map; and applying pooling layers to produce the key vector.

[0143] Some examples of the methods, apparatuses, and non-transient computer-readable media described above further include: combining a key vector and a feature map to generate an attention map. Some examples also include: combining an attention map and a feature map to generate a combined feature map. Some examples further include: decoding the combined feature map to generate a refined segmentation prediction, wherein the foreground text portion is based on a refined segmentation prediction identifier.

[0144] Some examples of the methods, apparatuses and non-transient computer-readable media described above also include: modifying the texture of a foreground text portion to produce a modified image.

[0145] A method for training a neural network for text segmentation is described. An embodiment of the method is configured to: identify a training set comprising training images and ground-based segmentation information for each training image, wherein the ground-based segmentation information identifies foreground text portions and background portions; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each training image; compute a first loss function based on a comparison of the segmentation prediction and the ground-based segmentation information; identify a key vector representing features of the foreground text portions based on the segmentation prediction; classify each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each training image; compute a second loss function based on a comparison of the refined segmentation prediction and the ground-based segmentation information; and update the parameters of the neural network based on the first and second loss functions.

[0146] An apparatus for training a neural network for text segmentation is described. The apparatus includes a processor, a memory in electronic communication with the processor, and instructions stored in the memory. The instructions are operable to cause the processor to: identify a training set including training images and ground-based segmentation information for each training image, wherein the ground-based segmentation information identifies foreground text portions and background portions; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each training image; compute a first loss function based on a comparison of the segmentation prediction and the ground-based segmentation information; identify a key vector representing features of the foreground text portions based on the segmentation prediction; classify each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each training image; compute a second loss function based on a comparison of the refined segmentation prediction and the ground-based segmentation information; and update the parameters of the neural network based on the first and second loss functions.

[0147] A non-transient computer-readable medium is described, storing code for training a neural network for text segmentation. In some examples, the code includes processor-executable instructions to: identify a training set comprising training images and ground-based segmentation information for each training image, wherein the ground-based segmentation information identifies foreground text portions and background portions; classify each pixel of the training images as foreground text or background to generate a segmentation prediction for each training image; compute a first loss function based on a comparison of the segmentation prediction and the ground-based segmentation information; identify a key vector representing features of the foreground text portions based on the segmentation prediction; classify each pixel of the training images as foreground text or background based on the key vector to generate a refined segmentation prediction for each training image; compute a second loss function based on a comparison of the refined segmentation prediction and the ground-based segmentation information; and update the parameters of the neural network based on the first and second loss functions.

[0148] Some examples of the methods, apparatuses, and non-transient computer-readable media described above also include: encoding images of training images to generate feature maps. Some examples also include: decoding the feature maps to generate segmentation predictions.

[0149] Some examples of the methods, apparatuses, and non-transient computer-readable media for decoding feature maps include applying a convolutional layer to the feature map. Some examples also include applying a first bias to the output of the convolutional layer. Some examples further include applying a first softmax to the output of the first bias.

[0150] Some examples of the methods, apparatuses, and non-transient computer-readable media used for identifying key vectors include: computing cosine similarity of segmentation predictions; applying a second bias based on the cosine similarity; applying a second softmax to the output of the second bias; combining the second softmax with a feature map; and applying pooling layers to produce the key vector.

[0151] Some examples of the methods, apparatuses, and non-transient computer-readable media described above further include: combining a key vector and a feature map to generate an attention map. Some examples also include: combining an attention map and a feature map to generate a combined feature map. Some examples further include: decoding the combined feature map to generate a refined segmentation prediction, wherein the foreground text portion is based on a refined segmentation prediction identifier.

[0152] In some examples, the combined feature map includes color information for each pixel of the image.

[0153] Some examples of the methods, apparatus, and non-transient computer-readable media described above further include: identifying initial training images. Some examples further include: for each initial training image in the initial training images, identifying foreground text portions and background portions. Some examples further include: modifying the initial training images based on the identified foreground text portions to generate additional training images, wherein the training set includes the initial training images and the additional training images.

[0154] Some examples of the methods, apparatuses and non-transient computer-readable media used to modify initial training images include replacing background text portions while retaining foreground text portions.

[0155] An apparatus for text segmentation is described. Embodiments of the apparatus include: an encoder configured to encode an image to generate a feature map, wherein the image includes a foreground text portion and a background portion; a first decoder configured to decode the feature map to generate a segmentation prediction; a key generator configured to identify a key vector representing features of the foreground text portion based on the segmentation prediction; an attention component configured to generate a combined feature map based on the key vector; and a second decoder configured to generate a refined segmentation prediction based on the combined feature map.

[0156] A method for providing an apparatus for text segmentation is described. The method includes providing: an encoder configured to encode an image to generate a feature map, wherein the image includes a foreground text portion and a background portion; a first decoder configured to decode the feature map to generate a segmentation prediction; a key generator configured to identify a key vector representing features of the foreground text portion based on the segmentation prediction; an attention component configured to generate a combined feature map based on the key vector; and a second decoder configured to generate a refined segmentation prediction based on the combined feature map.

[0157] In some examples, the encoder includes a ResNet architecture. In some examples, the first decoder includes convolutional layers, a first bias, and a first softmax. In some examples, the key generator includes a cosine similarity function, a second bias, a second softmax, a first product function, and a pooling layer. In some examples, the attention component includes a second product function and a concatenated function. In some examples, the second decoder includes convolutional layers, a batch normalization layer, and a ReLU layer.

[0158] The descriptions and accompanying drawings herein represent exemplary configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or otherwise modified. Furthermore, structures and devices may be represented in block diagram form to illustrate relationships between components and to avoid confusion with the described concepts. Similar components or features may have the same name but may have different reference numerals corresponding to different drawings.

[0159] Some modifications to this disclosure may be apparent to those skilled in the art, and the principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is given the broadest scope consistent with the principles and novel features disclosed herein.

[0160] The described methods can be implemented or performed by devices including general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. A processor can also be implemented as a combination of computing devices (e.g., a combination of DSPs and microprocessors, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Therefore, the functions described herein can be implemented in hardware or software and can be executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on a computer-readable medium in the form of instructions or code.

[0161] Computer-readable media include non-transient computer storage media and communication media, including any medium that facilitates the transfer of code or data. Non-transient storage media can be any available medium that can be accessed by a computer. For example, non-transient computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc (CD) or other optical disc storage devices, magnetic disk storage devices, or any other non-transient medium used to carry or store data or code.

[0162] Furthermore, the connecting components can be appropriately referred to as computer-readable media. For example, if code or data is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology are included in the definition of media. Combinations of media are also included within the scope of computer-readable media.

[0163] In this disclosure and the following claims, the word "or" indicates a list of inclusion properties, such that a list of, for example, X, Y, or Z represents X or Y or Z or XY or XZ or YZ or XYZ. Furthermore, the phrase "based on" is not used to indicate a conditionally closed set. For example, a step described as "based on condition A" could be based on conditions A and B. In other words, the phrase "based on" should be interpreted as meaning "at least partially based on". Moreover, the words "a" or "an" indicate "at least one".

Claims

1. A method for text segmentation, comprising: Receive an image, the image including a foreground text portion and a background portion; The image is encoded to generate a feature map; Based on the feature map, a segmentation prediction of the image is generated; A key vector representing the features of the foreground text portion is generated based on the segmentation prediction and the feature map; The key vector and the feature map are combined to generate an attention map; as well as Using a neural network, based on the attention map and the feature map, a refined segmentation prediction is generated by classifying each pixel of the image as foreground text or background.

2. The method according to claim 1, further comprising: The feature map is decoded to generate the segmentation prediction; as well as The key vector is identified based on the segmentation prediction.

3. The method according to claim 2, wherein decoding the feature map comprises: Apply the convolutional layer to the feature map; Apply the first bias to the output of the convolutional layer; as well as Apply the first softmax to the output of the first bias.

4. The method of claim 2, wherein identifying the key vector comprises: Calculate the cosine similarity of the segmentation prediction; The second bias is applied based on the cosine similarity; Apply the second softmax to the output of the second bias; Combine the second softmax with the feature map; as well as A pooling layer is applied to generate the key vector.

5. The method according to claim 1, further comprising: The attention map and the feature map are combined to generate a combined feature map; as well as The combined feature map is decoded to generate the refined segmentation prediction, wherein the foreground text portion is identified based on the refined segmentation prediction.

6. The method according to claim 1, further comprising: Modify the texture of the foreground text portion to produce a modified image.

7. A method for text segmentation, comprising: Receive an image, the image including a foreground text portion and a background portion; The image is encoded to generate a feature map; The feature map is decoded to generate a segmentation prediction; Based on the segmentation prediction identifier key vector, wherein the key vector represents the features of the foreground text portion; The key vector and the feature map are combined to generate an attention map; The attention map and the feature map are combined to generate a combined feature map; The combined feature map is decoded to generate a refined segmentation prediction; as well as The foreground text portion is identified based on the refined segmentation prediction.

8. The method according to claim 7, further comprising: The image is classified into foreground text or background using a neural network, which refines the segmentation prediction using the key vector.

9. The method of claim 7, wherein decoding the feature map comprises: Apply the convolutional layer to the feature map; Apply the first bias to the output of the convolutional layer; as well as Apply the first softmax to the output of the first bias.

10. The method of claim 7, wherein identifying the key vector comprises: Calculate the cosine similarity of the segmentation prediction; The second bias is applied based on the cosine similarity; Apply the second softmax to the output of the second bias; Combine the second softmax with the feature map; as well as A pooling layer is applied to generate the key vector.

11. The method of claim 7, further comprising: The segmentation prediction is re-thresholded to obtain a modified segmentation prediction, wherein the key vector is identified based on the modified segmentation prediction.

12. The method of claim 11, further comprising: A weighted sum is calculated based on the feature map and the modified segmentation prediction, wherein the key vector is identified based on the weighted sum.

13. The method according to claim 11, wherein: The modified segmentation prediction includes foreground prediction of the foreground text portion.

14. An apparatus for text segmentation, comprising: One or more processors; as well as One or more memories, including instructions executable by the one or more processors to: Receive an image, the image including a foreground text portion and a background portion; The image is encoded to generate a feature map; Based on the feature map, a segmentation prediction of the image is generated; A key vector representing the features of the foreground text portion is generated based on the segmentation prediction and the feature map; The key vector and the feature map are combined to generate an attention map; as well as Based on the attention map and the feature map, a refined segmentation prediction is generated by classifying each pixel of the image as foreground text or background.

15. The apparatus of claim 14, wherein the instructions are further executable to: Decode the feature map to generate the segmentation prediction; and The key vector is identified based on the segmentation prediction.

16. The apparatus of claim 15, wherein decoding the feature map comprises: Apply the convolutional layer to the feature map; Apply the first bias to the output of the convolutional layer; as well as Apply the first softmax to the output of the first bias.

17. The apparatus of claim 15, wherein identifying the key vector comprises: Calculate the cosine similarity of the segmentation prediction; The second bias is applied based on the cosine similarity; Apply the second softmax to the output of the second bias; Combine the second softmax with the feature map; as well as A pooling layer is applied to generate the key vector.

18. The apparatus of claim 14, wherein the instructions are further executable to: Combine the attention map and the feature map to generate a combined feature map; and The combined feature map is decoded to generate the refined segmentation prediction, wherein the foreground text portion is identified based on the refined segmentation prediction.

19. The apparatus of claim 14, wherein the instructions are further executable to: Modify the texture of the foreground text portion to produce a modified image.

Images