A scene text magnification method based on text segmentation

Abstract

A scene text magnification method based on text segmentation belongs to the technical field of image text processing. It includes the following steps: S1: text segmentation, the text is segmented and extracted from the given scene image to obtain the text image and text position information; S2: scene text erasing, the specified scene text is erased from the given scene image as a scene background image; S3: text magnification, using the text image and text position information obtained in step S1, specifying the text to be magnified, keeping the relative position unchanged, and magnifying by a certain proportion to obtain the magnified text image; S4: image fusion, that is, embedding the magnified text image into the scene background image according to certain rules to obtain the output image. The specified text in the scene is effectively magnified without affecting the background, which can help special groups with myopia or reading disabilities better understand the scene, and can be used for augmented reality applications.

Description

Technical Field

[0001] The present invention belongs to the technical field of text processing of images, and in particular relates to a scene text magnification method based on text segmentation that uses semantic segmentation as an auxiliary task to obtain text position information of real scene text images. Background Art

[0002] Text is an essential visual element in everyday life, providing rich information across diverse scenarios. For example, information such as guide signs, product prices and advertisements, and posters and book titles aids comprehension. However, text located far from the capture location is relatively small, making it difficult to read. Small characters printed on newspapers and other documents are difficult for people with myopia to discern. This challenge is even greater for people with dyslexia. Therefore, enlarging text in images can help improve understanding and prevent omissions and misunderstandings.

[0003] Due to the uniqueness of text image features, the current scene text editing methods mainly process text images based on three levels.

[0004] The first level is based on text image editing in the text domain and is suitable for scene text erasure. Nakamura et al. proposed a neural network based on a single-scale sliding window to erase scene text in "Scene text eraser". Liu et al. proposed EraseNet in "EraseNet: End-to-end text removal in the wild" by introducing a coarse-to-fine architecture and an additional segmentation head to help locate text. Tang et al. proposed predicting the strokes of text on word images in "Stroke-based scene text erasing using synthetic data for training". Both the strokes and the image are input into the image inpainting network proposed in "Image inpainting for irregular holes using partial convolutions" to generate text erasure results.

[0005] The second level involves editing text at the character level. In "STEFANN: Scene text editor using font adaptive neural network," Roy et al. proposed a method for font-adaptive scene text image editing, pioneering the direct editing of scene text images. Although Roy et al.'s work was limited to replacing only one character at a time, the method is highly insightful, pioneering the direct replacement of text within scene text images and representing the most representative work on character-based text replacement methods for scene text images.

[0006] The third level involves editing text at the word level. The SRNet method proposed by Wu et al. in "Editing Text in the Wild" pioneered a method for editing text images at the word level, using the word skeleton to control the main structure of the word. This work is divided into three submodules: a text conversion module, a background image inpainting module, and a foreground and background fusion module. Although Wu et al.'s method works well for simpler text effect images, it produces poor results when the text in the scene has complex effects.

[0007] Nakamura et al. proposed a text magnification method for scene text images in "Scene text magnifier". However, the text image of this method is obtained by comparing the original scene text image and the scene text erased image, and the text position information cannot be obtained. The implemented text magnification requires magnifying the text in the entire image and changes the original structure of the text, so it cannot be used for complex text styles. Summary of the Invention

[0008] The present invention aims to address the aforementioned issues with existing technologies by providing a method for scene text magnification based on text segmentation. This method, utilizing text location information obtained through semantic segmentation, can magnify specific text without altering the original structure of the text image. This method is applicable to scenarios with complex scene text styles. This method can help people with myopia or dyslexia better understand the scene and can be used in applications such as augmented reality.

[0009] A method for scene text magnification based on text segmentation includes the following steps:

[0010] S1: Text segmentation, segmenting and extracting text from a given scene image to obtain text image and text location information;

[0011] S2: scene text erasure, erasing the specified scene text from the given scene image as the scene background image;

[0012] S3: Text magnification: using the text image and text position information obtained in step S1, specify the text to be magnified, keep the relative position unchanged, and magnify it by a certain ratio to obtain an enlarged text image;

[0013] S4: Image fusion, that is, embedding the enlarged text image into the scene background image according to certain rules to obtain the output image.

[0014] The invention uses a four-stage process, with training samples from each stage processed through a pipeline that takes the original image as input and produces a text-erased image, a text image, an enlarged text image, and a text-enlarged scene image as output. This effectively enlarges scene text without affecting the background, enabling individuals with myopia or dyslexia to better understand the scene and potentially enabling applications such as augmented reality.

[0015] In step S1: In the text region segmentation task, the text region detector of the attention text detection module adopts a pyramid mask text detector, and uses an attention mechanism for text region detection, which can better achieve text segmentation.

[0016] Furthermore, DeeplabV3+, based on Resnet101, performs segmentation. By using different receptive fields and upsampling, it can achieve multi-scale feature extraction, which is suitable for the presence of text of different sizes in images. DeeplabV3+ uses depthwise separable convolution, which can reduce the number of parameters and improve the computational efficiency of the network.

[0017] Furthermore, due to the particularity of text symbols, a refined enhancement process is performed during the segmentation process, using a small-scale multi-layer perceptron to independently predict each point in the generated mask boundary area, thereby generating a smoother and more accurate mask.

[0018] Furthermore, the Loss of the text segmentation part is L sem , as shown below:

[0019] L sem =L sem1 +αL out +βL tri (1)

[0020] Among them, the input image is x, the features are extracted by DeeplabV3+ backbone network, and the predicted segmentation map x is obtained by activation function sem1 , and the true segmentation map x gt The cross entropy loss is as follows:

[0021] L sem1 =CrossEntropy(x sem1 , xgt ) (2)

[0022] In addition, the predicted segmentation map obtained by the enhancement module is x out , and the true segmentation map x gt The cross entropy loss is as follows:

[0023] L out =CrossEntropy(x out , x gt ) (3)

[0024] Similarly, the boundary of the text is very important to the segmentation effect, so here L tri Indicates a weighted cross entropy loss using text boundaries, where w tri is the binary mapping value of the text boundary area in the image. The WCE operation is the cross entropy loss between x and y with w as the weight.

[0025] L tri =WCE(x out , x gt , w tri ) (4)

[0026]

[0027] In step S2: the real scene image and the text mask image are input, and the output is a text-free scene background image, in which all text stroke pixels, including text shadows caused by lighting, are replaced with appropriate textures.

[0028] Among them, the global context module is adopted as the self-attention module, and the self-attention module is used for scene text erasure. Four loss functions are introduced to measure the structural and texture differences between the output image and the real text-free scene background image. The spatial smoothness during the training of the text erasure module is also taken into account, including pixel reconstruction loss, perceptual loss, style loss and total change loss.

[0029] In step S3: the steps of enlarging the text area are as follows:

[0030] 1) Based on the text image and text position information obtained from the text segmentation network, the text in the continuous area is grouped together, and the position of each text area is determined, including the coordinates of the upper left corner, lower left corner, and upper right corner of the rectangular box of the area;

[0031] 2) Draw a rectangle of the text area, determine the center point, and select 1.5 times, 2 times, 3 times, and other values ​​to enlarge the text area. After enlarging, the text area needs to be judged. If the text area is out of bounds, the enlarged position needs to be processed. Otherwise, it is processed based on the center point to obtain an enlarged text image.

[0032] In step S4: OpenCV is used to implement image fusion. The fusion method is as follows:

[0033] Fusion image: I_out = I_b*α+I_t*β+γ

[0034] This function implements the weighted sum of the linear coefficients of the two input images; the first parameter: I_b, represents the background image object for the weighted operation;

[0035] The second parameter: α, represents the weighting coefficient of the background image;

[0036] The third parameter: I_t, represents the text image object for weighted operation;

[0037] The fourth parameter: β, represents the weighting coefficient of the text image;

[0038] The fifth parameter: γ, represents a scalar acting on the weighted sum image, which is the offset of the weighted sum image;

[0039] Fused image: I_out, which represents the weighted sum of the two images. Its size and image type are the same as I_b and I_t.

[0040] The present invention edits text at the word level without changing the original structure of the text. It is applicable to scenarios with complex scene text styles. By using the text position information obtained by text segmentation, the present invention can achieve the effect of magnifying specified text. The first two stages of the present invention are based on a neural network model of deep learning, and the last two stages are based on the OpenCV method, which is improved for the scenarios used in the present invention. Effectively magnifying the specified text in the scene without affecting the background can help special groups with myopia or dyslexia better understand the scene, and can be used for applications such as augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of an embodiment of the present invention.

[0042] Figure 2 4 is a flow chart of a text segmentation module according to an embodiment of the present invention.

[0043] Figure 3 Schematic diagram of visualization results of each stage of text segmentation according to an embodiment of the present invention.

[0044] Figure 4 2 is a flow chart of a text erasing module according to an embodiment of the present invention.

[0045] Figure 5 3 is a schematic diagram of the visualization results of the scene text erasing stage in an embodiment of the present invention.

[0046] Figure 6 2 is a schematic diagram of image fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to more clearly illustrate the method flow in the embodiment of the present invention, the following embodiments will further illustrate the present invention with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0048] The present invention can amplify the specified text content of an input image based on text position information while retaining the background and overall structural style of the input image, including the steps of:

[0049] S1: Text segmentation: First, the text is segmented and extracted from the given scene image to obtain the text image and text location information;

[0050] S2: Scene text erasure, erasing the specified scene text from the given scene image as the scene background image;

[0051] S3: Text magnification: using the text image and text position information obtained in step S1, specify the text to be magnified, keep the relative position unchanged, and magnify it by a certain ratio to obtain an enlarged text image;

[0052] S4: Image fusion, that is, embedding the enlarged text image into the scene background image according to certain rules to obtain the output image.

[0053] Example 1

[0054] refer to Figure 1 and combined Figure 3 and 5 Visualization result example, Figure 1 This is a flowchart of a method for scene text magnification based on text segmentation. In the figure, "Step 1" is S1. The text pixel instance is the text image after the input image passes through the text segmentation module and the scene text background is removed. At the same time, we also obtain text position information. The "magnified text instance" is a text image of the same size that maintains the style and relative position of the text in the input image but is relatively magnified. The details are as follows:

[0055] S1: Text segmentation, that is, detecting the area where the text is located from the input image to obtain the text image and text location information.

[0056] In this embodiment, the original scene text image contains the characters "Danshan Road", "Xinhua Road" and the English translation "Danshan Road", "Xinhua Road", and the scene text is segmented to serve as a text image.

[0057] The process of the text segmentation module is as follows Figure 2 In this embodiment, the original scene image contains multiple lines of text with different color pixels. Figure 2 The text detection module, Deeplabv3+, and enhancement module shown in the figure are processed to obtain a binary image containing only black and white, where the text pixels are white and the background pixels are black. Then, the text pixels are extracted separately from the scene image by passing the input image and the binary image through the text cutout module to obtain a text image.

[0058] Please refer to Figure 2 and combined Figure 3 The present invention can achieve text segmentation for scene images of different sizes and scene text in multiple languages. To better utilize the performance of the neural network, based on the structure and characteristics of different languages, the present invention proposes a bilingual dataset, training method, and loss function for the neural network example of the present invention, including:

[0059] The public text segmentation dataset TextSeg contains scene text images, semantic labels, and text location information, all of which are manually segmented and extracted from real scene images. It contains text of different sizes and dimensions, but this dataset only contains English and numeric text, not Chinese text.

[0060] Prepare a Chinese dataset, which contains semantic labels and text location information manually annotated from real Chinese text scene images, as well as a large amount of synthetic data obtained by synthesizing text pixels and text-free background images proposed in TextLogo3K.

[0061] The text line area in the real scene is used as the input of the text transfer subnet, and based on the existing annotation of the text content, a text image with the same text content is generated as the content input of the text transfer subnet;

[0062] Calculate the pixel-level loss between the binary image obtained by text segmentation and the result image. The loss of the text segmentation part is L sem , as shown below:

[0063] L sem =L sem1 +αL out +βL tri (1)

[0064] Among them, the input image is x, the features are extracted by DeeplabV3+ backbone network, and the predicted segmentation map x is obtained by activation function sem1 , and the true segmentation map x gt The cross entropy loss is as follows:

[0065] L sem1 =CrossEntropy(x sem1 , x gt ) (2)

[0066] In addition, the predicted segmentation map obtained by the enhancement module is x out , and the true segmentation map x gt The cross entropy loss is as follows:

[0067] L out =CrossEntropy(x out , x gt ) (3)

[0068] Similarly, the boundary of the text is very important to the segmentation effect, so here L tri Indicates a weighted cross entropy loss using text boundaries, where w tri is the binary mapping value of the text boundary area in the image. The WCE operation is the cross entropy loss between x and y with w as the weight.

[0069] L tri =WCE(x out , x gt , w tri ) (4)

[0070]

[0071] Use gradient descent algorithm for optimization.

[0072] S2: Text erasure, that is, erasing the image in the scene text to obtain a scene image without text.

[0073] The flowchart of the scene text erasing module is as follows Figure 4 Considering that the previous text segmentation module has obtained the text mask image, in this module, the input is the real scene image and the text mask image output is the text-free scene background image, in which all the text stroke pixels, including the text shadows caused by lighting, are replaced with appropriate textures.

[0074] Training loss: Four loss functions are introduced to measure the structural and texture differences between the output image and the ground-truth text-free scene background image, while also taking into account the spatial smoothness during the training of the text erasing module, including pixel reconstruction loss, perceptual loss, style loss, and total variation loss.

[0075] Use gradient descent algorithm for optimization.

[0076] Please refer to Figure 4 and combined Figure 5 The present invention can erase text from scene images of different environments to obtain text-free scene images.

[0077] S3: Text enlargement, i.e. enlarging the text pixel instance size while maintaining the original style and text structure.

[0078] There are two ways to achieve text magnification. The first method is to magnify individual characters one by one, which brings about overlapping problems, especially when the character spacing is narrow, and when the characters are irregular, it will change the original structure of the word.

[0079] The second method is to enlarge the text area of ​​the segmented text image by a certain ratio. Through this process, the text can still maintain its original structure and there will be no problem of character overlap.

[0080] The present invention adopts the second method, which uses the traditional method to achieve the enlargement of the text area. The specific steps are as follows:

[0081] 1) Based on the text image and text position information obtained from the text segmentation network, the text in the continuous area is grouped together, and the position of each text area is determined, including the coordinates of the upper left corner, lower left corner, and upper right corner of the rectangular box of the area;

[0082] 2) Draw a rectangle of the text area, determine the center point, and select 1.5 times, 2 times, 3 times, and other values ​​to enlarge the text area. After enlarging, the text area needs to be judged. If the text area is out of bounds, the enlarged position needs to be processed. Otherwise, it is processed based on the center point to obtain an enlarged text image.

[0083] S4: Image fusion, that is, linearly weighting the scene background image and the enlarged text pixel image to obtain the output result.

[0084] Fusion diagram Figure 6 As shown. Use OpenCV to implement image fusion, the fusion method is as follows:

[0085] Fusion image: I_out = I_b*α+I_t*β+γ

[0086] This function implements the weighted sum of the linear coefficients of the two input images; the first parameter: I_b, represents the background image object for the weighted operation;

[0087] The second parameter: α, represents the weighting coefficient of the background image;

[0088] The third parameter: I_t, represents the text image object for weighted operation;

[0089] The fourth parameter: B, represents the weighting coefficient of the text image;

[0090] The fifth parameter: γ, represents a scalar acting on the weighted sum image, which is the offset of the weighted sum image;

[0091] Fused image: I_0ut, which represents the weighted sum of the two images. Its size and image type are the same as I_b and I_t.

[0092] The invention uses a four-stage process, with training samples from each stage processed through a pipeline that takes the original image as input and produces a text-erased image, a text image, an enlarged text image, and a text-enlarged scene image as output. This effectively enlarges scene text without affecting the background, enabling individuals with myopia or dyslexia to better understand the scene and potentially enabling applications such as augmented reality.

[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A scene text magnification method based on text segmentation, characterized in that The steps include: S1: Text segmentation, segmenting and extracting text from a given scene image to obtain text image and text location information; Segment and extract text from a given scene image to obtain text images and text location information. This consists of three modules: an attention text detection module, a DeeplabV3+ text segmentation module based on Resnet101, and a text cutout module. The attention text detection module is used to highlight areas that may contain text; The DeeplabV3+ text segmentation module based on Resnet101 is used to extract text features of the text area from the input image, obtain binary images of the text and background, and text position information; The text cutout module is used to extract text pixels from the binary image obtained by the network, filter out the background part, and obtain a text image; The segmentation loss is L sem , as shown below: L sem =L sem1 +αL out +βL tri (1) Among them, the input image is x, the features are extracted by DeeplabV3+ backbone network, and the predicted segmentation map x is obtained by activation function sem1 , and the true segmentation map x gt The cross entropy loss is as follows: L sem1 =CrossEntropy(x sem1 ,x gt ) (2) In addition, the predicted segmentation map obtained by enhancement is x out , and the true segmentation map x gt The cross entropy loss is as follows: L out =CrossEntropy(x out ,x gt ) (3) The boundary of the text is very important to the segmentation effect. tri Indicates a weighted cross entropy loss using text boundaries, where w tri is the binary mapping value of the text boundary area in the image; the WCE operation is the cross entropy loss between x and y with w as the weight; L tri =WCE(x out ,x gt ,w tri ) (4) S2: scene text erasure, erasing the specified scene text from the given scene image as the scene background image; S3: Text magnification: using the text image and text position information obtained in step S1, specify the text to be magnified, keep the relative position unchanged, and magnify it by a certain ratio to obtain an enlarged text image; S4: Image fusion, that is, embedding the enlarged text image into the scene background image according to certain rules to obtain the output image.

2. A scene text magnification method based on text segmentation as claimed in claim 1, characterized in that: The text region detector of the attention text detection module adopts a pyramid mask text detector and uses an attention mechanism to perform text region detection to better achieve text segmentation.

3. The scene text magnification method based on text segmentation according to claim 1, characterized in that: The DeeplabV3+ text segmentation module based on Resnet101 includes segmentation using DeeplabV3+ based on Resnet101, and multi-scale feature extraction is achieved by using different receptive fields and upsampling to meet the situation where texts of different sizes exist in the image; DeeplabV3+ uses depthwise separable convolution to reduce the number of parameters and improve the computational efficiency of the network.

4. A scene text magnification method based on text segmentation as claimed in claim 2, characterized in that The segmentation process is refined and enhanced by using a small-scale multi-layer perceptron to independently predict each point in the generated mask boundary area, thereby generating a smoother and more accurate mask.

5. The scene text magnification method based on text segmentation as claimed in claim 1, characterized in that In step S2, the scene text erasure inputs a real scene image and a text mask image, and outputs a text-free scene background image, in which all text stroke pixels, including text shadows caused by lighting, are replaced with appropriate textures.

6. A scene text magnification method based on text segmentation as claimed in claim 5, characterized in that The scene text erasure proposed in this paper adopts a global context module as a self-attention module, uses the self-attention module for scene text erasure, and introduces four loss functions to measure the structural and texture differences between the output image and the true text-free scene background image, while also taking into account the spatial smoothness during the training of the text erasure module, including pixel reconstruction loss, perceptual loss, style loss, and total change loss.

7. A scene text magnification method based on text segmentation as claimed in claim 1, characterized in that In step S3, the steps of text amplification are as follows: (1) Based on the text image and text position information obtained from the text segmentation network, the text of the continuous area is grouped together, and the position of each group of text areas, the coordinates of the upper left corner and lower left corner of the area rectangle, and the coordinates of the upper right corner and lower right corner of the rectangle; (2) Draw a rectangle of the text area, determine the center point, and select 1.5 times, 2 times, 3 times, etc. to enlarge the text area. After enlarging, the text area needs to be judged. If the text area is out of bounds, the enlarged position needs to be processed. Otherwise, it needs to be processed based on the center point to obtain an enlarged text image.

8. The method for scene text magnification based on text segmentation as claimed in claim 1, characterized in that In step S4: OpenCV is used to implement image fusion. The fusion method is as follows: Fusion image: I_out = I_b*α+I_t*β+γ This function implements the weighted sum of the linear coefficients of the two input images; the first parameter: I_b, represents the background image object for the weighted operation; The second parameter: α, represents the weighting coefficient of the background image; The third parameter: I_t, represents the text image object for weighted operation; The fourth parameter: β, represents the weighting coefficient of the text image; The fifth parameter: γ, represents a scalar acting on the weighted sum image, which is the offset of the weighted sum image; Fused image: I_out, which represents the weighted sum of the two images. Its size and image type are the same as I_b and I_t.

Images