An image redirection method based on text image content perception

Abstract

The application discloses a kind of based on text image content perception's image redirection method, it is related to the field of image processing in computer vision, which comprises the following steps: step 1, input text image to be adjusted;Step 2, respectively obtain the gradient graph and text center probability graph of the text image;Step 3, the gradient graph and the text center probability graph are fused to generate text energy graph by water flow diffusion method;Step 4, the text energy graph is used as the energy graph of joint carving process, finds minimum energy joint, and according to the target size of pre-established, carries out the insertion or deletion of the minimum energy joint, obtains redirection image;Step 5, output the redirection image.

Description

Technical Field

[0001] This invention relates to the field of image processing in computer vision, and more particularly to an image redirection method based on text image content awareness. Background Technology

[0002] With the widespread adoption of mobile devices and the development of the internet, the number of user-generated text images has increased dramatically. The text within these images often contains rich and important semantic information; for example, the recognition of traffic signs and license plates is crucial for autonomous driving and intelligent monitoring tasks. When displaying these text images on different devices and platforms, it is essential to better preserve the information in the text regions during resizing. In Optical Character Recognition (OCR) tasks, preserving more accurate text information within smaller image sizes is also critical. For instance, convolutional networks need to resize the original image to a fixed, uniform size (e.g., 640×640) before processing graphics. Preserving the text information within this limited size is of great significance for downstream tasks such as character recognition, character detection, and image processing.

[0003] Traditional image resizing methods, such as interpolation and cropping, while capable of adjusting image size, fail to specifically protect text information within the image, leading to decreased or lost text readability. The seam carving method proposed by Shai Avidan et al. can preserve some visually more important details during scaling, but in scenes with complex textures, this method ignores glyphs. This method can focus on parts of the text's edges, but can disrupt the glyph geometry due to cracks penetrating the font's interior.

[0004] In recent years, with the development of artificial intelligence technology, deep learning has continuously empowered the field of image retargeting. Building upon classical adjustment methods, neural networks can extract semantic information from images more comprehensively and accurately to guide the image retargeting process. Related scholars have achieved certain results in both the construction of importance assessment maps and the combined application of various classical methods. Guo Yingchun et al., in their Chinese invention patent application "An Image Retargeting Method Based on Relative Saliency Detection" (application number: CN202111228342.5), extracted a relative saliency map, detected image edges to obtain an edge map, linearly fused the edge map and the relative saliency map to obtain an importance map, and finally learned the displacement mapping from the input image to the target image, achieving image retargeting through the displacement map on the input image. This method mainly focuses on the visual saliency of the image; however, for the special scenario of text images, its saliency detection module may not fully understand the semantics and structure of the text, resulting in the text information not being prioritized for protection during the retargeting process.

[0005] Text images contain rich semantic information, but text in natural scenes presents challenges such as irregular shapes, diverse font formats, and complex backgrounds. Currently, few scholars have researched repositioning methods for text images. Existing methods cannot accurately distinguish the differences in feature distribution between text and non-text; therefore, textual information is easily destroyed during resizing, affecting its accuracy in downstream tasks.

[0006] Therefore, those skilled in the art are dedicated to developing a new image retargeting method for text images to solve the aforementioned problems in the prior art. Summary of the Invention

[0007] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to accurately distinguish the feature distribution differences between text and non-text, so as to effectively maintain the text content in the image during the resizing of the text image.

[0008] To achieve the above objectives, the present invention provides an image retargeting method based on text image content awareness, comprising the following steps:

[0009] Step 1: Input the text image to be adjusted;

[0010] Step 2: Obtain the gradient map and text center probability map of the text image respectively;

[0011] Step 3: Fuse the gradient map and the text center probability map using the water flow diffusion method to generate a text energy map;

[0012] Step 4: Use the text energy map as the energy map to guide the seam engraving process, find the minimum energy seam, and insert or delete the minimum energy seam according to the preset target size to obtain the retargeting image;

[0013] Step 5: Output the redirected image.

[0014] Furthermore, step 2 includes the following sub-steps:

[0015] Step 2.1: Apply the gradient operator convolution to obtain the gradient map;

[0016] Step 2.2: Apply a character-level text detection network to obtain the text center probability map.

[0017] Furthermore, the gradient operator in step 2.1 includes, but is not limited to, the Sobel operator, the Prewitt operator, or the Laplacian operator.

[0018] Furthermore, step 3 includes the following sub-steps:

[0019] Step 3.1: Normalize the gradient map and the text center probability map, then add them together with weights to obtain the initial distribution of the water flow:

[0020]

[0021] in, This is the merged water flow distribution map. This is the text center probability map. It is the gradient map, The fusion operation represents the diffusion of water flow. The weights of the text center probability graph are... The weights of the gradient graph, For normalization processing;

[0022] Step 3.2, according to The pixel values ​​in the data are sorted from high to low, and then the water level of all pixels is updated in the sorted order. One iteration is completed after all pixels have been updated.

[0023] Step 3.3 Middle position The direction of water level adjustment for a pixel is the pixel with the largest water level difference among its four neighboring pixels, that is:

[0024]

[0025] in, It is a location The water level height of the pixel at that location; It is the water level height of a certain pixel within its four neighboring pixels. , The values ​​are -1, 0, and 1; It is the water level height of the pixel with the largest difference in water level height among its four neighboring pixels. Indicates all possible In the combination, find the expression within the parentheses that maximizes the result. combination, It makes When the maximum value is obtained combination;

[0026] When finding the pixel with the largest difference in water level among its four neighbors, the water levels of the two pixels are made uniform. The position in the t-th iteration... The adjustment value for the water level height at that location is:

[0027]

[0028] After adjustment, the water level height of the two pixels is updated as follows:

[0029]

[0030] Each iteration is equivalent to performing a four-neighbor diffusion operation;

[0031] Step 3.4: Repeat the iterations in Steps 3.2 and 3.3 until the water level gradually stabilizes, then stop the iterations, i.e., when the condition is met. The text energy map is obtained, wherein, This is a pre-set iterative convergence threshold.

[0032] Further, in step 3.1, the weights of the text center probability graph... The weights greater than those in the gradient graph .

[0033] Furthermore, weight Weight .

[0034] Furthermore, the energy function in step 4 Defined as:

[0035]

[0036] The size of the input text energy map is Define vertical joints respectively and horizontal joints are :

[0037]

[0038]

[0039] Among them, vertical joints It is a pixel sequence from top to bottom of an image, with a horizontal seam. It is a sequence of pixels from left to right in an image; for an n×m image n is the number of rows, m is the number of columns, and the vertical seam is... Composed of n pixels, with a horizontal seam Composed of m pixels, It is the pixel of the vertical seam in the i-th row. x(i) is the pixel coordinate of the vertical seam in the i-th row, with column index x(i) and row index i. It is the pixel in the j-th column of the horizontal seam. It is the pixel coordinate of the horizontal seam in column j, column index j, row index j. , This means that for all i, the following condition is satisfied: This ensures that the column index change between adjacent rows does not exceed 1. This ensures that the row index change of adjacent columns does not exceed 1.

[0040] Furthermore, the minimum energy seam in step 4 is to use a dynamic programming algorithm to find a path with the minimum sum of energy values ​​from the vertical or horizontal seams of the text energy map, that is, the seam with the minimum energy.

[0041] For vertical joints, define From top to The cumulative energy of the minimum energy path at a location, and for horizontal seams, rotating the text energy map 90° to treat it as a vertical seam, processing it, and then rotating it back.

[0042] Furthermore, if it is the backward energy method, M The expression form is:

[0043]

[0044] pass The recursive formula is used to calculate the cumulative energy map of the entire text energy map, and finally the point with the minimum cumulative energy is found from the bottom row. The path of the entire seam is obtained by backtracking, which is the minimum energy seam. This represents the pixel value at the i-th row and j-th column position of the text energy map.

[0045] Furthermore, if the forward energy method is used, the cost increase due to the new adjacency relationships added after removing the current pixel is as follows:

[0046]

[0047] in, From The increased cost after removing the crack from the left side. From The increased cost after removing the crack coming from the right side. From The increased cost after removing the cracks coming from the upper side; This represents the pixel value at the (i)th row and (j+1)th column of the text energy map. This represents the pixel value at the i-th row and j-1-th column of the text energy map. This represents the pixel value at the (i-1)th row and jth column of the text energy map;

[0048] Therefore, the cumulative cost matrix of the forward energy method for:

[0049]

[0050] pass The recursive formula is used to calculate the cumulative energy map of the entire text energy map, and finally find the point with the minimum cumulative energy from the bottom row. The path of the entire seam is obtained by backtracking, which is the minimum energy seam.

[0051] The image retargeting method based on text image content awareness provided by this invention has at least the following technical effects:

[0052] 1. The technical solution provided by this invention can accurately protect text content. By introducing text energy map and character-level text detection model, it can accurately identify the importance of text regions and prioritize the protection of text content during image size adjustment, avoiding accidental deletion or distortion of text, and significantly improving the quality of text image retargeting.

[0053] 2. The technical solution provided by this invention offers an innovative feature map fusion method. By combining feature maps using the water flow diffusion method and drawing on the John Stokes equations of fluid mechanics, it can better simulate the energy distribution of text regions and generate text energy maps that better conform to the text structure. Compared with traditional methods, this method can more accurately guide image retargeting operations;

[0054] 3. The technical solution provided by this invention can provide an efficient and universal solution. Thanks to the shape robustness of character-level text detection, it performs well on various types of text images such as natural scene text and document images, and has wide applicability. At the same time, the computational complexity of this method is relatively low, making it suitable for scenarios with high real-time requirements and giving it high practical value.

[0055] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0056] Figure 1 This is a flowchart of a preferred embodiment of the present invention;

[0057] Figure 2 This is a method overview diagram of a preferred embodiment of the present invention;

[0058] Figure 3 yes Figure 1 A schematic diagram of the water diffusion fusion process of the preferred embodiment is provided. Detailed Implementation

[0059] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0060] This invention is specifically designed for text images. By introducing a text energy map, it precisely protects text information. When adjusting the image size, a non-uniform size adjustment is performed based on the text energy map, prioritizing the preservation of text areas and preventing accidental deletion or distortion of text. It is particularly suitable for scenarios such as natural scene text and document images, and has significant application value in the field of image scaling, especially in scenarios where maintaining the integrity of image content is required.

[0061] Example 1

[0062] like Figure 1 and Figure 2 As shown, an image retargeting method based on text image content awareness provided in an embodiment of the present invention includes the following steps:

[0063] Step 1: Input the text image to be adjusted;

[0064] Step 2: Obtain the gradient map and text center probability map of the text image respectively;

[0065] Step 3: Fuse the gradient map and the text center probability map using the water flow diffusion method to generate a text energy map;

[0066] Step 4: Use the text energy map as the energy map to guide the seam carving process, find the minimum energy seam, and insert or delete the minimum energy seam according to the pre-set target size to obtain the retargeting image;

[0067] Step 5: Output the redirected image.

[0068] Specifically, step 2 includes the following sub-steps:

[0069] Step 2.1: Apply gradient operators to convolution to obtain the gradient map; wherein, the gradient operators include, but are not limited to, the Sobel operator, the Prewitt operator, or the Laplacian operator.

[0070] Step 2.2: Apply a character-level text detection network to obtain a text center probability map. The core of the character-level text detection network CRAFT (Character Region Awareness for Text Detection) is its region awareness capability. Its network structure is as follows: the feature extraction layer uses a pre-trained VGG16 network for downsampling to extract deep semantic features. Then, a network structure similar to U-Net is used to concatenate the deep features with the shallow features. Image resolution is restored through convolution and deconvolution operations. Finally, a score map of the text center region is output, where the score of each pixel represents the probability that the point belongs to the text region.

[0071] Specifically, the character-level text detection network CRAFT is trained offline as an independent module when applied in the method of this embodiment, and its output is used in subsequent processes.

[0072] Specifically, CRAFT's training strategies generally include:

[0073] 1) Weakly supervised learning: Real datasets usually only have word-level annotations. CRAFT uses a weakly supervised learning method to generate character-level annotations from word-level annotations.

[0074] 2) Data augmentation: Apply techniques such as cropping, rotation, and color transformation during training to improve the robustness of the model;

[0075] 3) Training datasets: including SynthText from the pre-training phase and real datasets such as ICDAR 2013, ICDAR 2015, and ICDAR 2017.

[0076] Example 2

[0077] Building upon Example 1, a text center probability map and a gradient map are fused to obtain a text energy map. The gradient map is analogous to the ground in the water flow process, while the text center probability map is the water falling on that ground. A higher value at a pixel in the text energy map indicates a higher initial water level at that pixel. During the flow, the water follows the principle of "water flows downhill," and the new water height distribution formed after the flow region stabilizes is the fused text energy map.

[0078] The specific integration steps are as follows: Figure 3 As shown. Step 3 includes the following sub-steps:

[0079] Step 3.1: Normalize the gradient map and the text center probability map, then add them together with weights to obtain the initial distribution of the water flow:

[0080]

[0081] in, This is the merged water flow distribution map. It is a text center probability graph. It is a gradient map. The fusion operation represents the diffusion of water flow. The weights for the text center probability graph. The weights of the gradient graph, For normalization processing; weights of text center probabilities and gradient weights It is used to flexibly adjust the emphasis of text details on other details;

[0082] Specifically, based on practical experience, the weights of the text center probability graph... It is usually slightly larger than the weights of the gradient map. Specifically, setting weights. Weight .

[0083] Step 3.2: In each iteration, according to The pixel values ​​in the data are sorted from high to low, and then the water level of all pixels is updated in the sorted order. One iteration is completed after all pixels have been updated.

[0084] Step 3.3: In each iteration, Middle position The direction of water level adjustment for a pixel is the pixel with the largest water level difference among its four neighboring pixels, that is:

[0085]

[0086] in, It is a location The water level height of the pixel at that location; It is the water level height of a certain pixel within its four neighboring pixels. , The values ​​are -1, 0, and 1; It is the water level height of the pixel with the largest difference in water level height among its four neighboring pixels. Indicates all possible In the combination, find the expression within the parentheses that maximizes the result. combination, It makes When the maximum value is obtained combination;

[0087] When finding the pixel with the largest difference in water level among its four neighbors, the water levels of the two pixels are made uniform. The position in the t-th iteration... The adjustment value for the water level height at that location is:

[0088]

[0089] This adjustment is to ensure that the water level does not fall below the terrain height. After the adjustment, the water level of the two pixels is updated as follows:

[0090]

[0091] Each iteration is equivalent to performing a four-neighbor diffusion operation. The water level adjustment here uses the simplest and easiest-to-implement form—averaging the water levels of two pixels. However, more complex variations are possible, such as adjustments with finer granularity.

[0092] Step 3.4: Repeat the iterations in steps 3.2 and 3.3 until the water level gradually stabilizes, then stop the iterations, i.e., when the condition is met. The text energy map is obtained, where, This is a pre-set iterative convergence threshold.

[0093] Example 3

[0094] Based on Example 2, the energy function in step 4 Defined as:

[0095]

[0096] Energy function It is the pixel value at the i-th row and j-th column of the energy map.

[0097] The size of the input text energy map is Define vertical joints respectively and horizontal joints are :

[0098]

[0099]

[0100] in, It is a sequence of pixels from the top to the bottom of an image. It is a sequence of pixels from left to right of the image. The condition in the two formulas is to ensure that the seam is smooth and coherent in the horizontal / vertical direction, that is, every two consecutive pixels differ by at most one unit in the x / y direction.

[0101] Find the path with the lowest energy so that removing this path has the least impact on the overall image. Step 4, finding the minimum energy seam, involves using a dynamic programming algorithm to find the path with the minimum sum of energy values ​​from the vertical seams (from top to bottom) or horizontal seams (from left to right) of the text energy map; that is, the seam with the minimum energy.

[0102] For vertical joints, define From top to The cumulative energy of the minimum energy path at a location, and for horizontal seams, rotate the text energy map 90° to treat it as a vertical seam s, process it, and then rotate it back.

[0103] If it is the backward energy method, M The expression form is:

[0104]

[0105] pass The recursive formula is used to calculate the cumulative energy map of the entire text energy map, and finally, the point with the minimum cumulative energy is found from the bottom row. By backtracking, the path of the entire seam is obtained, which is the minimum energy seam. The energy function is then used. It is the pixel value at the i-th row and j-th column of the energy map.

[0106] Example 4

[0107] Forward energy calculation is an improved energy calculation method that considers not only the energy of the current pixel but also the energy changes of new neighboring pixels after the removal of the current pixel when constructing the cumulative cost matrix. This method can more accurately assess the importance of each pixel, thereby selecting more appropriate seams to remove or insert to maintain the structural integrity of the image.

[0108] Based on Example 3, if the forward energy method is used, the cost increase caused by the new adjacency relationships added after removing the current pixel is as follows:

[0109]

[0110] in, From The increased cost after removing the crack from the left side. From The increased cost after removing the crack coming from the right side. From The increased cost after removing the cracks coming from the upper side; It is the pixel value at the i-th row and j+1-th column position of the energy map. It is the pixel value at the i-th row and j-1-th column position of the energy map. It is the pixel value at the (i-1)th row and jth column of the energy map.

[0111] Therefore, the cumulative cost matrix of the forward energy method for:

[0112]

[0113] pass The recursive formula is used to calculate the cumulative energy map of the entire text energy map, and finally find the point with the minimum cumulative energy from the bottom row. By backtracking, the path of the entire seam is obtained, which is the seam with the minimum energy.

[0114] Once the seam with the lowest energy value is found, it can be removed or inserted. For image reduction, the seam is removed; for image enlargement, new pixels are inserted at the seam. This process is repeated until the image reaches the desired size.

[0115] Example 5

[0116] One application example of the technical solution provided by this invention in natural scene text processing is specifically for a road sign image (1024×768 pixels) width compression scenario:

[0117] The Sobel operator is used to extract gradient maps, forming high gradient barriers in the edge area of ​​the "speed limit 60" text and other prominent texture areas;

[0118] The CRAFT character-level detection network generates a text center probability map. The pixels in the vicinity of the center of each of the four characters "Speed ​​Limit 60" have Gaussian distributed score values. The higher the pixel value, the higher the probability that the area is far from the text center.

[0119] With the water flow diffusion parameters set as α=0.6 (text center probability weight), β=0.4 (gradient weight), and 200 iterations, after the fluid diffusion process, the gradient map and the text center probability map are merged into a text energy map that takes into account both text region and background texture information. Due to the diffusion process, the original Gaussian distribution of the text center probability map is more uniformly distributed under the terrain constraint of high gradient at the text edge, which also improves the method's ability to protect the character structure of the text region.

[0120] When performing seam sculpting using the forward energy method, pixel seams in low-energy areas such as the sky and roads are deleted first.

[0121] After being compressed to a width of 800 pixels, the text area of ​​"Speed ​​Limit 60" is more complete, and the background area is compressed more, balancing the aesthetics of the image and the readability of the text.

[0122] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An image retargeting method based on text-image content awareness, characterized in that, Includes the following steps: Step 1: Input the text image to be adjusted; Step 2: Obtain the gradient map and text center probability map of the text image respectively; Step 3: Fuse the gradient map and the text center probability map using the water flow diffusion method to generate a text energy map; Step 4: Use the text energy map as the energy map to guide the seam engraving process, find the minimum energy seam, and insert or delete the minimum energy seam according to the preset target size to obtain the retargeting image; Step 5: Output the redirected image; Step 3 includes the following sub-steps: Step 3.1: Normalize the gradient map and the text center probability map, then add them together with weights to obtain the initial distribution of the water flow. ; in, It is the merged water flow distribution map, i.e., the text energy map. This is the text center probability map. It is the gradient map, The fusion operation represents the diffusion of water flow. The weights of the text center probability graph are... The weights of the gradient graph, For normalization processing; Step 3.2, according to The pixel values ​​in the data are sorted from high to low, and then the water level of all pixels is updated in the sorted order. One iteration is completed after all pixels have been updated. Step 3.3 Middle position The water level adjustment direction of a pixel is the pixel with the largest water level difference in its four neighboring areas. When the pixel with the largest water level difference in its four neighboring areas is found, the water level of the two pixels is made uniform. Each iteration is equivalent to performing a four-neighboring area diffusion operation. Step 3.4: Repeat the iterations in Steps 3.2 and 3.3 until the water level gradually stabilizes, then stop the iterations, i.e., when the condition is met. The text energy map is obtained, wherein, It is a location Pixels at that location Water level at any given time It is a location Pixels at that location Water level at any given time This is a pre-set iterative convergence threshold.

2. The image redirection method based on text image content awareness as described in claim 1, characterized in that, Step 2 includes the following sub-steps: Step 2.1: Apply the gradient operator convolution to obtain the gradient map; Step 2.2: Apply a character-level text detection network to obtain the text center probability map.

3. The image redirection method based on text image content awareness as described in claim 2, characterized in that, The gradient operator in step 2.1 includes the Sobel operator, the Prewitt operator, or the Laplacian operator.

4. The image redirection method based on text image content awareness as described in claim 1, characterized in that, In step 3.1, the weights of the text center probability graph The weights greater than those in the gradient graph .

5. The image redirection method based on text image content awareness as described in claim 4, characterized in that, Weight Weight .

6. The image redirection method based on text image content awareness as described in claim 1, characterized in that, energy function in step 4 Defined as: in, The energy function value at the i-th row and j-th column is equal to the text energy map at the corresponding position. In position The pixel value at that location; The size of the input text energy map is Define vertical joints respectively and horizontal joints are : Among them, vertical joints It is a pixel sequence from top to bottom of an image, with a horizontal seam. It is a sequence of pixels from left to right in an image; for an n×m image n is the number of rows, m is the number of columns, and the vertical seam is... Composed of n pixels, with a horizontal seam Composed of m pixels, It is the pixel of the vertical seam in the i-th row. x(i) is the pixel coordinate of the vertical seam in the i-th row, with column index x(i) and row index i. It is the pixel in the j-th column of the horizontal seam. It is the pixel coordinate of the horizontal seam in column j, column index j, row index j. , This means that for all i, the following condition is satisfied: This ensures that the column index change between adjacent rows does not exceed 1. This ensures that the row index change of adjacent columns does not exceed 1.

7. The image redirection method based on text image content awareness as described in claim 6, characterized in that, The minimum energy seam in step 4 is to use a dynamic programming algorithm to find a path with the minimum sum of energy values ​​from the vertical or horizontal seams of the text energy map, that is, the seam with the minimum energy. For vertical joints, define From top to The cumulative energy of the minimum energy path at a location, and for horizontal seams, rotating the text energy map 90° to treat it as a vertical seam, processing it, and then rotating it back.

8. The image redirection method based on text image content awareness as described in claim 7, characterized in that, If it is the backward energy method, M The expression form is: pass The recursive formula is used to calculate the cumulative energy map of the entire text energy map, and finally find the point with the minimum cumulative energy from the bottom row. The path of the entire seam is obtained by backtracking, which is the minimum energy seam.

9. The image redirection method based on text image content awareness as described in claim 7, characterized in that, If using the forward energy method, the cost increase resulting from the addition of new adjacency relationships after removing the current pixel is as follows: in, From The increased cost after removing the crack from the left side. From The increased cost after removing the crack coming from the right side. From The increased cost after removing the cracks coming from the upper side; This means that the pixel value at the i-th row and j+1-th column of the text energy map is equal to the text energy map at the corresponding position. In position The pixel value at that location, This means that the pixel value at the i-th row and j-1-th column of the text energy map is equal to the text energy map at the corresponding position. In position The pixel value at that location, This means that the pixel value at the (i-1)th row and jth column of the text energy map is equal to the text energy map at the corresponding position. In position The pixel value at that location; Therefore, the cumulative cost matrix of the forward energy method for: pass The recursive formula is used to calculate the cumulative energy map of the entire text energy map, and finally find the point with the minimum cumulative energy from the bottom row. The path of the entire seam is obtained by backtracking, which is the minimum energy seam.

Images