Image redirection method based on text image content perception

Abstract

The invention discloses an image redirection method based on text image content perception, and relates to the field of image processing in computer vision, and the method comprises the following steps: 1, inputting a to-be-adjusted text image; 2, respectively obtaining a gradient graph and a text center probability graph of the text image; 3, fusing the gradient graph and the text center probability graph through a water flow diffusion method to generate a text energy graph; 4, taking the text energy diagram as an energy diagram of a seam engraving process, finding a minimum energy seam, and inserting or deleting the minimum energy seam according to a preset target size to obtain a redirected image; and step 5, outputting the redirected image.

Description

Technical Field

[0001] The present invention relates to the field of image processing in computer vision, and in particular, to an image redirection method based on text image content perception. Background Art

[0002] With the popularization of mobile devices and the development of the Internet, the number of text images generated by users has increased sharply, and the text in images often contains rich and important semantic information. For example, the recognition of traffic signs and license plate numbers is very important for autonomous driving and intelligent monitoring tasks. When displayed on different devices and platforms, these text images need to better protect the information in the text area during the process of resizing. In the optical character recognition (OCR) task, it is also very important to retain more and more accurate text information in a smaller image size. For example, when a convolutional network processes a graphic, it needs to first resize the original image size to a fixed unified size (such as 640×640). Protecting the text information well within the limited size is of great significance for the processing of downstream tasks (such as text recognition, text detection, image processing, etc.).

[0003] Traditional image resizing methods, such as interpolation and cropping, can resize the image, but cannot specifically protect the text information in the image, resulting in a decrease or loss of text readability. The seam carving method proposed by Shai Avidan et al. can retain some visually more important details during the scaling process, but in scenes with complex textures, this method will ignore the glyphs. This method can focus on some edges of the text, but it will damage the geometric structure of the glyphs because the seams pass through the inside of the font.

[0004] In recent years, with the development of artificial intelligence technology, deep learning has continuously empowered the field of image redirection. Based on classical adjustment methods, neural networks can extract semantic information in images more comprehensively and accurately to guide the image redirection process. Related scholars have achieved certain results in both the construction method of the importance evaluation map and the combined application of various classical methods. Guo Yingchun et al. in the Chinese invention patent application "An Image Redirection Method Based on Relative Salience Detection" (application number: CN202111228342.5) obtained an edge map by extracting a relative salience map and detecting the image edge, linearly fused the edge map with the relative salience map to obtain an importance map, and finally learned the displacement mapping from the input image to the target image, and realized image redirection on the input image through the shift map. This method mainly focuses on the visual salience of the image, but for the special scenario of text images, its salience detection module may not fully understand the semantics and structure of the text, resulting in the text information not being preferentially protected during the redirection process.

[0005] Text images contain rich semantic information, but text in natural scenes has difficulties such as irregular shapes, diverse font formats, and complex backgrounds. Currently, few scholars have studied redirection methods for text images. Various current methods cannot accurately distinguish the distribution differences between text and non-text features. Therefore, in the size adjustment, text information is easily damaged, which affects its accuracy in downstream tasks.

[0006] Therefore, those skilled in the art are committed to developing a new image redirection method for text images to solve the above problems existing in the prior art. Summary of the Invention

[0007] In view of the above defects of the prior art, the technical problem to be solved by the present invention is how to accurately distinguish the distribution differences between text and non-text features, so as to effectively preserve the text content in the image during the size adjustment of the text image.

[0008] To achieve the above object, the present invention provides an image redirection method based on text image content perception, including the following steps: Step 1: Input the text image to be adjusted; Step 2: Obtain the gradient map and text center probability map of the text image respectively; Step 3: Fuse the gradient map and the text center probability map through the water flow diffusion method to generate a text energy map; Step 4: Use the text energy map as the energy map to guide the seam carving process, find the minimum energy seam, and insert or delete the minimum energy seam according to the preset target size to obtain the redirected image; Step 5: Output the redirected image.

[0009] Further, the Step 2 includes the following sub-steps: Step 2.1: Apply gradient operator convolution to obtain the gradient map; Step 2.2: Apply a character-level text detection network to obtain the text center probability map.

[0010] Further, the gradient operator in the Step 2.1 includes but is not limited to Sobel operator, Prewitt operator or Laplacian operator.

[0011] Further, the Step 3 includes the following sub-steps: Step 3.1: Normalize the gradient map and the text center probability map and then add them with weights to obtain the initial distribution of the water flow: Wherein, is the fused water flow distribution map, is the text center probability map, is the gradient map, represents a fusion operation for water flow diffusion, is the weight of the text center probability map, is the weight of the gradient map, is the normalization process; Step 3.2: Sort the pixel values in from high to low, and then update the water levels of all pixels in the sorted order. One iteration is completed when all pixels are updated; Step 3.3: The direction of adjusting the water level height of the pixel at position in is the pixel with the largest water level height difference in its four-neighborhood, that is: is the water level height of the pixel at position ; is the water level height of a certain pixel in its four-neighborhood, , take values of -1, 0, 1; is the water level height of the pixel with the largest water level height difference in its four-neighborhood, represents finding the combination that makes the expression in the brackets reach the maximum value among all possible combinations, is the combination when reaches the maximum value; When finding the pixel in the direction with the largest water level height difference in its four-neighborhood, make the water levels of the two pixels uniform. The adjustment value of the water level height at position in the t-th iteration is: After adjustment, the water level heights of the two pixels are updated to: Each iteration is equivalent to performing a four-neighborhood diffusion operation; Step 3.4: Repeat the iterations in Step 3.2 and Step 3.3 until the change in water level height gradually stabilizes, then stop the iteration, that is, satisfy , and obtain the text energy map, where is a pre-set iteration convergence threshold.

[0012] Furthermore, in Step 3.1, the weight The weight greater than the gradient map .

[0013] Furthermore, the weight , the weight .

[0014] Furthermore, the energy function in step 4 is defined as: The size of the input text energy map is , and the vertical seam and the horizontal seam are defined as : Among them, the vertical seam is a sequence of pixels from the top to the bottom of an image, and the horizontal seam is a sequence of pixels from the left to the right of an image; for an n×m image , n is the number of rows, m is the number of columns, the vertical seam consists of n pixels, and the horizontal seam consists of m pixels, is the pixel of the vertical seam in the i-th row, is the pixel coordinate of the vertical seam in the i-th row, the column index x(i), and the row index i; is the pixel of the horizontal seam in the j-th column, is the pixel coordinate of the horizontal seam in the j-th column, the column index j, and the row index , means that for all i, the following conditions are satisfied, means to ensure that the change in the column index between adjacent rows does not exceed 1, means to ensure that the change in the row index between adjacent columns does not exceed 1.

[0015] Furthermore, the minimum energy seam in step 4 is to find a path with the minimum sum of energy values from the vertical seam or the horizontal seam of the text energy map using the dynamic programming algorithm, that is, the seam with the minimum energy; For the vertical seam, define as the cumulative energy of the minimum energy path from the top to the position, and for the horizontal seam, rotate the text energy map by 90° and treat it as a vertical seam, and then rotate it back after processing.

[0016] Furthermore, if it is the backward energy method, the expression form of M is: Through the recurrence formula, the cumulative energy map of the entire text energy map is calculated, and finally the point with the minimum cumulative energy is found from the bottom row, and the path of the entire seam is obtained through backtracking, that is, the minimum energy seam. represents the pixel value at the position of the j-th column in the i-th row of the text energy map.

[0017] Furthermore, if it is the forward energy method, after removing the current pixel, the increases in the costs brought by the new adjacent relationships are respectively: Among them, is the increased cost after removing the crack from the left side direction, is the increased cost after removing the crack from the right side direction, is the increased cost after removing the crack from the upper side direction; represents the pixel value at the position of the (j + 1)-th column in the i-th row of the text energy map, represents the pixel value at the position of the (j - 1)-th column in the i-th row of the text energy map, represents the pixel value at the position of the j-th column in the (i - 1)-th row of the text energy map; Therefore, the cumulative cost matrix of the forward energy method is: Through the recurrence formula, the cumulative energy map of the entire text energy map is calculated, and finally the point with the minimum cumulative energy is found from the bottom row, and the path of the entire seam is obtained through backtracking, that is, the minimum energy seam.

[0018] An image redirection method based on text image content perception provided by the present invention has at least the following technical effects: 1. The technical solution provided by the present invention can accurately protect the text content. By introducing the text energy map and the character-level text detection model, the importance of the text area can be accurately identified, and the text content is preferentially protected during the image size adjustment process, avoiding the text from being accidentally deleted or distorted, and significantly improving the quality of text image redirection; 2. The technical solution provided by the present invention provides an innovative way of feature map fusion. Through the feature map combination method of the water flow diffusion method, learning from the John Stokes equation of fluid mechanics, it can better simulate the energy distribution of the text area and generate a text energy map that more conforms to the text structure. Compared with the traditional method, this method can more accurately guide the image redirection operation; 3. The technical solution provided by the present invention can provide an efficient and general solution. Thanks to the shape robustness of character-level text detection, it performs excellently on various types of text images such as natural scene text and document images, and has wide applicability. At the same time, the computational complexity of this method is relatively low, suitable for scenarios with high real-time requirements, and has high practical value.

[0019] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, features and effects of the present invention. Brief Description of the Drawings

[0020] Figure 1 is the flowchart of the method of a preferred embodiment of the present invention; Figure 2 is the overview diagram of the method of a preferred embodiment of the present invention; Figure 3 is Figure 1 the schematic diagram of the water flow diffusion method fusion process of the provided preferred embodiment. Detailed Embodiments

[0021] The following introduces multiple preferred embodiments of the present invention with reference to the accompanying drawings of the specification to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0022] The embodiments of the present invention are specifically designed for text images. By introducing a text energy map, text information is accurately protected. When adjusting the image size, non-uniform size adjustment is performed according to the text energy map, giving priority to retaining the text area to avoid the text being accidentally deleted or distorted. It is particularly suitable for scenarios such as natural scene text and document images, and has important application value in the field of image scaling, especially suitable for scenarios that require maintaining the integrity of image content.

[0023] Embodiment 1 As Figure 1 and Figure 2 shown, a method for image redirection based on text image content perception provided by an embodiment of the present invention includes the following steps: Step 1: Input the text image to be adjusted; Step 2: Obtain the gradient map and text center probability map of the text image respectively; Step 3: Generate a text energy map by fusing the gradient map and the text center probability map through the water flow diffusion method; Step 4: Use the text energy map as the energy map to guide the seam carving process, find the minimum energy seam, and insert or delete the minimum energy seam according to the preset target size to obtain the redirected image; Step 5: Output the redirected image.

[0024] Specifically, Step 2 includes the following sub-steps: Step 2.1: Apply gradient operator convolution to obtain a gradient map; where the gradient operator includes, but is not limited to, Sobel operator, Prewitt operator, or Laplacian operator.

[0025] Step 2.2: Apply a character-level text detection network to obtain a text center probability map. The core of the character-level text detection network CRAFT (Character Region Awareness for Text Detection) is its region awareness ability. Its network structure is as follows: The feature extraction layer uses a pre-trained VGG16 network for downsampling to extract deep semantic features, and then adopts a network structure similar to U-Net to splice deep features and shallow features, and restores the image resolution through convolution and deconvolution operations. Finally, a score map of the text center region is output, and the score of each pixel in the map represents the probability that the point belongs to the text region.

[0026] Specifically, when the character-level text detection network CRAFT is applied in the method of this embodiment, it is trained offline as an independent module, and its output result is used for subsequent processes.

[0027] Specifically, the training strategy of CRAFT generally includes: 1) Weakly supervised learning: The real dataset usually only has word-level annotations. CRAFT generates character-level annotations from word-level annotations through weakly supervised learning methods; 2) Data augmentation: Apply techniques such as cropping, rotation, and color change during training to improve the robustness of the model; 3) Datasets for training: Include SynthText in the pre-training stage and real datasets such as ICDAR 2013, ICDAR2015, and ICDAR 2017.

[0028] Embodiment 2 Based on Embodiment 1, fuse the text center probability map and the gradient map to obtain a text energy map. Among them, the gradient map is analogized to the ground in the water flow process, and the text center probability map is the water falling on the ground. The higher the value at a certain pixel in the text energy map, the higher the water level of the pixel in the initial state. During the flow process, the water flow follows the principle of "water flows to lower places", and the new height distribution of the water formed after the flow area stabilizes is the fused text energy map.

[0029] Specific fusion steps are as Figure 3 shown. Step 3 includes the following sub-steps: Step 3.1: Normalize the gradient map and the text center probability map, and then perform weighted summation to obtain the initial distribution of the water flow: Among them, is the fused water flow distribution map, is the text center probability map, is the gradient map, represents the fusion operation of water flow diffusion, is the weight of the text center probability map, is the weight of the gradient map, is the normalization process; the weight of the text center probability and the weight of the gradient are used to flexibly adjust the emphasis on text details and other details; In particular, according to practical experience, the weight of the text center probability map is usually slightly greater than the weight of the gradient map. Specifically, set the weight , and the weight .

[0030] Step 3.2: In each iteration, sort the pixel values in from high to low, and then update the water levels of all pixels in the sorted order. Completing the update of all pixels is one iteration; Step 3.3: In each iteration, the direction of adjusting the water level height of the pixel at position in is the pixel with the largest water level height difference in its four-neighborhood, that is: is the water level height of the pixel at position ; is the water level height of a certain pixel in its four-neighborhood, , take values of -1, 0, 1; is the water level height of the pixel with the largest water level height difference in its four-neighborhood, represents finding the combination that makes the expression in the parentheses reach the maximum value among all possible combinations, is the combination when reaches the maximum value; When finding the pixel with the largest water level height difference direction in its four-neighborhood, equalize the water levels of the two pixels. The adjustment value of the water level height at position in the t-th iteration is: The adjustment here is to ensure that the water surface height will not be lower than the terrain height. After the adjustment, the water level heights of the two pixels are updated to: Each iteration is equivalent to performing a diffusion operation in the four-neighborhood. The water level adjustment amount here adopts the simplest and easiest-to-implement form - averaging the water levels of the two pixels. There can also be other more complex variants, which can be adjustment rules with finer adjustment granularity.

[0031] Step 3.4: Repeat the iterations in Step 3.2 and Step 3.3 until the change in the water level height gradually tends to be stable, then stop the iteration, that is, satisfy , and obtain the text energy map, where is a pre-set iteration convergence threshold.

[0032] Embodiment 3 Based on Embodiment 2, the energy function in Step 4 is defined as: The energy function is the pixel value at the position of the i-th row and j-th column in the energy map.

[0033] The size of the input text energy map is , and the vertical seam and the horizontal seam are defined as : where is a pixel sequence from the top to the bottom of an image, is a pixel sequence from the left to the right of an image. The conditions of the two formulas are to ensure that the seam is smooth and coherent in the horizontal / vertical direction, that is, the difference between every two consecutive pixels is at most one unit in the x / y direction.

[0034] Find a path with the lowest energy, so that when this path is removed, the impact on the overall image is the smallest. The minimum energy seam in Step 4 is to find a path with the minimum sum of energy values from the vertical seam (from the top to the bottom) or the horizontal seam (from the left to the right) of the text energy map using the dynamic programming algorithm, that is, the seam with the minimum energy; For the vertical seam, define as the cumulative energy of the minimum energy path from the top to the position, and for the horizontal seam, rotate the text energy map 90° to be regarded as the vertical seam s, and then rotate it back after processing.

[0035] If it is the backward energy method, M is expressed as: Through the recurrence formula, calculate the cumulative energy map of the entire text energy map, and finally find the point with the minimum cumulative energy from the bottom row. Through backtracking, obtain the path of the entire seam, that is, the minimum energy seam. The energy function is the pixel value at the position of the i-th row and j-th column of the energy map.

[0036] Example 4 The forward energy method is an improved energy calculation method. When constructing the cumulative cost matrix, the forward energy method not only considers the energy of the current pixel but also the energy change of the new neighbor pixels after removing the current pixel. This method can more accurately evaluate the importance of each pixel, so as to select more appropriate seams for removal or insertion to maintain the structural integrity of the image.

[0037] Based on Example 3, if it is the forward energy method, then after removing the current pixel, the increased costs brought by the new adjacent relationships are respectively: Among them, is the increased cost after removing the crack from the left direction, is the increased cost after removing the crack from the right direction, is the increased cost after removing the crack from the upper direction; is the pixel value at the position of the i-th row and j + 1-th column of the energy map, is the pixel value at the position of the i-th row and j - 1-th column of the energy map, is the pixel value at the position of the i - 1-th row and j-th column of the energy map.

[0038] Therefore, the cumulative cost matrix of the forward energy method is: Through the recurrence formula, calculate the cumulative energy map of the entire text energy map, and finally find the point with the minimum cumulative energy from the bottom row. Through backtracking, obtain the path of the entire seam, that is, the minimum energy seam.

[0039] Once the seam with the minimum energy value is found, the operation of removing or inserting the seam can be carried out. For image reduction, remove the seam; for image enlargement, insert new pixels at the seam. This process will be repeated until the image reaches the required size.

[0040] Example 5 Regarding an application example of the technical solution provided by the present invention in natural scene text processing, specifically for the width compression scenario of a road sign image (1024×768 pixels): The Sobel operator is used to extract the gradient map, forming high-gradient barriers in the edge regions of the text "Speed Limit 60" and other prominent texture regions; The text center probability map is generated through the character-level detection network CRAFT. The pixel values in the vicinity of the center of each of the four characters of "Speed Limit 60" have Gaussian-distributed score values. The higher the pixel value, the higher the probability that the region is close to the text center; Set the water flow diffusion parameters α = 0.6 (text center probability weight), β = 0.4 (gradient weight), and the number of iterations to 200. After the fluid diffusion process, the gradient map and the text center probability map are fused into a text energy map that takes into account both text region and background texture information. Due to the diffusion process, the original Gaussian distribution of the text center probability map achieves a more uniform distribution under the topographic constraint of high gradients at the text edges, which also improves the ability of this method to protect the glyph structure of the text region; When performing seam carving through the forward energy method, the pixel seams in low-energy regions such as the sky and the road are preferentially deleted.

[0041] After compressing to a width of 800 pixels, the integrity of the text region of "Speed Limit 60" is higher, and the compression rate of the background region is higher, taking into account both the aesthetics of the picture and the readability of the text.

[0042] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative labor. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field based on the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. An image redirection method based on text image content perception, characterized in that It includes the following steps: Step 1: Input the text image to be adjusted; Step 2: Obtain the gradient map and the text center probability map of the text image respectively; Step 3: Fuse the gradient map and the text center probability map by the water flow diffusion method to generate a text energy map; Step 4: Use the text energy map as the energy map to guide the seam carving process, find the minimum energy seam, and insert or delete the minimum energy seam according to the preset target size to obtain a redirected image; Step 5: Output the redirected image.

2. The image redirection method based on text image content perception according to claim 1, characterized in that Step 2 includes the following sub-steps: Step 2.1: Apply gradient operator convolution to obtain the gradient map; Step 2.2: Apply a character-level text detection network to obtain the text center probability map.

3. The image redirection method based on text image content perception according to claim 2, wherein, The gradient operator in Step 2.1 includes but is not limited to Sobel operator, Prewitt operator or Laplacian operator.

4. The image redirection method based on text image content perception according to claim 1, wherein Step 3 includes the following sub-steps: Step 3.1: Normalize the gradient map and the text center probability map and then add them with weights to obtain the initial distribution of the water flow: Among them, is the fused water flow distribution map, is the text center probability map, is the gradient map, represents the fusion operation of water flow diffusion, is the weight of the text center probability map, is the weight of the gradient map, is the normalization process; Step 3.2: Sort the pixel values in from high to low, and then update the water levels of all pixels in the sorted order one by one. One iteration is completed after updating all pixels;​​ Step 3.3, the position at which the water level height adjustment direction of the pixel is the pixel with the largest water level height difference in its four-neighborhood, that is: Among them, is the water level height of the pixel at position ; is the water level height of a certain pixel within its four-neighborhood, , take values of -1, 0, 1; is the water level height of the pixel with the largest water level height difference within its four-neighborhood, represents finding the combination that maximizes the expression within the parentheses among all possible combinations, is the combination when reaches the maximum value; When finding the pixel in the direction with the largest water level height difference within its four-neighborhood, the water levels of the two pixels are equalized, and the adjustment value of the water level height at position at the t-th iteration is: After adjustment, the water level heights of two pixels are updated to: Each iteration is equivalent to performing a four-neighborhood diffusion operation; Step 3.

4. Repeat the iteration in the said Step 3.2 and the said Step 3.3 until the change in water level height gradually tends to be stable, then stop the iteration, that is, when it satisfies , and obtain the text energy diagram, where is a preset iteration convergence threshold.

5. The image redirection method based on text image content perception according to claim 4, wherein In the step 3.1, the weight of the text center probability map is greater than the weight of the gradient map .

6. The image redirection method based on text image content perception according to claim 5, wherein Weight and weight .

7. The image redirection method based on text image content perception according to claim 4, characterized in that, The energy function of step 4 is defined as: The size of the input text energy diagram is , respectively defining a vertical seam and a horizontal seam as : Among them, the vertical seam is a pixel sequence from the top to the bottom of an image, and the horizontal seam is a pixel sequence from the left to the right of an image; for an n×m image , n is the number of rows and m is the number of columns. The vertical seam consists of n pixels, and the horizontal seam consists of m pixels, is the pixel of the vertical seam in the i-th row, is the pixel coordinate of the vertical seam in the i-th row, the column index x(i), and the row index i; is the pixel of the horizontal seam in the j-th column, is the pixel coordinate of the horizontal seam in the j-th column, the column index j, and the row index , means that for all i, the following conditions are satisfied, means to ensure that the change in the column index between adjacent rows does not exceed 1, means to ensure that the change in the row index between adjacent columns does not exceed 1.

8. The image redirection method based on text image content perception according to claim 7, wherein The minimum energy seam in Step 4 is to find a path with the minimum sum of energy values from the vertical seams or horizontal seams of the text energy map using the dynamic programming algorithm, that is, the seam with the minimum energy; For a vertical seam, define as the cumulative energy of the minimum energy path from the top to the position. For a horizontal seam, rotate the text energy map by 90° to treat it as a vertical seam, and then rotate it back after processing.

9. The method for image redirection based on text image content perception according to claim 8, characterized in that, If it is the backward energy method, M is expressed in the form of: Through Using the recurrence formula, calculate the cumulative energy diagram of the entire text energy diagram, and finally find the point with the minimum cumulative energy from the bottom row. By backtracking, obtain the path of the entire seam, which is the minimum energy seam represents the pixel value at the position of the i-th row and j-th column of the text energy diagram 10. The method for image redirection based on text image content perception according to claim 8, wherein, If it is the forward energy method, after removing the current pixel, the increases in costs brought by the new adjacent relationships added are respectively: Among them, is the increased cost after removing the crack coming from the left direction, is the increased cost after removing the crack coming from the right direction, is the increased cost after removing the crack coming from the upper direction; represents the pixel value at the position of the (i, j + 1)-th column of the text energy map, represents the pixel value at the position of the (i, j - 1)-th column of the text energy map, represents the pixel value at the position of the (i - 1, j)-th column of the text energy map; Therefore, the cumulative cost matrix of the forward energy method is as follows: Through the recurrence formula, calculate the cumulative energy diagram of the entire text energy diagram, and finally find the point with the minimum cumulative energy from the bottom row, and obtain the path of the entire seam through backtracking, that is, the minimum energy seam.

Images