A method for optimizing text positioning in screen images

Abstract

The present invention discloses a method for optimizing the positioning of text in a screen image, comprising the following steps: S1, for text to be positioned, requiring the user to provide a set of sample text and the screen image containing the text in the current software operating environment; wherein the text to be positioned has a certain type of rendering feature; S2, using a text rendering configuration recognition algorithm to identify the specific rendering configuration of the sample text in a rendering configuration space composed of a large number of parameters; S3, for other text with the same type of rendering feature, rendering the text into an image according to the automatically recognized rendering configuration, and using a template matching algorithm to match the image on the screen to locate the position of the text. The present invention avoids the problem of unsuccessful matching caused by changes in the text rendering results due to changes in the environment after the template matching switches the software operating environment. At the same time, it can adapt to different text rendering features, thereby improving the adaptability of text positioning.

Description

Technical Field

[0001] The present invention relates to the field of computer visualization, and in particular to a method for optimizing the positioning of text in a screen image. Background Art

[0002] In fields such as visual information-based software testing and software interface retrieval, it is often necessary to locate text within the screen images generated during software runtime. This can be done to determine the location of interface controls or to determine whether the interface matches certain text retrieval criteria. Locating text within screen images has become a common requirement in software engineering.

[0003] Existing technologies for locating text from screen images include template matching and OCR (optical character recognition) text recognition matching, each of which has some advantages and disadvantages. Template matching matches a given rendered sub-image of text in the entire screen image to locate the small area where the text is located. This method has a fast positioning speed and high accuracy when the environment remains unchanged, but it has poor adaptability. After switching the software operating environment (such as changes in the operating system or browser software), the rendering of the given text may change with the changes in the environment, resulting in a sharp drop in the accuracy of matching the original rendering in the screen image. OCR text recognition matching first performs OCR text recognition on the screen image to obtain all the text and its appearance location information, and then locates its position through string matching of the text. OCR text recognition matching has strong adaptability, but is slow and is affected by the effect of OCR itself, and may not guarantee accuracy. Summary of the Invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a method for optimizing positioning of text in a screen image that can quickly and adaptively locate a given text position in the screen image.

[0005] Technical solution: The text optimization positioning method of the present invention includes the following steps:

[0006] S1: For the text to be located, the user is required to provide a set of sample text and the screen image of the text in the current software running environment; wherein the text to be located has a certain type of rendering feature;

[0007] S2, in the rendering configuration space composed of a large number of parameters, a text rendering configuration recognition algorithm is used to identify the specific rendering configuration of the sample text;

[0008] S3, for other characters with the same type of rendering features, the characters are rendered into images according to the automatically identified rendering configuration, and the image is matched on the screen using a template matching algorithm to locate the position of the characters.

[0009] Furthermore, in step S2, the specific rendering configuration of the sample text is identified by using a text rendering configuration recognition algorithm, including the following steps:

[0010] S21, using an OCR text recognition tool, obtain a small image that closely contains the sample text;

[0011] S22, using three-variable quadratic polynomial regression to fit the inferred function P size , by the inference function P size Estimate the approximate font size of the sample text;

[0012] S23, uses a deep learning-based image classification algorithm to intelligently predict the font type, font style, font weight, and rendering engine of the sample text;

[0013] S24, determining the front and back background colors of the text sample image by an image processing method;

[0014] S25,uses a priority queue and heuristic search method to find the optimal,rendering configuration for the sample text.

[0015] Furthermore, in step S21, when there are other texts around the sample text and the OCR text recognition tool cannot accurately obtain the boundary area where the sample text is located, the following steps are taken:

[0016] First, a small image that closely contains the sample text is extracted. Then, a contour recognition algorithm is used to obtain the contours in the small image. For each contour, it is determined whether it is a text frame. If the outer frame is a rectangle, the contour area is basically the same as the outer frame area, and the width-to-length ratio meets the characteristics of text, the current contour is considered to be a text frame.

[0017] Further, in step S22, the function P is estimated size The steps to estimate the approximate font size of the sample text are as follows:

[0018] S221, by building a training library, obtain the h corresponding to each training word px 、b up 、b down Features and font size f size , as training data; where h px is the pixel height of the text; b up Is the text containing raised letters? If so, b up =1, otherwise b up =0;b down Indicates whether the text contains a sunken letter. If it does, b down =1, otherwise b down =0;

[0019] S222, based on the training data, use ternary quadratic polynomial regression to fit the inference function P size , the inferred function P size The input is h px 、b up 、b down , the output is f size ;

[0020] S223, through contour recognition, obtain the pixel height of the text in the small image obtained in step 21; px 、b up 、b down Substitute into the inference function P size , get the approximate font size of the sample text.

[0021] Furthermore, in step S23, the steps for implementing intelligent prediction of the font type, font style, font weight, and rendering engine of the sample text are as follows:

[0022] S231, establishing a text image training sample library for each font type, font style, font weight, and rendering engine;

[0023] S232: Build a deep learning prediction model based on the residual network ResNet50. Based on the sample library, train the deep learning prediction model from text and images to rendering parameters.

[0024] S233, using a deep learning prediction model to predict the sample text image obtained in step 21 to obtain a classification probability vector, where each value of the classification probability vector represents the possibility of belonging to the classification.

[0025] Furthermore, in step S24, the steps for determining the front and back background colors of the text sample image are as follows:

[0026] S241, grayscale the image by image processing method, and classify the various color types into two categories: black text on white background and white text on gray background; Figure 2 Value, extract the pixel values ​​around the image on the binary image, and calculate the mean m of the pixel values ​​around the image:

[0027] m=(mean(top_row)+mean(bottow_row)+mean(left_colum)+mean(right_colun)) / 4

[0028] Where, mean() is the mean function, mean(top_row) represents the mean of the pixel values ​​in the top row of the image, mean(bottow_row) represents the mean of the pixel values ​​in the bottom row of the image, mean(left_colum) represents the mean of the pixel values ​​in the leftmost column of the image, and mean(right_colum) represents the mean of the pixel values ​​in the rightmost column of the image;

[0029] S242, judging based on the mean m: if the mean m>255 / 2, it means that the image is more than half white around, and it is inferred that the binary image has a white background and black text, which belongs to the category of black text on a white background; otherwise, it means that the image is more than half black around, and the binary image has a black background and white text, which belongs to the category of white text on a gray background.

[0030] Furthermore, in step S25, the steps for finding the optimal rendering configuration of the sample text are as follows:

[0031] S251: According to step 22, the approximate size of the sample text is obtained as a, and the font size space is set to {a-1, a, a+1}, and the corresponding priority is set; according to the result returned by step 23, the value space and corresponding probability of the font type, font style, font weight, and rendering engine are obtained, and the probability of each value is used as its corresponding priority; according to step 24, the value space of the front and back background colors is obtained as {black text on white background, white text on gray background}, and the corresponding priority is set;

[0032] Take the Cartesian product of the values ​​of all parameters of the sample text to get a priority queue <r1,r2,…,r n >, r i is a specific rendering configuration, i = 1, 2, ..., n, where n is the total number of possible rendering configurations; its corresponding priority is the sum of the priorities of each parameter;

[0033] Step 252: Select the 200 highest priority rendering configurations in the priority queue to form a new priority queue. <r1,r2,…,r 200 >, each time, take the rendering configuration with the highest priority from the queue, use this rendering configuration to render the sample text into an image, and perform template matching with the sample text image obtained in step 21. If the similarity reaches the set threshold, the configuration is considered to be the rendering configuration of the sample text, and the text rendering configuration recognition algorithm is exited; otherwise, the next configuration in the queue is taken until the configuration with the highest similarity is obtained;

[0034] In step 253, if the optimal configuration has a low priority due to a prediction error and cannot exit from step 252, a new configuration to be tested is added using a heuristic search method based on a genetic algorithm until a rendering configuration with a similarity that reaches a set threshold is found. The specific steps are as follows:

[0035] Step 2531: convert {r1,…,r 200} Render configuration as initial population;

[0036] Step 2532: Use each rendering configuration in the population to render the sample text into an image, and perform template matching with the screen image. The similarity obtained is the fitness of an individual.

[0037] Step 2533: select individuals with high fitness from the current population based on the fitness of the individuals in the population;

[0038] In step 2534, a crossover operation is performed on the rendering configuration selected in step 2533 to generate a new rendering configuration; the new rendering configuration is added to the population, and step 2532 is repeated until an individual whose fitness reaches the set threshold is found, and the search is stopped.

[0039] Compared with the prior art, the present invention has the following significant effects:

[0040] 1. This invention proposes a text rendering configuration recognition algorithm that quickly and automatically recognizes text rendering configurations, renders text into images based on the configuration, and then locates text based on template matching, achieving faster positioning speed and higher accuracy than methods based on OCR text recognition.

[0041] 2. After automatically identifying the rendering configuration, the present invention performs text rendering and matching according to the current environment, thus avoiding the problem of unsuccessful matching caused by changes in text rendering results as the environment changes after the template matching software operating environment is switched. Compared with the algorithm directly based on template matching, the adaptability of text positioning is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flow chart of the text optimization positioning method of the present invention;

[0043] Figure 2 This is a schematic diagram of the example text and the screen image on which it appears in the present invention;

[0044] Figure 3 Flowchart of the text rendering configuration recognition algorithm of the present invention;

[0045] Figure 4 This is a schematic diagram of the OCR optimization application of the present invention;

[0046] Figure 5 This is a schematic diagram of the background colors before and after the text sample image judged by the present invention;

[0047] Figure 6 This is a flow chart of the supplementary configuration to be tested based on the genetic algorithm idea of ​​the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in further detail below with reference to the accompanying drawings and specific implementations.

[0049] The present invention designs a method for optimizing the positioning of text in screen images. The core idea is: based on a small amount of given sample text, the rendering configuration of the relevant text in the current software running environment is automatically inferred based on the OCR positioning results, and the target text to be located is rendered into an image using the obtained rendering configuration to perform template matching to obtain the position of the target text on the screen; this method has the advantages of fast template matching speed and high accuracy, and different environments are rendered according to different rendering configurations, which overcomes the disadvantage of weak adaptability of the template matching method and provides a faster, more accurate and better adaptable method for text positioning in screen images.

[0050] like Figure 1 As shown, the detailed implementation steps of the present invention are as follows:

[0051] Step 1: For the text with a certain type of rendering features that needs to be located, the user is first required to provide a set of sample texts and the screen images in the current software running environment. Figure 2 As shown in the figure, the user provides the sample text "Mainpage" and the screen image of the "Main page" in the current software running environment, which is used to locate all texts with the same rendering configuration as the Main page;

[0052] Step 2: Use a text rendering configuration recognition algorithm to quickly identify the specific rendering configuration of the sample text within a large number of rendering configuration parameters. For example, the text rendering configuration recognition algorithm identifies the specific configuration of the Main page in the current software environment as: 12px, Arial font, DirectWrite rendering engine, black text on a white background, 400 pt, and normal style.

[0053] like Figure 3 The figure below is a flowchart of the text rendering configuration recognition algorithm. The steps for quickly identifying the specific rendering configuration of a sample text in a rendering configuration space composed of a large number of parameters are as follows:

[0054] Step 21: Get a small image that closely contains the sample text. The specific steps are as follows:

[0055] Step 211, use OCR text recognition tool to perform whole-image recognition on the screen image where the sample text is located, obtain all the text in the image and its location, the location is described by the boundary area. Find the boundary area where the sample text is located among all the text and its location in the obtained image, and cut out a small image that closely contains the sample text according to the boundary area. For example, use OCR text recognition tool to Figure 2Perform full-image recognition to obtain all text and its location. Find the boundary area of ​​"Main page" in the recognition results and extract the small image that closely contains "Main page" based on the boundary area.

[0056] Step 212: When there are other texts around the sample text, the OCR text recognition tool cannot accurately obtain the boundary area of ​​the sample text. For example, if the sample text is Wikipedia, and the words "Welcome to Wikipedia" are connected in the screen image, the OCR text recognition tool can often only obtain the boundary area of ​​the entire "Welcome to Wikipedia" but cannot obtain the accurate boundary area of ​​Wikipedia. Figure 4 As shown, the present invention first extracts a small image that closely encompasses the phrase "Welcome to Wikipedia." Then, a contour recognition algorithm is used to obtain the contours within the small image, and each contour is determined to be a text frame. If the outer frame is a rectangle, the contour area is roughly the same as the outer frame area, and the aspect ratio conforms to the characteristics of text (the length, width, and aspect ratio are not too large or too small), the current contour is considered a text frame. After the contour determination is complete, the boundary region formed by the tenth to eighteenth text frame contours is the boundary region where the sample text "Wikipedia" is located. Based on this boundary region, the small image that closely encompasses "Wikipedia" is extracted.

[0057] Step 22: Estimate the approximate font size of the sample text. The specific steps are as follows:

[0058] Step 221: A webpage text training library is established, which includes HTML descriptions of webpages, webpage images, etc. The font size of each webpage text in the training library is obtained through HTML, the pixel height of the webpage text is obtained through contour recognition on the webpage image, and the character content is used to determine whether it includes capital letters, raised letters such as b and l, and lowered letters such as g and j. Based on the above information, the corresponding (h px ,b up ,b down ) Features and font size f size , as training data; where h px is the pixel height of the text, b up Is the text containing raised letters (if it contains raised letters, b up =1, otherwise b up =0), b down Whether the text contains a sunken letter (if it contains a sunken letter, b down =1, otherwise b down =0).

[0059] Step 222: Based on the training data, a three-variable quadratic polynomial regression is used to fit the inference function P from the height of the text pixels, whether it contains raised letters, whether it contains sunken letters to the text size. size , where the inference function P size The input is h px 、b up 、b down , the inferred function P size The output is f size .

[0060] Step 223: Use the contour recognition algorithm to obtain the pixel height of the text in the small image obtained in step 21, and determine whether the sample text contains raised letters and sunken letters based on the content of the sample text. px 、b up 、b down Substitute into the inference function P size , get the approximate font size of the sample text. For example, using contour recognition, get the pixel height of "Main page" in the small picture as 13. According to the content of "Main page", determine whether the sample text contains raised letters (M) and sunken letters (p, g). Substitute the obtained parameters into the inference function P size , and get the approximate font size of "Main page" to be 12.

[0061] Step 23: Use a deep learning-based image classification algorithm to intelligently predict the font type, font style, font weight, and rendering engine of the sample text.

[0062] Step 231: Establish a text image training sample library for rendering parameters such as font type, font style, font weight, and rendering engine. The sample library for font prediction includes nine representative font categories (Bell MT, Times New Roman, Book Antiqua, Arail, Agency FB, Verdana, Courier, Consolas, and Inconsolata) under the three major categories of serif, sans serif, and monospaced fonts, serving as targets for font type prediction. The sample library for font style prediction includes two categories, normal and italic, serving as targets for font style prediction. The sample library for font weight prediction includes four commonly used weight categories of 100, 400, 600, and 800, serving as targets for weight prediction. The sample library for rendering engine prediction includes nine representative rendering engine categories, serving as targets for rendering engine prediction. (To render text into an image, a rendering engine is required. Different rendering engines have different display and layout characteristics. Examples include Microsoft DirectWrite and Python's Pillow library.)

[0063] Step 232, based on the text image training sample library, train a deep learning prediction model from text images to rendering parameters. The deep learning model is a classification model based on the residual network ResNet50, which can determine which specific rendering parameters are used from the text image. The input of the prediction model is a text image, and the output is a classification probability vector. Each value of the classification probability vector represents the possibility of belonging to that classification. For example, to predict which font weight category a 64×64 RGB color text image belongs to, the model will return a classification probability vector [p1, p2, p3, p4], where the four values ​​represent the probabilities of font weights of 100, 400, 600, and 800, respectively.

[0064] In step 233, the prediction model is used to predict the sample text image obtained in step 2.1, resulting in a classification probability vector. Each value in the vector represents the probability of belonging to that classification. For example, if a sample text image is input and the weight of the text in the image is predicted, the algorithm will return a classification probability vector [0.2, 0.9, 0.7, 0.3], where the four values ​​represent the probabilities of a weight of 100, 400, 600, and 800, respectively.

[0065] Step 24: Determine the front and back background colors of the text sample image.

[0066] By using image processing technology to grayscale the image, various colors can be classified into two types: black text on a white background and white text on a gray background. Figure 2Binarization: extract the pixel values ​​around the image on the binary image and calculate the mean of the pixel values ​​around it:

[0067] m=(mean(top_row)+mean(bottow_row)+mean(left_colum)+mean(right_colun)) / 4 (1)

[0068] Where mean() is the mean function, mean(top_row) represents the mean of the top row of pixels, mean(bottow_row) represents the mean of the bottom row of pixels, mean(left_colum) represents the mean of the leftmost column of pixels, and mean(right_colum) represents the mean of the rightmost column of pixels. If the mean m of the surrounding pixels is greater than 255 / 2, then the image is more than half white, and thus the binary image has a white background and black text, indicating that this sample has black text on a white background. Otherwise, the image is more than half black, the binary image has a black background and white text, indicating that this sample has white text on a gray background.

[0069] like Figure 5 As shown, the small image containing the sample text "Main page" is binarized and the binarized Figure 4 Substituting the surrounding pixel values ​​into formula (1) yields m=255. m>255 / 2 indicates that the surrounding areas of the image are more than half white, thus inferring that the background of the binary image is white and the text is black, and the current sample belongs to the case of black text on a white background.

[0070] Step 25: Use a priority queue and heuristic search method to quickly find the optimal rendering configuration for the sample text. The specific steps are as follows:

[0071] Step 251: According to step 22, the approximate size of the sample text is a. Based on this, the font size space is set to {a-1, a, a+1}, and the corresponding priorities are set to 0.3, 0.4, and 0.3 respectively; according to the result returned by step 23, the value space and corresponding probability of the font type, font style, font weight, and rendering engine can be obtained, and the probability of each value is used as its corresponding priority; according to step 24, the value space of the front and back background colors can be obtained as {black text on a white background, white text on a gray background}. Assuming that step 24 determines that the current text is black text on a white background, the corresponding priority is set to {0.9, 0.1}; otherwise, the corresponding priority is set to {0.1, 0.9}.

[0072] Take the Cartesian product of the values ​​of all parameters of the sample text to get a priority queue <r1,r2,…,r n >, r iFor a specific rendering configuration (such as: 14px, Arial font, DirectWrite rendering engine, black text on white background, 400 pounds, italic style), its corresponding priority is the sum of the priorities of each parameter, i = 1, 2, ..., n, where n is the total number of possible rendering configurations.

[0073] Step 252: Select the 200 highest priority rendering configurations in the priority queue to form a new priority queue. <r1,r2,…,r 200 >, each time the rendering configuration with the highest priority is taken out from the queue, and the sample text is rendered into an image using the rendering configuration and template matching is performed with the sample text image obtained in step 21. If the similarity reaches the set threshold, the configuration is considered to be the rendering configuration of the sample text, and the text rendering configuration recognition algorithm is exited; otherwise, the next configuration in the queue is taken until the configuration with the highest similarity is obtained.

[0074] In step 253, in some cases, there may be a prediction error, resulting in the optimal configuration having a low priority, and it is impossible to exit from step 252. In view of this situation, the present invention uses a heuristic search idea based on a genetic algorithm to add new configurations to be tested until a rendering configuration with a similarity that reaches a set threshold is found. Figure 6 The specific steps are as follows:

[0075] Step 2531, initialize the rendering configuration population. Each individual in the population represents a specific rendering configuration (e.g., 12px, Arial font, DirectWrite rendering engine, black text on white background, 400 pt, normal style); and each factor in the configuration represents a gene, such as font size (12px). Replace {r1,…,r 200}Rendering configuration as the initial population.

[0076] Step 2532: Fitness calculation (population evaluation): Use each rendering configuration in the population to render the sample text into an image, and perform template matching with the screen image. The similarity obtained represents the fitness of an individual.

[0077] Step 2533: Select an operation. Based on the fitness of the individuals in the population, select individuals with high fitness from the current population. In other words, select rendering configurations that do not meet the set threshold but have relatively high similarity.

[0078] Step 2534, crossover operation. A crossover operation is performed on the rendering configuration selected in step 2533 to generate a new rendering configuration. This step uses a certain mating probability threshold (pc, generally 0.4 to 0.99) to randomly select crossover points, perform gene exchange, and generate new crossover individuals. For example, the rendering configurations <14px, Arial font, DirectWrite rendering engine, black text on a white background, 400 pounds, italic style> and <16px, Consolas font, pillow rendering engine, white text on a gray background, 600 pounds, normal style> are subjected to multi-point crossover. Assuming the randomly selected positions are 2, 4, and 6, the new rendering configurations <14px, Consolas font, DirectWrite rendering engine, white text on a gray background, 400 pounds, normal style> and <16px, Arial font, pillow rendering engine, black text on a white background, 600 pounds, italic style> are obtained. The new rendering configuration is added to the population, and step 2532 is repeated until an individual whose fitness reaches the set threshold is found (ie, a rendering configuration whose similarity reaches the set threshold is found), and the search is stopped.

[0079] Step 3: For other characters with the same rendering characteristics, render the characters into images according to the automatically identified rendering configuration, and use the template matching algorithm to match the images on the screen to locate the position of the characters. For example, Figure 2 The Chinese text "Recent changes" and the sample text "Main page" share the same rendering characteristics. Based on the automatically identified rendering configuration, the DirectWrite rendering engine renders "Recent changes" into an image using a 12px, black text on a white background, Arial font size 400 pt, Normal style. A template matching algorithm then matches this image to the screen to locate the location of the recent changes.

[0080] The text involved in the present invention can be text on various interfaces such as web pages, system interfaces, and app interfaces.

[0081] Experiments using this invention located text on 3,000 web pages hosted on 100 prominent websites in various fields, including Wikipedia, Amazon, and Sina News. The text was found in a variety of applications, including popular science, online shopping, and entertainment. The results showed that 97% of the text was successfully located, with an average time of approximately 0.3 seconds on a 4k image. This demonstrates that the proposed method can meet the needs of text location in most screen images.

[0082] The embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for optimizing the positioning of text in a screen image, characterized in that: The following steps are involved: S1: For the text to be located, the user is required to provide a set of sample texts and the screen images of the text in the current software running environment; wherein the text to be located has a certain type of rendering feature; S2, in the rendering configuration space composed of a large number of parameters, a text rendering configuration recognition algorithm is used to identify the specific rendering configuration of the sample text; S3, for other characters having the same type of rendering features, rendering the characters into images according to the automatically identified rendering configuration, and using a template matching algorithm to match the images on the screen to locate the positions of the characters; In step S2, the specific rendering configuration of the sample text is identified by using a text rendering configuration recognition algorithm, including the following steps: S21, using an OCR text recognition tool, obtain a small image that closely contains the sample text; S22, using three-variable quadratic polynomial regression to fit the inferred function P size , by the inference function P size Estimate the approximate font size of the sample text; S23, uses a deep learning-based image classification algorithm to intelligently predict the font type, font style, font weight, and rendering engine of the sample text; S24, determining the front and back background colors of the text sample image by an image processing method; S25,uses a priority queue and heuristic search method to find the optimal,rendering configuration for the sample text.

2. The method for optimizing and positioning text in a screen image according to claim 1, wherein: In step S21, when there are other texts around the sample text and the OCR text recognition tool cannot accurately obtain the boundary area where the sample text is located, the following steps are taken: First, a small image that closely contains the sample text is extracted. Then, a contour recognition algorithm is used to obtain the contours in the small image. For each contour, it is determined whether it is a text frame. If the outer frame is a rectangle, the contour area is basically the same as the outer frame area, and the width-to-length ratio meets the characteristics of text, the current contour is considered to be a text frame.

3. The method for optimizing and positioning text in a screen image according to claim 1, wherein: In step S22, the function P is estimated size The steps to estimate the approximate font size of the sample text are as follows: S221, by building a training library, obtain the h corresponding to each training word px 、b up 、b down Features and font size f size , as training data; where h px is the pixel height of the text; b up Is the text containing raised letters? If so, b up =1, otherwise b up =0;b down Indicates whether the text contains a sunken letter. If it does, b down =1, otherwise b down =0; S222, based on the training data, use ternary quadratic polynomial regression to fit the inference function P size , the inferred function P size The input is h px 、b up 、b down , the output is f size ; S223, through contour recognition, obtain the pixel height of the text in the small image obtained in step 21; px 、b up 、b down Substitute into the inference function P size , get the approximate font size of the sample text.

4. The method for optimizing and positioning text in a screen image according to claim 1, wherein: In step S23, the steps for intelligently predicting the font type, font style, font weight, and rendering engine of the sample text are as follows: S231, establishing a text image training sample library for each font type, font style, font weight, and rendering engine; S232: Build a deep learning prediction model based on the residual network ResNet50. Based on the sample library, train the deep learning prediction model from text and images to rendering parameters. S233, using a deep learning prediction model to predict the sample text image obtained in step 21 to obtain a classification probability vector, where each value of the classification probability vector represents the possibility of belonging to the classification.

5. The method for optimizing and positioning text in a screen image according to claim 1, wherein: In step S24, the steps for determining the front and back background colors of the text sample image are as follows: S241, grayscale the image using an image processing method, classifying the various color types into two categories: black text on a white background and white text on a gray background. At the same time, binarize the sample text image, extract the pixel values ​​around the image on the binarized image, and calculate the mean m of the surrounding pixel values: m=(mean(top_row)+mean(bottow_row)+mean(left_colum)+mean(right_colun)) / 4 Where, mean() is the mean function, mean(top_row) represents the mean of the pixel values ​​in the top row of the image, mean(bottow_row) represents the mean of the pixel values ​​in the bottom row of the image, mean(left_colum) represents the mean of the pixel values ​​in the leftmost column of the image, and mean(right_colum) represents the mean of the pixel values ​​in the rightmost column of the image; S242, judging based on the mean m: if the mean m>255 / 2, it means that the image is more than half white around, and it is inferred that the binary image has a white background and black text, which belongs to the category of black text on a white background; otherwise, it means that the image is more than half black around, and the binary image has a black background and white text, which belongs to the category of white text on a gray background.

6. The method for optimizing and positioning text in a screen image according to claim 1, wherein: In step S25, the steps for finding the optimal rendering configuration for the sample text are as follows: S251: According to step 22, the approximate size of the sample text is obtained as a, and the font size space is set to {a-1, a, a+1}, and the corresponding priority is set; according to the result returned by step 23, the value space and corresponding probability of the font type, font style, font weight, and rendering engine are obtained, and the probability of each value is used as its corresponding priority; according to step 24, the value space of the front and back background colors is obtained as {black text on white background, white text on gray background}, and the corresponding priority is set; Take the Cartesian product of the values ​​of all parameters of the sample text to get a priority queue <r1,r2,…,r n >, r i is a specific rendering configuration, i = 1, 2, ..., n, where n is the total number of possible rendering configurations; its corresponding priority is the sum of the priorities of each parameter; Step 252: Select the 200 highest priority rendering configurations in the priority queue to form a new priority queue. <r1,r2,…,r 200 >, each time, take the rendering configuration with the highest priority from the queue, use this rendering configuration to render the sample text into an image, and perform template matching with the sample text image obtained in step 21. If the similarity reaches the set threshold, the configuration is considered to be the rendering configuration of the sample text, and the text rendering configuration recognition algorithm is exited; otherwise, the next configuration in the queue is taken until the configuration with the highest similarity is obtained; In step 253, if the optimal configuration has a low priority due to a prediction error and cannot exit from step 252, a new configuration to be tested is added using a heuristic search method based on a genetic algorithm until a rendering configuration with a similarity that reaches a set threshold is found. The specific steps are as follows: Step 2531: convert {r1,…,r 200 } Render configuration as initial population; Step 2532: Use each rendering configuration in the population to render the sample text into an image, and perform template matching with the screen image. The similarity obtained is the fitness of an individual. Step 2533: select individuals with high fitness from the current population based on the fitness of the individuals in the population; In step 2534, a crossover operation is performed on the rendering configuration selected in step 2533 to generate a new rendering configuration; the new rendering configuration is added to the population, and step 2532 is repeated until an individual whose fitness reaches the set threshold is found, and the search is stopped.

Images