Cross-platform OCR (optical character recognition) method based on dynamic template matching

Abstract

The invention relates to the technical field of image recognition, and provides a cross-platform OCR recognition method based on dynamic template matching, and the method comprises the steps: inputting a to-be-recognized screenshot and a preset platform feature template library, and enabling the platform feature template library to comprise a platform feature template graph; cutting a platform feature region in the to-be-identified screenshot based on the layout features of the platform feature template graph to obtain a platform feature screenshot; the platform feature template graph is zoomed, the similarity between the zoomed platform feature template graph and the platform feature screenshot is calculated, and whether subsequent steps are executed or not is judged based on the similarity and the zooming proportion; and identifying the to-be-identified screenshot corresponding to the platform feature screenshot through an OCR (Optical Character Recognition) technology. According to the method, the accuracy of automatic recognition of the platform can be improved, the recognition recall rate of the text content is improved, and missed text information is effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a cross-platform OCR recognition method based on dynamic template matching. Background Art

[0002] Existing OCR technology has many shortcomings in multi-platform screenshot recognition applications. First, conventional OCR algorithms often only focus on the text information in the screenshot and ignore the platform attributes corresponding to the screenshot, which leads to inaccurate analysis of the screenshot content. Secondly, images captured by different platforms vary greatly in terms of resolution, interface layout, font style, etc. A single model is difficult to adapt to all these changes, which limits the generalization ability of the model and results in poor recognition results on some platforms. Finally, current multimodal methods have not yet been able to effectively integrate image and text information to determine the platform of the screenshot source and classify the content, resulting in cross-platform recognition tasks still facing the problems of low accuracy and poor robustness. Summary of the Invention

[0003] In order to help solve the above technical problems, this application provides a cross-platform OCR recognition method based on dynamic template matching, which adopts the following technical solutions:

[0004] A cross-platform OCR recognition method based on dynamic template matching, wherein the cross-platform OCR recognition method based on dynamic template matching includes:

[0005] S1: Input the screenshot to be identified and the preset platform feature template library, which includes the platform feature template map;

[0006] S2: cropping the platform feature area in the to-be-identified screenshot based on the layout features of the platform feature template to obtain a platform feature screenshot;

[0007] S3: scaling the platform feature template image, calculating the similarity between the scaled platform feature template image and the platform feature screenshot, and determining whether to execute step S4 based on the similarity and the scaling ratio;

[0008] S4: Identify the screenshot to be identified corresponding to the platform feature screenshot using OCR technology.

[0009] Preferably, step S3 includes:

[0010] S31: performing grayscale processing on the platform feature template image and the platform feature screenshot;

[0011] S32: scaling the size of the platform feature template map using a preset step size, where each size corresponds to a similarity heat map;

[0012] S33: For each scaled platform feature template image of different sizes, slide it over the image to be detected in a preset order, extract an image window of the same size as the platform feature template image, and calculate a similarity score;

[0013] S34: Each pixel value in the heat map represents the similarity score between the template and the corresponding position of the image. The highest similarity score is found in the heat map, and it is determined whether to execute step S4 based on the highest similarity score and the scaling ratio.

[0014] Preferably, the step S33 includes: calculating the similarity score by:

[0015]

[0016] Among them, T(u,v) is the gray value of the template pixel, is the grayscale mean of the template, I(x+u,y+v) is the grayscale value of the image at position (x+u,y+v), and Ix,y is the grayscale mean of the image window of the same size as the template.

[0017] Preferably, the step S34 includes: if the highest similarity score is greater than or equal to a preset similarity score threshold, it is considered that the platform feature template image of the current size and the platform feature screenshot are matched successfully, and step S4 is executed; if the highest similarity score is less than the preset similarity score threshold and the scaling ratio of the platform feature template image is less than the scaling ratio threshold, the platform feature screenshot is re-matched based on the next platform feature template image, and step S33 is re-executed; if the highest similarity score is less than the preset similarity score threshold and the scaling ratio of the platform feature template image is greater than or equal to the scaling ratio threshold, step S3 is exited, the corresponding screenshot to be identified is marked as abnormal, and manual review is triggered.

[0018] Preferably, the step S2 includes: detecting the background color depth of the screenshot to be identified, and if the background is dark, performing color inversion processing on the platform feature template image.

[0019] Preferably, the platform features of the platform feature template diagram at least include the navigation bar position and function button distribution.

[0020] In summary, compared with the prior art, this application has the following beneficial effects:

[0021] 1. The accuracy of the platform's automatic recognition has been significantly improved;

[0022] 2. Improved the recognition recall rate of text content and effectively reduced missed text information;

[0023] 3. In a multi-platform mixed application environment, compared with traditional solutions that rely solely on OCR text, the model of this application shows stronger adaptability and stability, and the overall recognition performance has been significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flowchart of an embodiment of a cross-platform OCR recognition method based on dynamic template matching in the present application;

[0025] Figure 2 A schematic diagram of an embodiment of the platform feature template diagram of the present application;

[0026] Figure 3 A schematic diagram of an embodiment of a platform feature screenshot of the present application. DETAILED DESCRIPTION

[0027] The present invention will be further described below with reference to the accompanying drawings. The structure and principle of the present invention will be very clear to those skilled in the art. It should be understood that the specific embodiments described herein are only intended to explain the present invention and are not intended to limit the present invention.

[0028] Figure 1 This is a flow chart of an embodiment of a cross-platform OCR recognition method based on dynamic template matching in this application. Figure 2 This is a schematic diagram of an embodiment of the platform feature template diagram of this application. Figure 3 A schematic diagram of an embodiment of a platform feature screenshot of the present application.

[0029] Cross-platform OCR recognition methods based on dynamic template matching include:

[0030] S1: Input the screenshot to be identified and the preset platform feature template library, which includes the platform feature template image. In step S1, input the screenshot to be identified and the preset platform feature template library (including typical layout, icon style, color distribution and other features of different platforms).

[0031] S2: Based on the layout features of the platform feature template, the platform feature area in the screenshot to be identified is cropped to obtain the platform feature screenshot. In step S2, the background color depth of the screenshot to be identified is first detected. If the background is dark, the platform feature template is color-inverted.

[0032] Specifically, step S2 may include two steps.

[0033] Step S21: Background color determination

[0034] Detect the color depth of the screenshot background. If the background is light (such as white / light gray), proceed directly to the next step. If the background is dark, invert the color of the template (such as black and white) to ensure that the template is compatible with the screenshot background.

[0035] Step S22: Crop key areas

[0036] According to the layout features of the platform template (such as the location of the navigation bar and the distribution of function buttons), the area containing the feature template in the screenshot is automatically cropped to remove the interfering background.

[0037] S3: Scale the platform feature template image, calculate the similarity between the scaled platform feature template image and the platform feature screenshot, and determine whether to execute step S4 based on the similarity and the scaling ratio. In step S3, specifically, step S3 includes:

[0038] S31: grayscale processing is performed on the platform feature template image and the platform feature screenshot.

[0039] S32: scaling the size of the platform feature template map by a preset step size, where each size corresponds to a similarity heat map.

[0040] S33: For each scaled platform feature template image of different sizes, slide it over the image to be detected in a preset order, extract an image window of the same size as the platform feature template image, and calculate the similarity score. The similarity score is calculated in the following way:

[0041]

[0042] Among them, T(u,v) is the gray value of the template pixel, is the grayscale mean of the template, I(x+u,y+v) is the grayscale value of the image at position (x+u,y+v), and Ix,y is the grayscale mean of the image window of the same size as the template.

[0043] Normalization fixes the score range to [-1, 1], where 1 indicates a perfect match (positive correlation), 0 indicates no correlation, and -1 indicates negative correlation (the template is the negative of the window). Subtracting the mean eliminates overall brightness differences, and dividing by the respective standard deviations achieves scale-independence and contrast-independence.

[0044] S34: Each pixel value in the heat map represents the similarity score between the template and the corresponding position of the image. The highest similarity score is found in the heat map, and whether to execute step S4 is determined based on the highest similarity score and the scaling ratio. Step S34 includes: if the highest similarity score is greater than or equal to the preset similarity score threshold, it is considered that the platform feature template image of the current size and the platform feature screenshot are matched successfully, and step S4 is executed. If the highest similarity score is less than the preset similarity score threshold (0.9) and the scaling ratio of the platform feature template image is less than the scaling ratio threshold, the platform feature screenshot is re-matched based on the next platform feature template image, and step S33 is re-executed. If the highest similarity score is less than the preset similarity score threshold and the scaling ratio of the platform feature template image is greater than or equal to the scaling ratio threshold, step S3 is exited, and the corresponding screenshot to be identified is marked as abnormal, triggering manual review.

[0045] Specifically, a scaling template is applied to the cropped screenshot, and the similarity between the template and the screenshot is calculated. The color channels are separated and template matching is performed on each color channel. Finally, the matching results of all channels are averaged and merged to determine whether there is a high degree of match: if the match is successful, the OCR recognition stage is entered; if the match fails and the scaling ratio does not exceed the threshold (such as ±20%), the template scaling ratio is adjusted and the matching is repeated; if the scaling ratio exceeds the limit and still cannot be matched, it is marked as abnormal data and triggers manual review.

[0046] In step S3, the image to be detected is first converted to grayscale, removing the color dimension and retaining only the brightness information to reduce computational complexity. The template image is also converted to grayscale. Template matching utilizes only intensity information, so converting from color to grayscale does not compromise positioning accuracy. Using a preset step size of scale_step (0.1), the template is scaled to different sizes. A similarity heatmap is generated for each scale.

[0047] S4: Use OCR technology to identify the screenshots to be identified that correspond to the platform feature screenshots. Step S4 is used for multimodal joint output, fusing the OCR results with the original image features (such as table boundaries and icon positions) to generate structured text output (such as JSON format) that preserves the layout information.

[0048] Step S41: Raw data analysis

[0049] The function parses the raw OCR results returned by the AI ​​model, cleans non-standard characters (such as Chinese character interference), extracts valid dates and text, converts the string into a dictionary using json.loads, and processes the key-value pair mapping.

[0050] Step S42: Spatial information association

[0051] Dynamically bind the coordinates obtained by template matching (such as the location of the like icon) to the adjacent text extracted by OCR (such as the number of likes). The region coordinate field is retained in the JSON (requires extending AIOcrModel to add the position field).

[0052] Step S43: Structured output generation

[0053] Compared with the prior art, this application has the following beneficial effects:

[0054] 1. The platform's automatic recognition accuracy has been significantly improved, reaching a high accuracy of approximately 98%;

[0055] 2. The recognition recall rate of text content has been increased to approximately 90%, effectively reducing missed text information;

[0056] 3. In a multi-platform mixed application environment, compared with traditional solutions that rely solely on OCR text, the model of this application shows stronger adaptability and stability, and the overall recognition performance has been significantly improved.

Claims

1. A cross-platform OCR recognition method based on dynamic template matching, characterized in that: Cross-platform OCR recognition methods based on dynamic template matching include: S1: Input the screenshot to be identified and the preset platform feature template library, which includes the platform feature template map; S2: cropping the platform feature area in the to-be-identified screenshot based on the layout features of the platform feature template to obtain a platform feature screenshot; S3: scaling the platform feature template image, calculating the similarity between the scaled platform feature template image and the platform feature screenshot, and determining whether to execute step S4 based on the similarity and the scaling ratio; S4: Identify the screenshot to be identified corresponding to the platform feature screenshot using OCR technology.

2. The cross-platform OCR recognition method based on dynamic template matching according to claim 1, characterized in that: The step S3 comprises: S31: performing grayscale processing on the platform feature template image and the platform feature screenshot; S32: scaling the size of the platform feature template map using a preset step size, where each size corresponds to a similarity heat map; S33: For each scaled platform feature template image of different sizes, slide it over the image to be detected in a preset order, extract an image window of the same size as the platform feature template image, and calculate a similarity score; S34: Each pixel value in the heat map represents the similarity score between the template and the corresponding position of the image. The highest similarity score is found in the heat map, and it is determined whether to execute step S4 based on the highest similarity score and the scaling ratio.

3. The cross-platform OCR recognition method based on dynamic template matching according to claim 2, characterized in that: The step S33 includes: calculating the similarity score by: Among them, T(u,v) is the gray value of the template pixel, is the grayscale mean of the template, I(x+u,y+v) is the grayscale value of the image at position (x+u,y+v), and Ix,y is the grayscale mean of the image window of the same size as the template.

4. The cross-platform OCR recognition method based on dynamic template matching according to claim 2, characterized in that: The step S34 includes: if the highest similarity score is greater than or equal to the preset similarity score threshold, it is considered that the platform feature template image of the current size and the platform feature screenshot are matched successfully, and step S4 is executed; if the highest similarity score is less than the preset similarity score threshold and the scaling ratio of the platform feature template image is less than the scaling ratio threshold, the platform feature screenshot is re-matched based on the next platform feature template image, and step S33 is re-executed; if the highest similarity score is less than the preset similarity score threshold and the scaling ratio of the platform feature template image is greater than or equal to the scaling ratio threshold, step S3 is exited, the corresponding screenshot to be identified is marked as abnormal, and manual review is triggered.

5. The cross-platform OCR recognition method based on dynamic template matching according to claim 1, characterized in that: The step S2 includes: detecting the background color depth of the screenshot to be identified, and if the background is dark, performing color inversion processing on the platform feature template image.

6. The cross-platform OCR recognition method based on dynamic template matching according to claim 1, characterized in that: The platform features of the platform feature template diagram at least include the navigation bar position and function button distribution.

Images