Text recognition method and text recognition apparatus
By acquiring window structure tree data to identify and remove target image elements, the input image is purified. Combined with OCR recognition technology, this solves the problem of text recognition errors in complex interfaces and achieves higher accuracy and precision.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2026-01-30
- Publication Date
- 2026-06-05
AI Technical Summary
Existing optical character recognition (OCR) technology cannot distinguish window node attributes when recognizing text on a screen, resulting in content interference between adjacent or similar text, leading to incorrect recognition results and affecting user experience.
By acquiring the window structure tree data of the target interface, target image elements are identified and removed to purify the input image. An optical character recognition engine is used to identify text content and location, and the target text and location information are matched with the window structure tree data.
It improves the accuracy of OCR recognition, eliminates interference from non-textual information, and enables precise extraction and positioning of specific text information in complex graphical user interfaces.
Smart Images
Figure CN122157219A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of text recognition processing, and in particular to a text recognition method and a text recognition device. Background Technology
[0002] Recognizing and extracting text content on a screen is typically done using Optical Character Recognition (OCR), which converts text from screenshots into text format. Currently, OCR usually takes the entire screen image as input, without distinguishing viewport node attributes. This type of OCR is prone to errors; for example, text in parts of the image or similar text can interfere with the recognition results of adjacent nodes, leading to incorrect results and impacting the user experience. Summary of the Invention
[0003] This application provides a text recognition method and a text recognition device.
[0004] On one hand, embodiments of this application provide a data processing method, including:
[0005] Get a screenshot of the first screen of the target interface; Identify the target image elements in the first screenshot that belong to the target type; Remove the target image element from the first screenshot to obtain the second screenshot; Optical character recognition is performed on the second interface screenshot to obtain recognition results, which include text content recognition results and text location recognition results. Target information is determined based on the recognition result. The target information includes target text and the target position information of the target text in the target interface. The target text corresponds to a portion of the text in the text content recognition result.
[0006] Optionally, determining the target image element in the first screenshot includes: Obtain the window structure tree data corresponding to the target interface, wherein the window structure tree data contains the attribute information of each view node in the target interface; Based on the attribute information of each view node, the first view node in the target interface is determined; The target image element in the first interface screenshot is determined based on the first view node.
[0007] Optionally, determining the target view node in the target interface based on the attribute information includes: Based on at least one of the control type property or view type property of each view node, determine the view node whose property is at least one of image, icon, button or custom drawing area as the target view node.
[0008] Optionally, removing the target image element from the first screenshot includes: The view position of the target view node in the target interface is obtained based on the attribute information of the target view node; Remove the target image element from the first interface screenshot based on the view position.
[0009] Optionally, determining the target information based on the recognition result includes: Based on the attribute information of each view node, a second view node in the target interface is determined, and the second view node is different from the first view node. The target text is determined based on the attribute information of the second view node, wherein the attribute information of the second view node includes the text information in the second view node; The target location information is determined based on the target text and the text location recognition result.
[0010] Optionally, determining the target location information based on the target text and the text location recognition result includes: Based on the target text, a matching text is determined from the text content recognition result, and the matching text and the target text satisfy a similarity condition; The location information of the matching text in the first interface screenshot is determined based on the text location recognition result; The target location information is determined based on the location information of the matched text in the screenshot of the first interface.
[0011] Optionally, determining the target image element in the first screenshot includes: The first screenshot is segmented to obtain multiple segmentation elements; The segmentation elements are identified by type, and the segmentation elements belonging to the target type are determined as the target image elements.
[0012] Optionally, determining the target information based on the recognition result includes: The target text is determined from the text content recognition results; The location information of the target text is determined as the target location information based on the text location recognition result.
[0013] Optionally, after determining the target information based on the recognition result, the method further includes at least one of the following: Based on the target location information, display the identification information corresponding to the target text on the target interface; The target information is output to the target application, which is able to perform the target task based on the generative language model and the target information.
[0014] On the other hand, embodiments of this application also provide a text recognition device, including: The acquisition module is used to capture a screenshot of the first screen of the target interface. The determination module is used to determine the target image elements in the first screenshot that belong to the target type; The first processing module is used to remove the target image element from the first interface screenshot to obtain a second interface screenshot. The recognition module is used to perform optical character recognition on the second interface screenshot to obtain recognition results, which include text content recognition results and text location recognition results. The second processing module is used to determine target information based on the recognition result. The target information includes target text and target location information of the target text in the target interface. The target text corresponds to a portion of the text in the text content recognition result. Attached Figure Description
[0015] Figure 1 This is a flowchart of the text recognition method according to an embodiment of this application; Figure 2 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S200; Figure 3 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S300; Figure 4 Examples of embodiments of this application Figure 1 A flowchart of one embodiment of step S500; Figure 5 Examples of embodiments of this application Figure 4 A flowchart of an embodiment of step S530; Figure 6 Examples of embodiments of this application Figure 1 A flowchart of another embodiment of step S200; Figure 7 Examples of embodiments of this application Figure 1 A flowchart of another embodiment of step S500; Figure 8This is a schematic diagram of the structure of the text recognition device according to an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0016] Various embodiments and features of this application are described herein with reference to the accompanying drawings.
[0017] It should be understood that various modifications can be made to the embodiments described herein. Therefore, the above description should not be considered as limiting, but merely as an example of embodiments. Other modifications within the scope and spirit of this application will be apparent to those skilled in the art.
[0018] The accompanying drawings, which are included in and form part of this specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.
[0019] These and other features of this application will become apparent from the following description of preferred forms of embodiments given as non-limiting examples, with reference to the accompanying drawings.
[0020] It should also be understood that although this application has been described with reference to some specific examples, those skilled in the art can certainly implement many other equivalent forms of this application.
[0021] The above and other aspects, features and advantages of this application will become more apparent when taken in conjunction with the accompanying drawings and in view of the following detailed description.
[0022] Specific embodiments of this application are described thereafter with reference to the accompanying drawings; however, it should be understood that the claimed embodiments are merely examples of this application, which can be implemented in various ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that could obscure the application. Therefore, the specific structural and functional details claimed herein are not intended to be limiting, but merely serve as the basis and representative basis for the claims to teach those skilled in the art to use this application in a variety of substantially any suitable detailed structures.
[0023] This specification may use the phrases “in one embodiment,” “in another embodiment,” “in yet another embodiment,” or “in other embodiments,” all of which may refer to one or more of the same or different embodiments according to this application.
[0024] Figure 1 A flowchart of a text recognition method according to an embodiment of this application is shown. An embodiment of this application provides a text recognition method, such as... Figure 1 As shown, it includes: S100, obtain a screenshot of the first screen of the target interface; The text recognition method of this embodiment is applied to electronic devices, which can be mobile terminals, tablets, computers, or other smart devices with display screens and computing capabilities. This method is applicable to extracting text from any graphical user interface currently displayed on the electronic device, i.e., the target interface. The target interface can be the entire physical display screen of the device, the operating system desktop, the main window or pop-up of an application, the content of a tab in a web browser, or a virtual interface presented through screen mirroring, remote desktop, or other methods. Specifically, a complete pixel image of the current target interface, i.e., a first interface screenshot, can be obtained by calling the screenshot interface provided by the operating system of the electronic device. The obtained first interface screenshot can be a bitmap format containing RGB or RGBA color channels, such as BMP or PNG, represented in memory, with a resolution consistent with the display resolution of the target interface.
[0025] S200, determine the target image element in the first interface screenshot that belongs to the target type; In this embodiment, the target type can refer to the type of non-text visual element that may interfere with the optical character recognition (OCR) process, and may include at least one of the following: icons, photos, graphic buttons, logos, decorative graphics, and graphic areas generated by custom drawing, etc. The target type can also be an element type that is manually identified as affecting the recognition of the main text content in the interface. Image areas belonging to the target type, i.e., target image elements, are located from the first interface screenshot. Target image elements can be non-text types in the first interface screenshot that may interfere with the subsequent optical character recognition (OCR) process. Specifically, the view hierarchy structure data of the target interface can be determined by obtaining window structure tree data. The window structure tree data, displayed in a tree structure, describes in detail the attributes of each view node in the interface, such as: node class name, resource identifier, boundary coordinates on the screen, text content, visibility, clickable state, etc. By parsing the window structure tree data, traversing all view nodes, nodes belonging to the target type are filtered based on the node's attribute information. The specific filtering method can be based on the node's control type attribute or view type attribute. For each identified node belonging to the target type, its position information on the screen, such as the coordinates of the rectangular bounding box, is extracted from its attributes. Based on this coordinate information, it is mapped onto the first screenshot of the interface, and the image content within that rectangular area can be identified as the target image element. The window structure tree is a tree-like structure that records the type, position, content, and other attributes of elements in the interface, as well as the relationships between elements.
[0026] S300, Remove the target image element from the first interface screenshot to obtain a second interface screenshot; In this embodiment, the identified target image elements are visually eliminated or weakened to reduce their interference with the subsequent optical character recognition process, thereby purifying the input image. Based on the precise location information of each target image element, image processing operations are performed on the corresponding areas of the first interface screenshot. Specific removal or processing methods may include pixel replacement, blurring, or cropping. Pixel replacement replaces all pixel values within the target image element area with a uniform background color, allowing the processed area to blend naturally with the surrounding background. Blurring eliminates the original clear graphic outlines and details within the target image element area, making its content unrecognizable and preventing misidentification as characters by OCR. Region cropping invalidates the region markings corresponding to the target image element, effectively removing the interfering area of the target image element. After processing with any or a combination of pixel replacement, blurring, and cropping, the resulting image is the second interface screenshot, which visually eliminates the direct influence of non-textual interference elements.
[0027] S400, Optical character recognition is performed on the second interface screenshot to obtain recognition results, the recognition results including text content recognition results and text position recognition results; In this embodiment, the screenshot of the second interface is input into an Optical Character Recognition (OCR) engine for processing. The OCR engine can accurately perform text detection, character segmentation, and character recognition. The recognition results include text content recognition results and text location recognition results. The text content recognition results can be a collection of all text content recognized by the OCR engine from the screenshot of the second interface. The text content recognition results are usually organized in the form of a list or sequence, where each item corresponds to a recognized text unit (such as a text block, a line of text, or a word). The content of each item is the recognized string. The text location recognition results are associated with each item in the text content recognition results and are used to describe the position of the text content in the two-dimensional image space of the screenshot of the second interface. This position information is usually represented in the form of bounding box coordinates, for example, using the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of a rectangle to define the area occupied by the text in the image. In this embodiment, the second screenshot of the interface is a cleaned image, in which interference sources such as icons and button graphics that may be misidentified as characters have been removed. Therefore, the probability of such misidentification by the OCR engine is greatly reduced; the contrast between the text area and the background is clearer, which helps the OCR engine to more accurately delineate the text bounding box.
[0028] S500, determine target information based on the recognition result, the target information including target text and target position information of the target text in the target interface, the target text corresponding to a portion of the text in the text content recognition result.
[0029] In this embodiment, by combining the recognition results of optical character recognition with the window structure tree data, the target text of interest to the user and its precise position in the target interface are located and extracted. Specifically, firstly, view nodes with text control types are searched from the window structure tree data; the attributes of these nodes are parsed to obtain their text attribute values. Specific information fragments are selected or parsed from these text contents as target text using preset rules (such as regular expression matching, keyword recognition, entity recognition models) or according to specific application requirements. The target text is matched with each text fragment in the text content recognition results returned by OCR, which can be based on string similarity. The text fragment that best matches or contains the target text is found in the OCR recognition results and marked as the corresponding text. Based on the text position recognition results returned by OCR, the precise bounding box coordinates of the corresponding text in the second interface screenshot are obtained. The second interface screenshot and the first interface screenshot (i.e., the visual mapping of the original target interface) are consistent in the coordinate system; these bounding box coordinates reflect the actual display area of the target text on the screen, and this coordinate information is the target position information.
[0030] This application, through the aforementioned method, identifies and removes target image elements based on view attributes, providing a purified input image for OCR recognition, eliminating interference from non-textual information, and improving the accuracy of OCR recognition. It integrates the text content provided by the view tree with the location information provided by OCR recognition, matching the target text parsed from the window structure tree with the location information provided by OCR recognition, thus enabling the extraction and location of specific text information in complex graphical user interfaces, such as the extraction and location of key information like time and location information.
[0031] In one embodiment of this application, such as Figure 2 As shown, determining the target image element in the first interface screenshot includes: S210, obtain the window structure tree data corresponding to the target interface, the window structure tree data containing the attribute information of each view node in the target interface; In this embodiment, the operating system of the electronic device can provide an interface to access the current application interface structure. By calling this interface, the system can return a structured data set, namely, window structure tree data. The window structure tree data describes the hierarchical relationship and attributes of all view nodes in the current interface. Each view node contains a set of attributes describing its state and characteristics. Attribute information may include: control type (text edit box, image button); coordinate position; text content contained in the node; and the node's unique identifier.
[0032] S220, Based on the attribute information of each view node, determine the first view node in the target interface; In this embodiment, after obtaining the window structure tree data corresponding to the target interface, the window structure tree data is parsed, and each view node in the window structure tree data is traversed. For the attribute information of each view node, it is determined whether it belongs to the target type, that is, whether the view node contains non-text element types that interfere with OCR. The first view node can be a view node identified as belonging to the target type. The specific method for determining the first view node can be based on the attribute information of the view node for filtering. For example, the control type attribute of the view node can be checked. If the control type attribute of the view node belongs to a non-text control, the view node can be determined as the first view node. If the control type attribute of the view node belongs to a text control, the view node is not determined as the first view node.
[0033] S230, determine the target image element in the first interface screenshot based on the first view node.
[0034] In this embodiment, for each determined first view node, position information, which can be coordinates, is extracted from its attributes. These coordinates are then mapped as a rectangular region onto the first interface screenshot. All pixels within the rectangular region of the first interface screenshot can be identified as target image elements. Specifically, coordinate information can be extracted from the attributes of the determined first view node. This coordinate information can be rectangular coordinates, defining the position of the image button on the screen. These rectangular coordinates are then mapped to the corresponding position in the first interface screenshot. The image content within the coordinate region in the first interface screenshot can be identified as the target image element.
[0035] For example, when a hotel booking application is opened on an electronic device, its current interface displays a hotel information card, including: the hotel name "Beijing Wangfujing Chunhao Hotel Main Branch" displayed in a text edit box control; and a "right arrow" icon for switching hotels displayed in an image button control, adjacent to the right side of the text edit box. Based on the attributes of the image button control, the "right arrow" icon is identified as the first view node. From the attributes of the first view node, its rectangular coordinates [860,240,940,260] are extracted. These coordinates represent a rectangular area that defines the position of the image button on the screen, with the top-left corner coordinates (860,240) and the bottom-right corner coordinates (940,260). The rectangular coordinates [860,240,940,260] are mapped to the corresponding position in the screenshot of the first interface. In the screenshot of the first interface, the image content located within the area of coordinates [860,240,940,260] is determined as the target image element, namely the "right arrow" icon.
[0036] In one embodiment of this application, determining the target view node in the target interface based on the attribute information includes: Based on at least one of the control type property or view type property of each view node, determine the view node whose property is at least one of image, icon, button or custom drawing area as the target view node.
[0037] In this embodiment, a window structure tree describing the current interface is obtained through a system interface. This structure tree contains multiple attributes for each view node. Based on the view node's control type attribute or view type attribute, it is determined whether the view node belongs to one of the following categories: image, icon, button, or custom drawing area. The control type attribute can be a low-level technical attribute describing the essence of a node in the window structure tree, typically corresponding to the category name of its implementation class. The view type attribute can be a semantic classification attribute or a logical type derived from the control type attribute. For example, the program iterates through all parsed view nodes, extracting the attribute information of each node. If a node's attribute is a button image, it can be determined that the node is a target view node; if another node's attribute is a custom drawing view, it can be determined that the node is a target view node; if another node's attribute is a text input control, it can be excluded from the target view node list.
[0038] In one embodiment of this application, such as Figure 3 As shown, removing the target image element from the first interface screenshot includes: S310, Obtain the view position of the target view node in the target interface based on the attribute information of the target view node; S320, Remove the target image element from the first interface screenshot according to the view position.
[0039] In this embodiment, after determining the target image element in the first interface screenshot, the target image element is removed from the first interface screenshot. First, the view position of the target view node in the target interface is obtained based on the attribute information of the target view node. Specifically, the attribute information of the selected target view nodes is parsed, and the spatial position information of the target view node in the target interface is extracted from the attribute information. The spatial position information of the target view node in the target interface can be determined based on the view structure tree data corresponding to the target view node. The spatial position information, i.e., the coordinate information corresponding to the target view node, is extracted from the view structure tree data corresponding to the target view node. After obtaining the view position of the target view node in the target interface, the target image element is removed from the first interface screenshot based on the view position, visually removing or weakening the image content corresponding to that position. Pixel replacement, blurring, or region cropping operations can be performed on the target image element to obtain the second interface screenshot, eliminating visual interference from non-text elements. The view position of the target view node in the target interface includes the positions of all vertices in the area where the target view node is located, or the position of a single vertex in the area where the target view node is located. If the view position in the target interface is the position of a single vertex in the area where the target view node is located, the attribute information of the target view node also includes the length and width information of the area where the target view node is located. Based on the view position, the area range where the target view node is located in the target interface can be determined, and then this area range can be processed to remove the target image element.
[0040] For example, in a hotel booking application interface, a non-text target view node is identified. The node representing the "right arrow" toggle button is determined as the target view node, with the attribute of an image button control. The rectangular coordinates [860, 240, 940, 260] are extracted from the view structure tree data corresponding to the target view node. These coordinates represent a rectangular area that defines the position of the image button on the screen: top-left corner coordinates (860, 240), bottom-right corner coordinates (940, 260). The rectangular coordinates [860, 240, 940, 260] are mapped to the corresponding position in the first screenshot. In the first screenshot, the image content located within the coordinate area [860, 240, 940, 260] is identified as the target image element, namely the "right arrow" icon. Pixel replacement, blurring, or region cropping operations can be performed on the "right arrow" icon.
[0041] In one embodiment of this application, such as Figure 4 As shown, determining the target information based on the recognition result includes: S510, based on the attribute information of each view node, determine the second view node in the target interface, the second view node being different from the first view node; In this embodiment, based on the attribute information of each view node, a second view node in the target interface is determined. The second view node differs from the first view node; the first view node can represent non-text elements, such as icons and buttons, while the second view node can represent text content elements. Nodes containing text content can be extracted based on the view structure tree data corresponding to the view nodes. Specifically, based on the attribute information of the view nodes, all view nodes in the view structure tree data are traversed, the attributes of each view node are parsed, and nodes whose attributes belong to the text control category are determined as second view nodes. For example, in a hotel booking application interface, a text edit box control node displaying the hotel name is identified in the display interface and determined as a second view node; and image button nodes identified as first view nodes are excluded.
[0042] S520, determine the target text based on the attribute information of the second view node, wherein the attribute information of the second view node includes the text information in the second view node; In this embodiment, after determining the second view node in the target interface, the target text is determined based on the attribute information of the second view node. The attribute information of the second view node includes the text information within the second view node, and the content of the target text can originate from the text attributes of the second view node. Specifically, after determining the second view node, the specific text fragments required by the user can be parsed from the second view node. For example, the text attribute information of the second view node may correspond to a sentence or a paragraph, and the required text information can be extracted based on the view structure tree data corresponding to the second view node.
[0043] S530, determine the target location information based on the target text and the text location recognition result.
[0044] In this embodiment, after determining the target text based on the attribute information of the second view node, the target position information of the target text in the target interface is determined. Specifically, an OCR engine can be used to identify the text position of the target text, thereby determining the target position information of the target text in the target interface. For example, after extracting the required target text based on the window structure tree data corresponding to the second view node, the OCR engine can be used to identify the coordinate information of the target text. The coordinate information of the target text can represent the actual display area of the target text on the target interface, and this coordinate information can be determined as the target position information of the target text.
[0045] For example, on the confirmation page of a hotel booking application, the text attribute of the second view node is: "Order Number: ORD123456, Total Price: ¥1,288". The user needs the text information of the order number. Based on the view structure tree data corresponding to the second view node, the string "ORD123456" can be extracted as the target text. Then, using an OCR engine, the coordinates of the target text are identified as [200, 400, 350, 430]. The target text "ORD123456" is then matched with the OCR recognition result to obtain its coordinates [200, 400, 350, 430]. The final target information is: {Target Text: "ORD123456", Target Location Information [200, 400, 350, 430]}.
[0046] Since the text content recognized by OCR may contain errors, but the text location recognition is relatively accurate, and the text displayed by the second view node recorded from the attribute information of the second view node is accurate, the text content of the target text can be directly determined from the attribute information of the second view node, and the location information of the target text can be matched from the recognition result of OCR, resulting in a more accurate result.
[0047] In one embodiment of this application, such as Figure 5 As shown, determining the target location information based on the target text and the text location recognition result includes: S5301, determine matching text from the text content recognition result based on the target text, wherein the matching text and the target text satisfy a similarity condition; S5302, determine the position information of the matching text in the first interface screenshot based on the text position recognition result; S5303, determine the target location information based on the location information of the matched text in the screenshot of the first interface.
[0048] In this embodiment, after determining the target text in the second view node using the window structure tree data, OCR recognition is performed on the second interface screenshot to obtain a recognition result containing all recognized text and their positions. Matching text is then determined from the text content recognition result based on the target text, provided that the matching text and target text meet a similarity condition. This similarity condition can be a configurable rule, specifically determined by matching strings. That is, among a large number of text fragments recognized by OCR, matching text corresponding to the target text is identified. Specifically, the string corresponding to the target text obtained from the window structure tree data is compared with the string recognized by OCR, and the portion of the OCR-recognized string that is identical to the string corresponding to the target text is determined as the matching text.
[0049] After determining the matching text from the text content recognition results based on the target text, the position information of the matching text in the first interface screenshot is determined based on the text position recognition results. The text position recognition results output by the OCR method correspond one-to-one with the text content recognition results. The corresponding position coordinates can be extracted from the matching text position recognition results, which is the position information of the matching text in the first interface screenshot. After determining the position information of the matching text in the first interface screenshot based on the text position recognition results, the target position information is determined based on the position information of the matching text in the first interface screenshot. The second interface screenshot is obtained by locally modifying the pixels of the first interface screenshot. This process does not involve any image geometric transformations that would cause pixel coordinate misalignment; the second interface screenshot and the first interface screenshot share the same pixel coordinate system. The position information of the matching text in the second interface screenshot is consistent with its position information in the first interface screenshot. The first interface screenshot is a direct mapping of the target interface, and the position information of the matching text in the first interface screenshot can be determined as the final target position information of the target text. Specifically, the corresponding position coordinates of the matching text in the first interface screenshot can be determined as the final target position information.
[0050] For example, on the confirmation page of a hotel booking application, the text attribute of the second view node is: "Order Number: ORD123456, Total Price: ¥1,288". The user needs the text information of the order number. Based on the view structure tree data corresponding to the second view node, the string "ORD123456" can be extracted as the target text. Then, using the OCR engine, a list of text strings is generated, including ["Order Number:", "ORD123456", "Total Price:", "¥1,288"]. "ORD123456" is compared with each item in the OCR recognition results. Since "ORD123456" completely matches the second item in the OCR result list, the second item in the OCR result list can be determined as the matching text. The text location recognition result output by the OCR engine corresponds one-to-one with the text content recognition result. The coordinate information [200, 400, 350, 430] corresponding to the matching text can be extracted. In image space, the matching text "ORD123456" is recognized within a rectangular area with the upper left corner at (200, 400) and the lower right corner at (350, 430). The target text determined by the window structure tree data and the matching text obtained by OCR recognition are confirmed to be the same content. The position of the matching text on the screen is the position of the target text on the screen. The coordinate information [200, 400, 350, 430] in the OCR recognition result can be determined as the target position information of the target text "ORD123456" on the original target screen interface; this coordinate information represents the display area of the characters "ORD123456" on the screen.
[0051] In one embodiment of this application, such as Figure 6 As shown, determining the target image element in the first interface screenshot includes: S240, the first interface screenshot is segmented to obtain multiple segmentation elements; S250, perform type identification on the segmented elements, and determine the segmented elements belonging to the target type as the target image elements.
[0052] In this embodiment, the target image element in the first interface screenshot can be determined by performing computer vision analysis on the screenshot. For example, when it is impossible to parse the image using the window structure tree data, image analysis can be used to identify the target image element. Specifically, the first interface screenshot is segmented to obtain multiple segmentation elements. That is, the screenshot is decomposed into multiple semantically independent visual units. Segmentation can be the process of dividing the input image into multiple regions, each region corresponding to a visual object. A deep learning-based image segmentation model can be used to perform pixel segmentation on the screenshot and output a set of segmentation results. For example, using an image segmentation model, the application interface can be segmented into multiple independent segmentation elements such as: "number display area", "number buttons", "operator buttons", "equal sign buttons", "application title bar", and "background".
[0053] The first screenshot of the interface is segmented to obtain multiple segmented elements. These elements are then type-identified, and those belonging to the target type are identified as target image elements. From all the segmented elements in the first screenshot, non-textual interference elements that need processing are selected. Specifically, for each segmented element, its semantic type can be determined to ascertain whether it is a non-textual interference element. Specifically, the semantic type of each element can be directly determined using the segmentation results output by an image segmentation model; alternatively, an image classification model can be used to crop the image region corresponding to each segmented element and then use the image classification model to determine the semantic type of the segmented element. The image classification model can be used to distinguish interface categories such as "text blocks," "icons," "buttons," "images," and "decorative lines." All obtained segmented elements are traversed, and their types are checked. If a segmented element belongs to a non-textual visual element type that may interfere with OCR, such as "icons," "buttons," or "decorative graphics," it can be identified as a target image element.
[0054] In one embodiment of this application, such as Figure 7 As shown, determining the target information based on the recognition result includes: S540, determine the target text from the text content recognition result; S550, the location information of the target text is determined as the target location information based on the text location recognition result.
[0055] In this embodiment, when the target text content cannot be determined using the window structure tree data, all text recognized by OCR can be analyzed, and the target text can be identified and located by combining the text's positional information. First, the text information needed by the user is identified and filtered from all text fragments recognized by OCR. Specifically, the text content recognition result can be a set of all recognized text strings returned by the OCR engine after processing the screenshot of the second interface. The set of text strings recognized by OCR is then analyzed and semantically understood to determine the target text. For example, all text strings in the OCR recognition result can be traversed to determine whether they are the target text.
[0056] After identifying the target text from the text content recognition results, the target text's location information is determined based on the text location recognition results. The text location recognition results can be coordinate information that corresponds one-to-one with the text content recognition results; each coordinate describes the position of a text fragment in the second interface screenshot. The text content recognition results and text location recognition results are in a one-to-one correspondence, allowing the target text's location information to be directly determined based on the OCR recognition results. After identifying the string corresponding to the target text, its coordinate information is directly determined from the location information provided by the OCR recognition. For example, when a user browses a product page, they need to quickly obtain and locate the product price. The text content recognition result of OCR is: [“Smartwatch”, “Flagship”, “¥1,288”, “Buy Now”, “Add to Cart”]; the text location recognition result is: the corresponding coordinate list, where the coordinates of “¥1,288” are [485, 400, 580, 430]; traversing the OCR text list, the string “¥1,288” is identified as the target text; the coordinates [485,400, 580, 430] are extracted from the text location recognition result; the final output is: {Target text: “¥1,288”, Target location information: [485, 400, 580,430]}.
[0057] In one embodiment of this application, after determining the target information based on the identification result, the method further includes at least one of the following: Based on the target location information, display the identification information corresponding to the target text on the target interface; In this embodiment, displaying the corresponding identifier information of the target text on the target interface based on the target location information can provide a visual enhancement effect. Specifically, based on the coordinate information corresponding to the target information, a visual identifier is drawn on the target interface using the graphics API provided by the system through an overlay method. For example, for the date "2023-12-20", an orange underline is drawn below the text based on its coordinates [300, 240, 450, 260]. Through the graphics overlay API provided by the operating system, the above visual elements are drawn on the corresponding positions on the screen without interfering with the underlying application.
[0058] The target information is output to the target application, which is able to perform the target task based on the generative language model and the target information.
[0059] In this embodiment, target information is output to a target application. The target application can then perform target tasks based on a generative language model and the target information, achieving deep understanding and intelligent processing of information. Specifically, target information can be sent to a user-selected target application. Upon receiving the target information, the target application invokes a local or cloud-based Large Language Model (LLM) to perform reasoning based on the semantics of the extracted and classified target information, thereby executing the target task. The target task can refer to an intelligent operation with practical utility driven by the Large Language Model. For example, the "Smart Calendar Assistant" application inputs {target text: "2023-12-20", target text: "CA1234", context: "flight"} into the Large Language Model and prompts: "Please create a calendar event based on this information." The Large Language Model can parse the date and flight number, and may combine web searches to supplement flight departure and arrival times, automatically creating a 24-hour scheduled event titled "Flight CA1234" in the user's calendar.
[0060] Based on the same inventive concept, the second aspect of this application also provides a text recognition device corresponding to the text recognition method. Since the principle of the text recognition device in this application is similar to that of the text recognition method described above, the implementation of the text recognition device can refer to the implementation of the method, and the repeated parts will not be described again.
[0061] Figure 8 The diagram illustrates the structure of text recognition provided in this application embodiment, which may include: The acquisition module is used to capture a screenshot of the first screen of the target interface. The determination module is used to determine the target image elements in the first screenshot that belong to the target type; The first processing module is used to remove the target image element from the first interface screenshot to obtain a second interface screenshot. The recognition module is used to perform optical character recognition on the second interface screenshot to obtain recognition results, which include text content recognition results and text location recognition results. The second processing module is used to determine target information based on the recognition result. The target information includes target text and target location information of the target text in the target interface. The target text corresponds to a portion of the text in the text content recognition result.
[0062] In one embodiment of this application, the acquisition module is further configured as follows: Obtain the window structure tree data corresponding to the target interface, wherein the window structure tree data contains the attribute information of each view node in the target interface; Based on the attribute information of each view node, the first view node in the target interface is determined; The target image element in the first interface screenshot is determined based on the first view node.
[0063] In one embodiment of this application, the determining module is further configured as follows: Based on at least one of the control type property or view type property of each view node, determine the view node whose property is at least one of image, icon, button or custom drawing area as the target view node.
[0064] In one embodiment of this application, the first processing module is further configured as follows: The view position of the target view node in the target interface is obtained based on the attribute information of the target view node; Remove the target image element from the first interface screenshot based on the view position.
[0065] In one embodiment of this application, the determining module is further configured as follows: Based on the attribute information of each view node, a second view node in the target interface is determined, and the second view node is different from the first view node. The target text is determined based on the attribute information of the second view node, wherein the attribute information of the second view node includes the text information in the second view node; The target location information is determined based on the target text and the text location recognition result.
[0066] In one embodiment of this application, the second processing module is further configured as follows: Based on the target text, a matching text is determined from the text content recognition result, and the matching text and the target text satisfy a similarity condition; The location information of the matching text in the first interface screenshot is determined based on the text location recognition result; The target location information is determined based on the location information of the matched text in the screenshot of the first interface.
[0067] In one embodiment of this application, the second processing module is further configured as follows: The first screenshot is segmented to obtain multiple segmentation elements; The segmentation elements are identified by type, and the segmentation elements belonging to the target type are determined as the target image elements.
[0068] In one embodiment of this application, the determining module is further configured as follows: The target text is determined from the text content recognition results; The location information of the target text is determined as the target location information based on the text location recognition result.
[0069] In one embodiment of this application, the second processing module is further configured as follows: Based on the target location information, display the identification information corresponding to the target text on the target interface; The target information is output to the target application, which is able to perform the target task based on the generative language model and the target information.
[0070] Based on the same inventive concept, such as Figure 9 As shown, this embodiment also includes an electronic device, comprising: Memory, used to store executable programs; A processor is configured to execute the executable program to perform the following steps: Get a screenshot of the first screen of the target interface; Identify the target image elements in the first screenshot that belong to the target type; Remove the target image element from the first screenshot to obtain the second screenshot; Optical character recognition is performed on the second interface screenshot to obtain recognition results, which include text content recognition results and text location recognition results. Target information is determined based on the recognition result. The target information includes target text and the target position information of the target text in the target interface. The target text corresponds to a portion of the text in the text content recognition result.
[0071] The above embodiments are merely exemplary embodiments of this application and are not intended to limit this application. The scope of protection of this application is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to this application within its substance and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of this application.
Claims
1. A text recognition method, comprising: Get a screenshot of the first screen of the target interface; Identify the target image elements in the first screenshot that belong to the target type; Remove the target image element from the first screenshot to obtain the second screenshot; Optical character recognition is performed on the second interface screenshot to obtain recognition results, which include text content recognition results and text location recognition results. Target information is determined based on the recognition result. The target information includes target text and the target position information of the target text in the target interface. The target text corresponds to a portion of the text in the text content recognition result.
2. The method according to claim 1, wherein determining the target image element in the first interface screenshot includes: Obtain the window structure tree data corresponding to the target interface, wherein the window structure tree data contains the attribute information of each view node in the target interface; Based on the attribute information of each view node, the first view node in the target interface is determined; The target image element in the first interface screenshot is determined based on the first view node.
3. The method according to claim 2, wherein determining the target view node in the target interface based on the attribute information includes: Based on at least one of the control type property or view type property of each view node, determine the view node whose property is at least one of image, icon, button or custom drawing area as the target view node.
4. The method according to claim 2, wherein removing the target image element from the first interface screenshot comprises: The view position of the target view node in the target interface is obtained based on the attribute information of the target view node; Remove the target image element from the first interface screenshot based on the view position.
5. The method according to claim 2, wherein determining the target information based on the identification result includes: Based on the attribute information of each view node, a second view node in the target interface is determined, and the second view node is different from the first view node. The target text is determined based on the attribute information of the second view node, wherein the attribute information of the second view node includes the text information in the second view node; The target location information is determined based on the target text and the text location recognition result.
6. The method according to claim 5, wherein determining the target location information based on the target text and the text location recognition result includes: Based on the target text, a matching text is determined from the text content recognition result, and the matching text and the target text satisfy a similarity condition; The location information of the matching text in the first interface screenshot is determined based on the text location recognition result; The target location information is determined based on the location information of the matched text in the screenshot of the first interface.
7. The method according to claim 1, wherein determining the target image element in the first interface screenshot includes: The first screenshot is segmented to obtain multiple segmentation elements; The segmentation elements are identified by type, and the segmentation elements belonging to the target type are determined as the target image elements.
8. The method according to claim 1, wherein determining the target information based on the identification result includes: The target text is determined from the text content recognition results; The location information of the target text is determined as the target location information based on the text location recognition result.
9. The method according to claim 1, wherein after determining the target information based on the identification result, the method further comprises at least one of the following: Based on the target location information, display the identification information corresponding to the target text on the target interface; The target information is output to the target application, which is able to perform the target task based on the generative language model and the target information.
10. A text recognition device, comprising: The acquisition module is used to capture a screenshot of the first screen of the target interface. The determination module is used to determine the target image elements in the first screenshot that belong to the target type; The first processing module is used to remove the target image element from the first interface screenshot to obtain a second interface screenshot. The recognition module is used to perform optical character recognition on the second interface screenshot to obtain recognition results, which include text content recognition results and text location recognition results. The second processing module is used to determine target information based on the recognition result. The target information includes target text and target location information of the target text in the target interface. The target text corresponds to a portion of the text in the text content recognition result.