Interface element echoing method and electronic device

Abstract

The application provides an interface element echo method and an electronic device, and the method comprises the following steps: determining a target image element from a first image element set, wherein the first image element set is an element set captured from a first display interface; capturing a second image element set contained in a second display interface, wherein the element content contained in the first display interface and the second display interface is the same, and the size of the elements contained is the same or different; and determining a target area of the target image element in the second display interface according to the matching relationship between the target image element and the second image element set.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method for displaying interface elements and an electronic device. Background Technology

[0002] In Robotic Process Automation (RPA) technology, capturing and displaying elements in the application is a key technology for achieving process automation.

[0003] Current methods for displaying elements in an application typically utilize the operating system's low-level Application Programming Interface (API) or the characteristics of the programming language to parse the interface elements or interface structure into a Document Object Model Tree (DOM Tree). This DOM Tree is then stored as discrete traditional features, and the corresponding element is found by matching these features during subsequent display. While this method offers simple element feature descriptions, it suffers from significant matching errors when the interface structure changes. Summary of the Invention

[0004] The purpose of this application is to provide a method and electronic device for displaying interface elements, which can improve the situation where there are matching errors in the display of elements in the current technology.

[0005] In a first aspect, this application provides a method for displaying interface elements, comprising: determining a target image element from a first set of image elements, wherein the first set of image elements is a set of elements captured from a first display interface; capturing a second set of image elements contained in a second display interface, wherein the element content contained in the first display interface and the second display interface is the same; and determining a target area of ​​the target image element in the second display interface based on the matching relationship between the target image element and the second set of image elements.

[0006] In the above implementation, by capturing the image elements in the interface, more intuitive image elements in the display interface can be obtained. Compared with traditional features or DOM Tree, it can better and more intuitively express the various elements in the interface. In addition, the elements to be displayed can be determined based on the matching of image elements, which can also more intuitively determine the elements that need to be displayed, reduce the error of element matching in the element display process, and improve the accuracy of the display.

[0007] In an optional implementation, the relative positions of each element content of the first display interface and each element content of the second display interface are the same; determining the target area of ​​the target image element in the second display interface based on the matching relationship between the target image element and the second image element set includes: determining a reference element from the first image element set; determining a reference area of ​​the reference element in the second display interface based on the reference element; determining a candidate area of ​​the target image element based on the reference area; and matching the target image element with the candidate area to determine the target area of ​​the target image element in the second display interface.

[0008] In the above embodiments, by incorporating the limitation of reference elements, it is possible to more accurately locate the target element in the second display interface, and also to achieve more accurate element feedback.

[0009] In an optional implementation, determining the reference element from the first image element set includes: determining, based on the position of the target image element in the first display interface, image elements within the range defined by the target image element from the first image element set as reference elements.

[0010] In the above embodiments, by selecting reference elements that are related to the position of the target image element based on its position in the first display interface, the target element can be located more effectively based on the reference elements.

[0011] In an optional implementation, determining image elements within the defined range of the target image element from the first image element set as reference elements based on the position of the target image element in the first display interface includes: comparing the similarity of each image element in the first image element set with each image element in the second image element set to determine a first matching element set in the first image element set, wherein the first matching element set corresponds to a second matching element set in the second image element set, wherein the similarity between any first matching element in the first matching element set and the unique second matching element in the second matching element set is greater than a set threshold; and determining image elements within the defined range of the target image element from the first matching element set as reference elements based on the position of the target image element in the first display interface.

[0012] In the above implementation, the first image element set can be matched with the second image element set first, and reference elements can be selected from the elements that can be matched one-to-one with the image elements in the second image element set. This can make the reference elements more accurate on the second display interface, and thus make the target image element more accurately positioned on the second display interface.

[0013] In an optional implementation, matching the target image element with the candidate area to determine the target area of ​​the target image element in the second display interface includes: determining a target scaling ratio for matching the target image element in the second display interface based on the first image element set and the second image element set; adjusting the size of the target image element according to the target scaling ratio to obtain an adjusted element; and matching the adjusted element with the candidate area to determine the target area of ​​the target image element in the second display interface.

[0014] In an optional implementation, determining the target scaling ratio of the target image element on the second display interface based on the first image element set and the second image element set includes: randomly selecting an i-th sample element set from the first matching element set, where i is a positive integer greater than one and less than or equal to N, and N is a positive integer greater than one; constructing a size relationship function between the first display interface and the second display interface based on the i-th sample element set and the element set corresponding to the i-th sample element set in the second matching elements; determining the i-th scaling ratio corresponding to the target image element based on the target image element and the size relationship function; repeating the above steps to obtain N scaling ratios, and determining the target scaling ratio of the target image element on the second display interface based on the first scaling ratio to the Nth scaling ratio.

[0015] In the above implementation, it can be considered that there may be a size difference between the display interfaces in the two stages. For example, one display interface is full screen and the other display interface is a scaled-down interface. The size can be adjusted first according to the size difference and then the image is matched. This can meet the display requirements of more scenarios, improve the applicability of the interface element display method, and make the matching results more accurate.

[0016] In an optional implementation, determining the reference area of ​​the reference element in the second display interface based on the reference element includes: determining the image element corresponding to the reference element in the second image elements; and determining the reference area of ​​the reference element in the second display interface based on the position of the image element corresponding to the reference element in the second image elements in the second display interface.

[0017] In an optional implementation, determining the candidate region of the target image element based on the reference region includes: determining the relative positional relationship between the target image element and the reference element based on the position of the target image element in the first display interface and the position of the reference element in the first display interface; and selecting the candidate region of the target image element from the second display interface based on the relative positional relationship and the reference region.

[0018] In an optional implementation, matching the target image element with the candidate area to determine the target area of ​​the target image element in the second display interface includes: matching each sub-region of the target image element in the candidate area according to an image matching method to determine the image similarity between the target image element and each sub-region in the candidate area; and taking the sub-region with the highest image similarity to the target image element as the target area.

[0019] In the above implementation, after selecting a candidate area, the image similarity between the target image element and each sub-region in the candidate area can be compared by image matching, thereby selecting the specific area of ​​the target image element in the second display interface, thus achieving more accurate element display.

[0020] In an optional implementation, before determining the reference element from the first image element set, determining the target area of ​​the target image element in the second display interface based on the matching relationship between the target image element and the second image element set further includes: calculating the similarity between the target image element and each image element in the second image element set; if only one image element in the second image element set has a similarity greater than a set threshold with the target image element, then determining the image element in the second image element set with a similarity greater than the set threshold as the matching element of the target image element; and determining the target area in the second display interface based on the matching element.

[0021] In the above implementation, before using the relatively complex reference element-based echoing process, the similarity between the target image element and each image element in the second image element set can be calculated first. If a unique matching element can be obtained directly, the element can be echoed relatively quickly. If a unique matching element cannot be obtained, the target image element can still be echoed by using reference elements. Combining the two methods can improve both efficiency and the echoing success rate.

[0022] In an optional implementation, calculating the similarity between the target image element and each image element in the second image element set includes: extracting target image features of the target image element; extracting image features of each image element in the second image element set; and calculating the similarity between the target image element and each image element in the second image element set based on the target image features and the image features of each image element in the second image element set.

[0023] In an optional implementation, the target image features include: a first target feature and a second target feature; or, the target image features include the first target feature; or, the target image features include the second target feature; the extraction of target image features from the target image elements includes: performing feature extraction on the target image elements using a neural network algorithm to obtain the first target feature; and / or, identifying the second target feature carried in the target image elements through image recognition.

[0024] In the above embodiments, the combination of multiple types of image features can more comprehensively express the information of the target image elements, thereby making the determination of feature similarity more accurate, and thus enabling higher accuracy of the echo.

[0025] Secondly, this application provides an interface element display device, comprising: a first determining module, configured to determine a target image element from a first image element set, wherein the first image element set is an element set captured from a first display interface; a capturing module, configured to capture a second image element set contained in a second display interface, wherein the element content contained in the first display interface and the second display interface is the same, and the size of the elements contained therein is the same or different; and a second determining module, configured to determine the target area of ​​the target image element in the second display interface based on the matching relationship between the target image element and the second image element set.

[0026] Thirdly, this application provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method described in any of the foregoing embodiments.

[0027] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be executed by a processor to complete the above-described interface element echoing method. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some aspects of this application.

[0029] These examples should not be considered as limiting the scope. For those skilled in the art, other related figures can be obtained from these figures without any inventive effort.

[0030] Figure 1 A block diagram illustrating an electronic device provided in an embodiment of this application;

[0031] Figure 2 A flowchart of the interface element echoing method provided in the embodiments of this application;

[0032] Figure 3 A schematic diagram of the display interface in an example is shown;

[0033] Figure 4 An optional flowchart of step 230 of the interface element echoing method provided in the embodiments of this application;

[0034] Figures 5a to 5c This is a schematic diagram of the display interface provided in an embodiment of this application;

[0035] Figure 6 An optional flowchart of step 236 of the interface element echoing method provided in the embodiments of this application;

[0036] Figure 7 A schematic diagram of the functional modules of the interface element display device provided in the embodiments of this application. Detailed Implementation

[0037] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0038] It should be noted that similar labels and letters in the accompanying figures below indicate similar items; therefore, once...

[0039] If an item is defined in one of the accompanying drawings, it does not need to be further defined and explained in subsequent drawings. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0040] The inventors' research revealed that traditional element capture and display methods rely on the operating system's underlying API 5 or the characteristics of programming languages ​​to parse interface elements or the DOM Tree of the interface structure, storing discrete traditional features or the DOM Tree. For example, displaying elements in the interface can be achieved by parsing the content of HTML (Hypertext Markup Language) to extract node information, such as XPath. This node information is then used as features. When display is needed, the element features extracted from the HTML are matched to find the elements that need to be displayed.

[0041] The description of element features in the above methods is relatively simple. However, the success rate of element matching decreases when the structure, resolution, or dots per inch (DPI) of the display interface changes. Furthermore, this feature extraction method is somewhat vulnerable to attacks on application systems. In other cases, such as operating system or application software upgrades, the previously extracted features cannot be used for matching, requiring a new environment where the HTML content is parsed again to obtain the element features of the display interface. For example, if the display interface uses custom frameworks or special programming techniques during development, it may be impossible to parse the HTML to obtain the interface elements, and therefore, the elements cannot be displayed.

[0042] Based on the inventor's research on the current situation, this application provides an interface element display method and an electronic device. The method in this application only requires the image elements in the currently displayed interface and uses image recognition to complete the screen element matching. This can more intuitively determine the elements that need to be displayed, reduce the error of element matching during the element display process, and improve the accuracy of the display.

[0043] Before introducing the interface element display method and electronic device of this application, let's first introduce the terminology involved in the interface element display method of this application:

[0044] RPA Designer: Simulates user actions to design workflows for application operations, data retrieval, and information transmission. The RPA Designer utilizes low-code writing, intelligent element capture, user interface (UI) automation plugins, multi-language support, and built-in technologies such as Optical Character Recognition (OCR), Computer Vision (CV), and Natural Language Processing (NLP) to assist in business process design. RPA Executor: Supports local or remote deployment, replacing tedious manual operations and executing automated workflows under the scheduling and control of a central platform. Element Capture: Captures all elements of the display interface in either the RPA Designer or RPA Executor. Element Recall: Locates a captured interface element in the Executor. High-Fidelity Scenario: A high-fidelity scenario is defined as the environment presented in both the RPA Designer and RPA Executor stages being identical. This primarily refers to the same display resolution, DPI, and maximum / minimum states.

[0045] For example, the interface element display method of this application can be used in RPA technology to realize the display of interface elements during the automation of machine processes. RPA technology can simulate employees' daily work operations using a keyboard and mouse on a computer, replacing humans in performing tasks such as logging into systems, operating software, reading and writing data, downloading files, and reading emails. Using automated robots as virtual labor for enterprises can free employees from repetitive, low-value tasks, allowing them to focus on high-value-added work. This enables enterprises to reduce costs and increase efficiency while undergoing digital and intelligent transformation.

[0046] RPA (Robotic Process Automation) is a software robot that replaces manual tasks in business processes and interacts with computer front-end systems like a human. Therefore, RPA can be seen as a software-based program robot running on a personal PC or server, mimicking user actions on a computer to automate repetitive tasks such as retrieving emails, downloading attachments, logging into systems, and data processing and analysis—faster, more accurate, and more reliable. While both RPA and traditional physical robots address the speed and accuracy issues in human work through specific rules, traditional physical robots are hardware-software hybrids requiring specific hardware support and software to perform tasks. RPA robots, on the other hand, are purely software-based; once the appropriate software is installed, they can be deployed to any PC or server to complete the assigned tasks.

[0047] In other words, RPA is a method and related technologies that utilize "digital employees" to perform business operations in place of humans. Essentially, RPA uses software automation technology to simulate human operation of computer systems, software, web pages, and documents, acquiring business information, executing business actions, and ultimately achieving automated process processing, labor cost savings, and improved efficiency. As the description shows, one of the core technologies of RPA is the capture and display of interface elements. That is, after obtaining the location and category of interface elements, it locates the elements in the currently displayed interface, thus simulating human actions on the corresponding interface elements.

[0048] To facilitate understanding of this embodiment, the electronic device that performs the interface element display method disclosed in this application embodiment will first be introduced.

[0049] like Figure 1 The diagram shown is a block diagram of an electronic device. The electronic device 100 may include a memory 111, a memory controller 112, a processor 113, a peripheral interface 114, an input / output unit 115, and a display unit 116. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device 100. For example, the electronic device 100 may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0050] The aforementioned memory 111, memory controller 112, processor 113, peripheral interface 114, input / output unit 115, and display unit 116 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The aforementioned processor 113 is used to execute executable modules stored in the memory.

[0051] The memory 111 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory 111 stores programs, and the processor 113 executes these programs upon receiving execution instructions. The methods executed by the electronic device 100 as defined in any embodiment of this application can be applied to or implemented by the processor 113.

[0052] The aforementioned processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a digital signal processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor.

[0053] The peripheral interface 114 described above couples various input / output devices to the processor 113 and the memory 111. In some embodiments, the peripheral interface 114, the processor 113, and the memory controller 112 can be implemented on a single chip. In other instances, they can be implemented on separate chips.

[0054] The input / output unit 115 described above is used to provide user input data. The input / output unit 115 may be, but is not limited to, a mouse and keyboard, etc.

[0055] The aforementioned display unit 116 provides an interactive interface (e.g., a user interface) between the electronic device 100 and the user, or displays image data for the user's reference. In this embodiment, the display unit can be a liquid crystal display (LCD) or a touch display. If it is a touch display, it can be a capacitive touchscreen or a resistive touchscreen that supports single-point and multi-point touch operations. Supporting single-point and multi-point touch operations means that the touch display can sense touch operations generated simultaneously from one or more locations on the touch display and pass the sensed touch operations to the processor for calculation and processing.

[0056] In this embodiment, the display unit 116 can be used to display a display interface that needs to be echoed, and the display interface may include multiple image elements.

[0057] The electronic device 100 in this embodiment can be used to execute various steps in the various methods provided in the embodiments of this application. The implementation process of the interface element display method is described in detail below through several embodiments.

[0058] The following is in conjunction with the appendix Figure 2 The method for displaying interface elements provided in the embodiments of this application will be described. Please refer to [link / reference]. Figure 2 This is a flowchart of the interface element echoing method provided in the embodiments of this application. The following will describe... Figure 2 The specific process shown will be explained.

[0059] Step 210: Determine the target image elements from the first image element set.

[0060] The first image element set is a set of elements obtained from the first display interface. The first display interface can be an interface within an application.

[0061] Optionally, image recognition can be used to identify the various image elements contained in the first display interface. For example, in addition to element content, each image element may include its location and the relative positional relationships between different image elements. The element content can represent the shape, color distribution, etc., of the image. The relative positional relationships can be used to characterize elements to the left of the image element, elements to the right of the image element, elements above the image element, elements below the image element, etc.

[0062] In this embodiment, the determination of the first image element set can be image elements in the first display interface that have been pre-captured before executing steps 210 to 230. After the image element set is determined, it can be used for subsequent echoes as long as the content of the display interface remains unchanged. In one example, the first display interface can be the interface displayed in an RPA designer, which can determine the target image elements from the first image element set contained in the first display interface.

[0063] For example, the categories of each image element in the first image element set can be set to multiple categories according to the actual situation of the interface. For example, the display interface may include eight element categories: text, icons, images, scroll bars, tables, drop-down lists, buttons, and calendar controls.

[0064] like Figure 3 As shown, Figure 3 A schematic diagram of a display interface in an example is shown. The diagram shows the text category Te1 and the icon category Ic1.

[0065] Step 220: In the second display interface, capture the second set of image elements contained therein.

[0066] The first display interface and the second display interface contain the same elements, but the sizes of the elements may be the same or different. For example, the first display interface may be a full-screen display, while the second display interface is a scaled-down display. In this case, the image elements in the first display interface are larger than the image elements in the second display interface.

[0067] Alternatively, a second set of image elements contained in the second display interface can be captured by image recognition.

[0068] In this embodiment, the image elements in the second image element set correspond one-to-one with the image elements in the first image element set. For example, each element in the first image element set can be represented as: F1, F2, F3, ..., FM; each element in the second image element set can be represented as: S1, S2, S3, ..., SM; image element Fj in the first image element set corresponds to image element Sj in the second image element set, where j is a positive integer greater than or equal to one and less than or equal to M, and M is the total number of image elements contained in the first image element set.

[0069] Image element Fj in the first image element set corresponds to image element Sj in the second image element set, indicating that image element Fj and image element Sj have the same content. Image elements Fj and Sj may have the same or different dimensions. For example, if the string "abc" is a text element in the first image element set, then the corresponding element in the second image element set also has the string "abc". The difference might be the size of the "abc" string displayed on the first and second display interfaces.

[0070] In one instance, the second display interface can be the interface displayed in the RPA executor, which can find the target image element that needs to be displayed in the second display interface.

[0071] Step 230: Based on the matching relationship between the target image element and the second image element set, determine the target area of ​​the target image element in the second display interface.

[0072] Different categories of target image elements may result in different matching outcomes, therefore different display methods can be used for different categories of image elements.

[0073] For example, if an image element is categorized as an icon, then in the second image element set, there may be multiple image elements with the same icon but different positions. Similarly, if an image element is categorized as a button, then in the second image element set, there may also be multiple image elements with the same button shape but different positions. Finally, if an image element is categorized as a calendar control, then in the second image element set, there may only be one calendar control image element.

[0074] Therefore, in order to satisfy the display of different categories of image elements, the matching elements can be determined through the following steps 231 and 232; the matching elements can also be determined through the following steps 233 to 236; or the matching elements can be determined first through steps 231 and 232, and if the matching elements cannot be determined through steps 231 and 232, then the matching elements can be determined through steps 233 to 236.

[0075] For example, if the selected target image element belongs to the calendar control category, since there may only be one calendar control image element in the second image element set, when comparing the similarity with the second image element set through steps 231 and 232, a uniquely matching image element can be determined in the second image element set. Therefore, the matching element can be determined through steps 231 and 232. Of course, if the selected target image element belongs to a category similar to the calendar control category, and there is only one unique element in a display interface, then the matching element can be determined through steps 231 and 232.

[0076] For example, if the selected target image element belongs to the icon or button category, since there may only be multiple similar image elements in the second image element set, when comparing similarity with the second image element set through steps 231 and 232, it is impossible to determine a uniquely matching image element in the second image element set. Therefore, it is difficult to determine the matching element through steps 231 and 232. In this case, the matching element can be determined through steps 233 to 236. Of course, if the selected target image element belongs to a category similar to the icon category, and there are multiple similar elements in a display interface, the matching element can be determined through steps 233 to 236.

[0077] In this embodiment, as Figure 4 As shown, step 230 may include steps 231 and 232.

[0078] Step 231: Calculate the similarity between the target image element and each image element in the second image element set.

[0079] If there is only one image element in the second image element set whose similarity to the target image element is greater than a set threshold, then step 232 is executed.

[0080] Step 232: Determine the image elements in the second image element set whose similarity to the target image element is greater than the set threshold as matching elements of the target image element, and select the target area in the second display interface based on the matching elements.

[0081] If there is no image element in the second image element set whose similarity to the target image element is greater than a set threshold, or if there is more than one image element in the second image element set whose similarity to the target image element is greater than a set threshold, then step 233 can be executed.

[0082] In one embodiment, step 231 may include: extracting target image features of the target image element; extracting image features of each image element in the second image element set; and calculating the similarity between the target image element and each image element in the second image element set based on the target image features and the image features of each image element in the second image element set.

[0083] For example, the target image features can be represented as image vectors, and the similarity can represent the distance between two image vectors. For example, the distance can be Euclidean distance, Mahalanobis distance, cosine distance, etc.

[0084] To accommodate the different representational needs of image elements, the image features extracted from image elements can include two types of image features, or only one type of image feature.

[0085] The two types of image features can be: the first type of image features extracted by neural network algorithms, and the second type of image features extracted by other methods.

[0086] Optionally, the content of the second type of image feature can differ for different categories of image elements. For example, if the image element is a graphic category such as an icon, image, scroll bar, or button, then the second type of image feature can include Histogram of Oriented Gradient (HOG) features, color histogram features, etc. As another example, if the image element is a category containing strings such as text, table, drop-down list, or calendar control, then the second type of image feature can include character features recognized using Optical Character Recognition (OCR).

[0087] Optionally, the content of the second type of image feature is the same for image elements of different categories. For example, this could include the HOG feature, color histogram feature, and character features recognized using Optical Character Recognition (OCR) technology. For instance, if the image element is a graphic category such as an icon, image, scrollbar, or button, the second type of image feature could include a Histogram of Oriented Gradient (HOG) and color histogram features. Since the image element is a graphic category, it does not contain strings, so the character features for this graphic category can be replaced with zeros. As another example, if the image element is a category containing strings such as text, table, dropdown list, or calendar control, the second type of image feature could include the HOG feature, color histogram feature, and character features recognized using Optical Character Recognition (OCR) technology.

[0088] The target image features may include: a first target feature and a second target feature; the target image features may also include only the first target feature; or the target image features may include only the second target feature.

[0089] In this embodiment, when the target image feature includes a first target feature and a second target feature, the image features of each image element in the extracted second image element set also include both types of image features; when the target image feature only includes the first target feature, the image features of each image element in the extracted second image element set also only include the first type of image features; when the target image feature only includes the second target feature, the image features of each image element in the extracted second image element set also only include the second type of image features.

[0090] The first target feature belongs to the first type of image feature, and the second target feature belongs to the second type of image feature.

[0091] The first target feature of the target image element is obtained by extracting features from the target image element through a neural network algorithm.

[0092] The aforementioned neural network algorithm can be a classification network algorithm based on Convolutional Neural Networks (CNN).

[0093] This classification network algorithm can use a loss function to control the learning of inter-class distance and intra-class distance, and the output of the previous layer of the classification layer can be used as the first target feature.

[0094] The aforementioned second target feature can be identified by image recognition, which identifies the second target feature carried in the target image element.

[0095] For example, the image recognition method could be OCR technology.

[0096] The method used to extract the image features of each image element in the second image element set is the same as the method used to extract the target image elements.

[0097] In this embodiment, the relative positions of the elements in the first display interface and the elements in the second display interface are the same. For example, the image elements in the first image element set can be represented as: F1, F2, F3, ..., FM; and the image elements in the second image element set can be represented as: S1, S2, S3, ..., SM. Taking image elements F3 and S3 as examples, in the first display interface, the image element F3 is adjacent to F5 on its left, F2 on its right, F1 above it, and F9 below it. Since image element F3 in the first display interface corresponds to image element S3 in the second display interface, and their relative positions are the same, in the second display interface, the image element S3 is adjacent to S5 on its left, S2 on its right, S1 above it, and S9 below it.

[0098] Please refer to again Figure 4 As shown, step 230 may also include steps 233 to 236.

[0099] Step 233: Determine the reference element from the first image element set.

[0100] For example, the reference element may be an element located around the target element in the first display interface.

[0101] In one embodiment, step 233 described above may include: determining, based on the position of the target image element in the first display interface, image elements within the range defined by the target image element from the first set of image elements as reference elements.

[0102] For example, the defined range can be a range determined based on the size of the image element. For instance, the defined range can be a range within the first display interface where the distance to the target image element is no greater than the distance between two image elements. Alternatively, the defined range can be a range determined based on a specific numerical value. For instance, the defined range can be a range within the first display interface where the distance to the target image element is no greater than a set length, such as 1cm, 3cm, etc.

[0103] like Figures 5a to 5c The diagram shows a schematic of the display interface in one example. Figure 5a The example shown can be a schematic diagram of the first display interface. Figure 5a , Figure 5b and Figure 5c The example shown can be a schematic diagram of the second display interface. Figure 5a The example shown illustrates multiple icon elements (four different icon elements are shown in the figure), multiple text elements (A1 file, ..., A14 file, B1 text, ..., B4 text, C1 text, C2 text, D1 document), and a scrollbar element. Taking the icon element representing the target image element as "A12 file" as an example, the elements surrounding this "A12 file" icon element include: the text elements of "A12 file", the icon element of "A5 file", the text elements of "A5 file", the text elements of "A11 file", and the icon element of "A11 file", etc. Reference elements for the target image element can be determined from the elements surrounding the "A12 file" icon element.

[0104] In another embodiment, step 233 described above may include steps 2331 and 2332.

[0105] 2331. Compare the similarity of each image element in the first image element set with each image element in the second image element set to determine the first matching element set in the first image element set.

[0106] The first matching element set corresponds to the second matching element set in the second image element set, wherein the similarity between any first matching element in the first matching element set and the unique second matching element in the second matching element set is greater than a set threshold.

[0107] The number of elements in the first matching element set is less than the number of elements in the first image element set. The number of elements in the first matching element set is the same as the number of elements in the second matching element set.

[0108] For example, image features of each image element in the first image element set and image features of each image element in the second image element set can be extracted. The image feature extraction method can be the same as that involved in step 231 above; for details, please refer to the description in step 231, which will not be repeated here.

[0109] Alternatively, before calculating similarity, image features can be extracted from each image element in the first image element set and each image element in the second image element set. When calculating the similarity between two image elements, the image features of each image element are then retrieved.

[0110] From the perspective of image features, in Figure 5a In the example shown, since the icons of all files are identical, and the icons of all compressed files are also identical, the file icon elements in the first image element set may have a similarity greater than a set threshold with multiple image elements in the second image element set. Therefore, such image elements cannot be selected as the first matching element set in step 2331.

[0111] 2332, Based on the position of the target image element in the first display interface, determine the image elements within the range defined by the target image element from the first matching element set as reference elements.

[0112] exist Figure 5a In the example shown, although the elements surrounding the icon element of "A12 file" include: text elements of "A12 file", icon elements of "A5 file", text elements of "A5 file", text elements of "A11 file", icon elements of "A11 file", text elements of "A13 file", and icon elements of "A13 file", etc., the icon elements of "A5 file", "A11 file", and "A13 file" are all the same graphic and therefore cannot be selected as the first matching element set in step 2331. However, the text elements of "A12 file", "A5 file", "A11 file", and "A13 file" are all unique and therefore can be selected as the first matching element set. Figure 5a In the example shown, the reference elements of the target image element can be identified as: the text elements of "A12 file" surrounding the icon element of "A12 file", the text elements of "A5 file", the text elements of "A11 file", the text elements of "A13 file", etc.

[0113] Step 234: Determine the reference area of ​​the reference element in the second display interface based on the reference element.

[0114] The reference area can be the area of ​​the image elements of the second image element set corresponding to the reference element in the second display interface.

[0115] For example, the image element corresponding to the reference element in the second image element can be determined first; and the reference area of ​​the reference element in the second display interface can be determined based on the position of the image element corresponding to the reference element in the second image element in the second display interface.

[0116] For example, if the reference element can be represented as Fp, then the image element of the second image element set corresponding to the reference element is Sp. The reference area of ​​the reference element in the second display interface can represent the area of ​​the image element Sp in the second display interface.

[0117] exist Figure 5b The example shown can be a schematic diagram of the second display interface. When the reference elements of the selected target image element are the text elements of "A12 file", "A5 file", "A11 file", and "A13 file" around the icon element of "A12 file", then the reference area Re1 of each reference element in the second display interface can be the area shown by the solid line box.

[0118] Step 235: Based on the reference area, determine the candidate region of the target image element.

[0119] Optionally, step 235 may include: determining the relative positional relationship between the target image element and the reference element based on the position of the target image element in the first display interface and the position of the reference element in the first display interface; and selecting a candidate area of ​​the target image element from the second display interface based on the relative positional relationship and the reference area.

[0120] like Figure 5a As shown, the target image element is the icon element of "A12 file", and the icon element of "A12 file" is located between the text elements of "A5 file" and the text element of "A12 file". Therefore, the relative positional relationship between the target image element and the reference element can be determined as follows: the target image element is located between two reference elements, and the area between the reference areas of the two reference elements can be identified as the candidate area.

[0121] like Figure 5b In the example shown, since the icon element of "A12 file" is located between the text elements of "A5 file" and "A12 file", the area between the text elements of "A5 file" and "A12 file" can be determined as the area to be selected, Se1.

[0122] Step 236: Match the target image element with the candidate area to determine the target area of ​​the target image element in the second display interface.

[0123] from Figure 5b As can be seen from the example shown, the actual selected candidate area is larger than the target area of ​​the target image element in the second display interface.

[0124] Therefore, the graphic of the target image element can be matched in the candidate area to determine the local area with the highest similarity to the graphic of the target image element as the target area.

[0125] like Figure 5c As shown, the identified target region Te1 is a part of the candidate region Se1.

[0126] Optionally, step 236 above may include: matching the target image element with each sub-region of the candidate area according to the image matching method, and determining the image similarity between the target image element and each sub-region of the candidate area; and taking the sub-region with the highest image similarity to the target image element as the target area.

[0127] For example, if the first display interface and the second display interface have the same size, the graphic of the target image element can be compared with each sub-region of the candidate area. The image similarity between the target image element and each sub-region of the candidate area is determined; the sub-regions with an image similarity greater than the image similarity threshold with the target image element are determined as the target area.

[0128] For example, the image threshold can be a value such as 0.8, 0.85, or 0.9.

[0129] For example, such as Figure 6 As shown, step 236 may include steps 2361 to 2363.

[0130] Step 2361: Determine the target scaling ratio of the target image element in the second display interface based on the first image element set and the second image element set.

[0131] Since each image element in the display interface is scaled up or down proportionally when the display interface is zoomed in or out, the target scaling ratio of the target image element in the second display interface can be determined based on the scaling ratios of other elements in the first image element in the second display interface.

[0132] In one embodiment, an image element can be randomly selected from the first matching element set, and then the corresponding image element in the second matching element set can be found. The scaling ratio of the two image elements is calculated, and the scaling ratio is determined as the target scaling ratio for the target image element to match in the second display interface.

[0133] In one embodiment, multiple screenings can be performed, and the target scaling ratio of the target image element matching the second display interface can be determined based on the scaling ratio of the image elements selected multiple times on the second display interface.

[0134] For example, step 2361 may include: randomly selecting an i-th sample element set from the first matching element set; constructing a size relationship function between the first display interface and the second display interface based on the i-th sample element set and the element set corresponding to the i-th sample element set in the second matching elements; and determining the i-th scaling ratio corresponding to the target image element based on the target image element and the size relationship function. Here, i is a positive integer greater than one and less than or equal to N, and N is a positive integer greater than one.

[0135] Repeat the above steps to obtain N scaling ratios. Based on the first scaling ratio to the Nth scaling ratio, determine the target scaling ratio that the target image element matches on the second display interface.

[0136] For example, a size relationship function can be constructed using the least squares method. The independent variable of this size relationship function can be the size parameters of each sample in the i-th sample element set, and the dependent variable can be the size parameters of each element in the element set corresponding to the i-th sample element set in the second matching element.

[0137] Alternatively, the ransac method can be used to randomly select the i-th sample element set from the first matching element set.

[0138] For example, step 2361 may include: randomly selecting multiple sample elements from the first matching element set; calculating the scaling ratio of each sample element on the second display interface according to the element corresponding to the multiple sample elements in the second matching element; and determining the scaling ratio of the target image element on the second display interface according to the scaling ratio of each sample element on the second display interface.

[0139] For example, the scaling ratio with the highest frequency among multiple sample elements can be selected as the scaling ratio of the target image element on the second display interface. Alternatively, the average scaling ratio can be calculated based on the scaling ratios among multiple sample elements and used as the scaling ratio of the target image element on the second display interface.

[0140] Step 2362: Adjust the size of the target image elements according to the target scaling ratio to obtain the adjusted elements.

[0141] Step 2363: Match the adjustment element with the candidate area to determine the target area of ​​the target image element in the second display interface.

[0142] For example, the graphic of the adjustment element can be compared sequentially from left to right in the candidate area to determine the target area of ​​the target image element in the second display interface. Alternatively, the graphic of the adjustment element can be compared sequentially from right to left in the candidate area to determine the target area of ​​the target image element in the second display interface.

[0143] In this process, after each comparison of the adjusted element's graphic with a sub-region of the candidate region, it gradually moves to the right of the currently compared sub-region until it successfully matches a sub-region of the candidate region. If the image similarity between the adjusted element's graphic and a sub-region of the candidate region is greater than an image threshold, then the comparison with a sub-region of the candidate region is considered successful.

[0144] For example, the distance the graphic of an adjustment element moves each time can be set as needed, and this distance is less than the width of the graphic of the adjustment element. For example, the distance the graphic of the adjustment element moves each time can be one-tenth, one-twentieth, etc., of the width of the graphic of the adjustment element.

[0145] Considering that the display interface may change at different stages due to DPI variations, causing the target element size to scale, or that the interface has built-in scrollbars that scale the image elements; or that the element to be displayed may be one of multiple elements with the same appearance in the display interface, for example... Figure 5a The file icon shown makes element display difficult. However, through steps 231 to 236 above, when the element in the second image element set that uniquely corresponds to the target image element cannot be directly calculated using a simple image vector, the target image element can still be located based on the assistance of reference elements; thus, both the efficiency and success rate of image element location can be considered.

[0146] Based on the same application concept, this application also provides an interface element display device corresponding to the interface element display method. Since the principle of the device in this application is similar to that of the aforementioned interface element display method, the implementation of the device in this application can refer to the description in the above method embodiments, and the repeated parts will not be described again.

[0147] Please see Figure 7This is a functional module diagram of the interface element display device provided in this application embodiment. Each module in the interface element display device in this embodiment is used to execute the steps in the above method embodiments. The interface element display device includes: a first determining module 310, a capturing module 320, and a second determining module 330; the contents of each module are as follows:

[0148] The first determining module 310 is used to determine the target image element from the first image element set, wherein the first image element set is an element set captured from the first display interface;

[0149] The capture module 320 is used to capture a second set of image elements contained in the second display interface, wherein the element content contained in the first display interface is the same as that in the second display interface.

[0150] The second determining module 330 is used to determine the target area of ​​the target image element in the second display interface based on the matching relationship between the target image element and the second image element set.

[0151] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the interface element echoing method described in the above method embodiments.

[0152] The computer program product of the interface element display method provided in this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the interface element display method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods,

[0154] Other methods can also be used to achieve this. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable functions for implementing the specified logical function.

[0155] Instructions. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in a different order than those shown in the figures. For example, two consecutive boxes may actually be executed substantially in parallel, or sometimes in reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or using a combination of dedicated hardware and computer instructions.

[0156] 5. In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0157] If the aforementioned function is implemented as a software functional module and sold as an independent product, or if...

[0158] When in use, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application 0, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. And the preceding...

[0159] The storage media mentioned include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks or optical disks, and various other types.

[0160] A medium capable of storing program code. It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0162] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for displaying interface elements, characterized in that, include: The target image element is determined from the first image element set, wherein the first image element set is the element set captured from the first display interface; In the second display interface, a second set of image elements contained therein is captured, wherein the element content contained in the first display interface and the second display interface is the same; the relative positions of each element content in the first display interface and each element content in the second display interface are the same. Based on the matching relationship between the target image element and the second image element set, the target area of ​​the target image element in the second display interface is determined, including: comparing the similarity between each image element in the first image element set and each image element in the second image element set to determine a first matching element set in the first image element set, wherein the first matching element set corresponds to a second matching element set in the second image element set, and the similarity between any first matching element in the first matching element set and the unique second matching element in the second matching element set is greater than a set threshold; Based on the position of the target image element in the first display interface, image elements within the range defined by the target image element are determined from the first set of matching elements as reference elements; The reference area of ​​the reference element in the second display interface is determined based on the reference element; Based on the reference area, the candidate regions of the target image elements are determined; The target image element is matched with the candidate area to determine the target area of ​​the target image element in the second display interface.

2. The method according to claim 1, characterized in that, The step of matching the target image element with the candidate area to determine the target area of ​​the target image element in the second display interface includes: Based on the first image element set and the second image element set, the target scaling ratio of the target image element in the second display interface is determined; Adjust the size of the target image elements according to the target scaling ratio to obtain the adjusted elements; The adjustment element is matched with the candidate area to determine the target area of ​​the target image element in the second display interface.

3. The method according to claim 2, characterized in that, The step of determining the target scaling ratio of the target image element on the second display interface based on the first image element set and the second image element set includes: Randomly select the i-th sample element set from the first matching element set, where i is a positive integer greater than one and less than or equal to N, and N is a positive integer greater than one. Based on the i-th sample element set and the element set corresponding to the i-th sample element set in the second matching element, construct the size relationship function between the first display interface and the second display interface; Based on the relationship function between the target image element and the size, the i-th scaling ratio corresponding to the target image element is determined; Repeat the above steps to obtain N scaling ratios. Based on the first scaling ratio to the Nth scaling ratio, determine the target scaling ratio that the target image element matches on the second display interface.

4. The method according to claim 1, characterized in that, Determining the reference area of ​​the reference element in the second display interface based on the reference element includes: Determine the image element corresponding to the reference element in the second image element; Based on the position of the image element corresponding to the reference element in the second image element in the second display interface, the reference area of ​​the reference element in the second display interface is determined.

5. The method according to claim 1, characterized in that, The step of determining the candidate region of the target image element based on the reference region includes: Based on the position of the target image element in the first display interface and the position of the reference element in the first display interface, the relative positional relationship between the target image element and the reference element is determined. Based on the relative positional relationship and the reference area, the candidate area of ​​the target image element is selected from the second display interface.

6. The method according to claim 1, characterized in that, The step of matching the target image element with the candidate area to determine the target area of ​​the target image element in the second display interface includes: The target image element is matched with each sub-region of the candidate region according to the image matching method to determine the image similarity between the target image element and each sub-region of the candidate region; The sub-region with the highest similarity to the target image element is taken as the target region.

7. The method according to claim 1, characterized in that, Before determining image elements within the defined range of the target image element from the first set of matching elements as reference elements, the step of determining the target area of ​​the target image element in the second display interface based on the matching relationship between the target image element and the second set of image elements further includes: Calculate the similarity between the target image element and each image element in the second image element set; If there is only one image element in the second image element set whose similarity to the target image element is greater than a set threshold, then the image element in the second image element set whose similarity to the target image element is greater than the set threshold is determined to be the matching element of the target image element. Based on the matching elements, the target area in the second display interface is determined.

8. The method according to claim 7, characterized in that, The calculation of the similarity between the target image element and each image element in the second image element set includes: Extract the target image features of the target image elements; Extract the image features of each image element in the second image element set; Based on the target image features and the image features of each image element in the second image element set, the similarity between the target image element and each image element in the second image element set is calculated.

9. The method according to claim 8, characterized in that, The target image features include: a first target feature and a second target feature; or, the target image features include the first target feature; or, the target image features include the second target feature. The extraction of target image features from the target image elements includes: The target image elements are feature extracted using a neural network algorithm to obtain the first target feature; and / or, The second target feature carried in the target image element is identified by image recognition.

10. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores machine-readable instructions executable by the processor, wherein when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method as described in any one of claims 1 to 9.

Images