Interface processing method, device, equipment and storage medium

By defining the business semantics and target location of interface elements, cross-platform automated operations are achieved, solving the problem that different platforms require different scripts and reducing maintenance costs and coupling.

CN112052005BActive Publication Date: 2024-11-22ALIBABA GROUP HOLDING LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201910492978.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-06-06
Publication Date
2024-11-22
Estimated Expiration
2040-05-21

AI Technical Summary

Technical Problem

Existing technologies require writing different automation scripts for different platforms, which increases maintenance costs. Furthermore, the automation scripts are coupled with the content displayed on the interface, resulting in frequent script modifications when the interface changes.

Method used

By defining the business semantics of interface elements and determining the target location based on the business semantics, cross-platform automated operations are achieved, avoiding hard coding based on the content displayed on the interface.

Benefits of technology

This allows the same script to be used on different platforms, reducing maintenance costs, decreasing the coupling between the script and the interface content, and extending the script's lifespan.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112052005B_ABST
    Figure CN112052005B_ABST
Patent Text Reader

Abstract

The present invention discloses an interface processing method, device, equipment and storage medium, the method comprising: determining the business semantics of a first interface element; determining the target position of the first interface element based on the business semantics; locating the first interface element on the interface according to the target position to operate the first interface element. According to an embodiment of the present invention, the same set of scripts can be applied to different platforms for automated processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computers, and in particular relates to an interface processing method, device, equipment and storage medium. Background Art

[0002] Automation is an important capability support in the continuous integration process of products, such as automated testing or automated analysis of products, thus providing important guarantees for the continuous iteration of products. Automation capabilities at different levels are adapted to different scenarios, and interface-based automation has unique application areas.

[0003] Since different platforms have different technical implementations, scripts need to be written based on the details of the technical implementations. Therefore, different scripts are written for platforms with different technical implementations. For example, the writing languages ​​of the Android system and the Apple mobile device operating system (iOS) are different, and the development technology stacks of the Android system and iOS are different. It is necessary to write a set of scripts for the Android system to achieve automation, and another set of scripts for iOS to achieve automation. Faced with multiple sets of scripts, the cost of maintaining scripts is increased. Some current solutions only use one programming language to write scripts for different platforms, but still need to write scripts based on the technical implementation details of the platform, and it is impossible to achieve one set of scripts for multiple platforms.

[0004] In addition, since the automation script is hard-coded based on the content displayed on the interface, the automation script is coupled with the content displayed on the interface, that is, the automation script is associated with the content displayed on the interface. Therefore, if the content displayed on the interface changes, the script needs to be additionally maintained. Summary of the invention

[0005] The embodiments of the present invention provide an interface processing method, device, equipment and storage medium, which can realize the application of a set of automation scripts to different platforms, and realize automated operation even if the content of the interface changes.

[0006] In one aspect, an embodiment of the present invention provides an interface processing method, comprising:

[0007] Determine the business semantics of the first interface element;

[0008] Based on the business semantics, determining a target position of the first interface element;

[0009] The first interface element is located on the interface according to the target position, so as to operate the first interface element.

[0010] On the other hand, an embodiment of the present invention provides an interface processing device, including:

[0011] A semantic determination module, used to determine the business semantics of the first interface element;

[0012] A position determination module, used to determine a target position of the first interface element based on the business semantics;

[0013] The operation execution module is used to locate the first interface element on the interface according to the target position, so as to operate the first interface element.

[0014] In another aspect, an embodiment of the present invention provides a computer device, the device comprising: a processor and a memory storing computer program instructions;

[0015] The processor implements the interface processing method when executing the computer program instructions.

[0016] On the other hand, an embodiment of the present invention provides a computer storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the interface processing method is implemented.

[0017] The interface processing method, device, equipment and storage medium of the embodiments of the present invention can realize automatic processing based on the business semantics of the content displayed on the interface. Even for different platforms, as long as the business semantics of the displayed content of different platforms are the same, the same set of scripts can be applied to different platforms for automatic processing. Moreover, there is no need to hard-code the content displayed on the interface in the script, which solves the coupling between the script and the content displayed on the interface. Even if the content displayed on the interface changes or the display position changes, as long as the business semantics do not change, there is no need to re-modify the script, thereby realizing cross-platform / cross-end automatic processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram showing a flow chart of an interface processing method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic flow chart of an interface processing method provided by another embodiment of the present invention is shown;

[0021] Figure 3 A schematic flow chart of an interface processing method provided by another embodiment of the present invention is shown;

[0022] Figure 4A schematic diagram of a terminal interface provided by an embodiment of the present invention is shown;

[0023] Figure 5 A schematic flow chart of an interface processing method provided by another embodiment of the present invention is shown;

[0024] Figure 6 A schematic flow chart of an interface processing method provided by another embodiment of the present invention is shown;

[0025] Figure 7 A block diagram of an interface processing device provided by an embodiment of the present invention is shown;

[0026] Figure 8 A schematic diagram of the hardware structure of a computing device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0028] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0029] In order to solve the problems in the prior art, the embodiments of the present invention provide an interface processing method, device, equipment and storage medium. The interface processing method provided by the embodiments of the present invention is first introduced below.

[0030] Figure 1 FIG. 1 is a flow chart of an interface processing method provided by an embodiment of the present invention. Figure 1 As shown, the interface processing method includes:

[0031] S101, determining the business semantics of a first interface element.

[0032] Among them, the keywords of the first interface element in the script can be used as the business semantics of the first interface element.

[0033] S102: Determine a target position of the first interface element based on the business semantics of the first interface element.

[0034] Wherein, S102 includes: according to the correspondence between the business semantics of the interface element and the interface position, determining the position corresponding to the business semantics of the first interface element as the above-mentioned target position.

[0035] For example, on the flight search interface, the business semantics of the interface elements may include: departure city, arrival city, and departure date. The above correspondences include: the correspondence between the departure city and the position of the departure city input box on the interface, the correspondence between the arrival city and the position of the arrival city input box on the interface, and the correspondence between the departure date and the position of the departure date input box on the interface.

[0036] The interface position in the corresponding relationship is a region coordinate or a position index.

[0037] S103: locate the first interface element on the interface according to the target position to operate the first interface element.

[0038] Wherein, the first interface element is located on the interface to be located.

[0039] It should be noted that the number of the first interface elements to be operated can be one or more. If the number of the first interface elements is multiple, these multiple first interface elements can be a user interface (UI) component. The interface elements include but are not limited to at least one of the following: text, image, icon.

[0040] There are many wireless client automation solutions based on interface drive, but due to the differences in the original technology stack and technical implementation details of different platforms, the use of automation brings multiple maintenance costs. Reducing maintenance costs has always been a goal pursued by automation solutions. Cross-end is one of the breakthrough directions. Some technical solutions have unified the automation scripting languages ​​of different platforms and use the same language for different ends. The automation framework adds abstraction layers and conversion layers to shield the differences between different platforms. The positioning of interface elements still adopts the ViewTree positioning solution. Because the details of the same UI in the technical implementation of both ends are definitely inconsistent, this solution still cannot achieve the goal of a set of scripts compatible with both ends in theory and technology.

[0041] Some other automation solutions take a different approach to finding and locating interface elements, starting from the final graphic image level presented in the interface, locating and searching elements based on the final graphic image presented by the elements, which can naturally shield the specific technical implementation and realize that the interfaces with different technical implementations that show the same display are driven by a set of scripts. AirTest, which combines OpenCV with the ViewTree solution, is a representative of this technical route.

[0042] ViewTree: The software view is composed of a tree structure, and each view element is a node on the view tree. OpenCV is a cross-platform computer vision library released under the BSD license (open source), which implements many common algorithms in image processing and computer vision. AirTest is an open source UI automation testing framework.

[0043] In the embodiment of the present invention, it is possible to realize automatic processing based on the business semantics of the content displayed on the interface. Even for different platforms, as long as the business semantics of the content displayed on different platforms are the same, the same set of scripts can be applied to different platforms for automatic processing, solving the problem of writing scripts applicable to the platform based on the technical implementation details of the platform. Among them, automation refers to the process in which a machine, software system or process (production, management process) achieves the expected goal through automatic detection, information processing, analysis and judgment, and manipulation and control without or with little direct human participation according to human requirements.

[0044] Moreover, in the embodiment of the present invention, the operation object of the script is business semantics. If the content displayed on the interface changes or the displayed position changes, if the business semantics do not change, the script does not need to be re-modified, which avoids the need to rewrite the script due to changes in the position of the interface elements or the text of the interface elements, prolongs the life cycle of the script, and reduces the maintenance cost of the script. In the script, there is no need to hard-code the text or interface elements displayed on the interface, which solves the coupling between the script and the content displayed on the interface, and greatly reduces the degree of correlation between the automatic processing of the script and the specific content displayed on the interface.

[0045] Figure 2 FIG. 2 is a flow chart of an interface processing method provided by another embodiment of the present invention. Figure 2 As shown, the interface processing method includes:

[0046] S201, obtaining at least one set of information displayed on the interface and the position of each set of information on the interface.

[0047] As an example, the position of each set of information on the interface may be the region coordinates of each set of information on the interface, or the position of each set of information on the interface may be the position index of each set of information on the interface.

[0048] S202, determining the business semantics of each group of information.

[0049] As an example, multiple groups of information are multiple groups of texts; the texts displayed on the interface are segmented, and multiple groups of texts are obtained according to the segmentation results, wherein each group of texts includes at least one word; each group of texts is classified, and machine learning is performed on the classified multiple groups of texts to obtain the business semantics of each group of texts.

[0050] The text displayed on the interface may be acquired through optical character recognition (OCR).

[0051] AirTest, which combines OpenCV with the ViewTree solution, uses a set of scripts to drive interfaces implemented with different technologies to show the same display. However, this solution still cannot cover the situation of pure text images well, especially for most apps that provide business services. The business information on the interface is expressed in a large amount of text, and many displayed information changes dynamically and cannot be located by a certain graphic image.

[0052] The embodiment of the present invention adds recognition for text at the graphic image level and identifies the text therein from the final interface display. On the one hand, since it is still based on the final interface presentation, it can naturally shield the specific implementation method. On the other hand, it can also well cover the situation of pure text images.

[0053] S203, for each group of information, the business semantics of the same group of information is matched with the position on the interface to establish a corresponding relationship between at least one business semantic and at least one position, wherein at least one business semantic corresponds to at least one position one by one.

[0054] For example, for a group of information, the business semantics of the information is matched with the position of the information on the interface. In the case of multiple groups of information, each group of information has business semantics and positions, and multiple groups of information have multiple business semantics and multiple positions in total, and a correspondence between multiple business semantics and multiple positions is established.

[0055] S204: Determine a target position corresponding to the business semantics of the first interface element according to the constructed corresponding relationship.

[0056] S205: locate the first interface element on the interface according to the target position to perform an operation on the first interface element.

[0057] As an example, after obtaining the target position, the interface element corresponding to the target position on the interface is used as the first interface element.

[0058] The embodiment of the present invention is described in detail below by taking an example.

[0059] Set on the current interface of air ticket search, the script needs to automatically trigger the input box of the departure city, that is, the first interface element to be operated in the script is the input box of the departure city.

[0060] The text displayed on the current interface is recognized through OCR; the recognized text is segmented to obtain the following multiple words: Beijing, Hangzhou, April 18, search; the segmented words are classified, that is, "Beijing" and "Hangzhou" are classified as cities, "April 18" is classified as time, and "search" is classified as action.

[0061] For the classified words, through machine learning, we can find that the business semantics of "Beijing" is the departure city, the business semantics of "Hangzhou" is the arrival city, the business semantics of "April 18" is the departure time, and the business semantics of "search" is air ticket search.

[0062] The departure city corresponds to the regional coordinate M1 of "Beijing" on the interface, the arrival city corresponds to the regional coordinate M2 of "Hangzhou" on the interface, the departure time corresponds to the regional coordinate M3 of "April 18" on the interface, and the air ticket search corresponds to the regional coordinate M4 of "Search" on the interface. In this way, a correspondence between multiple business semantics and multiple regional coordinates is established.

[0063] Since the operation to be executed is to trigger the input box of the departure city, the business semantics of the operation to be executed is the departure city. According to the corresponding relationship established above, it can be obtained that the position corresponding to the departure city is the regional coordinate M1.

[0064] After obtaining the regional coordinates M1, the interface element of the regional coordinates M1 on the interface is the input box of the departure city, and an operation is performed on the interface element of the regional coordinates M1, so that the script automatically triggers the input box of the departure city.

[0065] In the embodiment of the present invention, the script can automatically determine the business semantics of the content displayed on the interface without the participation of technical personnel, thereby reducing the workload of technical personnel.

[0066] Figure 3 FIG. 2 is a flow chart of an interface processing method provided by another embodiment of the present invention. Figure 3 As shown, the interface processing method includes:

[0067] S301, loading a business semantic configuration file.

[0068] Among them, the business semantic configuration file includes a first tree structure corresponding to the first interface, the interface elements in the first interface are leaf nodes of the first tree structure, and in the multiple block images divided by the first interface, the block image where the leaf node is located is the parent node of the leaf node.

[0069] The business semantics configuration file also includes the correspondence between the business semantics of the interface elements in the first tree structure and the position index of the interface elements.

[0070] S302: For a first interface element to be operated, determine the business semantics of the first interface element.

[0071] As an example, the business semantics of the first interface element is obtained by analyzing the keywords in the script of the operation to be executed. For example, if the script of the first interface element of the operation to be executed contains the keyword "departure city", the business semantics of the first interface element is the departure city.

[0072] As another example, the business semantics of the first interface element is determined by a machine learning algorithm.

[0073] S303, determining the position index corresponding to the business semantics of the first interface element according to the correspondence between the business semantics of the interface elements in the business semantics configuration file and the position index, hereinafter referred to as "the position index corresponding to the business semantics of the first interface element" as "the position index of the first interface element".

[0074] S304: Search in the first tree structure according to the position index of the first interface element to obtain the parent node of the first interface element.

[0075] S305, dividing the interface to be located into multiple block images to obtain a second tree structure corresponding to the interface to be located; wherein, the interface elements in the interface to be located are leaf nodes of the second tree structure, and in the multiple block images divided from the interface to be located, the block image where the leaf node is located is the parent node of the leaf node.

[0076] S306: Search in the second tree structure according to the parent node of the first interface element to obtain the parent node with the highest similarity to the parent node of the first interface element.

[0077] S307: Search for the first interface element in the leaf nodes of the parent node with the highest similarity according to the position index of the first interface element.

[0078] For example, the position index of the first interface element is 2, and the interface element with a position index of 2 is found in the leaf node of the parent node with the highest similarity, which is the first interface element.

[0079] The following still takes the interface for searching air tickets as an example to illustrate the embodiment of the present invention.

[0080] A screenshot is taken of the first interface for searching air tickets, and the first interface is segmented using a segmentation algorithm to construct a first tree structure corresponding to the first interface, and a correspondence between the business semantics of the interface elements in the first tree structure and the position index of the interface elements is established.

[0081] The segmentation algorithms include:

[0082] S311. Using image morphological algorithms, such as blurring, binarization, gradient, straight line detection, etc., combined with the design characteristics of mobile application UI, the first interface is first divided into regions to obtain a set of block images of the first interface.

[0083] S312: Based on the obtained set of block images, image morphology is used again to perform detailed division to obtain valid information nodes (valid information nodes are interface elements) in each block image, and these valid information nodes and their parent block images are organized into a first tree structure.

[0084] The result of segmenting the first interface is as follows: Figure 4 As shown in the figure, each solid line image is a block image. The dashed lines are interface elements. Each block image is a parent node, and the interface elements in the block image are leaf nodes of the block image. For example, Figure 4 The block image A in the figure is a parent node, and the leaf nodes of the parent node include: the text "Beijing", the text "Hangzhou" and an icon for switching between the departure place and the arrival place.

[0085] S313: The user assigns business semantics to the interface elements in the first tree structure, records the position index of the interface element and its parent node, and establishes a correspondence between the business semantics of the interface elements in the first tree structure and the position index of the interface elements. The first tree structure corresponding to the first interface and the established correspondence are saved in the business semantics configuration file.

[0086] After obtaining the business semantic configuration file, the script is run to load the business semantic configuration file. If the operation that the script needs to perform on the current interface (ie, the interface to be located) is to click the search button, the search button is located on the current interface through the following search algorithm.

[0087] The search algorithms include:

[0088] S314 , query the correspondence between the business semantics of the interface elements in the business semantics configuration file and the position index of the interface elements to obtain the position index of the “search button”.

[0089] S315. According to the position index of the “search button”, search in the first tree structure to obtain the parent node of the “search button”.

[0090] S316: Perform a segmentation process on the current interface to obtain a second tree structure corresponding to the current interface. The process of obtaining the second tree structure is similar to the process of obtaining the first tree structure, and will not be repeated here.

[0091] S317: traverse the second tree structure, and use a structural similarity algorithm to find a parent node in the second tree structure that is most similar to the parent node of the “search button”.

[0092] S318. If the most similar parent node is not found in S317, the entire search process fails.

[0093] S319, if the most similar parent node is found in S317, then all leaf nodes of the most similar parent node are traversed, and according to the algorithm, the node with the most similar size and position to the interface element of the "search button" is found. If the most similar node is found, the found node is the interface element that needs to be operated, otherwise, the search process fails.

[0094] To summarize briefly, the segmentation algorithm reconstructs the interface screenshot into a two-level tree structure. Each text, image, icon, etc. is a leaf node, and multiple related leaf nodes constitute their parent nodes. The defined business semantics is to give special business meanings to leaf nodes. The search algorithm first finds similar parent nodes in the interface to be located, and then finds similar child nodes in similar parent nodes.

[0095] Figure 2 and Figure 3 There are two ways to implement the interface processing method. Figure 2 It automatically analyzes business semantics to locate interface elements. Figure 3 Business semantic configuration files are used to locate interface elements.

[0096] for Figure 3 In the business semantic configuration file, the position of the interface element can be a position index or area coordinates.

[0097] Preferably, the position of the interface element is a position index. The following describes why the position of the interface element is preferably a position index.

[0098] If the aspect ratio of the terminal screen when the business semantic configuration file is formed is different from the aspect ratio of the terminal screen when the script is run. For example, the script is entered through mobile phone A, and a screenshot is taken using mobile phone A when the script is entered, and the business semantic configuration file is generated based on the screenshot (i.e., the first interface mentioned above). In fact, the script is run by mobile phone B, and the aspect ratio of the screen of mobile phone A is 16:9, and the aspect ratio of the screen of mobile phone B is 16:10.

[0099] If the position is in regional coordinates, the same interface element will be displayed in different regional coordinates on phone A and phone B. For example, the search button on phone A displays regional coordinates M1, while the search button on phone B displays regional coordinates slightly above M1.

[0100] In this way, when the aspect ratio of the terminal is different from that of the terminal running the script when the business semantic configuration file is formed, the position of the located interface element may be inaccurate.

[0101] From the above analysis, it can be seen that if the position in the business semantic configuration is the regional coordinate, the script needs to be used on a terminal with the same aspect ratio of the screen.

[0102] To address the above problem, the position in the business semantic configuration file is a position index, which has nothing to do with the specific area coordinates. By positioning according to the index position, a set of scripts can be used on terminals with different screen aspect ratios.

[0103] for Figure 2 In the automatic analysis of business semantics, regardless of whether the screen size of the terminal where the script is entered is the same as the screen size of the terminal where the script is run, the position of the interface element can be a position index or a regional coordinate. Figure 2 The position of the interface elements in the scheme can be described by regional coordinates.

[0104] Since the correspondence between the position of the interface element and the business semantics is established based on the interface to be positioned, and the correspondence is not established in advance before running the script, the aspect ratio of the terminal screen for establishing the correspondence is the same as the aspect ratio of the terminal screen when the correspondence is used. There is no situation where the aspect ratio of the terminal screen for establishing the correspondence is different from the aspect ratio of the terminal screen when the correspondence is used, and thus the accurate positioning of the interface elements can be achieved according to the regional coordinates.

[0105] Figure 5 FIG. 2 is a flow chart of an interface processing method provided by another embodiment of the present invention. Figure 5 As shown, the interface processing method includes:

[0106] S401, obtaining at least one group of texts displayed on the interface and the position of each group of texts on the interface.

[0107] As an example, the text displayed on the interface to be located is recognized by OCR, and the displayed text is segmented, and one word is regarded as a group of text.

[0108] S402: For the operation to be performed, obtain target text matching a keyword of the operation to be performed from at least one group of texts.

[0109] S403, locate the interface element at the position of the target text and perform an operation.

[0110] In an embodiment of the present invention, if an interface element includes text, the text can be identified from the content displayed on the interface to locate the interface element and implement automated operation. Since script automation is implemented based on the content displayed on the interface, it is not based on the technical implementation details of the platform. Even if the technical implementation details of the platform are different, the displayed content is the same, so that the same set of scripts can be applied to different platforms. The problem of writing scripts applicable to the platform based on the technical implementation details of the platform is solved.

[0111] Figure 6 FIG. 2 is a flow chart of an interface processing method provided by another embodiment of the present invention. Figure 6 As shown, the interface processing method includes:

[0112] S501, record or write automation scripts.

[0113] In the process of recording or writing the automation script, a business semantic configuration file is generated.

[0114] The business semantic configuration file includes a first tree structure corresponding to the first interface, the interface elements in the first interface are leaf nodes of the first tree structure, and in the multiple block images divided by the first interface, the block image where the leaf node is located is the parent node of the leaf node.

[0115] The business semantics configuration file also includes the correspondence between the business semantics of the interface elements in the first tree structure and the position index of the interface elements.

[0116] S502, triggering an automation script to run on a computer device.

[0117] S503, start the object to be executed.

[0118] S504, determine whether there is a business semantic configuration file, if the determination result is yes, execute S505, if the determination result is no, execute S506.

[0119] S505: Load the business semantic configuration file.

[0120] It should be noted that the business semantic configuration file is used in S511 to S514 of the following steps.

[0121] S506, for the operation to be executed, determine the method of finding and locating the interface element. There are five methods, the first method is the automatic business semantic method, namely S507 to S510 below; the second method is the configuration business semantic method, namely S511 to S514 below; the third method is the image text method, namely S515 to S517 below; the fourth method is the image matching method, namely S518 below; the fifth method is the traditional method, namely S519 below.

[0122] S507, obtaining at least one set of text displayed on the interface and the position of the text on the interface.

[0123] S508, machine learning the business semantics of each group of texts.

[0124] S509: Establish a correspondence between business semantics and locations.

[0125] S510: Obtain a position corresponding to the business semantics of the operation to be executed according to the first corresponding relationship.

[0126] S511, obtaining a position index of a first interface element according to a business semantic configuration file.

[0127] S512: Obtain the parent node of the first interface element according to the position index of the first interface element.

[0128] S513, segmenting the interface to be positioned to obtain a corresponding second tree structure.

[0129] S514: Search the second tree structure for the first interface element to locate the first interface element.

[0130] The specific implementation of S511 to S514 has been described above and will not be repeated here.

[0131] S515, obtaining at least one set of text displayed on the interface and the position of the text on the interface.

[0132] S516: Obtain target text matching the keyword of the operation to be performed from at least one group of texts.

[0133] S517, locate the position of the target text.

[0134] S518, locating the target position of the image through the Open Source Computer Vision Library (OpenCV).

[0135] S519: Locate the position of the interface element according to the information of the interface element.

[0136] For example, the position of an interface element is located according to its ID, name or label.

[0137] S520, perform operations according to the obtained position, verify the operation results, and run the next script.

[0138] Figure 7 FIG. 1 is a block diagram of an interface processing device provided by an embodiment of the present invention. Figure 7 As shown, the interface processing device 600 includes a semantic determination module 601 , a position determination module 602 and an operation execution module 603 .

[0139] The semantics determination module 601 is used to determine the business semantics of the first interface element.

[0140] The position determination module 602 is used to determine the target position of the first interface element based on the business semantics of the first interface element.

[0141] The operation execution module 603 is used to locate the first interface element on the interface according to the target position to operate the first interface element.

[0142] In one embodiment of the present invention, the location determination module 602 includes:

[0143] The target position determination module is used to determine the interface position corresponding to the business semantics of the first interface element as the target position according to the correspondence between the business semantics of the interface element and the interface position.

[0144] In one embodiment of the present invention, the interface processing device 600 further includes:

[0145] The determination module is used to determine at least one set of information displayed on the interface to be located and the position of each set of information on the interface.

[0146] The machine learning module is used to machine learn the business semantics of each set of information.

[0147] A corresponding relationship establishment module is used to correspond the business semantics of the same group of information with the position on the interface for each group of information to obtain the corresponding relationship.

[0148] In one embodiment of the present invention, the at least one set of information is at least one set of text; the determining module includes:

[0149] The word segmentation module is used to segment the text displayed on the interface to be located, obtain the at least one group of texts according to the word segmentation results, so as to classify each group of texts in the at least one group of texts, and perform machine learning on the at least one group of texts after classification to obtain the business semantics of each group of texts.

[0150] Wherein, each group of texts includes at least one word.

[0151] In one embodiment of the present invention, the operation execution module 603 includes:

[0152] An element positioning module is used to use the interface element corresponding to the target position on the interface as the first interface element.

[0153] In one embodiment of the present invention, the interface position in the corresponding relationship is a regional coordinate.

[0154] In one embodiment of the present invention, the interface position in the corresponding relationship is a position index.

[0155] In one embodiment of the present invention, the interface processing device 600 further includes:

[0156] The file loading module is used to load business semantic configuration files.

[0157] Among them, the business semantic configuration file includes a first tree structure corresponding to the first interface, the interface elements in the first interface are leaf nodes of the first tree structure, and in the multiple block images divided by the first interface, the block image where the leaf node is located is the parent node of the leaf node.

[0158] The business semantics configuration file also includes the corresponding relationship, which is the corresponding relationship between the business semantics of the interface elements in the first tree structure and the position index of the interface elements.

[0159] In one embodiment of the present invention, the target position is a position index of the first interface element; the operation execution module 603 includes:

[0160] A first parent node query module, used to query in the first tree structure according to the position index of the first interface element to obtain the parent node of the first interface element;

[0161] A tree structure construction module, used for dividing the interface to be located into a plurality of block images to obtain a second tree structure corresponding to the interface to be located; wherein the interface elements in the interface to be located are leaf nodes of the second tree structure, and among the plurality of block images divided from the interface to be located, the block image where the leaf node is located is the parent node of the leaf node;

[0162] The second parent node query module is used to query in the second tree structure according to the parent node of the first interface element to obtain the parent node with the highest similarity to the parent node of the first interface element.

[0163] The child node query module is used to find the first interface element in the leaf nodes of the parent node with the highest similarity according to the position index of the first interface element.

[0164] In one embodiment of the present invention, the keywords of the first interface element in the script are used as the business semantics of the first interface element.

[0165] In one embodiment of the present invention, the interface processing device 600 further includes:

[0166] The position acquisition module is used to acquire at least one group of texts displayed on the interface and the position of each group of texts on the interface.

[0167] The matching module is used to obtain target texts matching the keywords to be operated in the script from the at least one group of texts.

[0168] The element operation module is used to locate the interface element at the position of the target text and perform operations.

[0169] Figure 8 A schematic diagram of the hardware structure of a computing device provided by an embodiment of the present invention is shown.

[0170] The computing device may include a processor 701 and a memory 702 storing computer program instructions.

[0171] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0172] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 702 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid-state memory. In a specific embodiment, the memory 702 includes a read-only memory (ROM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.

[0173] The processor 701 implements any one of the interface processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 702 .

[0174] In one example, the computing device may further include a communication interface 703 and a bus 710. Figure 8 As shown, the processor 701, the memory 702, and the communication interface 703 are connected via a bus 710 and communicate with each other.

[0175] The communication interface 703 is mainly used to implement the communication between the modules, devices, units and / or equipment in the embodiment of the present invention.

[0176] Bus 710 includes hardware, software or both, and the parts of computing device are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 710 may include one or more buses. Although the embodiment of the present invention describes and shows a specific bus, the present invention considers any suitable bus or interconnection.

[0177] The computing device can execute the interface processing method in the embodiment of the present invention, thereby realizing the combination Figure 1 and Figure 2 The interface processing method and device described.

[0178] In addition, in combination with the interface processing method in the above embodiment, the embodiment of the present invention can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the interface processing methods in the above embodiment is implemented.

[0179] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0180] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0181] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0182] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. An interface processing method, comprising: Loading a business semantic configuration file; wherein the business semantic configuration file includes a first tree structure corresponding to a first interface, the interface elements in the first interface are leaf nodes of the first tree structure, and among a plurality of block images divided by the first interface, the block image where the leaf node is located is the parent node of the leaf node, and the business semantic configuration file also includes a corresponding relationship, which is a corresponding relationship between the business semantics of the interface elements in the first tree structure and the position index of the interface elements; For a first interface element to be operated, determining the business semantics of the first interface element; Determine, according to the correspondence between the business semantics of the interface element and the position index of the interface position, the interface position corresponding to the business semantics of the first interface element as the target position; The first interface element is located on the interface according to the target position, so as to operate the first interface element.

2. The method according to claim 1, wherein: The method further comprises: Determine at least one set of information displayed on the interface and the position of each set of information on the interface; Machine learning the business semantics of each set of information; For each group of information, the business semantics of the same group of information is matched with the position on the interface to obtain the corresponding relationship.

3. The method according to claim 2, wherein: The at least one set of information is at least one set of text; the at least one set of information displayed on the determination interface includes: Segmenting the text displayed on the interface, obtaining the at least one group of texts according to the segmentation results, so as to classify each group of texts in the at least one group of texts, and performing machine learning on the at least one group of texts after classification to obtain the business semantics of each group of texts; Wherein, each group of texts includes at least one word.

4. The method according to claim 1, wherein: The step of locating the first interface element on the interface according to the target position includes: The interface element corresponding to the target position on the interface is used as the first interface element.

5. The method according to claim 1, wherein: The target position is a position index of the first interface element; Positioning the first interface element on the interface according to the target position includes: According to the position index of the first interface element, query in the first tree structure to obtain the parent node of the first interface element; The interface to be located is divided into a plurality of block images to obtain a second tree structure corresponding to the interface to be located; wherein the interface elements in the interface to be located are leaf nodes of the second tree structure, and in the plurality of block images divided from the interface to be located, the block image where the leaf node is located is the parent node of the leaf node; According to the parent node of the first interface element, searching in the second tree structure to obtain the parent node with the highest similarity to the parent node of the first interface element; According to the position index of the first interface element, the first interface element is searched for in the leaf nodes of the parent node with the highest similarity.

6. The method according to claim 1, wherein: The keywords of the first interface element in the script are used as the business semantics of the first interface element.

7. The method according to claim 1, wherein: The method further comprises: Obtain at least one set of text displayed on the interface and the position of each set of text on the interface; Acquire target text matching the keyword to be operated in the script from the at least one group of texts; The interface element at the position of the target text is located and an operation is performed.

8. An interface processing device, comprising: A file loading module, used to load a business semantic configuration file; wherein the business semantic configuration file includes a first tree structure corresponding to a first interface, the interface elements in the first interface are leaf nodes of the first tree structure, and among the multiple block images divided by the first interface, the block image where the leaf node is located is the parent node of the leaf node, and the business semantic configuration file also includes a corresponding relationship, which is a corresponding relationship between the business semantics of the interface elements in the first tree structure and the position index of the interface elements; A semantic determination module, used to determine the business semantics of a first interface element to be operated; A position determination module, used to determine, according to the correspondence between the business semantics of the interface element and the position index of the interface position, the interface position corresponding to the business semantics of the first interface element as the target position; The operation execution module is used to locate the first interface element on the interface according to the target position, so as to operate the first interface element.