Page processing method and device, medium and program product

Abstract

The embodiment of the invention discloses a page processing method and device, a medium and a program product. The method comprises the following steps: in a process of executing a robot process automation program, searching a matching element corresponding to a target element from a webpage based on a path of the target element; if the matched element cannot be found based on the path of the target element, searching a candidate element consistent with the tag name of the target element from the webpage; and performing code similarity matching on the target element and the candidate element, determining a matching element corresponding to the target element according to a matching result, and executing the current execution code line based on the element information of the matching element. According to the scheme, the change of the elements in the current page can be detected in real time in the process of executing the robot process automation program, and the information related to the elements in the robot process automation program is updated in time under the condition that the elements in the current page are changed; and normal execution of a robot process automation program is ensured.

Description

Technical Field

[0001] The present application relates to the field of page processing technology, and in particular to a page processing method, device, medium and program product. Background Art

[0002] As a leader and pioneer in the "automation-first" era, RPA (Robotic Process Automation) is widely used by businesses and industry sectors in finance, accounting, and other fields. RPA page processing uses robotic process automation technology to simulate manual web page operations, including automatically opening browsers, locating page elements, filling out forms, clicking buttons, and other actions. This technology can replace manual labor in repetitive web page operation tasks, such as data collection, information entry, and system integration, and is particularly suitable for business processes that require operations across multiple web systems. Its core principle is to identify element features within the web page's DOM structure and programmatically simulate mouse and keyboard operations, achieving interface-level automation without the need for API integration.

[0003] However, RPA technology is highly sensitive to changes in webpage content. If webpage elements change due to website upgrades, the RPA program will fail, potentially disrupting automated business operations. To improve the stability of all RPA operations and its resilience to unexpected events, it's crucial to monitor changes in target system page elements, adaptively adjust RPA scripts during runtime, and ensure stable RPA processes. Summary of the Invention

[0004] The embodiments of the present application provide a page processing method, device, medium and program product to timely detect page changes during the execution of a robotic process automation program to ensure the normal execution of the robotic process automation program.

[0005] According to one aspect of the present application, a page processing method is provided, the method comprising:

[0006] During the execution of the robotic process automation program, for a target element targeted in a currently executed line of code, searching for a matching element corresponding to the target element from a web page based on a path of the target element;

[0007] If the matching element cannot be found based on the path of the target element, searching for a candidate element with the same tag name as the target element from the web page;

[0008] The target element is matched with the candidate elements for code similarity, and a matching element corresponding to the target element is determined according to the matching result, so as to execute the currently executed code line based on the element information of the matching element.

[0009] According to one aspect of the present application, a page processing device is provided, the device comprising:

[0010] A matching element search module is used to search for a matching element corresponding to a target element in a currently executed code line based on a path of the target element in a web page during the execution of a robotic process automation program;

[0011] A candidate element search module is configured to search for a candidate element with the same tag name as the target element from the web page if the matching element cannot be found based on the path of the target element;

[0012] The matching element determination module is used to perform code similarity matching between the target element and the candidate element, determine the matching element corresponding to the target element according to the matching result, and execute the current execution code line based on the element information of the matching element.

[0013] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor. The computer program is executed by at least one processor so that the at least one processor can execute the page processing method of any embodiment of the present application.

[0017] According to another aspect of the present application, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the page processing method of any embodiment of the present application when executed.

[0018] According to another aspect of the present application, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the page processing method of any embodiment of the present application is implemented.

[0019] The technical solution of the embodiment of the present application is that during the execution of the robotic process automation program, for the target element targeted in the current execution code line, a matching element corresponding to the target element is searched from the web page based on the path of the target element, thereby detecting the elements in the web page in real time during the execution of the robotic process automation program and promptly determining the changes in the elements in the web page. If the matching element cannot be found based on the path of the target element, a candidate element with the same tag name as the target element is searched from the web page, and the candidate element with the same tag name as the target element can be further searched, thereby narrowing the range of elements that may match the target element, matching the target element with the candidate element for code similarity, determining the matching element corresponding to the target element based on the matching result, and executing the current execution code line based on the element information of the matching element, thereby accurately determining the matching element corresponding to the target element through code similarity matching, and replacing and updating the element information of the target element based on the changed element information of the matching element presented on the web page, thereby avoiding execution problems caused by still executing according to the original element information of the target page during the execution of the robotic process automation program, improving the success rate of robotic automation program execution, and more effectively applying to dynamically changing web pages.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flowchart of a page processing method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a page processing method provided in another embodiment of the present application;

[0024] Figure 3 A flowchart of a page processing method provided in another embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a page processing device provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0028] It should be noted that the terms "first", "second", "third", "fourth", "actual", "preset", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Figure 1 This is a flow chart of a page processing method provided in an embodiment of the present application. The embodiment of the present application can be applied to the case where a robotic process automation program is executed when there are changes in elements on a web page. The method can be executed by a page processing device, which can be implemented in the form of hardware and / or software, and the page processing device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0030] S110 . During execution of a robotic process automation program, for a target element targeted in a currently executed line of code, searching a web page for a matching element corresponding to the target element based on a path of the target element.

[0031] Among them, RPA programs can simulate manual web page operations through RPA technology, such as automatically opening a browser, locating page elements, filling out forms, and clicking buttons. This technology can replace manual repetitive web page operation tasks, such as data collection, information entry, and system integration, and is particularly suitable for business processes that require operations across multiple web systems. A target element is a pre-determined page element in the RPA program. A page element is a basic component of a web page or software interface, representing the smallest unit with which users can interact or visually perceive it. Within the RPA program, elemental information for the target element is stored in a page information file. The process for initially storing the target element's elemental information within the RPA program involves using a browser plug-in to obtain the elemental information for the page elements on the web page and saving it in a page information file. This page information file is subsequently used to monitor page element changes, run the process, and perform version management. A sample page information file format is shown in Table 1.

[0032] Table 1

[0033]

[0034]

[0035] In the "Page Information File," the "Page URL" is used to locate the page, the "Element Path XPath" is used to locate the HTML element on the page, the "Element HTML Code" is used to find elements with high similarity, the "Preset Attributes" are used to mark key attributes of the HTML code, and the "Screenshot Path" is used to load screenshot files for use in the image recognition and matching algorithm. Every modification to the "Page Information File" is committed to the repository, recording the version and modification time.

[0036] For example, when a robotic process automation program is executed, it will follow the lines of code in the robotic process automation execution program. The lines of code may contain operations targeting some target elements. For the target element targeted in the currently executed line of code, the path of the target element can be obtained from the page information file. The path of the target element is used as a parameter of the element search function, and the element search function is called to search for a matching element corresponding to the target element from the web page. During the development of the robotic process automation program, the web page targeted when obtaining the element information of the target element is the same page as the web page targeted during the current execution of the robotic process automation program. Assuming that the elements in the web page have not changed, it is actually possible to find an element on the web page that is identical to the target element, that is, a corresponding matching element, belonging to the same element and having the same element information. If the elements in the web page have changed, the matching element corresponding to the target element found on the web page will not be consistent with the element information of the target element.

[0037] S120: If the matching element cannot be found based on the path of the target element, search the web page for a candidate element with the same tag name as the target element.

[0038] For example, if the elements in the web page have not changed, that is, the elements in the current web page are consistent with the information of the elements when the robotic process automation program was developed, then the matching elements corresponding to the target elements can be found based on the path of the target elements saved in the robotic process automation program. However, if the elements in the web page have changed, that is, the elements in the current web page are inconsistent with the information of the elements when the robotic process automation program was developed, then the matching elements corresponding to the target elements cannot be found from the web page based on the path of the target elements saved in the robotic process automation program. In this case, candidate elements that are consistent with the tag name of the target element can be found from the web page. The tag name of an element is a keyword used in HTML to define the element type, enclosed in angle brackets (such as 、 ), which determines how the browser parses and renders the content. Tag names are divided into start tags (such as <h1> ) and the closing tag (such as< / h1> ), some empty elements can be abbreviated (such as ).For example, Indicates a hyperlink,

[0039]

[0040]

[0041]

[0042]

[0043] Figure 2

[0044]

[0045]

[0046]

[0047] ), attributes or text content constitute elements, e.g. <a href＝"url"> Link The character combination in defines the structure and function of the hyperlink element. Its core features include grammatical rule constraints (such as angle brackets enclosing tag names) and semantic expressions (such as Definition table, the tag name is directly related to the function and display mode of the element, and is the basic identifier of the HTML structured document. When the elements in the web page change, generally the position, size and other features of the elements presented in the web page change, but the substantial features of the elements will not change, such as the tag name will not change. Therefore, the candidate element with the same tag name as the target element can be found from the web page, that is, the candidate element that may be the same element as the target element can be found. S130, the target element and the candidate element are matched for code similarity, and the matching element corresponding to the target element is determined according to the matching result, so as to execute the currently executed code line based on the element information of the matching element. Exemplarily, the code of the matching element that matches the target element and the code of the target element should be consistent or similar in theory. The target element and the candidate element can be matched for code similarity, so as to determine the matching element corresponding to the target element in the web page according to the matching result. Exemplarily, the code of the target element can be obtained from the RPA program, and the code of the candidate element can be obtained from the underlying file of the currently presented web page. The code of the target element is matched with the code of each candidate element for similarity, thereby determining the candidate element with the highest code similarity to the target element and meeting the requirements, which can be used as the matching element corresponding to the target element. In an embodiment of the present application, when the codes of the target element and the candidate element are both HTML code, the code similarity matching is performed, that is, HTML code similarity matching is performed. In an embodiment of the present application, if a matching element corresponding to the target element is determined, the element information of the matching element can be replaced with the element information of the target element in the RPA program, thereby updating the element information of the target element, so that the RPA program is effectively executed based on the latest element information of the target element, avoiding execution errors. The technical solution of the embodiment of the present application, during the execution of the RPA program, for the target element in the currently executed code line, the matching element corresponding to the target element is searched from the web page based on the path of the target element, thereby detecting the elements in the web page in real time during the execution of the RPA program, and promptly determining the changes in the elements in the web page.If the matching element cannot be found based on the path of the target element, a candidate element with the same tag name as the target element is searched from the web page, and the candidate element with the same tag name as the target element can be further searched, thereby narrowing the range of elements that may match the target element. The target element and the candidate element are matched for code similarity, and the matching element corresponding to the target element is determined according to the matching result. The currently executed code line is executed based on the element information of the matching element, so as to accurately determine the matching element corresponding to the target element through code similarity matching, and replace and update the element information of the target element based on the changed element information presented on the web page of the matching element, thereby avoiding execution problems caused by still executing according to the original element information of the target page during the execution of the robot process automation program, improving the success rate of the execution of the robot automation program, and more effectively applying to dynamically changing web pages. Figure 2 A flowchart of a page processing method provided for another embodiment of the present application, the embodiment of the present application is optimized based on the above embodiment, and the solutions not fully described in the embodiment of the present application can be found in the above embodiment. As shown, the method of the embodiment of the present application specifically includes the following steps: S210. In the process of executing the robotic process automation program, for the target element targeted in the currently executed code line, a matching element corresponding to the target element is searched from the web page based on the path of the target element. S220. If the matching element cannot be found based on the path of the target element, if the matching element cannot be found based on the path of the target element, then a candidate element that is consistent with the tag name of the target element is searched from the web page. S230. A code character sequence is obtained from the code of the target element and a code character sequence is obtained from the code of the candidate element respectively. Wherein, a code character sequence refers to a string of characters (such as letters, numbers, symbols) arranged in a regular pattern in a programming or markup language, which is used to represent program instructions, data or document structure. In HTML, such sequences are represented by tags (such as <h1>( represents a first-level title). For example, during the code similarity matching process, similarity matching can be performed on code character sequences. The code of a target element is obtained from a robotic process automation program, and the code character sequence is obtained from the code of the target element. The code of a candidate element is obtained from the underlying code of a webpage, and the code character sequence is obtained from the code of the candidate element.

[0048] S240: Perform similarity matching between the code character sequence of the target element and the code character sequence of the candidate element.

[0049] Exemplarily, the code character sequence of the target element and the code character sequence of the candidate element can be matched for similarity. Specifically, the same sequence and different sequence of the code character sequence of the target element and the code character sequence of the candidate element can be compared based on the code character sequence matching algorithm to determine the length of the same sequence. The code character sequence matching algorithm can adopt the MersDiff algorithm. The ratio of the length of the same sequence to the length of the code character sequence of the candidate element can be used as the similarity matching result of the code character sequence of the target element and the code character sequence of the candidate element.

[0050] S250: Acquire preset attributes from the code of the target element and acquire preset attributes from the code of the candidate element respectively.

[0051] Preset attributes can be pre-determined attributes that reflect the functionality of an element. Attributes such as "Please enter your username," "username," "Please enter your password," "password," and "Login" can reflect the functionality of an element. If the target element and a candidate element all have the same preset attributes, it can be considered to some extent that the candidate element is likely to be the same as the target element. Preset attributes can be obtained from the target element's code and from the candidate element's code, respectively.

[0052] S260: Perform similarity matching based on the preset attributes of the target element and the preset attributes of the candidate elements.

[0053] Exemplarily, the preset attributes of the target element and the preset attributes of the candidate elements may be similarly matched. Specifically, the similarity matching result of the preset attributes of the target element and the preset attributes of the candidate elements may be determined based on the number of matched preset attributes.

[0054] In an embodiment of the present application, similarity matching is performed based on the preset attributes of the target element and the preset attributes of the candidate element, including:

[0055] For each candidate element, each preset attribute in the candidate element is matched with the preset attribute of the target element;

[0056] According to the matching result of the preset attributes, a value is assigned to each of the preset attributes in the candidate elements, and the values ​​of each preset attribute are weighted and summed to determine the similarity of the preset attributes between the target element and the candidate elements.

[0057] Exemplarily, in the process of similarity matching based on the preset attributes of the target element and the preset attributes of the candidate elements, each preset attribute in the candidate element can be matched with the preset attributes of the target element for each candidate element. A value is assigned to each preset attribute based on the matching result of each preset attribute. Exemplarily, if a preset attribute in the candidate element matches a preset attribute in the target element, the preset attribute is assigned a value of 1; if there is a preset attribute in the candidate element that does not match the preset attributes of the target element, the preset attribute is assigned a value of 0. For all the preset attributes in the candidate element, a weighted sum is taken based on the values ​​of the preset attributes to obtain the similarity between the preset attributes of the candidate element and the target element. Exemplarily, assuming that the candidate element contains n preset attributes in total, the value of the i-th preset attribute is e i , the corresponding weight is w i , then the preset attribute similarity of the candidate element is

[0058] S270: Determine a matching element corresponding to the target element according to the matching result, and execute the currently executed code line based on element information of the matching element.

[0059] Exemplarily, the execution scheme of the above-mentioned S230-S240 and S250-S260 is that S230-S240 and / or S250-S260 can be executed, that is, the matching result between the candidate element and the target element can be determined only by the similarity matching of the code character sequence, or the matching result between the candidate element and the target element can be determined only according to the similarity matching of the preset attribute, or the matching result between the candidate element and the target element can be determined according to the similarity matching of the code character sequence and the similarity matching of the preset attribute.

[0060] In the embodiment of the present application, the process of determining the matching result includes:

[0061] The code character sequence similarity obtained by similarity matching according to the code character sequence and the preset attribute similarity obtained by similarity matching according to the preset number are normalized and then weighted summed to obtain the matching result between the target element and the candidate element.

[0062] For example, if the similarity matching result between the candidate element and the target element is determined based on the code character similarity matching and the preset element similarity matching, the code character sequence similarity can be determined based on the code character similarity matching, i.e., the process of S230-S240, and the preset attribute similarity matching result can be determined based on the preset attribute similarity matching result, i.e., the scheme of S350-S260. The code character similarity matching result and the preset attribute similarity matching result are weighted and summed to obtain the matching result between the candidate element and the target element. The weights of the code character sequence similarity and the preset attribute similarity can be determined based on the specific situation.

[0063] In an embodiment of the present application, a code character sequence is obtained from the code of the target element and a code character sequence is obtained from the code of the candidate element respectively; the code character sequence of the target element and the code character sequence of the candidate element are similarly matched; and / or, preset attributes are obtained from the code of the target element and the code of the candidate element respectively; similarity matching is performed based on the preset attributes of the target element and the preset attributes of the candidate element, and the code character sequence is obtained from the code of the candidate element to determine the similarity with the code character sequence of the target element, and / or the preset attributes are obtained from the code of the candidate element to determine the similarity with the preset attributes of the target element, so that within the range of the candidate elements, the matching element corresponding to the target element is accurately matched based on the code, which solves the problem that the elements in the current web page change and the element information change, but the robot process automation program fails to detect the element change in time during execution and still executes with the original element information, resulting in execution failure.

[0064] Figure 3 This is a flow chart of a page processing method provided by another embodiment of the present application. The present embodiment is optimized based on the above embodiment. For solutions not fully described in the present embodiment, please refer to the above embodiment. Figure 3 As shown, the method of the embodiment of the present application specifically includes the following steps:

[0065] S310 . During execution of a robotic process automation program, for a target element targeted in a currently executed line of code, searching a web page for a matching element corresponding to the target element based on a path of the target element.

[0066] S320: If the matching element cannot be found based on the path of the target element, search the web page for a candidate element with the same tag name as the target element.

[0067] S330: Perform code similarity matching between the target element and the candidate elements.

[0068] S340: If there is a candidate element whose code similarity value with the target element is greater than a preset threshold, the candidate element corresponding to the highest code similarity value is used as the matching element corresponding to the target element.

[0069] For example, during the code similarity matching process, a code similarity value can be calculated. For example, the code similarity value can be the code character sequence similarity value, or the preset attribute similarity value can be used as the code similarity value, or the code similarity value can be obtained by weighted summing the code character similarity value and the preset attribute similarity value. If there is a candidate element and the target element whose code similarity value is greater than a preset threshold, the candidate element corresponding to the highest code similarity value is used as the matching element corresponding to the target element. The preset threshold can be determined based on actual conditions, for example, it can be determined to be 50%.

[0070] S350: Otherwise, matching the target element image of the target element with the full-screen image of the webpage is performed to determine a matching element corresponding to the target element.

[0071] Among them, a full-screen screenshot of a web page can be taken through a browser Action plug-in to take a screenshot of the web page and saved in a full-screen image. Exemplarily, if there is no candidate element with a code similarity value greater than a preset threshold value with the target element, it reflects that a matching element corresponding to the target element cannot be found based on the code similarity value, and then further matching can be performed based on the target element image of the target element and the full-screen image of the web page to determine the matching element corresponding to the target element based on image matching.

[0072] In the embodiment of the present application, determining a matching element corresponding to the target element by matching the target element image of the target element with the full-screen image of the webpage includes:

[0073] Performing a moving traversal matching operation on the target element image of the target element in the full-screen image of the webpage according to a preset step length, and determining a region image in the full-screen image that matches the target element image;

[0074] The webpage elements contained in the region image are determined as matching elements corresponding to the target element.

[0075] For example, the target element image of the target element can be moved and matched in the full-screen image of the web page according to a preset step size. For example, starting from the upper left corner of the full-screen image, the upper left corner of the target element image is aligned with the upper left corner of the full-screen image, and the regional image starting from the upper left corner of the full-screen image and consistent with the target element image is matched with the target element image. The target element image is then moved horizontally according to a preset step size for matching. If it moves to the rightmost edge of the full-screen image, it continues to move downward by the preset step size and then continues to move horizontally according to a preset compensation, traversing various regions in the full-screen image for matching. The regional image with the highest matching similarity is used as the regional image that matches the target element image. The web page elements contained in the regional image that matches the target element image are used as matching elements corresponding to the target element. In addition, in another possible implementation scheme, "moving and traversing the target element image of the target element in the full-screen image of the web page according to a preset step size" can be specifically "matching the target element image of the target element in the full-screen image of the web page according to the Opencv template matching algorithm", so as to determine the regional image in the full-screen image that matches the target element image.

[0076] In the embodiment of the present application, if there is a candidate element with a code similarity value greater than a preset threshold value to the target element, the candidate element corresponding to the highest code similarity value is used as the matching element corresponding to the target element; otherwise, the matching element corresponding to the target element is determined based on the target element image of the target element and the full-screen image of the web page. The above scheme directly determines the matching element corresponding to the target element when the code similarity value meets the requirements, and continues to determine the matching image corresponding to the target element based on the comparison of the target element image and the full-screen image of the web page when the code similarity value does not meet the requirements, thereby improving the success rate of finding matching images.

[0077] As a non-limiting implementation manner, after determining the matching element corresponding to the target element according to the matching result, the method further includes:

[0078] Executing the currently executed code line based on the element information of the matching element, and determining whether the execution result reaches a preset result;

[0079] If yes, the matching element is determined as the final matching element corresponding to the target element;

[0080] Otherwise, the matching element is removed as a matching element of the target element.

[0081] For example, after finding a matching element corresponding to a target element on a webpage, element information of the matching element can be determined and used as the new element information of the target element. Based on the matching element's element information, the currently executed code line is executed to implement the operation on the target element. After executing the currently executed code line, the execution result can be determined to reflect whether the matching element found accurately corresponds to the target element. A determination can be made as to whether the execution result meets a preset result. If so, the matching element is determined to be the final matching element corresponding to the target element; otherwise, the matching element is removed as the matching element corresponding to the target element. For example, the process of determining whether the execution result meets the preset result can be as follows: assuming the target element is a login element, executing the login element's preset result achieves normal login, with the webpage redirecting to the post-login content display page. If, after executing the currently executed code line, the actual execution result matches the preset result, meaning normal login is achieved and the webpage redirects to the post-login content display page, the execution result is determined to have met the preset result. If normal login is not achieved and the webpage does not redirect to the post-login content display page, the execution result is determined to have not met the preset result.

[0082] As a non-limiting implementation manner, after determining the matching element corresponding to the target element according to the matching result, the method further includes:

[0083] Obtaining element information of the matching element from the web page;

[0084] The element information of the target element is replaced by the element information of the matching element as the updated element information of the target element.

[0085] Exemplarily, after determining the matching element corresponding to the target element, the element information of the matching element can be obtained from the web page. Specifically, the element information of the matching element can be obtained from the page information file corresponding to the underlying code of the web page, and the element information of the matching element can be replaced with the element information of the target element. That is, the element information of the target element in the robotic process automation program is replaced with the element information of the matching element, so as to update the element information of the target element in the robotic process automation program to match the current presentation of the target element in the web page.

[0086] This application provides a specific implementation process of a page processing method, which specifically includes:

[0087] 1. Page element collection

[0088] When generating a page information file for the first time, use a custom browser action plug-in during RPA process development to obtain information about page elements and take screenshots, saving them to the file. This file is subsequently used to monitor changes to page elements, run processes, and perform version management. Table 1 shows a sample format for the page information file.

[0089] In the "Page Information File," the "Page URL" is used to locate the page, the "Element Path XPath" is used to locate the HTML element on the page, the "Element HTML Code" is used to find elements with high similarity, the "Key Attributes" is used to identify key features of the HTML code, and the "Screenshot Path" is used to load screenshot files for use in the image recognition and matching algorithm. Every modification to the "Page Information File" is committed to the repository, recording the version and modification time.

[0090] 2. Page element change monitoring and RPA process script self-healing

[0091] The page element change monitoring and self-healing process is divided into 8 main steps.

[0092] SETP1: Read the "page information file". The RPA process obtains the configuration information of the target element of the current operation and sends it to the browser action plug-in.

[0093] SETP2: The browser action plug-in monitors changes in the current page elements and searches for matching elements. If no changes are detected, the target element's status is set to "Monitoring Normal" and STEP 5 is executed. After the status is updated, the RPA process performs subsequent operations. If changes are detected, the RPA process attempts to save the updated, valid target element to a temporary "page information file" and executes STEP 3 to verify the self-healing behavior level. Otherwise, the process is marked as "Abnormal" and proceeds to STEP 5 to perform operations and prepare for manual processing. This step is divided into the following steps.

[0094] (1) Determine whether the current element needs self-healing

[0095] The basic idea is to use the browser Action plug-in to match the target element, open the browser according to the "page information file" to obtain the target element's xpath path, as a parameter of the find_element function, and call the function to find the element.

[0096] If the function's return element is not empty, obtain the HTML code of the returned page element and compare it with the record in the "page information file." If they are completely consistent, no self-healing is required. Otherwise, execute STEP 3 to verify the behavior level of self-healing. If the function's return element is empty, it indicates that the xpath path does not correspond to the page element, and execute step (2) of this step.

[0097] (2) Rematch by similarity of element HTML code

[0098] Get the HTML code of the target element in the "page information file" and use Python to get the outermost tag name in the HTML code. Use the find_element_by_tag_name function to traverse the page elements and match candidate elements with the same tag.

[0099] If the above steps return a list of candidate elements, and the list is empty, it indicates that no candidate elements were matched, and manual processing is performed. If the list is not empty, it indicates that multiple candidate elements were matched, and the similarity of the candidate elements is then compared one by one.

[0100] In the candidate element list, get the HTML code character sequence of each candidate element through getAttribute("outerHtml"). Use the Myers Diff algorithm to compare the HTML code character sequences of the candidate element and the target element in turn to obtain the code character sequence similarity S.

[0101] In addition, the key attribute matching degree M of the HTML element is defined. The calculation method is as follows: first, the key attribute of the target element is obtained. Then, the keyword search method is used to count the number of key attributes present in the HTML code of the candidate element, and finally the weighted average is obtained. (The key attribute is the preset attribute in the above embodiment)

[0102] Assume that the candidate element contains n preset attributes, and the value of the i-th preset attribute is e i , the corresponding weight is w i , then the preset attribute similarity of the candidate element is

[0103] Finally, considering the code character sequence similarity and key attribute similarity, the formula for element similarity is given as follows:

[0104] Element similarity = code character sequence similarity S*a + key attribute similarity M*b, where a is the weight of code similarity and b is the weight of key attribute similarity.

[0105] The candidate elements with similarity greater than 50% are retained as the retained candidate elements. Among the retained candidate elements, the candidate element with the highest similarity is selected as the best matching element for the current target element.

[0106] If, in the above steps, the best matching element for the target element is obtained, the HTML code and corresponding XPath of the matching element are obtained and saved in a temporary "page information file", proceed to STEP 3 to perform self-healing behavior level verification.

[0107] If no element is matched, then proceed to step (3).

[0108] (3) Use image recognition technology to re-match the location

[0109] Read the screenshot path of the target element stored in the basic table, use image recognition technology to obtain the coordinates of the screenshot location in the webpage screenshot, click the coordinates and record the HTML tag and related XPath in a temporary "page information file". The main steps are as follows:

[0110] A. Use the browser Action plug-in to take a screenshot of the web page and save it as the original image;

[0111] B. Read the target image stored in the "page information file", that is, the image of the saved target element;

[0112] C. Read the original image, that is, a full-screen screenshot of the web page;

[0113] D. Process the original image, convert it into grayscale image, and mask the color difference;

[0114] E. Use the image matching algorithm to obtain the coordinates of the upper left vertex (x0, y0) of the target in the original image.

[0115] F. Calculate the target center position coordinates (x1, y1);

[0116] G. Control the mouse to click the target element at the coordinates (x1, y1);

[0117] H. The browser Action plug-in captures the click event and obtains the xpath path;

[0118] I. Save the xpath path and other information to a temporary "page information file".

[0119] After obtaining the height h and width w of the target element target image, the coordinates (x1, y1) of the center position of the target element target on the screen can be calculated using the following formula.

[0120] X1=X0+h / 2;

[0121] Y1=Y0+w / 2.

[0122] If a matching element for the target element is found in the above steps, proceed to STEP 3. Otherwise, mark it as "abnormal" and go to STEP 5 to perform the operation.

[0123] Step 3: Verify the behavior of the self-healing process. Use the browser Action plug-in to perform PRA operations on the newly replaceable element to determine whether the expected effect is achieved and the process runs smoothly. If so, the status is marked as "Self-healing successful."

[0124] STEP 4: The RPA process determines whether to update the element information in the "Page Information File" based on the execution status reported by the browser Action plug-in. If an update is required, the temporary "Page Information File" is used to update the "Page Information File" and update the XPath field, HTML code, and element screenshots for the corresponding target element.

[0125] Step 5: Determine the execution status of this step in the RPA process. Based on the status indicated in the processing logic, the status can be "normal monitoring," "abnormal," or "self-healing successful." If the status is "abnormal," the RPA process is interrupted and a human operator is notified. If the status is "self-healing successful," proceed to Step 6. If the status is "normal," proceed directly to Step 7.

[0126] STEP 6: Summarize the page element monitoring results and write them into the status field of the row where the current element is located in the "Page Information File".

[0127] STEP 7: Determine whether the RPA process has subsequent operations. If so, switch to the next row of elements and continue to STEP 2 to perform the operations.

[0128] Figure 4 This is a schematic diagram of the structure of a page processing device provided in an embodiment of the present application. The device can execute the page processing method provided in any embodiment of the present application and has the corresponding functional modules and beneficial effects of the execution method. Figure 4 As shown, the device includes:

[0129] A matching element search module 410 is configured to search for a matching element corresponding to a target element in a currently executed line of code based on a path of the target element in a web page during execution of a robotic process automation program;

[0130] A candidate element search module 420 is configured to search for a candidate element with the same tag name as the target element from the web page if the matching element cannot be found based on the path of the target element;

[0131] The matching element determination module 430 is configured to perform code similarity matching between the target element and the candidate elements, determine a matching element corresponding to the target element according to the matching result, and execute the currently executed code line based on the element information of the matching element.

[0132] In the embodiment of the present application, the matching element determination module 430 performs code similarity matching on the target element and the candidate element, including:

[0133] respectively obtaining a code character sequence from the code of the target element and a code character sequence from the code of the candidate element;

[0134] Performing similarity matching between the target element's code character sequence and the candidate element's code character sequence; and / or,

[0135] respectively obtaining a preset attribute from the code of the target element and obtaining a preset attribute from the code of the candidate element;

[0136] Similarity matching is performed based on the preset attributes of the target element and the preset attributes of the candidate elements.

[0137] In the embodiment of the present application, the matching element determination module 430 performs similarity matching based on the preset attributes of the target element and the preset attributes of the candidate element, including:

[0138] For each candidate element, each preset attribute in the candidate element is matched with the preset attribute of the target element;

[0139] According to the matching result of the preset attributes, a value is assigned to each of the preset attributes in the candidate elements, and the values ​​of each preset attribute are weighted and summed to determine the similarity of the preset attributes between the target element and the candidate elements.

[0140] In the embodiment of the present application, the matching element determination module 430 determines that the matching result includes:

[0141] The code character sequence similarity obtained by similarity matching according to the code character sequence and the preset attribute similarity obtained by similarity matching according to the preset number are normalized and then weighted summed to obtain the matching result between the target element and the candidate element.

[0142] In the embodiment of the present application, the matching element determination module 430 determines the matching element corresponding to the target element according to the matching result, including:

[0143] If there is a candidate element with a code similarity value greater than a preset threshold value with the target element, the candidate element corresponding to the highest code similarity value is used as the matching element corresponding to the target element;

[0144] Otherwise, matching the target element image of the target element with the full-screen image of the webpage is performed to determine a matching element corresponding to the target element;

[0145] Determining a matching element corresponding to the target element by matching the target element image of the target element with the full-screen image of the webpage includes:

[0146] Performing a moving traversal matching operation on the target element image of the target element in the full-screen image of the webpage according to a preset step length, and determining a region image in the full-screen image that matches the target element image;

[0147] The webpage elements contained in the region image are determined as matching elements corresponding to the target element.

[0148] In the embodiment of the present application, after determining the matching element corresponding to the target element according to the matching result, the apparatus further includes:

[0149] An execution result judgment module is used to execute the current execution code line based on the element information of the matching element to determine whether the execution result reaches a preset result;

[0150] a determination module, configured to, if yes, determine the matching element as the final matching element corresponding to the target element;

[0151] The cancellation module is used for canceling the matching element as a matching element of the target element otherwise.

[0152] In the embodiment of the present application, after determining the matching element corresponding to the target element according to the matching result, the apparatus further includes:

[0153] An element information acquisition module, configured to acquire element information of the matching element from the web page;

[0154] An updating module is used to replace the element information of the target element with the element information of the matching element as the updated element information of the target element.

[0155] A page processing device provided in an embodiment of the present application can execute a page processing method provided in any embodiment of the present application, and has functional modules and beneficial effects corresponding to the execution method.

[0156] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0157] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0158] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless page processing transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0159] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the page processing method.

[0160] In some embodiments, the page processing method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the page processing method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the page processing method in any other appropriate manner (for example, by means of firmware).

[0161] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable web processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0163] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0165] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0166] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0167] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the page processing method provided in any embodiment of the present application.

[0168] During implementation, the computer program product may be written in one or more programming languages ​​or a combination thereof to write computer program code for performing the operations of the present invention, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0169] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired information of the technical solution of this application can be achieved. This document is not limited here.

[0170] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.< / h1>

Claims

1. A page processing method, characterized in that: The method comprises: During the execution of the robotic process automation program, for a target element targeted in a currently executed code line, searching for a matching element corresponding to the target element from a web page based on a path of the target element; If the matching element cannot be found based on the path of the target element, searching for a candidate element with the same tag name as the target element from the web page; The target element is matched with the candidate elements for code similarity, and a matching element corresponding to the target element is determined according to the matching result, so as to execute the currently executed code line based on the element information of the matching element.

2. The method according to claim 1, characterized in that Performing code similarity matching on the target element and the candidate element includes: respectively obtaining a code character sequence from the code of the target element and a code character sequence from the code of the candidate element; Performing similarity matching between the target element's code character sequence and the candidate element's code character sequence; and / or, respectively obtaining a preset attribute from the code of the target element and obtaining a preset attribute from the code of the candidate element; Similarity matching is performed based on the preset attributes of the target element and the preset attributes of the candidate elements.

3. The method according to claim 2, characterized in that Performing similarity matching based on the preset attributes of the target element and the preset attributes of the candidate element includes: For each candidate element, each preset attribute in the candidate element is matched with the preset attribute of the target element; According to the matching result of the preset attributes, a value is assigned to each of the preset attributes in the candidate elements, and a weighted sum is performed on the values ​​of each preset attribute to determine the similarity of the preset attributes between the target element and the candidate elements.

4. The method according to claim 2, characterized in that The process of determining the matching result includes: The code character sequence similarity obtained by performing similarity matching according to the code character sequence and the preset attribute similarity obtained by performing similarity matching according to the preset number are normalized and then weighted summed to obtain the matching result between the target element and the candidate element.

5. The method according to claim 1, wherein Determining a matching element corresponding to the target element according to the matching result includes: If there is a candidate element with a code similarity value greater than a preset threshold value with the target element, the candidate element corresponding to the highest code similarity value is used as the matching element corresponding to the target element; Otherwise, matching the target element image of the target element with the full-screen image of the webpage is performed to determine a matching element corresponding to the target element; Determining a matching element corresponding to the target element by matching the target element image of the target element with the full-screen image of the webpage includes: Performing a moving traversal matching operation on the target element image of the target element in the full-screen image of the webpage according to a preset step length, and determining a region image in the full-screen image that matches the target element image; The webpage elements contained in the region image are determined as matching elements corresponding to the target element.

6. The method according to any one of claims 1 to 5, characterized in that After determining the matching element corresponding to the target element according to the matching result, the method further includes: Executing the currently executed code line based on the element information of the matching element, and determining whether the execution result reaches a preset result; If yes, the matching element is determined as the final matching element corresponding to the target element; Otherwise, the matching element is removed as a matching element of the target element.

7. The method according to any one of claims 1 to 5, characterized in that After determining the matching element corresponding to the target element according to the matching result, the method further includes: Obtaining element information of the matching element from the web page; The element information of the target element is replaced by the element information of the matching element as the updated element information of the target element.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the page processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the page processing method according to any one of claims 1 to 7 when executed.

10. A computer program product, characterized in that The method comprises a computer program, which implements the page processing method according to any one of claims 1 to 7 when executed by a processor.

Images