Operation automation system, operation automation device, operation automation method and program

Abstract

To perform support so that a web element designated as an operation object can be found before change even in the case that a description of a web page is changed.SOLUTION: A storage unit stores a scenario including a web page identifier, an operation object web element identifier, and a web operation to a web element of an operation object, and auxiliary information for specifying the operation object in a web page. A scenario execution unit reads the web page of the web page identifier included in the scenario, and determines whether there is the operation object web element identifier in the read web page. The scenario execution unit selects a new web element of an operation object by analyzing contents of the read web page by using the auxiliary information, and performs web operation written in the scenario to the selected web element in the case of determining that there is not the operation object web element identifier.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an operation automation system, an operation automation device, an operation automation method, and a program. [Background technology]

[0002] RPA (Robotic Process Automation), a type of automation tool, is a technology that can automatically execute routine tasks for applications that have been performed manually by operating the user interface of an information processing device such as a PC (personal computer). For example, RPA can reproduce tasks that have been performed manually by executing a scenario that describes the work procedure (see, for example, Patent Document 1).

[0003] On the other hand, many office tasks these days involve internal and external web page operations. Therefore, there is RPA that can automate web page operations (see, for example, Non-Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4883638 [Non-patent literature]

[0005] [Non-Patent Document 1] "WinActor Suite Library Browser Operation (Value Acquisition, 3)", [online], NTT Advanced Technology Corporation, retrieved July 20, 2023,<URL:https: / / winactor.biz / sweet / 2021 / 09 / 30_4617.html> Summary of the Invention [Problem to be solved by the invention]

[0006] RPA automates web operations by breaking down web operations into processes for web elements on a web page, describing them in advance, and then operating each web element according to that description. However, there are cases where the description of the web page to be operated is changed after the automated process is written. In such cases, conventional technology has the problem that the target web element cannot be found, making it impossible to execute the automated process.

[0007] In view of the above circumstances, the present invention aims to provide an operation automation system, an operation automation device, an operation automation method, and a program that can assist in finding web elements that were specified as operation targets before a change was made to the description of a web page. [Means for solving the problem]

[0008] According to one aspect of the present invention, a storage unit stores a scenario including a web page identifier for identifying a web page, an operation target web element identifier for identifying an operation target web element within the web page, and a web operation for the operation target web element, and stores analysis information including information on a description of the operation target web element and information on descriptions of surrounding web elements that are web elements surrounding the operation target web element, the analysis information being obtained based on a pre-update web page that is the web page at the time of creating the scenario, as auxiliary information for identifying an operation target on the web page; and a web browser that reads a web page with the web page identifier included in the scenario and writes the web operation target web element identifier into the read web page. an operation automation system comprising: a scenario execution unit that performs a determination process to determine whether or not there is a web element identifier to be operated; if the determination process determines that there is a web element identifier to be operated, performs the web operation on the web element to be operated on the web element identified by the web element identifier to be operated; if the determination process determines that the web element identifier to be operated on is not present, performs an analysis process to select a new web element to be operated from among the web elements included in the web page by analyzing the contents of the loaded web page using the auxiliary information; and performs the web operation on the web element to be operated on the new web element to be operated selected by the analysis process.

[0009] One aspect of the present invention is the above-mentioned operation automation system, wherein, in the analysis process, the scenario execution unit sets at least some of the web elements included in the updated web page, which is the web page at the time the scenario is executed, as candidates for new operation targets, generates analysis information for each candidate, including information about the description of the candidate web element and information about the descriptions of web elements surrounding the candidate web element, obtained based on the updated web page, calculates an index indicating the possibility that the candidate is a new operation target using the generated analysis information for the candidate and the analysis information for the web element to be operated read from the memory unit, narrows down the candidates based on the calculated index, and sets the candidate web element narrowed down to one by the analysis process, or a web element selected by the user from the multiple candidate web elements narrowed down by the analysis process, as the new operation target web element.

[0010] One aspect of the present invention is the above-mentioned operation automation system, wherein the scenario execution unit obtains tag or display position information of the web element to be operated from the analysis information of the web element to be operated, and selects as the candidate a web element from among the web elements included in the updated web page whose tag is the same as the tag of the web element to be operated or whose display position is within a predetermined range from the display position of the web element to be operated.

[0011] One aspect of the present invention is the above-mentioned operation automation system, wherein the analysis information includes identification information of the web page, an identifier of the web element to be operated, content surrounding the web element to be operated in the web page before the update, a surrounding web element identifier that identifies the surrounding web element, and content surrounding the surrounding web element in the web page before the update.

[0012] One aspect of the present invention is the above-mentioned operation automation system, wherein the surrounding web elements are other web elements that are close to the web element to be operated in the syntax tree of web elements included in the web page.

[0013] One aspect of the present invention is the above-mentioned operation automation system, wherein the auxiliary information further includes a model that calculates the index using analysis information of the candidate and the analysis information of the web element to be operated.

[0014] One aspect of the present invention is the above-mentioned operation automation system, further comprising a learning unit that learns the model using multiple sets of training data, each set consisting of analytical information for a web element to be operated, obtained based on a learning web page before the update, and analytical information for a new web element to be operated, obtained based on the updated learning web page.

[0015] One aspect of the present invention is the above-mentioned operation automation system, wherein the learning unit further uses training data sets of analytical information for web elements to be operated, obtained based on a learning web page, and analytical information for new web elements to be operated or new web elements that are not to be operated, obtained based on a web page updated by adding, moving, or deleting web elements from the learning web page at a predetermined probability, to learn the model.

[0016] One aspect of the present invention is the above-mentioned operation automation system, further comprising a scenario editing unit that generates analytical information for web elements to be operated that are included in an edited scenario, and writes the generated analytical information to the memory unit in association with the generated scenario.

[0017] According to one aspect of the present invention, a storage unit stores a scenario including a web page identifier for identifying a web page, an operation target web element identifier for identifying an operation target web element within the web page, and a web operation for the operation target web element, and stores analysis information including information on a description of the operation target web element and information on descriptions of surrounding web elements that are web elements surrounding the operation target web element, the analysis information being obtained based on a pre-update web page that is the web page at the time of creating the scenario, as auxiliary information for identifying an operation target on the web page; and a storage unit that loads a web page with the web page identifier included in the scenario using a web browser, and stores analysis information within the loaded web page. and a scenario execution unit that performs a determination process to determine whether the operation target web element identifier exists, and if the determination process determines that the operation target web element identifier exists, performs the web operation on the operation target web element on the web element identified by the operation target web element identifier, and if the determination process determines that the operation target web element identifier does not exist, performs an analysis process to select a new operation target web element from the web elements included in the web page by analyzing the contents of the loaded web page using the auxiliary information, and performs the web operation on the operation target web element on the new operation target web element selected by the analysis process.

[0018] According to one aspect of the present invention, a scenario including a web page identifier for identifying a web page, an operation target web element identifier for identifying a web element to be operated within the web page, and a web operation on the operation target web element is stored in a storage unit that stores analysis information, which is acquired based on a pre-update web page that is the web page at the time of creating the scenario, and includes information about a description of the operation target web element and information about descriptions of peripheral web elements that are web elements surrounding the operation target web element, as auxiliary information for identifying an operation target on the web page. The scenario is acquired from a storage unit that stores analysis information, which is acquired based on a pre-update web page that is the web page at the time of creating the scenario, as auxiliary information for identifying an operation target on the web page. The web page having the web page identifier included in the acquired scenario is read by a web browser, and the read web page is and a scenario execution step of performing the web operation on the web element to be operated on the web element specified by the operation target web element identifier if it is determined in the determination step that the operation target web element identifier is present, and performing an analysis process of selecting a new web element to be operated on from among the web elements included in the web page by analyzing the content of the loaded web page using the auxiliary information if it is determined in the determination step that the operation target web element identifier is not present, and performing the web operation on the web element to be operated on the new web element to be operated selected by the analysis process.

[0019] One aspect of the present invention is a program for causing a computer to function as the above-described operation automation system. [Effects of the Invention]

[0020] According to the present invention, even when a change is made to the description of a Web page, it is possible to assist in finding a Web element that was specified as an operation target before the change. [Brief explanation of the drawings]

[0021] [Figure 1]1 is an overall configuration diagram of an operation automation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of modeling a Web page according to the embodiment. [Figure 3] 10A and 10B are diagrams illustrating examples of an HTML representation and a syntax tree representation of Web page content according to the embodiment. [Figure 4] 10A and 10B are diagrams illustrating a representation of a web page content before the web page is updated according to the embodiment, and a display in a web browser. [Figure 5] 10A and 10B are diagrams illustrating a representation of a web page content after the web page is updated according to the embodiment, and a display in a web browser. [Figure 6] 10A and 10B are diagrams illustrating a representation of a web page content after the web page is updated according to the embodiment, and a display in a web browser. [Figure 7] FIG. 10 is a diagram showing an example of analytical information according to the embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a scenario according to the embodiment. [Figure 9] FIG. 10 is a diagram illustrating a configuration example of an RPA processing unit according to the embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of a scenario editing process performed by the operation automation device according to the embodiment. [Figure 11] FIG. 10 is a flowchart showing an example of a scenario execution process performed by the operation automation device according to the embodiment. [Figure 12] 10 is a flowchart showing an example of the operation of the operation target Web element determination unit according to the embodiment. FIG. [Figure 13] 10 is a flowchart showing an example of the operation of the operation target Web element determination unit according to the embodiment. FIG. [Figure 14] FIG. 10 is a flowchart showing an example of the operation of the analytical information generator according to the embodiment. [Figure 15] FIG. 2 is a diagram illustrating an example of the configuration of a model generating device according to the embodiment. [Figure 16] FIG. 10 is a flowchart showing an example of the operation of the pseudo teacher data generating unit according to the embodiment. [Figure 17]FIG. 10 is a flowchart showing an example of the operation of the pseudo teacher data generating unit according to the embodiment. [Figure 18] FIG. 10 is a flowchart showing an example of the operation of the normal teacher data generation unit according to the embodiment. [Figure 19] FIG. 2 is a diagram illustrating a hardware configuration of the operation automation device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0022] An embodiment of the present invention will be described in detail below with reference to the drawings. The operation automation device of this embodiment is equipped with RPA. RPA is a type of automation tool, and its functions include the automatic operation of web pages. Data describing the automatic operation of RPA is called a scenario. In this embodiment, if the description of the web page is changed after the processing of the automatic operation to be executed by RPA is described in the scenario, the system supports the discovery of the web element to be operated. This makes it easy to respond to changes in the web page.

[0023] To operate a web page with RPA, the web element to be operated (link, button, form, etc.) is directly identified as the operation target, and events (operations) such as clicks and character input are sent to the operation target. There are three main methods for identifying web elements:

[0024] (1) Specify the coordinates on the screen directly. (2) Obtain coordinates by performing image recognition on the screen. (3) From the html (Hypertext Markup Language) data describing the screen, the location in the html syntax (XPath, etc.) of the object to be operated is specified.

[0025] Of the above methods, method (3) is superior because it does not depend on the RPA execution environment, such as display size, and there is no need for recognition errors. However, if the description of the web page is changed after the operation content for the target web page has been determined and written in the scenario, methods (1) and (2) may be able to continue to execute the RPA operation as long as the appearance of the web page does not change, whereas method (3) will become inoperable if the logical structure is changed, even if the appearance of the web page does not change.

[0026] To deal with this situation, it is necessary to access the changed web page, understand the changes, and then modify the operation details described in the scenario. However, performing this manually is a cumbersome task, and it is difficult to deal with frequently changed web pages. In addition, it may be difficult to understand how the operation target has changed on the changed web page. Furthermore, after modifying the operation details, it is necessary to redo the operation described in the scenario from the beginning, which is costly. In this embodiment, by solving these problems, an RPA that is resilient to web page changes can be realized, thereby improving user convenience.

[0027] FIG. 1 is a diagram showing an example of the overall configuration of an operation automation system 10 according to an embodiment of the present invention. FIG. 1 illustrates only functional blocks related to this embodiment. The operation automation system 10 includes an operation automation device 1 and a model generation device 4. As shown in FIG. 1, the operation automation device 1 and the model generation device 4 are connected to a Web service providing device 8 via a network 7. The network 7 and the Web service providing device 8 can be implemented using any general-purpose technology. For example, the network 7 may be a public network such as the Internet, a private network such as a LAN (Local Area Network), or a combination thereof. The Web service providing device 8 provides a Web page P1 to the operation automation device 1 via the network 7. The Web page P1 is content data for the Web page. Although FIG. 1 illustrates only one operation automation device 1 and one Web service providing device 8, the number of operation automation devices 1 and Web service providing devices 8 is arbitrary.

[0028] The operation automation device 1 is, for example, a computer device. The operation automation device 1 includes an input unit 11, a display unit 12, a scenario storage unit 13, an analytical information storage unit 14, an RPA processing unit 15, a web browser unit 16, and a model storage unit 17.

[0029] The input unit 11 is a user interface operated by the user when inputting user instructions to the operation automation device 1. The input unit 11 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), buttons, and a touch panel. The display unit 12 displays data. The display unit 12 is an image display device such as a CRT display, a liquid crystal display, or an organic EL display. The display unit 12 may be configured as a touch panel integrated with the input unit 11.

[0030] The scenario storage unit 13 stores an RPA scenario R1. The scenario R1 is data written in a description format executable by the RPA processing unit 15. The scenario R1 indicates the execution order of operations, operation targets, and events to be performed on the operation targets. The events include the operations to be performed on the operation targets and the parameter values ​​used when performing the operations. If no parameters are used in the operations, the events do not include the parameter values. The operations indicated by the events are, for example, operations performed on the operation targets by the input unit 11. The operation targets include applications that realize predetermined functions of the operation automation device 1 and web browsers. When a web browser is the operation target, for example, the events indicate the acquisition of a web page P1 by specifying a URL (Universal Resource Locator), and web operations performed by the input unit 11 on web elements in the acquired web page P1.

[0031] The analytical information storage unit 14 stores a DB (database) of analytical information. The analytical information is used to estimate the destination of a web element in web page P1 that was described as a target for operation when a scenario was created and that element is moved to a different location in the HTML syntax of web page P1 when the scenario is executed. Hereinafter, a web element that is a target for operation by RPA will be referred to as a "target web element." Furthermore, when the description of the target web element is written in a different location in the HTML syntax of web page P1 after it is updated, it will be referred to as the target web element being moved.

[0032] The RPA processing unit 15 has two operating modes for creating and executing the scenario R1: a scenario creation mode and a scenario execution mode. In the scenario creation mode, the RPA processing unit 15 determines the event of which web element in the acquired web page P1 to operate based on instructions input by the scenario creator via the input unit 11, and describes the event in the scenario R1. Furthermore, the RPA processing unit 15 extracts analytical information about the operation target web element based on the information contained in the acquired web page P1, associates it with the scenario R1, and stores the extracted information in the analytical information DB of the analytical information storage unit 14. In the scenario execution mode, the RPA processing unit 15 acquires the web page P1 provided by the web service providing device 8 via the network 7 using a web browser provided by the web browser unit 16 in accordance with the contents described in the scenario R1, and performs a web operation on the acquired web page P1. If the RPA processing unit 15 determines that the operation target web element described in the scenario R1 has been moved, it selects a candidate web element from the web page P1 and generates analytical information for the selected web element. The RPA processing unit 15 selects a destination web element from among the destination candidates using the analysis information of the operation target web element and the analysis information of the destination candidate web elements, and performs the web element operation described in scenario R1 on the destination web element. Alternatively, the RPA processing unit 15 performs the web element operation described in scenario R1 on the destination web element selected by the user via the input unit 11 from among multiple destination candidates selected using the analysis information of the operation target web element and the analysis information of the destination candidate web elements.

[0033] The web browser unit 16 provides a web browser. The web browser unit 16 can be realized using any general-purpose technology. The web browser provided by the web browser unit 16 is a commonly used web browser with an extension function commonly used for web page development added. The web browser executed by the web browser unit 16 retrieves the web page P1 corresponding to the URL input from the RPA processing unit 15 from the web service providing device 8 and displays it on the display unit 12. The web browser executed by the web browser unit 16 also manipulates the web elements of the XPath input from the RPA processing unit 15. The web browser executed by the web browser unit 16 inputs the content included in the web page P1 and the web elements within that content, and obtains the XPath of the input web element from the input content. The web browser executed by the web browser unit 16 can also obtain the display coordinates of the specified web element on the web browser display screen by using an extension function. Furthermore, by sending a program from the RPA processing unit 15, the web browser executed by the web browser unit 16 can analyze the retrieved web page P1 and return the analysis results to the RPA processing unit 15.

[0034] Model storage unit 17 stores the web element learning model created by model generation device 4. The web element learning model is a model that receives analytical information of the operation target web element and analytical information of a candidate destination of the operation target web element as input, and calculates the probability that the candidate destination operation target web element is the destination of the operation target web element.

[0035] The model generation device 4, like the RPA processing unit 15, uses a web browser to acquire a web page P1 provided by the web service providing device 8 via the network 7. The model generation device 4 generates a web element learning model using the acquired web page P1. The model generation device 4 outputs the generated web element learning model to the operation automation device 1. The model generation device 4 operates independently of the RPA processing unit 15 of the operation automation device 1, but it is necessary for the model generation device 4 to operate before the RPA processing unit 15 executes scenario R1 and to store the web element learning model in the model storage unit 17.

[0036] FIG. 2 is a diagram showing a model of a Web page P1 used in this embodiment. This model of Web page P1 includes a Web page identifier P10 expressed as a URL and Web page content P20 written in HTML. The Web page content P20 has multiple Web elements P30-1 to P30-N (N is an integer equal to or greater than 2). FIG. 2 shows an example where N=2. Each Web element P30-n (n is an integer equal to or greater than 1 and equal to or less than N) is made up of HTML tags (a, div, p, etc.) and includes a Web element identifier P40-n written in XPath and coordinates P50-n indicating the display position of the Web element P30-n when it is displayed in a Web browser. Note that the a tag specifies the starting point of a link, the div tag indicates a separator, and the p tag indicates a paragraph.

[0037] FIG. 3 shows examples of an HTML representation P21 and a syntax tree representation P22 of the Web page content P20 shown in FIG. 2. The Web page content P20 of the Web page P1 acquired by the RPA processing unit 15 or the model generation device 4 in FIG. 1 is in the format of an HTML representation P21 as shown in FIG. 3(a). However, the RPA processing unit 15 and the model generation device 4 of this embodiment convert it to and from the format of a syntax tree representation P22 as shown in FIG. 3(b) as needed. Each node in the syntax tree representation P22 shown in FIG. 3(b) is described with a corresponding tag added in the HTML representation P21 shown in FIG. 3(a). Node numbers in the syntax tree representation P22 are assigned when converting the HTML representation P21 to the syntax tree representation P22. In this embodiment, the node numbers are assigned using a preorder traversal method. Hereinafter, a node with node number n will be referred to as node #n. Each node in the syntax tree representation P22 contains all the information about the portion of the html representation P21 that corresponds to that node, and it is possible to reconvert the syntax tree representation P22 to the html representation P21. For example, node #1 in the syntax tree representation P22 has information added to it that can be converted into " " at the corresponding portion in the html representation P21. During this conversion, the RPA processing unit 15 and the model generation device 4 also perform processing to facilitate the correspondence between the html representation P21 and the syntax tree representation P22 by embedding the node number as a comment attribute in the tag of each web element in the html representation P21.

[0038] 4 to 6, examples of changes in the representation of the Web page content shown in FIG. 3 and changes in the display of the Web browser when the Web page content is updated will be described.

[0039] FIG. 4 illustrates a representation of web page content before a web page update and a web browser display. FIG. 4(a) illustrates an HTML representation P21-1 of web page content P20 before the web page update, FIG. 4(b) illustrates a syntax tree representation P22-1 of the HTML representation P21-1 shown in FIG. 4(a), and FIG. 4(c) illustrates a display P23-1 of web page content P20 before the web page update in a web browser. The XPath of the operation target web element P31-1 in the HTML representation P21-1 in FIG. 4(a) is " / html / body / h1." In the syntax tree representation P22-1 in FIG. 4(b), the operation target web element P31-1 corresponds to the web element P32-1 at node #9. Furthermore, the operation target web elements P31-1 and P32-1 are displayed as an operation target web element P33-1 in the web browser display P23-1 shown in FIG. 4(c).

[0040] Figure 5 shows a representation of the web page content after the web page is updated and the display in the web browser. Figure 5(a) shows an html representation P21-2 after the web page update of the html representation P21-1 in Figure 4(a). Figure 5(b) shows a syntax tree representation P22-2 of the html representation P21-2 shown in Figure 5(a), and Figure 5(c) shows a display P23-2 of the web page content P20 in the web browser after the web page is updated.

[0041] Figure 6 shows a different representation of web page content after a web page update from Figure 5, and a display in a web browser. Figure 6(a) shows an html representation P21-3 after the web page update of the html representation P21-1 in Figure 4(a). Figure 6(b) shows a syntax tree representation P22-3 of the html representation P21-3 shown in Figure 6(a), and Figure 6(c) shows a display P23-3 of the web page content P20 in a web browser after the web page update.

[0042] The operation target Web element P31-1 in Fig. 4(a) is the operation target Web element P31-2 in the html representation P21-2 in Fig. 5(a), and the XPath has changed to " / html / body / ul / li[3] / h1". Also, the operation target Web element P31-1 in Fig. 4(a) is the operation target Web element P31-3 in the html representation P21-3 in Fig. 6(a), and the XPath has changed to " / html / body / div / h1".

[0043] In addition, in the syntax tree representation P22-1 shown in Figure 4(b), the web element P32-1 to be operated is node #9, but in the syntax tree representation P22-2 shown in Figure 5(b), the operation target has moved to web element P32-2 at node #10, and in the syntax tree representation P22-3 shown in Figure 6(b), the operation target has moved to web element P32-3 at node #10.

[0044] On the other hand, the web browser display P23-1 shown in Figure 4(c) is almost identical to the web browser display P23-2 shown in Figure 5(c) or the web browser display P23-3 shown in Figure 6(c), making it difficult for users to know that the web page content has been updated. Also, the XPath of the operation target web element P31-1 before the update was " / html / body / h1," and the XPaths of the operation target web elements P31-2 and P31-3 after the update were " / html / body / h1" and " / html / body / div / h1," respectively, but it is difficult to know this change using a simple algorithm.

[0045] The main purpose of this embodiment is to discover that the target web elements P31-1, P32-1, and P33-1 in each representation before the web page modification shown in Figure 4 have moved to the target web elements P31-2, P32-2, and P33-2 after the web page modification shown in Figure 5, or to the target web elements P31-3, P32-3, and P33-3 shown in Figure 6, and to obtain the XPath of the target web element P31-2 or the XPath of the target web element P31-3.

[0046] 7A and 7B are diagrams illustrating examples of analytical information used by the RPA processing unit 15 and the model generation device 4. Fig. 7A shows an example of analytical information A1, and Fig. 7B shows a syntax tree representation P22 of Web page content P20 from which analytical information A1 shown in Fig. 7A was obtained. The syntax tree representation P22 and the target Web element P32 shown in Fig. 7B correspond to the syntax tree representation P22-1 and the target Web element P32-1 shown in Fig. 4B.

[0047] 4 to 6, even if the change to the Web page content P20 is slight, the XPath of the Web elements changes irregularly, making it difficult to determine the XPath after the change. However, when the syntax tree representation P22-1 of the Web page content P20 shown in Fig. 4(b) is compared with the syntax tree representation P22-3 shown in Fig. 6(b), it is found that the syntax tree representation around the operation target Web element P32-1 and the syntax tree representation around the operation target Web element P32-3 are almost identical. Using this as a clue, it may be possible to determine that the operation target Web element P32-3 is a movement of the operation target Web element P32-1 before the update. However, when comparing the syntax tree representation P22-1 of the web page content P20 shown in Figure 4(b) with the syntax tree representation P22-2 shown in Figure 5(b), the syntax tree representation around the web element P32-1 to be operated and the syntax tree representation around the web element P32-2 to be operated are similar but not identical, and it is difficult to determine by a simple comparison process that the web element P32-2 is the moved version of the web element P32-1.

[0048] Therefore, in this embodiment, not only the target Web element but also information on its surroundings in the syntax tree is collected, and the similarity of the syntax trees is comprehensively determined using machine learning techniques. To achieve this, the syntax tree surrounding the target Web element must be converted into a format that facilitates machine learning. In this embodiment, this is called analytical information, and the RPA processing unit 15 creates analytical information A1 for the specified Web element based on the syntax tree representation P22 of the Web page content P20, as shown in FIG. 7(a). The Web element for which analytical information A1 is generated is referred to as the target Web element.

[0049] 7(a) includes information about the entire Web page P1, information about the target Web element, and information about surrounding Web elements. The surrounding Web elements are Web elements that are close to the target Web element on the syntax tree.

[0050] The web page information A11 is an example of information relating to the entire web page P1. The web page information A11 includes, for example, the URL of the web page P1 and the total number of nodes included in the syntax tree representation P22 of the web page content P20.

[0051] The target web element identifier A12 and the target web element peripheral information A13 are information about the target web element. The target web element identifier A12 is information that identifies the target web element and is indicated by XPath. The XPath is obtained from the web element identifier P40 of the web page P1. The target web element peripheral information A13 indicates the content surrounding the target web element in the web page P1. The target web element peripheral information A13 includes information about the node number, XPath, HTML tag name, and display coordinates of the target web element. The node number is obtained from the syntax tree representation P22, the XPath is obtained from the web element identifier P40 of the web page P1, the HTML tag name is obtained from the web element P30 of the web page P1, and the display coordinate information is obtained from the coordinates P50 of the web page P1.

[0052] The surrounding web element information indicates the content surrounding the surrounding web element on web page P1. The surrounding web elements are, for example, the parent, older brother, and younger brother web elements of the target web element. As shown in FIG. 7(b), when the target web element is node #9, the parent node is node #4, which is one level higher, and the older brother nodes are nodes #5 and #6, which have the same parent node as the target web element node and have a smaller node number than the target web element node number. Note that younger brother nodes are nodes that have the same parent node as the target web element node and have a larger node number than the target web element node number, but FIG. 7(b) does not show any younger brother nodes of the target web element. The web element corresponding to the parent node is the parent web element, the web element corresponding to the older brother node is the older brother web element, and the web element corresponding to the younger brother node is the younger brother web element.

[0053] 7(a) includes parent peripheral web element information A14, older brother 1 peripheral web element information A15, older brother 2 peripheral web element information A16, and younger brother peripheral web element information A17. Parent peripheral web element information A14 includes information on the node number, XPath, HTML tag name, and display coordinates of the parent web element, while older brother 1 peripheral web element information A15 and older brother 2 peripheral web element information A16 include information on the node number, XPath, HTML tag name, and display coordinates of the first older brother web element and the second older brother web element, respectively. Younger brother peripheral web element information A17 indicates that the younger brother web element does not exist on this web page, but if the younger brother web element does exist, the node number, XPath, HTML tag name, and display coordinates of the younger brother web element are set. Similar to the surrounding information of the target web element, the node number of the surrounding web element is obtained from the syntax tree representation P22, the XPath of the surrounding web element is obtained from the web element identifier P40 of the web page P1, the HTML tag name is obtained from the web element P30 of the web page, and the display coordinate information is obtained from the coordinate P50 of the web page P1.

[0054] FIG. 8 shows an example of scenario R1. Scenario R1 is composed of multiple records. Each record stores an execution order R11, an operation target R12, an operation R13, and parameters R14 for performing the operation. The operation target R12 is, for example, an application (APP), an RPA, or a web browser (WEB). By referencing the operation target R12, it is possible to uniquely determine whether the operation target is a web browser. This scenario R1 is linked to analytical information A1 contained in the analytical information DB of the analytical information storage unit 14. If the operation target R12 is a web browser and the description location of the operation target web element in the HTML syntax has moved since the scenario was created when the scenario was executed, analytical information A1 is used to estimate the XPath of the destination.

[0055] When the operation target R12 indicates a web browser (WEB), a reference point R15 of the web operation information is set in the operation R13. A web page identifier R16, an operation target web element identifier R17, and a web operation R18 are set in the web operation information in association with the reference point R15. The reference point R15 is a primary key. The web page identifier R16 is information that identifies the operation target web page P1 and is described by, for example, a URL. The operation target web element identifier R17 identifies the operation target web element among the web elements in the web page P1 identified by the web page identifier R16. The XPath of the operation target web element is set in the operation target web element identifier R17. The web operation R18 indicates an operation to be performed on the operation target web element identified by the web page identifier R16 and the operation target web element identifier R17. Note that the web page identifier R16, the operation target web element identifier R17, and the web operation R18 may be set in the operation R13 of the scenario R1.

[0056] FIG. 9 is a diagram illustrating an example of the configuration of the RPA processing unit 15, which is the core of this embodiment. The RPA processing unit 15 operates based on an input scenario file. The scenario file is a file that includes scenario R1. Whether the RPA processing unit 15 creates or executes a scenario file during operation depends on the input operation mode. The operation mode is input, for example, by the input unit 11. A web element learning model read from the model storage unit 17 is also input to the RPA processing unit 15. Furthermore, various setting values ​​are input to the RPA processing unit 15. Each setting value may be input by the input unit 11, or the RPA processing unit 15 may read these setting values ​​that are pre-stored in a setting value storage unit, a recording medium, or an external device (not shown). Each setting value will be described later.

[0057] The RPA processing unit 15 is configured to include a control unit 21, a scenario creation unit 22, a scenario execution unit 23, and a Web processing unit 24. The control unit 21 controls the entire RPA processing unit 15. The scenario creation unit 22 performs processing in a scenario creation mode. The scenario execution unit 23 performs processing in a scenario execution mode. The Web processing unit 24 performs Web-related processing.

[0058] Scenario creation unit 22 has a scenario editing unit 221. Scenario editing unit 221 creates or edits a scenario file according to information input by the user via input unit 11. When scenario editing unit 221 sets an operation on a target Web element in scenario R1 being edited, it instructs Web processing unit 24 to generate analytical information for the target Web element. Scenario editing unit 221 writes the scenario file in which edited scenario R1 is set into scenario storage unit 13, and writes the analytical information generated by Web processing unit 24 in the analytical information DB of analytical information storage unit 14 in association with the scenario file.

[0059] The scenario execution unit 23 includes an APP operation unit 231, an internal processing unit 232, a Web operation unit 233, an operation target Web element determination unit 234, and a two-class classification unit 235. When an operation target R12 indicating an external application is set in the record of scenario R1 in the input scenario file, the APP operation unit 231 processes the external application using the settings of the operation R13 and parameter R14 of that record. When an operation target R12 indicating an RPA is set in the record of scenario R1 in the input scenario file, the internal processing unit 232 processes the RPA using the settings of the operation R13 and parameter R14 of that record. When an operation target R12 indicating a web browser is set in the record of scenario R1 in the input scenario file, the Web operation unit 233 processes the web browser using the Web operation information of the reference point R15 indicated by the operation R13 of that record and the settings of the parameter R14.

[0060] The operation target web element determination unit 234 estimates and determines the destination XPath when the operation target web element is moved during scenario execution. The operation target web element determination unit 234 uses information about the web page being the target of scenario execution, as well as settings for the estimation mode, the analysis program to be sent to the web browser unit 16, the minimum probability for the operation target web element to be selected as a destination candidate, the display coordinate range for the operation target web element to be selected as a destination candidate, and the maximum number of destination candidates for the operation target web element. Note that an estimation mode value of "1" indicates that one destination web element is selected, while a value of 2 or greater indicates the number of destination web element candidates. The operation target web element determination unit 234 selects at least some of the web elements included in the updated web page P1 as destination candidates for the operation target web element and instructs the web processing unit 24 to generate analysis information for the destination candidate web elements to use in narrowing down the selected candidates.

[0061] The two-class classification unit 235 calculates the probability that the candidate destination of the operation target web element selected by the operation target web element determination unit 234 is the destination of the operation target web element using the input web element learning model, the analytical information of the operation target web element, and the analytical information of the candidate destination of the operation target web element. Based on the calculated probability, the operation target web element determination unit 234 selects the destination of the operation target web element from the candidate destination operation target web elements, or a candidate to be presented to the user to allow the user to select the destination of the operation target web element.

[0062] The Web processing unit 24 is called by both the scenario creation unit 22 and the scenario execution unit 23. The Web processing unit 24 includes a Web browser control unit 241, a Web identification determination unit 242, an HTML / syntax tree conversion unit 243, and an analytical information generation unit 244. The Web browser control unit 241 operates the Web browser and extended functions provided by the Web browser unit 16. The Web identification determination unit 242 determines whether a Web element with a specified Web element identifier is present in the Web page P1 read from the Web browser. The HTML / syntax tree conversion unit 243 bidirectionally converts the acquired Web page content P20 between an HTML representation P21 and a syntax tree representation P22. The analytical information generation unit 244 analyzes the Web page content P20 and generates analytical information. The analytical information generation unit 244 operates using information about the Web page for which a scenario is being created or executed, as well as a set value for the number of peripheral nodes for which peripheral Web element information is generated.

[0063] The following describes the processing of the operation automation device 1. Fig. 10 is a flowchart showing an example of scenario editing processing in the operation automation device 1. Error processing is omitted in Fig. 10. When the input operation mode is the scenario creation mode, the control unit 21 controls the RPA processing unit 15 to execute the editing processing shown in Fig. 10.

[0064] If the user inputs the creation of a new scenario (step S101: YES), the scenario editing unit 221 of the scenario creation unit 22 generates a new scenario file. On the other hand, if the user inputs information that identifies the scenario file to be edited (step S101: NO), the scenario editing unit 221 reads the scenario file to be edited from the scenario storage unit 13 (step S102).

[0065] When the user inputs editing details for the scenario using the input unit 11 (step S103), the scenario editing unit 221 edits the scenario R1 according to the input editing details (step S104). When the scenario editing unit 221 edits Web operation information for performing a Web operation on a target Web element (step S105), it notifies the Web processing unit 24 of the URL of the Web page indicated by the Web page identifier R16 of the Web operation information and the XPath indicated by the target Web element identifier R17, and instructs the Web processing unit 24 to generate analytical information. The Web processing unit 24 generates analytical information A1 based on the instruction (step S106).

[0066] The scenario editing unit 221 repeats the processing from step S103 (step S107) until the user inputs an end command via the input unit 11. When the user inputs an end command via the input unit 11, the scenario editing unit 221 saves a scenario file in which the edited scenario R1 is set in the scenario storage unit 13, and also saves the analysis information generated in step S106 in the analysis information DB of the analysis information storage unit 14 in association with the saved scenario file (step S108).

[0067] Fig. 11 is a flowchart showing an example of scenario execution processing in the operation automation device 1. Error processing is omitted in Fig. 11. When the input operation mode is the scenario execution mode, the control unit 21 controls the RPA processing unit 15 to execute the editing processing shown in Fig. 11.

[0068] The control unit 21 reads out the scenario file designated as the execution target from the scenario storage unit 13 and acquires the scenario R1 (step S201). The control unit 21 sets the initial value 1 to the variable m (step S202).

[0069] The control unit 21 reads a record in which the variable m is set in the execution order R11 from the scenario R1 (step S203). The record of the scenario R1 in which the variable m is set in the execution order R11 is described as record m. The control unit 21 determines whether or not the read record m exists (step S204). If the control unit 21 determines that the read record m exists (step S204: YES), it determines the operation target R12 of the record m (step S205).

[0070] If the control unit 21 determines that the operation target R12 is an application (APP), it instructs the APP operation unit 231 to execute the operation. The APP operation unit 231 refers to the record m and executes the operation on the application indicated by the operation R13 using the parameter value set in the parameter R14 (step S206).

[0071] In step S203, if the control unit 21 determines that the operation target R12 is an RPA, it instructs the internal processing unit 232 to execute the operation. The internal processing unit 232 refers to the record m and executes the operation indicated by the operation R13 using the parameter value set in the parameter R14 (step S207).

[0072] In step S203, if control unit 21 determines that operation target R12 is a web browser (WEB), it instructs execution to web operation unit 233. Web operation unit 233 reads out web operation information corresponding to reference point R15 indicated by operation R13 of record m. Web operation unit 233 determines whether the read out web operation information is a web operation for the operation target web element identified by XPath (step S208).

[0073] If Web operation unit 233 determines that the web operation is for the operation target web element (step S208: YES), it checks the XPath of the operation target web element (step S209). That is, Web operation unit 233 notifies operation target web element determination unit 234 of the web page identifier R16 and operation target web element identifier R17 in the web operation information as the URL of the operation target web page and the XPath of the operation target web element, and queries the XPath. If the operation target web element has not moved, operation target web element determination unit 234 notifies web operation unit 233 that the XPath has not changed. On the other hand, if the operation target web element has moved, operation target web element determination unit 234 notifies web operation unit 233 of the XPath of the destination of the operation target web element.

[0074] If Web operation unit 233 determines in step S208 that the Web operation is not for the operation target web element (step S208: NO), or after processing in step S209, it performs the Web operation in accordance with the Web operation information read in step S208 (step S210). However, if Web operation unit 233 receives the XPath of the destination of the operation target web element from operation target web element determination unit 234 in step S209, it performs the Web operation on the operation target web element identified by the destination XPath instead of the XPath described in operation target web element identifier R17.

[0075] When the processing of step S206, step S207, or step S210 ends, the control unit 21 adds 1 to the variable m (step S211). The control unit 21 repeats the processing from step S203. Note that in step S206, the APP operation unit 231 may set a predetermined value to the variable m in accordance with a predetermined condition, and execute the processing from step S203. Similarly, in step S207, the internal processing unit 232 may set a predetermined value to the variable m in accordance with a predetermined condition, and execute the processing from step S203. Similarly, in step S210, the Web operation unit 233 may set a predetermined value to the variable m in accordance with a predetermined condition, and execute the processing from step S203. Then, if the control unit 21 determines in step S204 that there is no record m that has been read (step S204: NO), the control unit 21 ends the processing of FIG. 11.

[0076] Fig. 12 is a flow diagram showing an example of the operation of operation target Web element determination unit 234 shown in Fig. 9. Operation target Web element determination unit 234 executes the process shown in Fig. 12 in step S209 of Fig. 11.

[0077] The operation target web element determination unit 234 receives the URL of the operation target web page, the XPath of the operation target web element, the web element learning model, the setting value estimation mode, the analysis program, the minimum probability for determining the operation target web element as a destination candidate, the display coordinate range for determining the operation target web element as a destination candidate, and the maximum number of destination candidates, and outputs the XPath of the destination of the operation target web element. This allows the web page P1 to be updated, and even if the operation target web element has been moved since the scenario was created, the operation target web element determination unit 234 receives the URL of the operation target web page and the XPath of the operation target web element output by the web operation unit 233 in step S208 of FIG. 11.

[0078] The operation target Web element determination unit 234 outputs the URL of the operation target Web page and the XPath of the operation target Web element to the Web processing unit 24, and confirms the existence of the XPath of the output operation target Web element (step S301). The Web browser control unit 241 acquires the Web page P1 of the URL received from the operation target Web element determination unit 234 using the Web browser provided by the Web browser unit 16. The Web identification determination unit 242 determines whether the Web element of the XPath received from the operation target Web element determination unit 234 exists in the Web page content P20 of the Web page P1 acquired by the Web browser control unit 241, and outputs the determination result to the operation target Web element determination unit 234.

[0079] If operation target Web element determination unit 234 receives a determination result from Web processing unit 24 that an operation target Web element exists (step S302: YES), it notifies Web operation unit 233 that there is no change in XPath and ends the process.

[0080] On the other hand, if the operation target Web element determination unit 234 receives a determination result from the Web processing unit 24 that the operation target Web element does not exist (step S302: NO), it references the analytical information DB stored in the analytical information storage unit 14 and checks whether analytical information exists for the Web element identified by the URL of the operation target Web page and the XPath of the operation target Web element (step S303). If the operation target Web element determination unit 234 determines that analytical information does not exist (step S304: NO), it outputs an error, such as displaying an error on the display unit 12 (step S305). For example, if analytical information does not exist for the operation target Web element because a scenario created by an RPA that does not have the functionality of this embodiment is being used, an error occurs.

[0081] If the target Web element determination unit 234 determines that analytical information exists (step S304: NO), it reads analytical information A1 of the target Web element confirmed in step S303 from the analytical information storage unit 14 and stores it in temporary memory (step S306).

[0082] The operation target Web element determination unit 234 uses the Web processing unit 24 to operate the Web browser and extended function, and acquires the HTML representation P21 and syntax tree representation P22 of the Web page content P20 included in the Web page P1 indicated by the URL of the operation target Web page. That is, the operation target Web element determination unit 234 outputs the URL of the operation target Web page to the Web processing unit 24. The Web browser control unit 241 of the Web processing unit 24 acquires the Web page P1 indicated by the received URL using the Web browser provided by the Web browser unit 16. The HTML / syntax tree conversion unit 243 acquires the HTML representation P21 of the Web page content P20 from the Web page P1 acquired by the Web browser control unit 241 and converts it into the syntax tree representation P22. The HTML / syntax tree conversion unit 243 embeds the node number in the syntax tree representation P22 in the HTML representation P21. The operation target Web element determination unit 234 temporarily stores the HTML representation P21 and the syntax tree representation P22 created by the HTML / syntax tree conversion unit 243 in memory (step S307).

[0083] Operation target Web element determination unit 234 notifies Web processing unit 24 of HTML representation P21 of Web page content P20 and instructs it to analyze. Web browser control unit 241 of Web processing unit 24 sends HTML representation P21 of Web page content P20 and an analysis program to Web browser unit 16 and instructs it to start analyzing Web page content P20 (step S308).

[0084] The web browser unit 16 executes the analysis program using an extended function of the web browser, selects destination candidate web elements from the web page content P20 described in the HTML representation P21, and temporarily stores a list of the selected destination candidate web elements (step S309). The candidate selection criteria are as follows: 1) the web element has the same tag as the target web element; 2) the display coordinates on the browser are within the range of the display coordinates of the target web element for the destination candidate. The tag and display coordinates of the target web element are read from the analytical information A1 temporarily stored in step S306 by the target web element determination unit 234, the web browser control unit 241, or the web browser unit 16. The web browser unit 16 associates the display coordinates and XPath information of the selected destination candidate web element read from the HTML representation P21 of the web page content P20 with the web element information, and stores the web element information.

[0085] Figure 13 is a flow diagram continuing from Figure 12. The operation target web element determination unit 234 sorts the list of destination candidate web elements selected in step S309 in order of distance from the display coordinates of the operation target web element in descending order (step S310). If the display coordinates are at the same distance, the operation target web element determination unit 234 sorts them based on the distance from the operation target web element in the syntax tree representation.

[0086] The operation target Web element determination unit 234 extracts the highest-ranked destination candidate Web element from the list of destination candidate Web elements that have not yet been selected, and then notifies the Web processing unit 24 of the URL of the operation target web page and the XPath of the destination candidate Web element, and instructs the Web processing unit 24 to generate analytical information for the extracted destination candidate Web element. The operation target Web element determination unit 234 temporarily stores analytical information A1 created by the analytical information generation unit 244 (step S311).

[0087] The operation target Web element determination unit 234 inputs analytical information A1 of the operation target Web element and analytical information A1 of the destination candidate Web element to the two-class classification unit 235 and instructs it to calculate the probability that the destination candidate is the destination of the operation target Web element. The two-class classification unit 235 reads the Web element learning model from the model storage unit 17, inputs analytical information A1 of the operation target Web element and analytical information A1 of the destination candidate Web element to the Web element learning model, and returns the output probability to the operation target Web element determination unit 234 (step S312). If the probability is less than the minimum value for determining the destination candidate, the operation target Web element determination unit 234 rejects the currently selected destination candidate; if the probability is equal to or greater than the minimum value, the operation target Web element determination unit 234 temporarily stores the node number of the currently selected destination candidate and the calculated probability in a candidate list (step S313).

[0088] The operation target Web element determination unit 234 makes an end determination (step S314). The end conditions are met when one of the following is satisfied. The first end condition is when the number of destination candidates selected in step S313 reaches the maximum number of destination candidates. The second end condition is when all destination candidates included in the list of destination candidate Web elements sorted in step S310 have been selected.

[0089] If the operation target Web element determination unit 234 determines that the termination condition is not met (step S314: NO), it repeats the process from step S311. If the operation target Web element determination unit 234 determines that the termination condition is met (step S314: YES), it determines whether the candidate destination selected in step S312 is included in the candidate list (step S315). If the operation target Web element determination unit 234 determines that the candidate destination is not included in the candidate list (step S315: NO), it outputs an error, such as displaying an error on the display unit 12 (step S316).

[0090] If the operation target web element determination unit 234 determines that there are destination candidate web elements in the candidate list (step S315: YES), it sorts the destination candidate web elements in descending order of the probability calculated in step S312. If the maximum number of destination candidate settings is 1, the operation target web element determination unit 234 selects the destination candidate web element with the highest probability as the destination web element. On the other hand, if the maximum number of destination candidate settings is 2 or greater, the operation target web element determination unit 234 displays the destination candidate web elements, as many as specified by the maximum value, starting with the one with the highest probability, on the display unit 12. The user inputs information about a destination web element selected from the displayed destination candidate web elements using the input unit 11 (step S317). If there is no suitable web element, the user inputs "No selection" using the input unit 11.

[0091] Operation target Web element determination unit 234 determines whether "no selection" has been input (step S318). If operation target Web element determination unit 234 determines that "no selection" has been input (step S318: YES), it outputs an error, such as by displaying an error on display unit 12, and terminates processing (step S316). If operation target Web element determination unit 234 determines that "no selection" has not been input (step S318: NO), it reads the XPath of the Web element selected in step S317 from analytical information A1 generated in step S311. Operation target Web element determination unit 234 outputs the read XPath to Web operation unit 233 as the XPath of the destination of the operation target Web element (step S319).

[0092] FIG. 14 is a flow diagram showing an example of the operation of analytical information generation unit 244 shown in FIG. 9. In scenario creation mode, analytical information generation unit 244 is invoked when scenario editing unit 221 describes an operation targeting a web element in a scenario. Analytical information generation unit 244 generates analytical information A1 related to the operation target web element described in the scenario. The generated analytical information A1 is associated with the scenario and stored in the analytical information DB stored in analytical information storage unit 14. In scenario execution mode, analytical information generation unit 244 is invoked to detect the destination of the web element when operation target web element determination unit 234 determines that the XPath of the operation target web element described in scenario R1 does not exist in the operation target web page. Operation target web element determination unit 234 generates analytical information A1 related to web elements that are candidate destinations for the operation target web element. The analytical information generation unit 244 inputs the URL of the target web page, the XPath of the target web element, the syntax tree representation of the web page content, the HTML representation of the web page content with embedded node numbers, and the number of surrounding nodes of the set value, and outputs analytical information A1.

[0093] The Web processing unit 24 receives the URL of the Web page and the XPath of the Web element to be operated from the scenario editing unit 221 in step S106 of Fig. 10. Alternatively, the Web processing unit 24 receives the URL of the Web page and the XPath of the Web element of the destination candidate from the Web processing unit 24 in step S311 of Fig. 13. The Web browser control unit 241 reads out the Web page P1 corresponding to the received URL using the Web browser provided by the Web browser unit 16. The HTML / syntax tree conversion unit 243 converts the HTML representation P21 of the acquired Web page content P20 into a syntax tree representation P22 and embeds a node number in the HTML representation P21.

[0094] The analytical information generation unit 244 inputs, as the URL of the target web page and the XPath of the target web element, either the URL of the web page and the XPath of the target web element output from the scenario editing unit 221, or the URL of the web page and the XPath of the destination candidate web element output from the web processing unit 24. The analytical information generation unit 244 also inputs the syntax tree representation P22 and the HTML representation P21 generated by the HTML / syntax tree conversion unit 243. The analytical information generation unit 244 also reads the number of neighboring nodes, which is a set value.

[0095] The analytical information generator 244 scans the entire syntax tree representation P22 of the Web page content P20 to determine the total number of nodes, and stores this together with the input URL of the Web page as Web page information A11 (Step S401). Furthermore, the analytical information generator 244 stores the XPath of the input target Web element as a target Web element identifier A12 (Step S402).

[0096] The analytical information generator 244 uses the input XPath of the target web element to identify the target node, which is the node of the target web element in the syntax tree representation P22 of Web page content P20. The analytical information generator 244 extracts information about the target node and saves it as target web element peripheral information A13 (step S403). Specifically, the analytical information generator 244 references the syntax tree representation P22 of Web page content P20, obtains a node number from the target node corresponding to the target web element, and saves the obtained node number and the XPath of the target web element as target web element peripheral information A13.

[0097] The analytical information generator 244 inputs the node number acquired in step S403 and the HTML representation P21 of the input Web page content P20 to the Web browser control unit 241. The Web browser control unit 241 acquires and returns the HTML tag name and display coordinates of the Web element identified by the input node number from the input HTML representation P21. The analytical information generator 244 adds the HTML tag name and coordinates of the target Web element returned from the Web browser control unit 241 to the target Web element peripheral information A13 and saves them (step S404).

[0098] The analytical information generator 244 references the syntax tree representation P22 of the Web page content P20 and acquires the node number from the parent node of the target node. The analytical information generator 244 inputs the acquired node number of the parent node and the HTML representation P21 of the Web page content P20 to the Web browser control unit 241. The Web browser control unit 241 acquires and returns the XPath, HTML tag name, and coordinates of the parent Web element identified by the input node number from the input HTML representation P21. The analytical information generator 244 saves parent-related Web element information A14, which sets the node number of the parent node and the XPath, HTML tag name, and coordinates of the parent Web element returned from the Web browser control unit 241 (step S405).

[0099] The analytical information generator 244 references the syntax tree representation P22 of the Web page content P20 and acquires the node number from the older node of the target node. For each acquired older node, the analytical information generator 244 performs the same process as for the parent node in step S405 and saves the node number of the older node and the older node's surrounding Web element information A15, A16, which sets the XPath, html tag name, and coordinates of the older node's Web element (step S406).

[0100] The analytical information generator 244 references the syntax tree representation P22 of the Web page content P20 and acquires the node numbers from the younger nodes of the target node. For each acquired younger node, the analytical information generator 244 performs the same process as for the parent node in step S405 and saves the younger brother's surrounding Web element information A17, which sets the younger brother node's node number and the younger brother's Web element's XPath, html tag name, and coordinates (step S407).

[0101] The analytical information generation unit 244 outputs analytical information A1, which includes the saved web page information A11, target web element identifier A12, target web element peripheral information A13, parent peripheral web element information A14, older brother peripheral web element information A15, A16, and younger brother peripheral web element information A17, to the calling scenario editing unit 221 or the operation target web element determination unit 234.

[0102] FIG. 15 is a diagram illustrating an example of the configuration of the model generation device 4 shown in FIG. 1. The model generation device 4 includes a control unit 41, a pseudo-teacher data generation unit 42, a normal teacher data generation unit 43, a learning device 44, a web browser control unit 45, an HTML / syntax tree conversion unit 46, an analytical information generation unit 47, a web browser unit 48, a positive example storage unit 51, a negative example storage unit 52, and an analytical information storage unit 53. The model generation device 4 receives the URL of a web page, the URL of the web page that was actually updated, the XPath of the web element to be operated before the web page was moved due to the update, the XPath of the web element to be operated after the web page was moved due to the update, and various setting values, and outputs a web element learning model. The input data and each setting value may be input via an input unit (not shown), or the model generation device 4 may read them from a storage unit, a recording medium, or an external device (not shown). The setting values ​​will be described later.

[0103] The model generation device 4 operates independently of the RPA processing unit 15 of the operation automation device 1 and generates a web element learning model. The web element learning model is used when the operation target web element determination unit 234 of the operation automation device 1 uses the two-class classification unit 235 to calculate the probability that a destination candidate web element is the actual destination in scenario execution mode. The generation of the web element learning model is carried out using the model generation device 4 over a sufficient amount of time during the development of the RPA executed by the RPA processing unit 15 of the operation automation device 1, and it is assumed that the generated web element learning model will be distributed simultaneously with the developed RPA program. The model generation device 4 generates learning data by analyzing actually updated web pages. However, since it is expected that sufficient learning data will not be obtained from this alone, the model generation device 4 also has a function to generate more learning data by pseudo-generating updated web pages from unupdated web pages.

[0104] The control unit 41 controls the entire model generation device 4. The pseudo teacher data generation unit 42 adds, moves, or deletes web elements to a web page with a predetermined probability to generate a pseudo updated web page. The pseudo teacher data generation unit 42 stores a set of analytical information that correctly reflects the source web element and the destination web element in the updated web page in a positive example database in the positive example storage unit 51, and stores a set of analytical information that incorrectly reflects the source web element and the destination web element in the updated web page in a negative example database in the negative example storage unit 52. The normal teacher data generation unit 43 stores a set of analytical information that reflects the source web element and the destination web element in the actual updated web page in the positive example database in the positive example storage unit 51. The learner 44 trains a web element learning model using the set of analytical information stored in the positive example database in the positive example storage unit 51 and the set of analytical information stored in the negative example database in the negative example storage unit 52.

[0105] The web browser control unit 45, the HTML / syntax tree conversion unit 46, the analytical information generation unit 47, the web browser unit 48, and the analytical information storage unit 53 have the same functions as the web browser control unit 241, the HTML / syntax tree conversion unit 243, the analytical information generation unit 244, the web browser unit 16, and the analytical information storage unit 14 of the operation automation device 1, respectively.

[0106] FIG. 16 is a flow diagram illustrating an example of the operation of pseudo-teacher data generation unit 42 shown in FIG. 15. Pseudo-teacher data generation unit 42 receives the URL of a web page, the number of trials for a set value, the probability of selecting a node to be operated, the type of node to be deleted, the probability of adding an A node, the probability of adding a div node, the probability of adding a p node, the probability of deleting a node, and the probability of selecting a negative example node. Pseudo-teacher data generation unit 42 generates a set of analytical information that correctly reflects the source and destination of the move and stores it in positive example storage unit 51. It also generates a set of analytical information that incorrectly reflects the source and destination of the move and stores it in negative example storage unit 52. Furthermore, pseudo-teacher data generation unit 42 stores analytical information generated using the XPath before the move in the analytical information DB of analytical information storage unit 53 for use by regular teacher data generation unit 43, as described below.

[0107] The pseudo teacher data generation unit 42 inputs the URL of the input web page to the web browser control unit 45. The web browser control unit 45 acquires the web page P1 indicated by the input web page URL using a web browser provided by the web browser unit 48 (step S501). The pseudo teacher data generation unit 42 inputs the html representation P21 of the web page content P20 of the web page P1 acquired in step S501 to the HTML / syntax tree conversion unit 46. The HTML / syntax tree conversion unit 46 converts the input html representation P21 into a syntax tree representation P22. The HTML / syntax tree conversion unit 46 assigns a node number to the syntax tree representation P22 and embeds the assigned node number in the html representation P21 (step S502).

[0108] The pseudo teacher data generation unit 42 follows the nodes of the syntax tree representation P22, selects an operation target node according to the operation target node selection probability, and then counts up the number of trials (step S503). If an operation target node is not selected in step S503 (step S504: NO), the pseudo teacher data generation unit 42 repeats the process from step S503.

[0109] If an operation target node is selected in step S503 (step S504: YES), and if the node type of the selected operation target node is a deletion target node type, the pseudo teacher data generation unit 42 deletes the selected operation target node from the syntax tree representation P22 of the Web page content P20 in accordance with the node deletion probability (step S505). At that time, if the selected operation target node has child or grandchild nodes, the pseudo teacher data generation unit 42 also deletes these nodes from the syntax tree representation P22 of the Web page content P20 at the same time. If the operation target node is deleted in step S505 (step S506: YES), the pseudo teacher data generation unit 42 repeats the processing from step S503.

[0110] If no node is deleted in step S503 (step S506: NO), the pseudo teacher data generation unit 42 adds an A node to the syntax tree representation P22 of the Web page content P20 according to the A node addition probability (step S507). The A node is a Web element using an a tag. If an A node is added in step S507 (step S508: YES), the pseudo teacher data generation unit 42 repeats the process from step S503.

[0111] If an A node is not added in step S507 (step S508: NO), the pseudo-teacher data generation unit 42 adds a div node to the syntax tree representation P22 of the Web page content P20 according to the div node addition probability (step S509). A div node is a Web element that uses a div tag. If a div node is added in step S509 (step S510: YES), the pseudo-teacher data generation unit 42 repeats the process from step S503.

[0112] If a div node is not added in step S509 (step S510: NO), the pseudo-teacher data generation unit 42 adds a p node to the syntax tree representation P22 of the Web page content P20 according to the p node addition probability (step S511). The p node is a Web element using a p tag.

[0113] The pseudo teacher data generation unit 42 determines whether the number of trials counted up in step S503 has reached the input set value of the number of trials (step S512). If the pseudo teacher data generation unit 42 determines that the number of trials has not been reached (step S512: NO), it repeats the processing from step S503. If the pseudo teacher data generation unit 42 determines that the number of trials has been reached (step S512: YES), it performs the processing shown in FIG.

[0114] Fig. 17 is a flow diagram showing a continuation of the process in Fig. 16. The pseudo-teacher data generation unit 42 re-assigns node numbers to the syntax tree representation P22 after the processes in steps S503 to S511 have been performed. The pseudo-teacher data generation unit 42 saves, as moved nodes, nodes whose node numbers have changed from before the processes in steps S503 to S511 have been performed (step S521).

[0115] The pseudo teacher data generation unit 42 inputs the updated syntax tree representation P22 to the HTML / syntax tree conversion unit 46. The HTML / syntax tree conversion unit 46 converts the input syntax tree representation into an HTML representation P21 and embeds node numbers. The HTML / syntax tree conversion unit 46 returns the HTML representation P21 of the updated Web page content P20 to the pseudo teacher data generation unit 42 (step S522).

[0116] The pseudo teacher data generation unit 42 outputs the HTML representation P21 of the Web page content P20 before the update, the HTML representation P21 of the Web page content P20 after the update, and the node number of the mobile node before the move and the node number after the move to the Web browser control unit 45. Using the Web browser provided by the Web browser unit 48, the Web browser control unit 45 obtains the XPath of the mobile node before the move from the HTML representation P21 of the Web page content P20 before the update, and the XPath of the mobile node before the move from the HTML representation P21 of the Web page content P20 after the update, and outputs them to the pseudo teacher data generation unit 42 (step S523).

[0117] Pseudo teacher data generation unit 42 instructs analytical information generation unit 47 to generate analytical information using the XPath of the mobile node before movement. That is, pseudo teacher data generation unit 42 outputs the URL of the input web page, the XPath of the mobile node before movement, and the syntax tree representation P22 and HTML representation P21 of the web page content P20 before movement to analytical information generation unit 47. Analytical information generation unit 47 uses this information as input and performs processing similar to that shown in FIG. 14 to generate analytical information A1. Analytical information generation unit 47 outputs the generated analytical information A1 to pseudo teacher data generation unit 42. Pseudo teacher data generation unit 42 stores the generated analytical information A1 in analytical information storage unit 53 (step S524). The generated analytical information A1 is used in processing by regular teacher data generation unit 43, which will be described later.

[0118] Pseudo teacher data generation unit 42 instructs analytical information generation unit 47 to generate analytical information using the XPath of the mobile node after it has moved. That is, pseudo teacher data generation unit 42 outputs the URL of the input web page, the XPath of the mobile node after it has moved, and the syntax tree representation P22 and HTML representation P21 of the web page content P20 after it has moved to analytical information generation unit 47. Analytical information generation unit 47 uses this information as input and performs processing similar to that shown in FIG. 14 to generate analytical information A1. Analytical information generation unit 47 outputs the generated analytical information A1 after the web page update to pseudo teacher data generation unit 42 (step S525).

[0119] The pseudo-teaching data generation unit 42 sets the analysis information A1 before the mobile node moves, obtained in step S514, and the analysis information A1 after the mobile node moves, obtained in step S515, and stores them as positive example teaching data in the positive example database of the positive example storage unit 51 (step S526).

[0120] The pseudo-teacher data generation unit 42 selects nodes that are not mobile nodes from the syntax tree representation P22 of the updated Web page content P20 in accordance with the input negative example node selection probability. The pseudo-teacher data generation unit 42 generates analytical information A1 for the selected nodes using a process similar to that used to generate analytical information for mobile nodes (step S527).

[0121] The pseudo teacher data generation unit 42 sets the analysis information A1 before the movement of the mobile node obtained in step S514 and the analysis information A1 generated in step S517 as a set, and stores them as negative example teacher data in the negative example database of the negative example storage unit 52 (step S528).

[0122] FIG. 18 is a flow diagram illustrating an example of the operation of the regular teacher data generation unit 43 shown in FIG. 15. The regular teacher data generation unit 43 receives the URL of the updated web page, the XPath of the target web element before the web page update, and the XPath of the target web element after the web page update, and outputs multiple sets of analytical information that correctly reflect the source and destination of the movement. The inputs to the regular teacher data generation unit 43 must be prepared separately from the input unit described in this embodiment. These inputs may be entered by a user via an input unit (not shown), read from a storage unit or recording medium (not shown), or received or read from another information processing device. The regular teacher data generation unit 43 also assumes that the analytical information of the web elements of the updated web page before the update has already been generated by the pseudo teacher data generation unit 42.

[0123] The normal teacher data generation unit 43 inputs the URL of the web page that has actually been updated to the web browser control unit 45. The web browser control unit 45 acquires the web page P1 indicated by the web page URL using the web browser provided by the web browser unit 48 (step S601). The normal teacher data generation unit 43 inputs the html representation P21 of the web page content P20 of the web page P1 acquired in step S601 to the HTML / syntax tree conversion unit 46. The HTML / syntax tree conversion unit 46 converts the input html representation P21 into a syntax tree representation P22 (step S602). The HTML / syntax tree conversion unit 46 embeds the node number in the syntax tree representation P22 into the html representation P21.

[0124] Normal teacher data generation unit 43 outputs the URL of the actually updated Web page, the XPath of the updated target Web element, syntax tree representation P22 of Web page content P20 generated in step S602, and syntax tree representation P22 of Web page content P20 into which the node number has been embedded in step S602 to analytical information generation unit 47. Analytical information generation unit 47 uses this information as input and performs processing similar to that shown in FIG. 14 to generate analytical information A1. Analytical information generation unit 47 outputs the generated analytical information A1 of the target Web element after the web page update to normal teacher data generation unit 43 (step S603).

[0125] Normal training data generation unit 43 searches the analytical information DB stored in analytical information storage unit 53 using the XPath of the pre-update target Web element as a key, and reads out analytical information A1 before the Web page update (step S604). Normal training data generation unit 43 associates analytical information A1 before the Web page update read in step S604 with analytical information A1 after the Web page update generated in step S603, and stores the resulting set in the positive case database of positive case storage unit 51 as positive training data.

[0126] 16 to 18, the learning device 44 uses the training data of positive examples stored in the positive example database of the positive example storage unit 51 and the training data of negative examples stored in the negative example database of the negative example storage unit 52 to input analytical information for each of the two web elements and learn a web element learning model that outputs the probability that one web element is the destination of the other web element. Specific examples of such learning processing include supervised learning for classification, such as a support vector machine, a random forest, or a neural network. The learning device 44 outputs the learned web element learning model. The operation automation device 1 stores the web element learning model output by the learning device 44 in the model storage unit 17. Furthermore, the operation automation device 1 may acquire and store a program for implementing the RPA 15 together with the web element learning model.

[0127] When a web element to be operated in an RPA scenario is specified by XPath, the logical structure of the web page may change, causing the XPath of the web element to be operated to change even though the appearance of the web page remains almost unchanged. In such cases, the operation automation device of this embodiment can automatically obtain a new XPath from the web page and continue the operation.

[0128] The functions of the operation automation device 1 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions of the RPA processing unit 15 and the web browser unit 16 may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. The program may also be provided over a network. Note that the term "computer system" herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include media that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system serving as a server or client. The program may also be designed to implement some of the above-described functions, or may be capable of implementing the above-described functions in combination with programs already stored in the computer system.

[0129] The operation automation device 1 may be realized by multiple computer devices connected to a network. In this case, it is possible to arbitrarily determine which of the multiple computer devices each functional unit of the operation automation device 1 is realized by. Furthermore, the same functional unit may be realized by multiple computer devices. For example, one or more of the scenario storage unit 13, the analytical information storage unit 14, and the model storage unit 17 may be realized by a computer device different from the computer devices on which the RPA processing unit 15 and the Web browser unit 16 of the operation automation device 1 are realized. Furthermore, the model storage unit 17 may be provided in a model storage device connected to the operation automation device 1. In this case, one model storage device may be connected to multiple operation automation devices 1.

[0130] Similarly, the model generation device 4 may be realized by a plurality of computer devices connected to a network. In this case, it is possible to arbitrarily select which of the plurality of computer devices each functional unit of the model generation device 4 is realized by. Furthermore, the same functional unit may be realized by a plurality of computer devices.

[0131] Furthermore, although operation automation device 1 and model generation device 4 are described above as separate devices, they may be integrated into one device. In this case, web browser control unit 241 may have the functionality of web browser control unit 45, HTML / syntax tree conversion unit 243 may have the functionality of HTML / syntax tree conversion unit 46, analytical information generation unit 244 may have the functionality of analytical information generation unit 47, and web browser unit 16 may also function as web browser unit 48.

[0132] FIG. 19 is a device configuration diagram showing an example of the hardware configuration of the operation automation device 1. The operation automation device 1 includes a processor 91, a storage unit 92, a communication interface 93, and a user interface 94. The processor 91 is a central processing unit that performs calculations and control. The processor 91 is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The processor 91 realizes the functions of the RPA processing unit 15 and the web browser unit 16 by reading and executing programs from the storage unit 92. The storage unit 92 realizes the scenario storage unit 13, the analytical information storage unit 14, and the model storage unit 17. The storage unit 92 further includes a work area when the processor 91 executes various programs. As described above, the storage unit 92 stores the program for realizing the RPA 15 and the web element learning model. The communication interface 93 is communicatively connected to other devices. The user interface 94 is an input device such as a keyboard, a pointing device (mouse, tablet, etc.), a button, or a touch panel, or a display device such as a display. A human operation is input through the user interface 94. The user interface 94 realizes the input unit 11 and the display unit 12.

[0133] 19. In this case, processor 91 reads and executes programs from storage unit 92, thereby implementing the functions of control unit 41, pseudo-teacher data generation unit 42, normal teacher data generation unit 43, learning device 44, web browser control unit 45, HTML / syntax tree conversion unit 46, analytical information generation unit 47, and web browser unit 48. Storage unit 92 implements positive example storage unit 51, negative example storage unit 52, and analytical information storage unit 53.

[0134] All or part of the functions of the operation automation device 1 and the model generation device 4 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array).

[0135] According to the embodiment described above, the operation automation system 10 can continue the automatic operation by estimating the changed web element even if the web page to be operated is changed after the automatic operation scenario is created. Furthermore, if the web page to be operated is changed after the automatic operation scenario is created and the web element to be operated cannot be found, the operation automation system 10 can continue the automatic operation at a minimum cost by suggesting an alternative web element. Furthermore, if the web page to be operated is changed after the automatic operation scenario is created, the operation automation system 10 can reduce the scenario modification cost by suggesting an alternative web element.

[0136] According to the above-described embodiment, the operation automation system has the function of operating a Web application directly or using a Web browser, which is an external system. The operation automation system also stores a Web page identifier and a Web element identifier for identifying a Web element to be operated, and the operation content for the Web element to be operated, as a scenario, and executes the scenario to automate the operation of the Web application. In addition to the Web page identifier and the Web element identifier, the operation automation system also stores auxiliary information for identifying the operation target. When executing a scenario, if the operation target indicated by the Web page identifier and the Web element identifier described in the scenario does not exist, the operation automation system analyzes the content of the Web page using the auxiliary information, creates new operation target candidates, and creates an indicator indicating the likelihood that the candidate is actually the operation target. The operation automation system then continues executing the scenario with a Web element selected by the user from the candidates as the new operation target. Alternatively, the operation automation system continues executing the scenario with a Web element selected from the candidates as the new operation target.

[0137] The operation automation system may use analytical information and a web element learning model as auxiliary information for identifying the operation target. The analytical information includes content surrounding the operation target web element on the web page at the time of scenario creation and is information summarizing information about the web page to be operated. The web element learning model is extracted from the entire web page. When the operation automation system executes a scenario, there is no operation target web element identified by the web page identifier and web element identifier described in the scenario. Therefore, when creating new operation target web element candidates, the operation automation system generates analytical information equivalent to that at the time of scenario creation for each web element included in the web page at the time of scenario execution, and calculates an index of the likelihood that each of these web elements is the operation target at the time of scenario creation that has moved due to a web page update, using the analytical information of the target web element at the time of scenario creation, the generated analytical information of each web element, and the web element learning model.

[0138] The analysis information may include information on the web page on which the operation target web element is located, a web element identifier of the operation target web element, information extracted from the content of web pages surrounding the operation target web element, and information extracted from the web element identifiers of web elements surrounding the operation target web element and the content of web pages surrounding the web elements. Furthermore, the operation automation system may select a predetermined number of web elements that are close to the operation target web element on the syntax tree when the content of the web page is represented by a syntax tree, and set them as the web elements surrounding the operation target web element.

[0139] The operation automation system may select, as a candidate for a new operation target, a web element that is close in coordinate position on the display screen or on the syntax tree of the web page when the scenario is executed to a web element to be operated in the scenario being executed, thereby reducing the processing time for generating candidates and improving accuracy.

[0140] Furthermore, the operation automation system may prepare a plurality of pairs of the content of a web page at a certain point in time and web elements added, deleted, or moved within the page due to subsequent updates of the web page, and may train the web element learning model using the prepared pairs.The operation automation system may also generate a pseudo-updated web page by adding, deleting, or moving predetermined types of nodes with a predetermined probability to a syntax tree generated from the content of the web page, and may train the web element learning model further using pairs of the original web page and the pseudo-updated web page.

[0141] The operation automation system may store analytical information, which is part of auxiliary information for identifying an operation target, in association with a scenario, and store the web element learning model together with the computer program itself.

[0142] Although the embodiments of the present invention have been described above with reference to the drawings, it is clear that the above embodiments are merely examples of the present invention and that the present invention is not limited to the above embodiments. Therefore, additions, omissions, substitutions, and other modifications of components may be made without departing from the technical spirit and scope of the present invention. [Explanation of symbols]

[0143] 1…Operation automation device 4...Model generation device 7. Network 8...Web service provider 10...Automated operation system 11...Input section 12...Display section 13...Scenario memory section 14…Analysis information storage unit 15...RPA processing unit 16...Web browser section 17...Model memory section 21...Control unit 22...Scenario Creation Department 23...Scenario Execution Department 24...Web processing unit 41...Control unit 42...Pseudo teacher data generation unit 43...Regular teacher data generation unit 44...Learning device 45...Web browser control unit 46...Syntax tree conversion unit 47…Analysis information generation unit 48…Web browser section 51...Positive example memory section 52...Negative example storage unit 53…Analysis information storage unit 91...Processor 92...Storage section 93...Communication interface 94...User Interface 221...Scenario Editorial Department 231...APP operation section 232...Internal processing unit 233…Web operation section 234...Operation target web element determination unit 235...2-class classification section 241...Web browser control unit 242...Web identification and judgment unit 243...Syntax tree conversion unit 244…Analysis information generation unit

Claims

1. a storage unit that stores a scenario including a web page identifier that identifies a web page, an operation target web element identifier that identifies an operation target web element within the web page, and a web operation on the operation target web element, and that stores analysis information that includes information regarding a description of the operation target web element and information regarding descriptions of surrounding web elements that are web elements surrounding the operation target web element, obtained based on a pre-update web page that is the web page at the time of creating the scenario, as auxiliary information for identifying the operation target on the web page; a scenario execution unit that loads a web page of the web page identifier included in the scenario using a web browser, performs a determination process to determine whether the operation target web element identifier is present in the loaded web page, and if it is determined by the determination process that the operation target web element identifier is present, performs the web operation on the operation target web element on the web element specified by the operation target web element identifier, and if it is determined by the determination process that the operation target web element identifier is not present, performs an analysis process to select a new operation target web element from the web elements included in the web page by analyzing the content of the loaded web page using the auxiliary information, and performs the web operation on the operation target web element on the new operation target web element selected by the analysis process; An automated operation system comprising:

2. In the analysis process, the scenario execution unit sets at least some web elements included in the updated web page, which is the web page at the time of execution of the scenario, as candidates for a new operation target, generates analysis information for each candidate, including information on the description of the candidate web element and information on the descriptions of web elements surrounding the candidate web element, obtained based on the updated web page, calculates an index indicating the possibility that the candidate is a new operation target using the generated analysis information of the candidate and the analysis information of the operation target web element read from the storage unit, narrows down the candidates based on the calculated index, and sets the candidate web element narrowed down to one by the analysis process, or a web element selected by the user from the multiple candidate web elements narrowed down by the analysis process, as the new operation target web element. The operation automation system according to claim 1 .

3. the scenario execution unit acquires tag or display position information of the web element to be operated from the analysis information of the web element to be operated, and selects, as the candidate, a web element from among the web elements included in the updated web page whose tag is the same as the tag of the web element to be operated or whose display position is within a predetermined range from the display position of the web element to be operated; The operation automation system according to claim 2 .

4. The analysis information includes identification information of the web page, an identifier of the operation target web element, content surrounding the operation target web element in the pre-update web page, a surrounding web element identifier that identifies the surrounding web element, and content surrounding the surrounding web element in the pre-update web page. The operation automation system according to claim 1 .

5. The neighboring web elements are other web elements that are close to the operation target web element in a syntax tree of web elements included in the web page. The operation automation system according to claim 1 .

6. The auxiliary information further includes a model for calculating the index using analytical information of the candidate and analytical information of the web element to be operated. The operation automation system according to claim 2 .

7. The system further includes a learning unit that learns the model using a plurality of sets of training data, each set being a set of analytical information on a web element to be operated that is obtained based on a learning web page before the update and analytical information on a new web element to be operated that is obtained based on the updated learning web page. The operation automation system according to claim 6.

8. the learning unit further uses training data of a set of analytical information of the web elements to be operated, obtained based on the learning web page, and analytical information of the web elements to be newly operated or the web elements not to be newly operated, obtained based on a web page updated by adding, moving, or deleting web elements from the learning web page at a predetermined probability, to learn the model; The operation automation system according to claim 7.

9. a scenario editing unit that generates analytical information for a web element to be operated that is included in the edited scenario, and writes the generated analytical information to the storage unit in association with the generated scenario; The operation automation system according to claim 1 .

10. a storage unit that stores a scenario including a web page identifier that identifies a web page, an operation target web element identifier that identifies an operation target web element within the web page, and a web operation on the operation target web element, and that stores analysis information that includes information regarding a description of the operation target web element and information regarding descriptions of surrounding web elements that are web elements surrounding the operation target web element, obtained based on a pre-update web page that is the web page at the time of creating the scenario, as auxiliary information for identifying the operation target on the web page; a scenario execution unit that loads a web page of the web page identifier included in the scenario using a web browser, performs a determination process to determine whether the operation target web element identifier is present in the loaded web page, and if the determination process determines that the operation target web element identifier is present, performs the web operation on the operation target web element on the web element specified by the operation target web element identifier, and if the determination process determines that the operation target web element identifier is not present, performs an analysis process to select a new operation target web element from the web elements included in the web page by analyzing the content of the loaded web page using the auxiliary information, and performs the web operation on the operation target web element on the new operation target web element selected by the analysis process; An automated operation device comprising:

11. a determining step of: acquiring the scenario from a storage unit that stores a scenario including a web page identifier that identifies a web page, an operation target web element identifier that identifies a web element to be operated within the web page, and a web operation on the operation target web element; and storing analysis information, acquired based on the pre-update web page that is the web page at the time of creating the scenario, including information regarding a description of the operation target web element and information regarding descriptions of surrounding web elements that are web elements surrounding the operation target web element, as auxiliary information for identifying the operation target on the web page; loading the web page with the web page identifier included in the acquired scenario using a web browser; and determining whether the operation target web element identifier is present in the loaded web page; a scenario execution step of performing the web operation on the web element identified by the operation target web element identifier when it is determined in the determination step that the operation target web element identifier exists, and performing an analysis process of selecting a new operation target web element from among the web elements included in the web page by analyzing the content of the loaded web page using the auxiliary information when it is determined that the operation target web element identifier does not exist, and performing the web operation on the operation target web element on the new operation target web element selected by the analysis process; An operation automation method comprising:

12. Computer, A program for causing the operation automation system according to any one of claims 1 to 9 to function.

Images