Operation automation device, web operation automation method, and program

Abstract

To provide an operation automation device that executes an operation on a web element in a web page even when the web page is changed.SOLUTION: A storage unit stores web operation information including a web page identifier, a first web element identifier, and an operation in association with a peripheral image of a web element. A web element identifier determination unit determines whether the first web element identifier is included in a web page identified by the web page identifier. When not included, an image recognition control unit acquires position information of an area in the web page that is similar to the peripheral image, and a web element identifier acquisition unit acquires a second web element identifier from within the area of the position information. A web browser control unit, when the web page includes the first web element identifier, commands operation on a web element identified by the first web element identifier and, when the web page does not include the first web element identifier, commands operation on a web element identified by the second web element identifier.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an operation automation device, a web operation automation method, and a program. [Background technology]

[0002] RPA (Robotic Process Automation), a type of automation tool, is a technology that can automatically execute routine tasks for applications that have been performed manually by operating the user interface of an information processing device such as a PC (personal computer). For example, RPA executes a scenario that describes the work procedure, thereby reproducing the work that has been performed manually (see, for example, Patent Document 1). RPA can also incorporate procedures for operating Web pages into the scenario (see, for example, Non-Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4883638 [Non-patent literature]

[0004] [Non-Patent Document 1] "WinActor Suite Library Browser Operation (Value Acquisition, 3)", [online], NTT Advanced Technology Corporation, retrieved July 20, 2023,<URL:https: / / winactor.biz / sweet / 2021 / 09 / 30_4617.html> Summary of the Invention [Problem to be solved by the invention]

[0005] Even if the web page has the same URL (Universal Resource Locator), the content of the web page may be updated. With conventional technology, when a web page specified as the operation target in an RPA scenario is changed, the web element described as the direct operation target in the scenario cannot be found when the scenario is executed, and the operation cannot be continued.

[0006] In view of the above circumstances, the present invention aims to provide an operation automation device, a web operation automation method, and a program that can perform operations on web elements in a web page even when the web page is changed. [Means for solving the problem]

[0007] One aspect of the present invention includes a storage unit that stores web operation information including a web page identifier that identifies a web page, a first web element identifier that identifies a web element to be operated within the web page, and an operation on the web element, in association with a peripheral image of the web element; a web element identifier determination unit that reads a web page having the web page identifier included in the web operation information from a web browser and determines whether the first web element identifier included in the web operation information is present in the read web page; and when the web element identifier determination unit determines that the first web element identifier is not present, a position of an area in the image of the web page that is similar to the peripheral image associated with the web operation information. The operation automation device includes an image recognition control unit that acquires information, a web element identifier acquisition unit that acquires a second web element identifier that identifies a web element displayed within an area indicated by the location information, and a web browser control unit that, when the web element identifier determination unit determines that the first web element identifier is present, performs a process of instructing the web browser to perform the operation included in the web operation information on the web element of the first web element identifier, and, when the web element identifier determination unit determines that the first web element identifier is not present, performs a process of instructing the web browser to perform the operation included in the web operation information on the web element of the second web element identifier acquired by the web element identifier acquisition unit.

[0008] One aspect of the present invention is the above-mentioned operation automation device, which further includes a data control unit that, when the web element identifier determination unit determines that the first web element identifier is not present, rewrites the first web element identifier set in the web operation information to the second web element identifier, and links a peripheral image of the web element of the second web element identifier on the web page to the web operation information and writes it to the memory unit.

[0009] One aspect of the present invention is the above-mentioned operation automation device, wherein when the web element identifier determination unit determines that the first web element identifier is present, the data control unit links a peripheral image of the web element of the first web element identifier on the web page to the web operation information and writes it to the memory unit.

[0010] One aspect of the present invention is the above-mentioned operation automation device, wherein the web element identifier acquisition unit presents the second web element identifier or the web element of the second web element identifier to a user, and receives input of confirmation information indicating that the web element identified by the second web element identifier is the correct operation target, or web element correction information indicating the correct web element to be operated.

[0011] One aspect of the present invention is the above-mentioned operation automation device, wherein the web browser control unit, when the web element identifier determination unit determines that the first web element identifier is not included and the web element identifier acquisition unit receives input of the confirmation information, performs processing to instruct the web browser to perform the operation included in the web operation information on the web element of the second web element identifier, and when the web element identifier determination unit determines that the first web element identifier is not included and the web element identifier acquisition unit receives input of the confirmation information, performs processing to instruct the web browser to perform the operation included in the web operation information on the web element indicated by the web element modification information,

[0012] One aspect of the present invention is the above-mentioned operation automation device, which further includes a data control unit that rewrites the first web element identifier set in the web operation information to the second web element identifier when the confirmation information is input, and rewrites it to a third web element identifier that identifies the web element indicated by the web element correction information when the web element correction information is input.

[0013] One aspect of the present invention is the above-mentioned operation automation device, wherein when the data control unit rewrites the first web element identifier set in the web operation information to the second web element identifier, the data control unit links a peripheral image of the web element of the second web element identifier in the web page to the web operation information and writes it into the memory unit, and when the data control unit rewrites the first web element identifier set in the web operation information to the third web element identifier, the data control unit links a peripheral image of the web element of the third web element identifier in the web page to the web operation information and writes it into the memory unit.

[0014] One aspect of the present invention includes a reading step of reading web operation information from a storage unit that associates and stores web operation information including a web page identifier that identifies a web page, a first web element identifier that identifies a web element to be operated within the web page, and an operation on the web element with a peripheral image of the web element; a web element identifier determination step of reading a web page having the web page identifier included in the web operation information from a web browser and determining whether the first web element identifier included in the web operation information is present in the read web page; and when it is determined in the web element identifier determination step that the first web element identifier is not present, a region in the image of the web page that is similar to the peripheral image associated with the web operation information. a web element identifier acquisition step of acquiring a second web element identifier that identifies a web element displayed within an area indicated by the position information; and a web browser control step of, if it is determined in the web element identifier determination step that the first web element identifier is present, performing a process of instructing the web browser to perform the operation included in the web operation information on the web element of the first web element identifier, and, if it is determined in the web element identifier determination step that the first web element identifier is not present, performing a process of instructing the web browser to perform the operation included in the web operation information on the web element of the second web element identifier acquired in the web element identifier acquisition step.

[0015] One aspect of the present invention is a program for causing a computer to function as the above-mentioned operation automation device. Effect of the Invention

[0016] According to the present invention, even if a Web page is changed, it is possible to execute operations on Web elements within the Web page. [Brief description of the drawings]

[0017] [Figure 1] 1 is an overall configuration diagram of an embodiment of the present invention; [Diagram 2] FIG. 13 is a diagram showing an example of a scenario according to the embodiment. [Diagram 3] FIG. 11 is a diagram showing an example of information stored in a database according to the embodiment. [Figure 4] FIG. 2 is a diagram showing an example of modeling a Web page according to the embodiment. [Diagram 5] FIG. 2 is a diagram illustrating a detailed configuration of an RPA execution unit according to the embodiment. [Figure 6] 13 is a flowchart showing a scenario execution process of the operation automation device according to the embodiment. [Figure 7] 13 is a flowchart showing the operation of a Web processing unit according to the embodiment. [Figure 8] FIG. 2 is a diagram illustrating a hardware configuration of the operation automation device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. The operation automation device of this embodiment is equipped with an RPA. RPA is a type of automation tool, and its functions include a function for automatically operating a web page. When a web page to be operated automatically is changed, a conventional RPA cannot find the web element to be directly operated, and therefore cannot continue the operation described in the scenario. The RPA equipped in the operation automation device of this embodiment can find the web element to be operated and continue the automatic operation according to the scenario, even if the contents of the web page described as the operation target in the created scenario are changed.

[0019] To operate a web page with RPA, the web element to be operated (link, button, form, etc.) is directly identified as the operation target, and events (operations) such as clicks and character input are sent to the operation target. There are three main methods for identifying web elements:

[0020] (1) Specify the coordinates on the screen directly. (2) Obtain coordinates by performing image recognition on the screen. (3) The HTML syntax position (XPath, etc.) of the object to be operated is specified from the HTML (Hypertext Markup Language) data describing the screen.

[0021] Method (1) depends on the RPA execution environment, such as the display size. Method (2) has a high image recognition cost. In addition, there is a problem that it cannot be used in some situations, such as when a screen similar to the web element to be operated is present on the screen. Therefore, method (3) is the most efficient. However, in method (3), when the web page to be operated is changed, the HTML structure may change significantly even if the appearance of the screen has not changed much, and the web element may not be identified. In this embodiment, this problem is solved to realize an RPA that is resistant to changes in web pages.

[0022] Fig. 1 is a diagram showing an example of the overall configuration of the present invention. As shown in Fig. 1, an operation automation device 1 is connected to a Web service providing device 4 via a network 3. The network 3 and the Web service providing device 4 can be realized by any general-purpose technology. For example, the network 3 may be a public network such as the Internet, a private network such as a LAN (Local Area Network), or a combination of these. The Web service providing device 4 provides a Web page P to the operation automation device 1 via the network 3.

[0023] The operation automation device 1 is, for example, a computer device. The operation automation device 1 includes an input unit 11, a display unit 12, a browser unit 13, a scenario storage unit 14, a database 15, an RPA execution unit 16, and an image recognition unit 17. The scenario storage unit 14, the database 15, and the RPA execution unit 16 realize the function of RPA.

[0024] The input unit 11 is a user interface operated by a user when inputting a user's instruction to the operation automation device 1. The input unit 11 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), a button, a touch panel, etc. The display unit 12 displays data. The display unit 12 is an image display device such as a CRT display, a liquid crystal display, an organic EL display, etc.

[0025] The browser unit 13 provides a Web browser. The browser unit 13 can be realized by any general-purpose technology. The browser unit 13 connects to the Web service providing device 4 via the network 3 in accordance with an operation performed by a user through the input unit 11, and displays a Web page P provided by the Web service providing device 4 on the display unit 12. The browser unit 13 also operates in accordance with an operation performed by a user through the input unit 11 on the Web page P displayed on the display unit 12.

[0026] The scenario storage unit 14 stores the RPA scenario R. The scenario R is data described in a description format executable by the RPA execution unit 16. The scenario R indicates the execution order of operations, operation targets, operations to be performed on the operation targets, and parameter values ​​used when executing the operations. The operation targets are applications that realize a specified function of the operation automation device 1, and the like. The operation targets include a web browser. The operations indicated in the scenario R are operations performed on the operation targets by the input unit 11. If the operation target is a web browser, the details of the operations are described in the database 15.

[0027] Database 15 stores data indicating details of operations on a Web browser. Database 15 and scenario storage unit 14 may be an integrated storage unit. For example, database 15 describes information that identifies the Web page and Web element on which the Web operation is to be performed, and the Web operation to be performed on the Web element. The Web operation is, for example, an operation (event) on a Web element, such as clicking on a Web element or inputting characters into a Web element.

[0028] The RPA execution unit 16 executes scenario R. The RPA execution unit 16 performs operations on an external system according to scenario R. When the operation target is a Web browser, the RPA execution unit 16 performs a Web operation read from the database 15 on a Web page P provided by the Web service providing device 4 via the network 3.

[0029] The image recognition unit 17 is an arbitrary general-purpose image recognition system. The image recognition unit 17 detects an area in an image that is similar to another image.

[0030] Fig. 2 is a diagram showing an example of scenario R shown in Fig. 1. Scenario R is composed of multiple records. Each record stores an execution sequence R1, an operation target R2, an operation R3, and parameters R4 for performing the operation. The operation target R2 is, for example, an application (APP), an RPA, a web browser (WEB), etc. By referring to the operation target R2, it is possible to uniquely determine whether the operation target is a web or not.

[0031] FIG. 3 is a diagram showing an example of information stored in the database 15 of FIG. 1. The database 15 stores Web operation information D1 and Web element peripheral image information D2. The Web operation information D1 has a record referenced by a reference point D11. In each record, the reference point D11, a Web page identifier D12, a Web element identifier D13, and a Web operation D14 are set in association with each other. The reference point D11 is a primary key. The Web page identifier D12 is information for identifying the Web page P to be operated. The Web element identifier D13 identifies the Web element to be operated among the Web elements in the Web page P identified by the Web page identifier. The Web operation D14 indicates an operation to be performed on the Web element identified by the Web element identifier D13. Furthermore, the record stores an image reference point D15 for referencing an image stored in the Web element peripheral image information D2.

[0032] The web element peripheral image information D2 has a record that associates an image reference point D21 with a peripheral image D22. The image reference point D21 is a primary key. The peripheral image D22 is a peripheral image of the web element to be operated.

[0033] FIG. 4 is a diagram showing an example of modeling of a web page P. The web page P is data described in HTML, for example. The web page P is referenced by a web page identifier P1. The web page P also has a page image P2, which is display information of the entire page. In addition, the web page P can include one or more web elements P3. The web element has a web element identifier P4. The web element can be referenced by the web element identifier P4. The web element identifier P4 is indicated by, for example, XPath. The web element P3 also has coordinates P5 indicating its position in the page image P2. The peripheral image of the web element P3 in the page image P2 is P6. The peripheral image P6 includes the display area of ​​the web element P3 and is an area larger than the web element P3. The image recognition unit 17 can obtain an area of ​​an image similar to the peripheral image P6 by performing image recognition on the page image P2. Hereinafter, the n-th (n is an integer equal to or greater than 1) web element P3 will be described as a web element P3-n. Moreover, the Web element identifier P4, the coordinates P5, and the surrounding image P6 of the Web element P3-n are respectively written as the Web element identifier P4-n, the coordinates P5-n, and the surrounding image P6-n.

[0034] FIG. 5 is a diagram showing a detailed configuration of the RPA execution unit 16 shown in FIG. 1. The RPA execution unit 16 has a control unit 21, a scenario reading unit 22, an APP (application) processing unit 23, an internal processing unit 24, and a Web processing unit 25. The control unit 21 controls the entire RPA execution unit 16. The scenario reading unit 22 reads a scenario R from the scenario storage unit 14. The APP processing unit 23 performs processing described in a record of the scenario R when the operation target R2 of the record is an application outside the RPA. The internal processing unit 24 performs processing described in the record when the operation target R2 of the record of the scenario R is the RPA itself. The Web processing unit 25 performs processing described in the record when the operation target R2 of the record of the scenario R is the Web.

[0035] The Web processing unit 25 includes a database control unit 251, a Web browser control unit 252, a Web element identifier determination unit 253, an image recognition control unit 254, and a Web element identifier acquisition unit 255. The database control unit 251 reads data from the database 15 and writes data to the database 15. The Web browser control unit 252 operates the Web browser provided by the browser unit 13. The Web element identifier determination unit 253 determines whether a Web element having a specified Web element identifier exists in the Web page P read from the Web browser. The image recognition control unit 254 acquires position information indicating at which coordinates in the image of the Web page P an area similar to the specified surrounding image exists, using the image recognition unit 17. The Web element identifier acquisition unit 255 acquires the Web element identifier of the Web element existing in the area in the Web page P indicated by the position information acquired by the image recognition control unit 254.

[0036] Fig. 6 is a flowchart showing the execution process of scenario R in the operation automation device 1. Error processing is omitted in Fig. 6.

[0037] The control unit 21 instructs the scenario reading unit 22 to read a scenario. The scenario reading unit 22 reads a scenario R to be executed from the scenario storage unit 14 (step S1). The control unit 21 sets an initial value of 1 to a variable m (step S2).

[0038] The control unit 21 reads a record in which the variable m is set in the execution order R1 from the scenario R (step S3). The record of the scenario R in which the variable m is set in the execution order R1 is described as record m. The control unit 21 determines whether or not the read record m exists (step S4). If the read record m exists, the control unit 21 determines the operation target R2 of the record m (step S5).

[0039] When the control unit 21 determines that the operation object R2 is an APP, it instructs the APP processing unit 23 to execute the operation. The APP processing unit 23 performs processing according to the record m (step S6). Specifically, the APP processing unit 23 executes the operation indicated by the operation R3 on the application indicated by the operation object R2, using the parameter value set in the parameter R4.

[0040] In step S3, when the control unit 21 determines that the operation target R2 is an RPA, it instructs the internal processing unit 24 to execute the operation. The internal processing unit 24 performs the process according to the record m (step S7). Specifically, the internal processing unit 24 executes the process indicated by the operation R3 using the parameter value set in the parameter R4.

[0041] In step S3, if the control unit 21 determines that the operation target R2 is WEB, it instructs the Web processing unit 25 to execute the operation. The Web processing unit 25 executes the Web operation specified by the operation R3 of the record m using the parameter value set in the parameter R4 (step S8).

[0042] When the process of step S6, step S7, or step S8 is completed, the control unit 21 adds 1 to the variable m (step S9). The control unit 21 repeats the process from step S3. In step S6, the APP processing unit 23 may set a predetermined value to the variable m according to a predetermined condition and execute the process from step S3. Similarly, in step S7, the internal processing unit 24 may set a predetermined value to the variable m according to a predetermined condition and execute the process from step S3. Similarly, in step S8, the Web processing unit 25 may set a predetermined value to the variable m according to a predetermined condition and execute the process from step S3. Then, when the control unit 21 determines that no record m has been read in step S4, it ends the process.

[0043] Fig. 7 is a flowchart showing the operation of the Web processing unit 25. Fig. 7 shows detailed processing of step S8 in Fig. 6. Error processing is omitted in Fig. 7.

[0044] The database control unit 251 reads the reference point set for operation R3 from record m of scenario R (step S101). The database control unit 251 uses the reference point read in step S101 as a search key to identify a record in Web operation information D1. The identified record is called record d. The database control unit 251 reads the Web page identifier D12, the Web element identifier D13, and the Web operation D14 from the identified record d (step S102).

[0045] Web browser control unit 252 reads from browser unit 13 the Web page P that browser unit 13 is causing display unit 12 to display and that is identified by Web page identifier D12 read in step S102 (step S103).

[0046] The Web element identifier determination unit 253 determines whether or not the Web page P read in step S103 contains a Web element P3-n (n is an integer equal to or greater than 1) identified by the Web element identifier D13 read in step S102 (step S104).

[0047] If Web element identifier determination unit 253 determines that Web element P3-n with Web element identifier D13 exists (step S105: YES), it causes Web browser control unit 252 to execute the process of step S106. That is, Web browser control unit 252 instructs browser unit 13 to perform, on Web element P3-n, the operation indicated by Web operation D14 read in step S102, using parameter R4 set in record m of scenario R (step S106).

[0048] The Web browser control unit 252 judges whether the operation instructed in step S106 was successful or unsuccessful (step S107). For example, even if a Web element with the Web element identifier D13 specified when the scenario was created exists, if the Web element has been changed to a different Web element from that when the scenario was created, the operation instructed in step S106 may fail. If the Web browser control unit 252 judges that the operation was successful (step S107: YES), it reads the peripheral image P6-n of the Web element P3-n from the page image P2 (step S108). For example, the Web browser control unit 252 may set the peripheral image P6 to an area obtained by enlarging the minimum rectangle that contains the entire display area of ​​the Web element P3 by a predetermined percentage on each of the upper, lower, left, and right sides with the rectangle as the center. Note that the method of determining the shape and size of the peripheral image P6 is not limited to this. The size of the peripheral image P6 may be determined in advance according to the size of the display area of ​​the Web element P3, or may be set by the user. Also, the shape of the peripheral image P6 may be determined according to the shape of the display area of ​​the Web element P3.

[0049] When the database control unit 251 adds a new record to the Web element peripheral image information D2, it stores the peripheral image P6-n read in step S108 in the peripheral image D22 of the added record. Furthermore, the database control unit 251 stores the image reference point D21 assigned to the stored peripheral image D22 in the added record. The database control unit 251 stores the image reference point D21 set in the added record of the Web element peripheral image information D2 in the image reference point D15 of record d of the Web operation information D1 (step S109). The Web processing unit 25 ends the process.

[0050] On the other hand, if the Web element identifier determination unit 253 determines that the Web element P3 with the Web element identifier D13 does not exist (step S105: NO), it instructs the database control unit 251 to execute the process of step S110. Also, if the Web browser control unit 252 determines that the operation instructed in step S106 has failed (step S107: NO), it instructs the database control unit 251 to execute the process of step S110. The database control unit 251, which has received the instruction from the Web element identifier determination unit 253 or the Web browser control unit 252, specifies a record in the Web element peripheral image information D2 using the image reference point D15 of the record d as a search key. The database control unit 251 reads the peripheral image D22 from the specified record in the Web element peripheral image information D2 (step S110).

[0051] The Web browser control unit 252 reads the page image P2 of the Web page P identified by the Web page identifier D12 read in step S102 (step S111). The image recognition control unit 254 inputs the page image P2 read in step S111 and the peripheral image D22 read in step S110 to the image recognition unit 17 to perform image recognition. The image recognition control unit 254 acquires the coordinates of an image area in the page image P2 that is similar to the peripheral image D22 (step S112). The image recognition control unit 254 outputs information on the acquired coordinates to the Web element identifier acquisition unit 255.

[0052] Web element identifier acquisition unit 255 acquires, from Web page P, Web element identifier P4-n' of Web element P3-n' located within the area indicated by the coordinate information input from image recognition control unit 254 (n' is an integer equal to or greater than 1). Web element identifier acquisition unit 255 stores acquired Web element identifier P4-n' as Web element identifier D13 of record d in Web operation information D1 (step S113). Web processing unit 25 performs processing from step S104. However, in step S104, Web element identifier P4-n' stored in record d in step S113 is used as Web element identifier D13. That is, Web element identifier determination unit 253 determines whether Web page P includes Web element P3-n' identified by Web element identifier P4-n' set in Web element identifier D13 (step S104). As a result, in step S105, the Web element identifier determination unit 253 determines that the Web element P3-n' with the Web element identifier D13 exists (step S105: YES).

[0053] The Web browser control unit 252 instructs the browser unit 13 to perform the operation indicated by the Web operation D14 read in step S102 on the Web element P3-n' using the parameter R4 set in the record m of the scenario R (step S106). If the operation is successful (step S107: YES), the Web browser control unit 252 reads the peripheral image P6-n' of the Web element P3-n' from the page image P2 (step S108). When the database control unit 251 adds a new record to the Web element peripheral image information D2, it stores the peripheral image P6-n' in the peripheral image D22 of the added record. Furthermore, the database control unit 251 stores the image reference point D21 assigned to the stored peripheral image D22 in the added record. The database control unit 251 stores the image reference point D21 set in the added record of the Web element peripheral image information D2 in the image reference point D15 of the record d (step S109). The Web processing unit 25 ends the process.

[0054] Before storing the Web element identifier P4-n' acquired in step S113 in record d in the Web operation information D1, the Web element identifier acquisition unit 255 may present the Web element P3-n' identified by the Web element identifier P4-n' or the acquired Web element identifier P4-n' to the user by displaying it on the display unit 12. For example, the Web element identifier acquisition unit 255 may display the page image P2 and further display the display area of ​​the Web element P3-n' in the page image P2 by surrounding it with a line.

[0055] If the user determines that the displayed Web element P3-n' or Web element identifier P4-n' is correct, the user inputs confirmation information indicating that it is the correct operation target via input unit 11. If the confirmation information is input, Web element identifier acquisition unit 255 stores Web element identifier P4-n' acquired in step S113 in record d in Web operation information D1. Web processing unit 25 performs the processes from step S104 as described above.

[0056] On the other hand, if the user determines that the displayed Web element P3-n' or Web element identifier P4-n' is incorrect, the user inputs information about the correct Web element P3-n" to be operated via the input unit 11 (n" is an integer equal to or greater than 1). For example, the Web element identifier acquisition unit 255 displays page image P2, and the user selects, via the input unit 11, the display area of ​​the Web element P3-n" to be the correct operation target within the page image P2. The Web element identifier acquisition unit 25 acquires the Web element identifier P4-n" of the Web element P3-n" selected by the input unit 11 from the Web page P, and stores it in record d within the Web operation information D1. The Web processing unit 25 performs the processes from step S104 onwards in the same manner as described above, using the Web element identifier P4-n" instead of the Web element identifier P4-n'.

[0057] That is, the Web element identifier determination unit 253 determines whether the Web page P contains a Web element P3-n" identified by the Web element identifier P4-n" (step S104). The Web element identifier determination unit 253 determines that the Web element P3-n" exists (step S105: YES). The Web browser control unit 252 instructs the browser unit 13 to perform the operation indicated by the Web operation D14 read in step S102 on the Web element P3-n" using the parameter R4 set in the record m of the scenario R (step S106). If the operation is successful (step S107: YES), the Web browser control unit 252 reads the peripheral image P6-n" of the Web element P3-n" from the page image P2 (step S108). When the database control unit 251 adds a new record to the Web element peripheral image information D2, it stores peripheral image P6-n'' in the peripheral image D22 of the added record. Furthermore, the database control unit 251 stores the image reference point D21 assigned to the stored peripheral image D22 in the added record. The database control unit 251 stores the image reference point D21 set in the added record of the Web element peripheral image information D2 in the image reference point D15 of record d (step S109). The Web processing unit 25 ends the processing.

[0058] The functions of the operation automation device 1 in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing the functions of the browser unit 13, the RPA execution unit 16, and the image recognition unit 17 may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The program may also be provided through a network. Note that the "computer system" here includes hardware such as an OS and peripheral devices. The "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into a computer system. Furthermore, the "computer-readable recording medium" may include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned functions, and may further be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0059] The operation automation device 1 may be realized by a plurality of computer devices connected to a network. In this case, it is possible to arbitrarily select which of the plurality of computer devices is used to realize each functional unit of the operation automation device 1. Also, the same functional unit may be realized by a plurality of computer devices. For example, one or more of the scenario storage unit 14, the database 15, and the image recognition unit 17 may be realized by a computer device different from the computer device in which the operation automation device 1 is realized.

[0060] FIG. 8 is a device configuration diagram showing a hardware configuration example of the operation automation device 1. The operation automation device 1 includes a processor 71, a storage unit 72, a communication interface 73, and a user interface 74. The processor 71 is a central processing unit that performs calculations and control. The processor 71 is, for example, a central processing unit (CPU) or a graphics processing unit (GPU). The processor 71 realizes the functions of the browser unit 13, the RPA execution unit 16, and the image recognition unit 17 by reading and executing a program from the storage unit 72. The storage unit 72 realizes the scenario storage unit 14 and the database 15. The storage unit 72 further has a work area when the processor 71 executes various programs. The communication interface 73 is connected to other devices so as to be able to communicate with them. The user interface 74 is an input device such as a keyboard, a pointing device (mouse, tablet, etc.), a button, a touch panel, etc., or a display device such as a display. A human operation is input through the user interface 74. The user interface 74 realizes the input unit 11 and the display unit 12 .

[0061] All or part of the functions of the operation automation device 1 may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA).

[0062] When a web element to be operated in an RPA scenario is specified by XPath, the logical structure of the web page may change, and the XPath of the web element to be operated may change even though the appearance of the web page remains almost unchanged. In such a case, the operation automation device of this embodiment can automatically obtain a new XPath from the web page and continue the operation.

[0063] According to the above-described embodiment, the operation automation device includes a storage unit, a web element identifier determination unit, an image recognition control unit, a web element identifier acquisition unit, and a web browser control unit. The storage unit stores web operation information including a web page identifier for identifying a web page, a first web element identifier for identifying a web element to be operated in the web page, and an operation on the web element, in association with a peripheral image of the web element. The storage unit corresponds to, for example, the scenario storage unit 14 and the database 15 in the embodiment. The web element identifier determination unit reads a web page of a web page identifier included in the web operation information from a web browser, and determines whether the first web element identifier included in the web operation information is present in the read web page. When the web element identifier determination unit determines that the first web element identifier is not present, the image recognition control unit obtains position information of an area in the image of the web page that is similar to the peripheral image associated with the web operation information. The web element identifier acquisition unit obtains a second web element identifier for identifying a web element displayed in the area indicated by the position information. When the web element identifier determination unit determines that a first web element identifier is present, the web browser control unit performs processing to instruct the web browser to perform an operation included in the web operation information on the web element having the first web element identifier, and when the web element identifier determination unit determines that the first web element identifier is not present, the web browser control unit performs processing to instruct the web browser to perform an operation included in the web operation information on the web element having the second web element identifier acquired by the web element identifier acquisition unit.

[0064] The operation automation device may further include a data control unit. The data control unit corresponds to, for example, database control unit 251 of the embodiment. When the web element identifier determination unit determines that the first web element identifier does not exist, the data control unit rewrites the first web element identifier set in the web operation information to the second web element identifier and writes a peripheral image of the web element of the second web element identifier in the web page in the storage unit linked to the web operation information. When the web element identifier determination unit determines that the first web element identifier exists, the data control unit may also write a peripheral image of the web element of the first web element identifier in the web page in the storage unit linked to the web operation information.

[0065] The web element identifier acquisition unit may present the second web element identifier or the web element of the second web element identifier to a user, and receive input of confirmation information indicating that the web element identified by the second web element identifier is the correct target for operation, or web element correction information indicating the correct web element to be operated.

[0066] The web browser control unit performs processing to instruct the web browser to perform an operation included in the web operation information on the web element having the second web element identifier when the web element identifier determination unit determines that the first web element identifier is not included and the web element identifier acquisition unit receives input of confirmation information, and performs processing to instruct the web browser to perform an operation included in the web operation information on the web element indicated by the web element modification information when the web element identifier determination unit determines that the first web element identifier is not included and the web element identifier acquisition unit receives input of confirmation information.

[0067] The operation automation device may further include a data control unit. The data control unit corresponds to, for example, the database control unit 251 of the embodiment. The data control unit rewrites the first web element identifier set in the web operation information to the second web element identifier when confirmation information is input, and rewrites it to a third web element identifier that identifies the web element indicated by the web element correction information when web element correction information is input. When the data control unit rewrites the first web element identifier set in the web operation information to the second web element identifier, the data control unit may write a peripheral image of the web element of the second web element identifier in the web page in association with the web operation information in the storage unit, and when the data control unit rewrites the first web element identifier set in the web operation information to the third web element identifier, the data control unit may write a peripheral image of the web element of the third web element identifier in the web page in association with the web operation information in the storage unit.

[0068] When an operation on a web element having a first web element identifier fails, the operation automation device may perform the same process as when the web element identifier determination unit determines that the first web element identifier does not exist. For example, when an operation that the first web browser control unit instructs the web browser to perform on the web element having the first web element identifier fails, the image recognition control unit acquires position information of an area in an image of the web page that is similar to a peripheral image linked to the web operation information. The web element identifier acquisition unit acquires a second web element identifier that identifies a web element displayed within the area indicated by the acquired position information, and the web browser control unit instructs the web browser to perform the operation included in the web operation information on the web element having the acquired second web element identifier.

[0069] Although the embodiment of the present invention has been described above with reference to the drawings, it is clear that the above embodiment is merely an example of the present invention and the present invention is not limited to the above embodiment. Therefore, addition, omission, substitution, and other modifications of components may be made without departing from the technical spirit and scope of the present invention. [Explanation of symbols]

[0070] 1…Operation automation device 3. Network 4. Web service provider 11...Input section 12...Display section 13…Browser section 14…Scenario memory section 15…Database 16…RPA Execution Department 17...Image recognition section 21...Control unit 22...Scenario loading section 23…APP processing section 24...Internal processing unit 25…Web processing unit 71…Processor 72...Storage section 73...Communication interface 74...User Interface 251...Database control unit 252…Web browser control unit 253…Web element identifier determination unit 254...Image recognition control unit 255…Web element identifier acquisition section

Claims

1. a storage unit that stores web operation information including a web page identifier that identifies a web page, a first web element identifier that identifies a web element to be operated on within the web page, and an operation on the web element, in association with a peripheral image of the web element; a web element identifier determination unit that reads a web page having the web page identifier included in the web operation information from a web browser and determines whether the first web element identifier included in the web operation information is present in the read web page; an image recognition control unit that acquires, when the web element identifier determination unit determines that the first web element identifier is not present, position information of an area in the image of the web page that is similar to the peripheral image associated with the web operation information; a web element identifier acquisition unit that acquires a second web element identifier that identifies a web element to be displayed within the area indicated by the position information; a web browser control unit that, when the web element identifier determination unit determines that the first web element identifier is present, performs a process of instructing the web browser to perform the operation included in the web operation information on the web element of the first web element identifier, and, when the web element identifier determination unit determines that the first web element identifier is not present, performs a process of instructing the web browser to perform the operation included in the web operation information on the web element of the second web element identifier acquired by the web element identifier acquisition unit; An automated operation device comprising:

2. a data control unit that, when the web element identifier determination unit determines that the first web element identifier does not exist, rewrites the first web element identifier set in the web operation information to the second web element identifier, and links a peripheral image of the web element of the second web element identifier on the web page to the web operation information and writes the image to the storage unit. The operation automation device according to claim 1 .

3. When the web element identifier determination unit determines that the first web element identifier is present, the data control unit writes a peripheral image of the web element of the first web element identifier in the web page to the web operation information in the storage unit. The operation automation device according to claim 2 .

4. The web element identifier acquisition unit presents the second web element identifier or the web element of the second web element identifier to a user, and receives input of confirmation information indicating that the web element identified by the second web element identifier is a correct operation target, or web element correction information indicating the correct operation target web element. The operation automation device according to claim 1 .

5. The web browser control unit, when it is determined by the web element identifier determination unit that the first web element identifier is not included and the web element identifier acquisition unit receives input of the confirmation information, performs a process of instructing the web browser to perform the operation included in the web operation information on the web element of the second web element identifier, and when it is determined by the web element identifier determination unit that the first web element identifier is not included and the web element correction information is received, performs a process of instructing the web browser to perform the operation included in the web operation information on the web element indicated by the web element correction information. The operation automation device according to claim 4.

6. a data control unit that rewrites the first web element identifier set in the web operation information to the second web element identifier when the confirmation information is input, and rewrites the first web element identifier set in the web operation information to a third web element identifier that identifies the web element indicated by the web element correction information when the web element correction information is input; The operation automation device according to claim 4.

7. When the data control unit rewrites the first web element identifier set in the web operation information to the second web element identifier, the data control unit writes a peripheral image of the web element of the second web element identifier in the web page linked to the web operation information in the memory unit, and when the data control unit rewrites the first web element identifier set in the web operation information to the third web element identifier, the data control unit writes a peripheral image of the web element of the third web element identifier in the web page linked to the web operation information in the memory unit. The operation automation device according to claim 6.

8. a reading step of reading out web operation information from a storage unit that stores web operation information including a web page identifier that identifies a web page, a first web element identifier that identifies a web element to be operated on in the web page, and an operation on the web element, and a peripheral image of the web element in association with the web operation information; a web element identifier determination step of reading a web page having the web page identifier included in the web operation information from a web browser and determining whether the first web element identifier included in the web operation information is present in the read web page; an image recognition control step of acquiring position information of an area similar to the peripheral image associated with the web operation information in the image of the web page when it is determined that the first web element identifier is not present in the web element identifier determination step; a web element identifier acquisition step of acquiring a second web element identifier that identifies a web element to be displayed within the area indicated by the position information; a web browser control step of performing a process of instructing the web browser to perform the operation included in the web operation information on the web element of the first web element identifier when it is determined in the web element identifier determination step that the first web element identifier is present, and performing a process of instructing the web browser to perform the operation included in the web operation information on the web element of the second web element identifier acquired in the web element identifier acquisition step when it is determined in the web element identifier determination step that the first web element identifier is not present; A web operation automation method comprising:

9. Computer, A program for causing the operation automation device according to any one of claims 1 to 7 to function.

Images