Information processing apparatus, method, program, and system

Abstract

To reduce the operational burden on a user.SOLUTION: A program according to an embodiment of the present disclosure causes a computer to function as: means for acquiring guidance information for a predetermined setting in a UI (User Interface); means for analyzing the acquired guidance information and recognizing an operation required for the predetermined setting; means for determining a pointer operation procedure in the UI based on the recognized operation; and means for analyzing an image of the UI displayed on a display unit, detecting an object to which an action defined in the operation procedure is applied for each determined operation procedure, and executing the action on the object.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a method, a program, and a system. [Background technology]

[0002] For example, when using IT (Information Technology) such as a personal computer (PC) or an application, users may not always be able to use the technology properly, resulting in problems such as the technology not performing as intended. In such cases, users may obtain knowledge to resolve the problem by using a help function or contacting a support center. However, users may not have the literacy to understand the obtained knowledge and perform appropriate operations, or even if they are able to understand the knowledge, they may find the operations complicated.

[0003] In recent years, task automation technologies, such as RPA (Robotic Process Automation), have become well known. For example, Patent Document 1 discloses a technical idea of ​​extracting instruction actions and object cases by natural language analysis of received text data, searching for an execution action command corresponding to the instruction action and an execution action target corresponding to the object case by referring to an execution action table and an object case table, and automatically generating program code based on the search results. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-190776 Summary of the Invention [Problem to be solved by the invention]

[0005] The technical idea of ​​Patent Document 1 requires that the execution action table and the object case table be properly defined in advance. However, since there are a huge number of objects that a user may operate, depending on the knowledge the user has, even if the object case can be extracted, it may not be possible to search for the corresponding execution action object, and program code may not be automatically generated.

[0006] An object of the present disclosure is to provide a technology that reduces the operational burden on the user. [Means for solving the problem]

[0007] A program according to one embodiment of the present disclosure causes a computer to function as a means for acquiring guidance information for a specified setting in a UI (User Interface), a means for analyzing the acquired guidance information and recognizing operations required for the specified setting, a means for determining a pointer operation procedure in the UI based on the recognized operation, a means for analyzing an image of the UI displayed on a display unit, and for each determined operation procedure, a means for detecting an object to which an action defined in the operation procedure is applied, and a means for executing the action on the object. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing the configuration of a client device according to the present embodiment. [Figure 3] FIG. 2 is a block diagram showing the configuration of a server according to the present embodiment. [Figure 4] FIG. 1 is an explanatory diagram of one aspect of the present embodiment. [Figure 5] FIG. 2 is a diagram illustrating a data structure of an FAQ database according to the present embodiment. [Figure 6] 10 is a flowchart of a UI operation automation process according to the present embodiment. [Figure 7] 10A to 10C are diagrams conceptually illustrating operation procedures determined in the UI operation automation process of the present embodiment. [Figure 8] 10A to 10C are diagrams illustrating an example of a screen displayed in the UI operation automation process of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0010] (1) Information processing system configuration The configuration of the information processing system will now be described with reference to Fig. 1, which is a block diagram showing the configuration of the information processing system according to this embodiment.

[0011] As shown in FIG. 1, the information processing system 1 includes a client device 10 and a server 30. The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW.

[0012] The client device 10 is an example of an information processing device that transmits a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer. The user of the client device 10 may be, for example, but is not limited to, a person who has experienced a problem related to IT technology. The problem related to IT technology may include a problem with the client device 10 or another IT device, a problem with software or an IT service used by the client device 10 or another IT device, etc.

[0013] The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request transmitted from the client device 10. The server 30 is, for example, a server computer.

[0014] An operator terminal (not shown) may be further connected to the server 30. The user of the operator terminal may be, for example, an operator engaged in support work related to IT technology (hardware, software, services, or a combination thereof) used by the user of the client device 10. In response to a request from the user of the client device 10, the operator may input text or voice into the operator terminal to guide the user through an operation to fulfill the user's request (e.g., to resolve a problem). Such text information or voice information may be transmitted from the operator terminal to the client device 10 via the server 30. However, such an operator and operator terminal may be replaced by a system that returns a rule-based response, or a system that returns a response using a trained model or a large-scale language model.

[0015] (1-1) Client device configuration The configuration of the client device will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the client device of this embodiment.

[0016] 2, the client device 10 includes a storage device 11, a processor 12, an input / output interface 13, and a communication interface 14. The client device 10 is connected to a display 21.

[0017] The storage device 11 is configured to store programs and data, and is, for example, a combination of a read-only memory (ROM), a random access memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0018] The programs include, for example, the following programs: OS (Operating System) programs Applications that process information (e.g., web browsers)

[0019] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0020] The processor 12 is a computer that implements the functions of the client device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following: ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Gate Array)

[0021] The input / output interface 13 is configured to acquire information (for example, a user's instruction) from an input device connected to the client device 10, and to output information (for example, an image) to an output device connected to the client device 10.

[0022] The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display 21, a speaker, or a combination thereof.

[0023] The communication interface 14 is configured to control communications between the client device 10 and an external device (eg, the server 30).

[0024] The display 21 is configured to display an image (a still image or a moving image). The display 21 is, for example, a liquid crystal display or an organic EL display.

[0025] (1-2) Server configuration The configuration of the server will now be described with reference to Fig. 3, which is a block diagram showing the configuration of the server according to this embodiment.

[0026] As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0027] The storage device 31 is configured to store programs and data, and is, for example, a combination of ROM, RAM, and storage (for example, flash memory or a hard disk).

[0028] The programs include, for example, the following programs: OS programs Application programs that perform information processing

[0029] The data includes, for example, the following data: Databases referenced in information processing - Results of information processing

[0030] The processor 32 is a computer that implements the functions of the server 30 by running a program stored in the storage device 31. The processor 32 is, for example, at least one of the following: ·CPU GPU ASIC FPGA

[0031] The input / output interface 33 is configured to acquire information (for example, a user's instruction) from an input device connected to the server 30, and to output information (for example, an image) to an output device connected to the server 30.

[0032] The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0033] The communication interface 34 is configured to control communication between the server 30 and an external device (eg, the client device 10 or an operator terminal).

[0034] (2) One aspect of the embodiment An example of this embodiment will now be described with reference to Fig. 4, which is an explanatory diagram of this example.

[0035] 4, the client device 10 acquires guidance information GI20 for a predetermined setting in a UI (User Interface) IN21 from, for example, the server 30. For example, if the predetermined setting is a connection setting for a wireless LAN (Local Area Network), the guidance information GI20 may be text information that sequentially describes operations of the UI IN21 required for the connection setting.

[0036] The client device 10 analyzes the guidance information GI20 and recognizes the operations required for the specified settings. Then, the client device 10 determines the operation procedure for the pointer PO22 in the UI IN21 based on the operations recognized by the analysis. The operation procedure may define an action and an object to which the action is applied.

[0037] The client device 10 analyzes the image of the UI IN21 displayed on the display 21 (an example of a "display unit"), and for each determined operation procedure, detects the object OB23 to which the action defined in the operation procedure is to be applied, and executes the action on the object OB23.

[0038] In this way, according to the client device 10 of this embodiment, the object OB23 to be operated on the UI IN21 is automatically detected, and the necessary action for the object OB23 is automatically executed and a predetermined setting is made, without any operation by the user US20 of the client device 10. In other words, according to the client device 10 of this embodiment, it is possible to reduce the operational burden on the user US20 that is required to make a predetermined setting.

[0039] (3) Database The databases of this embodiment will be described. The following databases are stored in the storage device 31.

[0040] (3-1) FAQ database The FAQ database of this embodiment will now be described with reference to Fig. 5, which shows the data structure of the FAQ database of this embodiment.

[0041] The FAQ database (table) stores FAQ information, which is information about frequently asked questions and their answers (for example, IT technology problems and their solutions).

[0042] 5, the FAQ database includes an "ID" field, a "question" field, and an "answer" field. Each field is associated with the other fields.

[0043] The "ID" field stores an FAQ ID, which is information that uniquely identifies FAQ information.

[0044] The "Question" field stores question information. Question information is an information element that constitutes FAQ information, and is information related to frequently asked questions (for example, questions about IT technology problems).

[0045] The "Answer" field stores answer information. Answer information is an information element that constitutes FAQ information, and is information about an answer to a corresponding question (for example, a setting operation in a UI to solve an IT technology problem).

[0046] The client device 10 can acquire the guidance information without using the FAQ database, and therefore the FAQ database can be omitted in this embodiment.

[0047] (4) Information processing The information processing of this embodiment will be described.

[0048] (4-1) UI operation automation processing The UI operation automation process of this embodiment will be described. Fig. 6 is a flowchart of the UI operation automation process of this embodiment. Fig. 7 is a diagram conceptually showing an operation procedure determined in the UI operation automation process of this embodiment. Fig. 8 is a diagram showing an example of a screen displayed in the UI operation automation process of this embodiment.

[0049] The UI operation automation process in FIG. 6 executes, on behalf of the user, operations of the UI (particularly, GUI (Graphical User Interface)) required for predetermined settings.

[0050] As shown in FIG. 6, the client device 10 acquires guidance information (S110). Specifically, the client device 10 acquires guidance information for a predetermined setting in a user interface (UI).

[0051] As a first example of obtaining guidance information (S110), the client device 10 obtains a request from a user and obtains guidance information in response to the request. The request may include, for example, a request to resolve an IT technology problem (e.g., a problem that occurred when the user was using a PC, smartphone, or other device). The client device 10 transmits the request to the server 30 and obtains guidance information corresponding to a response to the request from the server 30. The client device 10 may present the guidance information to the user, or may execute a subsequent step without presenting the guidance information to the user.

[0052] Here, the server 30 can acquire guidance information or information that serves as the basis for the guidance information using various techniques. As a first example, the server 30 may transmit to the client device 10 information generated by providing a large-scale language model with a prompt including a user request and an instruction for outputting information corresponding to the request, or guidance information based on the information. This large-scale language model may be pre-trained with, for example, FAQs related to IT problem solution cases. As a second example, the server 30 references an FAQ database (FIG. 5) and extracts setting operations in the UI associated with the problem corresponding to the user request. The server 30 transmits information on the extracted setting operations or guidance information based on the information to the client device 10. As a third example, the server 30 transmits information output by providing a model input based on the user request to a trained model, or guidance information based on the information to the client device 10. The trained model may be constructed, for example, by performing supervised learning on the correlation between questions and answers in FAQs related to IT problem solution cases. As a fourth example, the server 30 transmits the user's request to an operator terminal (not shown). In response to a request from a user, the operator inputs text or voice into the operator terminal to guide the user through setting operations in the UI to fulfill the user's request (for example, to solve a problem). The operator terminal transmits the text or voice information to the server 30, and the server 30 transmits the received information or guidance information based on it to the client device 10.

[0053] As a second example of obtaining guidance information (S110), the client device 10 obtains guidance information that the user has searched for on his or her own. For example, the user obtains information on setting operations in a UI to solve an IT problem that the user is currently having from a help page or knowledge community of an IT device, application, or SaaS (Software as a Service) that the user is using, and inputs the information into the client device 10 (which may include copying the relevant text information).

[0054] The guidance information typically includes text information. This text information may include text information obtained by performing a voice recognition process on voice information provided to the user of the client device 10. For example, the client device 10 may obtain voice information from the server 30 and obtain the guidance information (text information) by performing a voice recognition process on the voice information.

[0055] After step S110, the client device 10 analyzes the guidance information (S111). Specifically, the client device 10 analyzes the guidance information acquired in step S110 and recognizes operations required for the predetermined settings. As an example, the client device 10 analyzes the guidance information (more precisely, text information included in the guidance information) using a natural language model. As a result, the client device 10 recognizes, for example, a combination of an action (verb) constituting a setting operation in a UI described in the guidance information and an object (object) that is the target of the action, in the setting operation in the UI described in the guidance information. Depending on the guidance information, multiple pairs of actions and objects may be recognized.

[0056] After step S111, the client device 10 determines the operation procedure (S112). Specifically, the client device 10 determines a pointer operation procedure in the UI based on the analysis result in step S111 (i.e., the recognized operation required for a predetermined setting). As an example, the client device 10 determines one or more operation procedures according to the number of recognized operations (combinations of actions and objects), and each operation procedure can be expressed in a format such as "object #n" is "action #n" as shown in FIG. 7. However, although the operation procedures are expressed in natural language in FIG. 7, the operation procedures may also be expressed in a machine-readable language. When determining multiple operation procedures, the client device 10 can determine the execution order of these operation procedures based on the analysis result in step S111.

[0057] After step S112, the client device 10 selects a procedure (S113). Specifically, the client device 10 selects the uncompleted procedure that is earliest in the execution order from among the operation procedures determined in step S112.

[0058] After step S113, the client device 10 executes the action (S114). Specifically, the client device 10 executes the action defined in the procedure selected in step S113. As an example, the client device 10 analyzes an image of the UI displayed on the display 21 and detects an object on the UI to which the action defined in the operation procedure selected in step S113 is applied. For example, techniques such as image recognition used in RPA can be applied to the image analysis. For example, the client device 10 may search the UI for an object having visual characteristics matching "object #n." Then, while the UI is displayed on the display 21, the client device 10 moves a pointer to the position of the detected object. Then, the client device 10 executes an action on the detected object. In the example of FIG. 8 , the client device 10 analyzes the image of the UI IN31 and detects an object OB33 on the UI IN31. The client device 10 moves a pointer PO32 to the position of the detected object OB33 and executes an action on the object OB33.

[0059] After step S114, if all of the operation procedures determined in step S112 have been executed, the UI operation automation process of Fig. 6 ends. On the other hand, if any uncompleted operation procedures remain, the client device 10 returns to the step of selecting a procedure (S113). In this way, if multiple operation procedures have been determined in step S112, the client device 10 repeats steps S113 to S114 to sequentially execute actions in accordance with the order of the multiple operation procedures.

[0060] (5) Summary As described above, the client device 10 of this embodiment acquires guidance information for a predetermined setting in the UI, analyzes the acquired guidance information to recognize operations required for the predetermined setting, and determines a pointer operation procedure in the UI based on the recognized operations. The client device 10 analyzes the UI image displayed on the display 21, and for each determined operation procedure, detects an object to which an action defined in the operation procedure is applied, and executes the action on the object. As a result, even without an operation by the user of the client device 10, the object to be operated on in the UI is automatically detected, and the necessary action is automatically executed on the object, thereby performing the predetermined setting. In other words, the operational burden on the user required to perform the predetermined setting can be reduced.

[0061] The client device 10 may move the pointer to the position of the detected object for each determined operation procedure and then execute the action while the UI is displayed on the display 21. This allows the content of the setting operation that is automatically performed to be visually presented to the user.

[0062] The client device 10 may acquire a request from the user and acquire guidance information in response to the acquired request, thereby realizing the setting of the UI in response to the user request while reducing the operational burden on the user.

[0063] The client device 10 may acquire guidance information based on information corresponding to the settings associated with the requested trouble in a table that associates troubles related to IT technology with UI settings for resolving the troubles, thereby reducing the operational burden on the user when setting up the UI for resolving the trouble.

[0064] The client device 10 may obtain guidance information based on the information output by providing a model input based on the request to the trained model, thereby realizing a UI setting that is statistically estimated to be appropriate for the user's request while reducing the user's operational burden.

[0065] The means for acquiring guidance information may acquire guidance information based on information generated by providing a prompt, including a request and an instruction to output guidance information corresponding to the request, to a large-scale language model. This makes it possible to realize UI settings that are statistically estimated to be appropriate for various requests from users while reducing the operational burden on the user.

[0066] The guidance information may include text information, which allows for analysis using natural language processing.

[0067] The text information may include text information obtained by performing speech recognition processing on speech information provided to a computer user, thereby enabling analysis using natural language processing even when speech-based guidance information is obtained.

[0068] The client device 10 may analyze the text information included in the acquired guidance information using a natural language model and recognize the operations required for the predetermined settings, thereby obtaining a recognition result that is robust and appropriate for the expression of the text information.

[0069] When multiple operation procedures are determined, the client device 10 may execute actions sequentially in accordance with the order of the multiple operation procedures. This allows the setting of the UI to be realized while reducing the operational burden on the user, even when multiple steps are required for the setting of the UI.

[0070] (6) Other variations The storage device 11 may be connected to the client device 10 via a network NW. The display 21 may be integrated with the client device 10. The storage device 31 may be connected to the server 30 via the network NW.

[0071] In the above description, examples have been given in which steps in each process are executed in a specific order, but the execution order of each step is not limited to the described example as long as there is no dependency. Furthermore, a process described as being performed by one device may be executed by another device, and a process described as being performed by the interaction of multiple devices may be executed by a single device. For example, each step of the above information processing may be executed by any of the client device 10, the server 30, and the operator terminal.

[0072] In executing the action (S114), the client device 10 may present (for example, display on the display 21) information indicating the operation procedure determined in step S112 and the execution status of the operation procedure. The operation procedure may be expressed in natural language, for example, as shown in FIG. 7. This allows the user to understand the content of the setting operation that is automatically performed. In particular, if any of the operation procedures is not completed correctly, the client device 10 may present (for example, display on the display 21) information indicating the operation procedure determined in step S112 and the execution status of the operation procedure. This allows the user to understand which operation procedure to resume, even if the UI setting by the UI operation automation process is interrupted for some reason.

[0073] (7) Supplementary Notes The matters explained in the embodiment and the modified examples are additionally noted below.

[0074] (Appendix 1) A computer (10), A means for acquiring guidance information for a predetermined setting in a UI (User Interface) (S110); A means for analyzing the acquired guidance information and recognizing an operation required for the predetermined setting (S111); A means for determining a pointer operation procedure in the UI based on the recognized operation (S112); a means (S114) for analyzing an image of the UI displayed on a display unit (21), detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; A program that functions as a

[0075] (Appendix 2) the means for executing the action moves the pointer to the position of the detected object for each of the determined operation procedures while the UI is displayed on the display unit, and then executes the action. The program described in Appendix 1.

[0076] (Appendix 3) causing the computer to function as a means for obtaining a request from a user of the computer; the means for acquiring the guidance information acquires the guidance information in response to the acquired request. The program described in Appendix 1.

[0077] (Appendix 4) The means for acquiring guidance information acquires guidance information based on information corresponding to a setting associated with the trouble corresponding to the request in a table that associates troubles related to IT (Information Technology) techniques with settings in the UI for solving the troubles. The program described in Appendix 3.

[0078] (Appendix 5) The means for acquiring guidance information acquires guidance information based on information output by providing a model input based on the request to a trained model. The program described in Appendix 3.

[0079] (Appendix 6) the means for acquiring guidance information acquires guidance information based on information generated by providing a prompt including the request and an instruction for outputting guidance information in response to the request to a large-scale language model. The program described in Appendix 3.

[0080] (Appendix 7) The guidance information includes text information. A program according to any one of Supplementary Note 1 to Supplementary Note 6.

[0081] (Appendix 8) the text information includes text information obtained by performing a speech recognition process on speech information provided to a user of the computer; 1. The program described in Appendix 7.

[0082] (Appendix 9) the recognition means analyzes the text information included in the acquired guidance information using a natural language model and recognizes an operation required for the predetermined setting. 1. The program described in Appendix 7.

[0083] (Appendix 10) when a plurality of operation procedures are determined, the means for executing the action executes the action sequentially in accordance with the order of the plurality of operation procedures; The program described in Appendix 1.

[0084] (Appendix 11) causing the computer to function as a means for presenting information indicating the determined operation procedure and an execution status of the operation procedure; The program described in Appendix 1.

[0085] (Appendix 12) the presenting means presents information indicating the determined operation procedure and an execution status of the operation procedure when any of the operation procedures is not completed correctly. 11. The program described in Appendix 11.

[0086] (Appendix 13) A means for acquiring guidance information for a predetermined setting in a UI (User Interface) (S110); A means for analyzing the acquired guidance information and recognizing an operation required for the predetermined setting (S111); A means for determining a pointer operation procedure in the UI based on the recognized operation (S112); a means (S114) for analyzing an image of the UI displayed on a display unit (21), detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; An information processing device (10) comprising:

[0087] (Appendix 14) A computer (10) A step (S110) of acquiring guidance information for a predetermined setting in a UI (User Interface); A step (S111) of analyzing the acquired guidance information and recognizing an operation required for the predetermined setting; A step (S112) of determining a pointer operation procedure in the UI based on the recognized operation; a step (S114) of analyzing an image of the UI displayed on a display unit (21), detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; How to do it.

[0088] (Appendix 15) A system (1) including a plurality of information processing devices (10, 30), A means for acquiring guidance information for a predetermined setting in a UI (User Interface) (S110); A means for analyzing the acquired guidance information and recognizing an operation required for the predetermined setting (S111); A means for determining a pointer operation procedure in the UI based on the recognized operation (S112); a means (S114) for analyzing an image of the UI displayed on a display unit (21), detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; A system comprising:

[0089] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above-described embodiments. Furthermore, the above-described embodiments can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-described embodiments and modifications can be combined. [Explanation of symbols]

[0090] 1: Information processing system 10: Client device 11:Storage device 12: Processor 13: Input / output interface 14: Communication interface 21: Display 30: Server 31:Storage device 32: Processor 33: Input / output interface 34: Communication interface

Claims

1. Computer, means for acquiring guidance information for predetermined settings in a UI (User Interface); means for analyzing the acquired guidance information and recognizing an operation required for the predetermined setting; means for determining a pointer operation procedure in the UI based on the recognized operation; means for analyzing an image of the UI displayed on a display unit, detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; A program that functions as a

2. the means for executing the action moves the pointer to the position of the detected object for each of the determined operation procedures while the UI is displayed on the display unit, and then executes the action. The program according to claim 1.

3. causing the computer to function as a means for obtaining a request from a user of the computer; the means for acquiring the guidance information acquires the guidance information in response to the acquired request. The program according to claim 1.

4. the means for acquiring guidance information acquires guidance information based on information corresponding to a setting associated with the trouble corresponding to the request in a table that associates troubles related to IT (Information Technology) with settings in the UI for solving the troubles; The program according to claim 3.

5. The means for acquiring guidance information acquires guidance information based on information output by providing a model input based on the request to a trained model. The program according to claim 3.

6. the means for acquiring guidance information acquires guidance information based on information generated by providing a prompt including the request and an instruction for outputting guidance information in response to the request to a large-scale language model. The program according to claim 3.

7. The guidance information includes text information.

7. The program according to claim 1.

8. the text information includes text information obtained by performing a speech recognition process on speech information provided to a user of the computer; The program according to claim 7.

9. the recognition means analyzes the text information included in the acquired guidance information using a natural language model and recognizes an operation required for the predetermined setting. The program according to claim 7.

10. when a plurality of operation procedures are determined, the means for executing the action executes the action sequentially in accordance with the order of the plurality of operation procedures; The program according to claim 1.

11. causing the computer to function as a means for presenting information indicating the determined operation procedure and an execution status of the operation procedure; The program according to claim 1.

12. the presenting means presents information indicating the determined operation procedure and an execution status of the operation procedure when any of the operation procedures is not completed correctly. The program according to claim 11.

13. means for acquiring guidance information for a predetermined setting in a UI (User Interface); means for analyzing the acquired guidance information and recognizing an operation required for the predetermined setting; means for determining a pointer operation procedure on the UI based on the recognized operation; means for analyzing an image of the UI displayed on a display unit, detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; An information processing device comprising:

14. The computer acquiring guidance information for a predetermined setting in a UI (User Interface); analyzing the acquired guidance information and recognizing an operation required for the predetermined setting; determining a pointer operation procedure in the UI based on the recognized operation; analyzing an image of the UI displayed on a display unit, detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; How to do it.

15. A system including a plurality of information processing devices, means for acquiring guidance information for a predetermined setting in a UI (User Interface); means for analyzing the acquired guidance information and recognizing an operation required for the predetermined setting; means for determining a pointer operation procedure on the UI based on the recognized operation; means for analyzing an image of the UI displayed on a display unit, detecting an object to which an action defined in the operation procedure is applied for each of the determined operation procedures, and executing the action on the object; A system comprising:

Images