Information processing device, method, and program

Abstract

To provide easy-to-follow instructions for operation. [Solution] The information processing device includes: an acquisition unit that receives a video or a plurality of still images representing a process that includes performing at least one type of operation on the user terminal's operation screen; a pre-processing unit that acquires text representing the operation and an image representing the operation screen at the timing of the operation based on the received video or a plurality of still images; and a processing unit that inputs the text representing the operation, the image representing the operation screen at the timing of the operation, and the user terminal's operation screen into a generating AI model and executes the content of the operation.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, method, and program.

Background Art

[0002] Conventionally, RPA (Robotic Process Automation) has been known. RPA has problems in initial labor for registering work content using a dedicated tool, and in operation and maintenance where the RPA program must also be changed when the specifications of the target work are changed.

[0003] On the other hand, tools for replacing computer work with generative AI have also emerged (Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Another object of the present invention is to provide an information processing apparatus, method, and program that can easily instruct operations.

[0006] Another object of the present invention is to provide an information processing apparatus, method, and program that can instruct operations manually.

[0007] A third object of the present invention is to provide an information processing apparatus, method, and program that can appropriately perform operations even in domain-specific scenarios.

[0008] Furthermore, a fourth objective of the present invention is to provide an information processing device, method, and program that can easily create manuals. [Means for solving the problem]

[0009] A first aspect of this disclosure is an information processing device comprising: an acquisition unit that receives a video, a plurality of still images, or operation content representing a process that includes performing at least one type of operation on the operation screen of a user terminal; a pre-processing unit that acquires text representing the operation and an image representing the operation screen at the timing of the operation based on the received video, a plurality of still images, or operation content; and a processing unit that inputs the text representing the operation, the image representing the operation screen at the timing of the operation, and a new user terminal operation screen into a generating AI model to execute the content of the operation.

[0010] A second aspect of this disclosure is an information processing device comprising: an acquisition unit that receives a document represented by text or images describing a process that includes performing at least one type of operation on the operation screen of a user terminal; and a processing unit that inputs the received document and the operation screen of the user terminal into a generating AI model and executes the process.

[0011] A third aspect of this disclosure is an information processing device comprising: an acquisition unit that receives pairs of explanatory content for a process on the operation screen of a user terminal and content for executing the process; and a processing unit that inputs the received pairs, the operation screen of the user terminal, and a prompt instructing a process that includes performing at least one type of operation on the operation screen to a generating AI model and executes the instructed process.

[0012] A fourth aspect of this disclosure is an information processing device comprising: an acquisition unit that receives a video or a plurality of still images representing a process that includes performing at least one type of operation on the operation screen of a user terminal, and explanatory text describing the process that includes performing at least one type of operation; and a processing unit that inputs the received video or a plurality of still images, the explanatory text, and a prompt that instructs the creation of a document using the text or images describing the process represented by the video to a generating AI model to create the document.

[0013] A fifth aspect of this disclosure is an information processing method, wherein the computer receives a video, a set of still images, or an operation content representing a process that includes performing at least one type of operation on the operation screen of a user terminal; based on the received video, a set of still images, or an operation content, it obtains text representing the operation and an image representing the operation screen at the timing of the operation; and inputs the text representing the operation, the image representing the operation screen at the timing of the operation, and a new user terminal operation screen into a generating AI model to execute the content of the operation.

[0014] A sixth aspect of this disclosure is an information processing method in which a computer receives a document represented by text or images describing a process that includes performing at least one type of operation on the operation screen of a user terminal, and inputs the received document and the operation screen of the user terminal into a generating AI model to execute the process.

[0015] A seventh aspect of this disclosure is an information processing method, wherein a computer receives a pair of explanatory content for a process on the operation screen of a user terminal and content for executing the said process, inputs the received pair, the operation screen of the user terminal, and a prompt instructing a process that includes performing at least one type of operation on the operation screen, and executes the instructed process.

[0016] The eighth aspect of the present disclosure is an information processing method, which includes receiving a moving image or a plurality of still images representing a process including performing at least one type of operation on an operation screen of a user terminal, and an explanatory text explaining the process including performing the at least one type of operation, and inputting the received moving image or plurality of still images, the explanatory text, and a prompt for instructing to create a document using a sentence or an image explaining the process represented by the moving image into a generative AI model, and causing a computer to execute creating the document.

[0017] The ninth aspect of the present disclosure is an information processing program, which is a program for causing a computer to function as any one of the information processing apparatuses according to the first aspect to the fourth aspect.

Advantages of the Invention

[0018] According to the disclosed technology, an operation can be easily instructed.

Brief Description of the Drawings

[0019] [Figure 1] It is a block diagram showing the configuration of the information processing system of the present embodiment. [Figure 2] It is a schematic block diagram of an example of a computer functioning as the management server of the present embodiment. [Figure 3] It is a block diagram showing the functional configuration of the management server of the present embodiment. [Figure 4] It is a diagram showing an example of a screen display on a user terminal. [Figure 5] It is a flowchart showing the content of the operation execution processing routine of the management server according to the first embodiment. [Figure 6] It is a flowchart showing the content of the operation execution processing routine of the management server according to the second embodiment. [Figure 7] It is a flowchart showing the content of the operation execution processing routine of the management server according to the third embodiment. [Figure 8] It is a block diagram showing the configuration of the management server according to the fourth embodiment. [Figure 9] It is a flowchart showing the content of the manual creation process routine of the management server according to the fourth embodiment. [Figure 10] It is a diagram showing an example of a screen display on a user terminal.

Mode for Carrying Out the Invention

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0021] [First Embodiment] [Outline of this Embodiment] In this embodiment, following a person's operation, it imitates it as it is. Using not only the prompt but also a moving image representing the person's operation log itself, while abstracting and interpreting it with a generative AI (Artificial Intelligence), an execution program for imitation is created. In abstraction, for example, since the desktop size varies depending on the person, appropriate icon images and the like are remembered. Here, as in the third embodiment described later, even without giving the name of the icon, as long as the operation log is given, execution is possible. It may have a function of automatically correcting after executing the once-created execution program and identifying the locations where errors occur.

[0022] Also, it may have a function of modularizing and connecting the created execution programs. Modularize a person's operation and save it internally, enabling another operation from the middle.

[0023] For example, assuming there is an execution program consisting of steps 1 to 5, steps 1 to 3 and steps 4 to 5 are made into two modules. For example, steps 1 to 3 are for creating an email, and steps 4 to 5 are for sending and encrypting. After that, when a prompt instructing "Save as a draft after creating the email" is input, only steps 1 to 3 are executed, and then only the new process of "draft" is executed. Note that what is connected to the module may be text or another module.

[0024] Furthermore, instructions given only in text may not always be sufficient. It's difficult for humans to give instructions, and difficult for machines to understand. Conversely, operation logs alone can also be difficult for machines to understand.

[0025] In reality, even when giving text-based instructions to any web service using the computer use or OpenAI® Operator described in Non-Patent Document 1 above, it frequently fails to function as intended. In such cases, the text-based instructions are supplemented by having the system learn additional human operations.

[0026] <System Configuration of This Embodiment> As shown in Figure 1, the information processing system 100 according to the first embodiment includes a management server 10 installed on the service management company side and user terminals 24 operated by users. The management server 10 is an example of an information processing device. In Figure 1, for simplicity, an example is shown where two user terminals 24 are provided, but three or more user terminals 24 may be provided.

[0027] The management server 10 and the user terminal 24 are connected via a network 26 such as the Internet.

[0028] The user terminal 24 consists of smartphones, mobile phones, PDAs (Personal Digital Assistants), or notebook / book-type computer terminals, etc.

[0029] Figure 2 is a block diagram showing the hardware configuration of the management server 10 in this embodiment.

[0030] As shown in Figure 2, the management server 10 includes a CPU (Central Processing Unit) 11, ROM (Read Only Memory) 12, RAM (Random Access Memory) 13, storage 14, input unit 15, display unit 16, and communication interface (I / F) 17. Each component is connected to the others via a bus 19 so that they can communicate with each other.

[0031] The CPU 11 is a central processing unit that executes various programs and controls various components. Specifically, the CPU 11 reads a program from the ROM 12 or storage 14 and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculations according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores programs for performing various processes.

[0032] ROM12 stores various programs and data. RAM13 temporarily stores programs or data as a working area. Storage14 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs, including the operating system, and various data.

[0033] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used for various types of input.

[0034] The display unit 16 is, for example, a liquid crystal display and displays various information. The display unit 16 may also function as an input unit 15 by employing a touch panel system.

[0035] The communication interface 17 is an interface for communicating with other devices, and standards such as Ethernet®, FDDI, and Wi-Fi® can be used.

[0036] Next, the functional configuration of the management server 10 will be described. As shown in Figure 3, the management server 10 functionally comprises an acquisition unit 30, a preprocessing unit 32, a processing unit 34, a creation unit 36, a module creation unit 38, and a model storage unit 62.

[0037] The acquisition unit 30 receives a video from the user terminal 24 that represents a process involving performing at least one type of operation on the user terminal 24's operation screen. For example, the acquisition unit 30 receives a video that represents a process involving operating at least one type of application on the user terminal's operation screen.

[0038] Specifically, the acquisition unit 30 receives video recordings of user operations on the user terminal 24's operation screen. For example, it receives video recordings of a series of user operations, such as launching a calendar app and transferring today's schedule to a memo app.

[0039] The acquisition unit 30 may also receive multiple still images from the user terminal 24 that represent a process involving at least one type of operation on the user terminal 24's operation screen. For example, it may receive a collection of snapshots taken when a user action such as a click occurs as multiple still images.

[0040] Furthermore, the acquisition unit 30 also accepts explanatory text describing a process that includes performing at least one type of operation. For example, it accepts explanatory text such as "Transfer the schedule to a memo."

[0041] Furthermore, the acquisition unit 30 receives user input from the user terminal 24, which represents an instruction sentence for instructing the generation AI model 62A to perform processing including the operation of a new application using a module described later. Furthermore, the acquisition unit 30 receives explanatory text after executing the operation.

[0042] The preprocessing unit 32 acquires a processing execution memory that includes text representing the operation and an image representing the operation screen at the timing of the operation, based on the received video and explanatory text.

[0043] Specifically, the preprocessing unit 32 recognizes the boundaries between operations in the video and acquires a processing execution memory that includes text representing the operation and an image representing the operation screen at the timing of the operation, based on the boundaries between operations.

[0044] More specifically, the preprocessor 32 uses scene detection technology to detect scene divisions in the video, thereby recognizing the divisions between operations in the video and adding divisions to the video.

[0045] The preprocessing unit 32, at each segment, uses image recognition AI to verbalize the user operation represented by the video, associates it with the image at that moment, and stores it in the processing execution memory.

[0046] At this time, based on the user's actions, the system stores in the processing execution memory which actions were performed and on what target.

[0047] For example, if a series of user actions are (1) launch the calendar app, (2) select and copy today's schedule, (3) launch the notes app, and (4) paste into the notes app, the resulting verbal description would be "(1) launch the calendar app, (2) select and copy today's schedule, (3) launch the notes app, and (4) paste into the notes app."

[0048] Image recognition AI is an existing technology, and it can be a general-purpose image recognition AI, or an AI that has been further trained on image and text pairs in response to user actions.

[0049] Furthermore, when acquiring the processing execution memory, it is also determined which part of the explanatory text it corresponds to.

[0050] For example, if a series of user actions are (1) launching an app with the icon "C", (2) selecting and copying a range of text within the app, (3) launching a memo app, and (4) pasting into the memo app, it would be difficult to semantically interpret what (1) "C" represents and what (2) the text within the app represents.

[0051] In contrast, by adding the explanatory text "Transfer schedule to memo" to the input of the image recognition AI, the meanings of (1) and (2) are interpreted as "Launch the calendar app" and "Select and copy the schedule text," and stored in the processing execution memory, enabling robust operation.

[0052] Simply pressing "text within the app" could lead to errors, but by memorizing the text "schedule text," the probability of retrieving schedule-related text is increased.

[0053] Additionally, explanatory text such as "Transfer schedule to memo" may be added to the beginning of the processing execution memory. This allows the image recognition AI to interpret the content of each operation while taking the overall purpose into account, and add text to make it closer to a "manual." For example, a processing execution memory representing "Overall purpose → Explanation of step 1 "Launch the calendar app" + Content of step 1 → Explanation of step 2 "Select and copy the schedule text" + Content of step 2 → Step 3 → ..." can be obtained. Furthermore, the preprocessing unit 32 stores the processed execution memory, along with the obtained explanatory text, in the storage unit 14 so that it can be read later. At this time, the processing execution memory contains the entire history of trial and error, and may therefore contain unnecessary processing. Therefore, the preprocessing unit 32 may use the generated AI model 62A to separate the necessary processing from the unnecessary processing in the processing execution memory with the obtained explanatory text appended, and extract only the necessary processing. In addition, the preprocessing unit 32 obtains from the output of the generated AI model 62A the locations where errors occur when executing the processing in the processing execution memory with the obtained explanatory text appended, and if it is determined that it will operate normally, it stores the processing execution memory with the obtained explanatory text appended in the storage 14.

[0054] The processing unit 34 inputs the processing execution memory, which includes text representing the operation and an image representing the operation screen at the timing of the operation, as well as the operation screen of the new user terminal 24 (see left side of Figure 4), into the generated AI model 62A and executes the content of the operation.

[0055] Specifically, the generating AI model 62A refers to the processing execution memory and executes the process according to the procedure stored there. In the example above, the steps of launching the calendar app, selecting and copying today's schedule, launching the memo app, and pasting into the memo app are stored along with the image, so it executes them.

[0056] In this case, a prompt can be created to instruct the system to execute the process represented by the processing execution memory, and the operation screen of the user terminal 24 (see the left side of Figure 4) and the prompt (see the upper right side of Figure 4) can be input to the generated AI model 62A to execute the operation. Figure 4 shows an example of inputting the prompt "Create a questionnaire for the product "Telescope". Figure 4 also shows an example of displaying the new operation screen of the user terminal 24, the prompt, and the processing execution memory on the display screen of the user terminal 24.

[0057] For example, the processing unit 34 inputs the received pair, the operation screen of the user terminal 24, and a prompt instructing the AI ​​model to perform a process that includes operating at least one type of application on the operation screen, and then executes the instructed process.

[0058] The creation unit 36 ​​creates a program to execute a process corresponding to the content of the operation. For example, the creation unit 36 ​​creates a program to execute a process corresponding to the process executed by the generated AI model 62A.

[0059] The module creation unit 38 creates modules that execute partial processing within programs based on multiple programs created by the creation unit 36.

[0060] Furthermore, the processing unit 34 inputs the operation screen of the user terminal 24 and a new instruction prompt that instructs processing including new operations on the operation screen using the created module to the generated AI model 62A, and executes the instructed processing.

[0061] For example, the processing unit 34 inputs the operation screen of the user terminal 24 and a new instruction prompt that uses the created module to instruct the operation screen to perform a process that includes the operation of at least one new application, to the generated AI model 62A, and then executes the instructed process. Furthermore, if the processing unit 34 receives only explanatory text, it reads the processing execution memory containing the received explanatory text from the storage unit 14, inputs the read processing execution memory and the operation screen of the new user terminal into the generated AI model 62, and executes the operation.

[0062] The model storage unit 62 stores the generated AI model 62A. The generated AI model 62A may be stored on an external device separate from the management server 10. In this case, the management server 10 should send a prompt to the external device to obtain the output of the generated AI model 62A.

[0063] Here, an example of a generative AI model 62A is the generative AI of Claude®, among others.

[0064] In this embodiment, the case where a prompt is input to the generating AI model 62A was described as an example, but prompt input is not required. For example, the processing unit 34 may internally understand that a browser should be opened and then execute the operation.

[0065] <How information processing systems work> Next, the operation of the information processing system 100 according to this embodiment will be described.

[0066] When the management server 10 receives from the user terminal 24 a video representing a process that includes performing at least one type of operation on the user terminal's operation screen, and explanatory text describing the process that includes performing at least one type of operation, it executes the operation execution processing routine shown in Figure 5. The operation execution processing routine is an example of an information processing method.

[0067] In step S100, the acquisition unit 30 receives from the user terminal 24 a video image representing a process that includes performing at least one type of operation on the user terminal 24's operation screen, and explanatory text describing the process.

[0068] In step S101, the preprocessor 32 uses scene detection technology to detect scene divisions in the video, recognizes divisions for each operation in the video, and adds divisions to the video.

[0069] In step S102, the preprocessor 32 verbalizes the user operations represented by the video at each segment using image recognition AI, associates the image at that moment with the verbalization, and stores it in the processing execution memory. This obtains processing execution memory that includes text representing the operation and an image representing the operation screen at the moment of the operation. The preprocessor 32 also stores the obtained processing execution memory, with the explanatory text added, in the storage 14 so that it can be read later.

[0070] In step S104, the processing unit 34 inputs the processing execution memory, which includes text representing the operation and an image representing the operation screen at the timing of the operation, as well as the operation screen of the user terminal 24, into the generated AI model 62A and executes the content of the operation.

[0071] In step S106, the creation unit 36 ​​creates a program to execute the processing corresponding to the content of the operation.

[0072] In step S108, the module creation unit 38 creates a module that executes partial processing in a program based on the multiple programs created by the creation unit 36, and then terminates the operation execution processing routine.

[0073] Furthermore, the acquisition unit 30 receives user input from the user terminal 24, which represents an instruction sentence for instructing the generation AI model 62A to perform a new operation using the module.

[0074] Then, the processing unit 34 inputs the operation screen of the user terminal 24 and a new instruction prompt that instructs processing including new operations on the operation screen using the created module to the generated AI model 62A, and executes the instructed processing. Furthermore, if the processing unit 34 receives only explanatory text, it reads the processing execution memory containing the received explanatory text from the storage unit 14, inputs the read processing execution memory and the operation screen of the new user terminal into the generated AI model 62, and executes the operation.

[0075] As described above, according to the information processing system of the first embodiment, a video image representing a process that includes performing at least one type of operation on the user terminal's operation screen is received, text representing the operation and an image representing the operation screen at the timing of the operation are acquired, and the text representing the operation, the image representing the operation screen at the timing of the operation, and the user terminal's operation screen are input to a generating AI model to execute the content of the operation. This makes it easy to instruct the operation.

[0076] [Second Embodiment] Next, an information processing system according to the second embodiment will be described. Since the configuration of the information processing system according to the second embodiment is the same as that of the first embodiment, the same reference numerals are used and their description is omitted.

[0077] <Summary of this embodiment> In this embodiment, the operation is performed according to a document created by a person (e.g., a manual). Alternatively, the operation may be performed according to a manual created in the fourth embodiment described later.

[0078] Alternatively, the user may perform the operation according to the manual and point out any omissions or errors in the manual. The user can perform the operation according to a manual created by a human and identify where errors occur. Alternatively, the user can perform the operation according to the manual created in the fourth embodiment and identify where errors occur.

[0079] <System Configuration of This Embodiment>

[0080] The information processing system 100 according to this embodiment, as in the first embodiment, includes a management server 10 and a user terminal 24 operated by the user, as shown in Figure 1 above.

[0081] Next, the functional configuration of the management server 10 will be described. As shown in Figure 3 above, the management server 10 functionally comprises an acquisition unit 30, a preprocessing unit 32, a processing unit 34, a creation unit 36, a module creation unit 38, and a model storage unit 62.

[0082] The acquisition unit 30 receives a manual that describes a process involving performing at least one type of operation on the user terminal 24's operation screen, using text or images. For example, the acquisition unit 30 receives a manual that describes a process involving operating at least one type of application on the user terminal 24's operation screen, using text or images. For example, the manual might describe, using images and text, launching a calendar application, selecting and copying today's schedule, launching a memo application, and pasting into the memo application.

[0083] The acquisition unit 30 further receives explanatory text describing a process that includes performing at least one type of operation.

[0084] The preprocessing unit 32 acquires a processing execution memory that includes text representing the operation and an image representing the operation screen at the timing of the operation, based on the received manual and explanatory text.

[0085] Specifically, the preprocessing unit 32 recognizes the divisions between operations in the manual and acquires a processing execution memory that includes text representing the operation and an image representing the operation screen at the timing of the operation, based on the divisions between operations.

[0086] More specifically, if the manual is a PDF file with each page separated, the preprocessor 32 recognizes the separation between operations using page breaks. If a page contains multiple operations, it recognizes the separation using clues such as procedure numbers or text-to-image pairs that are expected to be demarcations. This separation recognition may be performed using a rule-based model or a model obtained through machine learning.

[0087] The preprocessing unit 32 saves the text information and image information described in the manual as processing execution memory at each section.

[0088] The processing unit 34 inputs the processing execution memory, which includes text representing the operation and an image representing the operation screen at the timing of the operation, as well as the operation screen of the user terminal 24, into the generating AI model 62A and executes the content of the operation.

[0089] In this case, the processing unit 34 may create a prompt instructing the user to execute the process represented by the processing execution memory, and input the user terminal 24's operation screen and the prompt into the generated AI model 62A to execute the operation.

[0090] The processing unit 34 further obtains from the output of the generated AI model 62A the locations where errors occur when executing the process manually.

[0091] In this embodiment, the case where a prompt is input to the generating AI model 62A was described as an example, but prompt input is not required. For example, the processing unit 34 may internally understand that a browser should be opened and then execute the operation.

[0092] <How information processing systems work> Next, the operation of the information processing system 100 according to this embodiment will be described.

[0093] When the management server 10 receives a manual from the user terminal 24 that describes a process including performing at least one type of operation on the user terminal's operation screen, and an explanatory text describing the process including performing at least one type of operation, it executes the operation execution processing routine shown in Figure 6. The operation execution processing routine is an example of an information processing method. For processes similar to those in the first embodiment, the same reference numerals are used and detailed explanations are omitted.

[0094] In step S200, the acquisition unit 30 receives from the user terminal 24 a manual that describes a process that includes performing at least one type of operation on the user terminal 24's operation screen, and explanatory text that describes the said process.

[0095] In step S201, the preprocessing unit 32 recognizes the divisions between operations in the manual and adds divisions to the manual.

[0096] In step S202, the preprocessing unit 32 saves the text information and image information described in the manual as processing execution memory for each section, thereby acquiring processing execution memory that includes text representing the operation and images representing the operation screen at the timing of the operation.

[0097] In step S204, the processing unit 34 inputs the processing execution memory, which includes text representing the operation and an image representing the operation screen at the timing of the operation, as well as the operation screen of the user terminal 24, into the generated AI model 62A and executes the content of the operation.

[0098] The processing unit 34 further obtains from the output of the generated AI model 62A the locations where errors occur when executing the process manually.

[0099] In step S205, it is determined whether or not an error occurred. If an error occurred, the operation execution processing routine is terminated. On the other hand, if no error occurred, the process proceeds to step S106.

[0100] In step S106, the creation unit 36 ​​creates a program to execute the processing corresponding to the content of the operation.

[0101] In step S108, the module creation unit 38 creates a module that executes partial processing in a program based on the multiple programs created by the creation unit 36, and then terminates the operation execution processing routine.

[0102] Furthermore, the acquisition unit 30 receives user input from the user terminal 24, which represents an instruction sentence for instructing the generation AI model 62A to perform a new operation using the module.

[0103] Then, the processing unit 34 inputs the operation screen of the user terminal 24 and a new instruction prompt that instructs processing including new operations on the operation screen using the created module to the generated AI model 62A, and executes the instructed processing.

[0104] As described above, according to the information processing system of the second embodiment, a manual is received which is represented using text or images that describe a process that includes performing at least one type of operation on the user terminal's operation screen. The received manual and a prompt instructing the user to perform the process described in the manual are input to the generating AI model, and the process is executed. This makes it possible to instruct the user to perform an operation using a manual.

[0105] [Third Embodiment] Next, an information processing system according to the third embodiment will be described. Note that the configuration of the information processing system according to the third embodiment is the same as that of the first embodiment, and therefore the same reference numerals are used, and their description is omitted.

[0106] <Summary of this embodiment> In this embodiment, proxy execution in a specific domain is made possible by having the generation AI learn arbitrary images and text. For example, by having the AI ​​learn the icon image of an in-house program and the name of that icon, the generation AI can be made to call the in-house program.

[0107] <System Configuration of This Embodiment> The information processing system 100 according to this embodiment, as in the first embodiment, includes a management server 10 and a user terminal 24 operated by the user, as shown in Figure 1 above.

[0108] Next, the functional configuration of the management server 10 will be described. As shown in Figure 3 above, the management server 10 functionally comprises an acquisition unit 30, a preprocessing unit 32, a processing unit 34, a creation unit 36, a module creation unit 38, and a model storage unit 62.

[0109] The acquisition unit 30 receives a video from the user terminal 24 that represents a process including performing at least one type of operation on the user terminal 24's operation screen. The acquisition unit 30 may also receive multiple still images from the user terminal 24 that represent a process including performing at least one type of operation on the user terminal 24's operation screen.

[0110] The acquisition unit 30 further receives explanatory text describing a process that includes performing at least one type of operation.

[0111] The acquisition unit 30 receives pairs of descriptions of processes on the user terminal 24's operation screen and the content of those processes. Specifically, the acquisition unit 30 receives pairs of processes that include performing at least one operation on any image area on the user terminal 24's operation screen and the name of that image area. For example, the acquisition unit 30 receives pairs of click operations on an application's icon image or menu display and the name of the application.

[0112] The processing unit 34 inputs the received pair, the processing execution memory containing the text representing the operation and the image representing the operation screen at the timing of the operation, and the operation screen of the user terminal 24 into the generated AI model 62A and executes the content of the operation.

[0113] In this case, the processing unit 34 may input the received pair, the user terminal's operation screen, and a prompt instructing the AI ​​model 62A to perform a process that includes performing at least one type of operation on the operation screen, and execute the instructed process.

[0114] For example, the processing unit inputs the received pair, the user terminal's operation screen, and a prompt instructing the user to perform a process that includes operating at least one type of application on the operation screen, into the generated AI model 62A, and executes the instructed process.

[0115] <How information processing systems work> Next, the operation of the information processing system 100 according to this embodiment will be described.

[0116] When the management server 10 receives from the user terminal 24 a pair of an arbitrary image area on the user terminal 24's operation screen and the name of that image area, a video representing a process that includes performing at least one type of operation on the user terminal 24's operation screen, and explanatory text describing the process that includes performing at least one type of operation, it executes the operation execution processing routine shown in Figure 7. The operation execution processing routine is an example of an information processing method.

[0117] In step S300, the acquisition unit 30 receives from the user terminal 24 a pair of an arbitrary image area on the user terminal 24's operation screen and the name of the image area, a video representing a process that includes performing at least one type of operation on the user terminal 24's operation screen, and explanatory text describing the process.

[0118] In step S101, the preprocessor 32 uses scene detection technology to detect scene divisions in the video, recognizes divisions for each operation in the video, and adds divisions to the video.

[0119] In step S302, the preprocessor 32 verbalizes the user operation represented by the video at each segment using image recognition AI, associates it with the image at that time, and stores it in the processing execution memory. This obtains processing execution memory containing text representing the operation and an image representing the operation screen at the time of the operation. At this time, a pair of an arbitrary image area on the operation screen of the received user terminal 24 and the name of that image area may be used.

[0120] In step S304, the processing unit 34 inputs a pair of an arbitrary image area on the user terminal 24's operation screen and the name of that image area, along with the processing execution storage and the user terminal 24's operation screen, into the generated AI model 62A and executes the operation.

[0121] In step S106, the creation unit 36 ​​creates a program to execute the processing corresponding to the content of the operation.

[0122] In step S108, the module creation unit 38 creates a module that executes partial processing in a program based on the multiple programs created by the creation unit 36, and then terminates the operation execution processing routine.

[0123] Furthermore, the acquisition unit 30 receives user input from the user terminal 24, which represents an instruction sentence for instructing the generation AI model 62A to perform a new operation using the module.

[0124] Then, the processing unit 34 inputs the operation screen of the user terminal 24 and a new instruction prompt that instructs processing including new operations on the operation screen using the created module to the generated AI model 62A, and executes the instructed processing.

[0125] As described above, according to the information processing system of the third embodiment, a pair of an arbitrary image area on the user terminal's operation screen and the name of the image area is received, and the received pair, the user terminal's operation screen, and a prompt instructing a process that includes performing at least one type of operation on the operation screen are input to the generating AI model, and the instructed process is executed. This enables appropriate operation even in domain-specific situations.

[0126] The acquisition unit 30 may also accept the address of a database that stores pairs of data, including a process that performs at least one type of operation on any image area on the user terminal's operation screen, and the name of the image area. In this case, the processing unit 34 should acquire the pairs of data, including a process that performs at least one type of operation on any image area on the user terminal's operation screen, and the name of the image area, from the received address.

[0127] Furthermore, the method described in the third embodiment above, which involves performing a process that includes performing at least one type of operation on an arbitrary image area on the user terminal's operation screen, and receiving a pair of this pair with the name of the image area, and inputting the received pair into the generation AI model, may also be applied to the first and second embodiments above.

[0128] [Fourth Embodiment] Next, an information processing system according to the fourth embodiment will be described. Note that parts having the same configuration as the information processing system according to the first embodiment will be denoted by the same reference numerals and their descriptions will be omitted.

[0129] <Summary of this embodiment> In this embodiment, instead of creating an executable program from video footage representing human actions, the process is described in words and images to create a document (e.g., a manual). At this time, the manual is created using explanatory text that describes the process, adding explanations of the overall purpose and each process.

[0130] <System Configuration of This Embodiment> The information processing system 100 according to this embodiment, as in the first embodiment, includes a management server 10 and a user terminal 24 operated by the user, as shown in Figure 1 above.

[0131] Next, the functional configuration of the management server 10 will be described. As shown in Figure 8, the management server 10 functionally comprises an acquisition unit 430, a processing unit 434, and a model storage unit 62.

[0132] The acquisition unit 430 receives a video image representing a process that includes performing at least one type of operation on the user terminal 24's operation screen. For example, the acquisition unit 430 receives a video image representing a process that includes operating at least one type of application on the user terminal 24's operation screen. The acquisition unit 430 may also receive multiple still images from the user terminal 24 that represent a process that includes performing at least one type of operation on the user terminal 24's operation screen.

[0133] The acquisition unit 430 further receives explanatory text describing a process that includes performing at least one type of operation.

[0134] The processing unit 434 inputs the received video and explanatory text, along with a prompt instructing the AI ​​model 62A to create a manual using the text or images that describe the process represented by the video, to create the manual.

[0135] For example, if there is explanatory text "Transfer schedule to memo," the manual will be created by adding "Transfer schedule to memo" at the beginning, and then, for each operation, the generating AI model 62A will interpret the content of each process based on the overall purpose and add explanatory text.

[0136] As an example, create a manual that follows this structure: "Overall objective → Explanation of step 1: 'Launch the calendar app' + Content of step 1 → Explanation of step 2: 'Select and copy the schedule text' + Content of step 2 → Step 3 → ..."

[0137] <How information processing systems work> Next, the operation of the information processing system 100 according to this embodiment will be described.

[0138] When the management server 10 receives from the user terminal 24 a video representing a process that includes performing at least one type of operation on the user terminal 24's operation screen, and explanatory text describing the process that includes performing at least one type of operation, it executes the manual creation processing routine shown in Figure 9. The manual creation processing routine is an example of an information processing method.

[0139] In step S400, the acquisition unit 30 receives from the user terminal 24 a video image representing a process that includes performing at least one type of operation on the user terminal 24's operation screen, and explanatory text describing the process.

[0140] In step S402, the processing unit 434 inputs the received video and explanatory text, along with a prompt instructing the generating AI model 62A to create a manual using text or images that explain the process represented by the video, and then creates the manual.

[0141] As described above, the information processing system according to the fourth embodiment receives a video image representing a process that includes performing at least one type of operation on the user terminal's operation screen, and explanatory text describing the process that includes performing at least one type of operation. The received video image, the explanatory text, and a prompt instructing the system to create a manual using the text or images describing the process represented by the video image are input to a generating AI model to create a manual. This makes it possible to easily create a manual.

[0142] Furthermore, the system may be implemented as an information processing system having the function of executing the processes described in each of the embodiments above. In this case, a display screen 500 as shown in Figure 10 can be displayed to the user terminal 24 to accept operations. In Figure 10, an example is shown where the display screen 500 includes an area 500A for displaying the manual and an area 500B for displaying the operation screen of the user terminal while processing is being executed using the generation AI model. In this case, when the button 516 is operated, the processing unit outputs the received manual and the operation screen of the user terminal while processing is being executed using the generation AI model, in association with each other. This allows the execution to proceed in increments of a certain amount (one PDF page in the example of Figure 10) while comparing it with the manual. This enables stable step-by-step execution of processing using the generation AI model and makes manual revision and operation verification easier.

[0143] Furthermore, Figure 10 shows an example in which the display screen 500 includes a button 510 that instructs the start of recording of moving images, a button 512 that instructs the display to check the manual, a button 514 that instructs the execution of processing by the processing unit, a button 518 that instructs the automatic correction of errors in the manual, and a button 520 that instructs the verification of the operation of the processing based on the manual.

[0144] Furthermore, Figure 10 shows an example where the display screen 500 includes a text box 522 for accepting input of variables related to processing, including new operations. Using this text box 522, the acquisition unit accepts input of variables related to processing, including new operations, and the processing unit inputs the user terminal's operation screen, the variables, and the new instruction prompt into the generating AI model and executes the instructed processing. In this way, the parts to be made into variables can be extracted and made freely inputtable. The extraction of the parts to be made into variables can be done by the user arbitrarily specifying a range of time or area within the operation, or the generating AI can automatically extract the parts to be made into variables, or it can automatically extract common operations from multiple operations. In addition, it may also be possible to instruct the attachment of files or changes to the processing itself in parts that are prone to change. [Explanation of Symbols]

[0145] 10 Management Server 11 CPU 14 Storage 15 Input section 16 Display 17 Communication Interface 24 User terminals 26 Network 30 Acquisition Department 32 Pre-processing section 34 Processing Unit 36 Creation Department 38 Module Creation Section 62 Model Memory Unit 62A Generative AI Model 100 Information Processing Systems 430 Acquisition Department 434 Processing Unit

Claims

1. An acquisition unit that receives a video, multiple still images, or operation details representing a process that includes performing at least one type of operation on the user terminal's operation screen, A preprocessing unit that acquires text representing the operation and an image representing the operation screen at the timing of the operation, based on the received video, multiple still images, or operation content. A processing unit inputs text representing the operation, an image representing the operation screen at the timing of the operation, and the operation screen of a new user terminal into a generating AI model and executes the content of the operation. Information processing device including

2. The acquisition unit further receives explanatory text describing a process that includes performing at least one of the operations. The information processing apparatus according to claim 1, wherein the preprocessing unit acquires text representing the operation and an image representing the operation screen at the timing of the operation, based on the received moving image, a plurality of still images, or operation content and the explanatory text.

3. A data acquisition unit that receives a document, which is represented using text or images, that describes a process that includes performing at least one type of operation on the user terminal's operation screen, A processing unit that inputs the received document and the user terminal's operation screen into a generated AI model and executes the process, Information processing device including

4. The acquisition unit further receives explanatory text describing a process that includes performing at least one of the operations. The information processing apparatus according to claim 3, wherein the processing unit inputs the received document, the explanatory text, and the user terminal operation screen into the generating AI model and executes the processing.

5. The information processing apparatus according to claim 3, further comprising the processing unit presenting, obtained from the output of the generated AI model, locations in the document where errors occur when performing the processing.

6. The information processing apparatus according to claim 3, wherein the processing unit further outputs the received document and the operation screen of the user terminal in which the processing is being executed using the generated AI model, in association with each other.

7. An acquisition unit that receives pairs of the description of a process on the user terminal's operation screen and the execution details of the said process, A processing unit inputs the received pair, the user terminal's operation screen, and a prompt instructing a process that includes performing at least one type of operation on the operation screen to a generated AI model and executes the instructed process, Information processing device including

8. The acquisition unit receives a database containing pairs of explanations of processes on the user terminal's operation screen and the execution details of those processes, which have been prepared in advance. The processing unit obtains pairs from the received database of the description of the process on the user terminal's operation screen and the execution details of the process. The information processing apparatus according to claim 7, which inputs the acquired pair, the user terminal's operation screen, and a prompt instructing a process that includes performing at least one type of operation on the operation screen to the generated AI model and executes the instructed process.

9. The description of the above process is the name of an arbitrary image region, The information processing apparatus according to claim 7, wherein the execution of the aforementioned process includes performing at least one type of operation on the arbitrary image region.

10. The acquisition unit further receives pairs of the explanation content of the process on the user terminal's operation screen and the execution content of the process, The information processing apparatus according to claim 1, wherein the processing unit inputs the received pair, the text representing the operation, the image representing the operation screen at the timing of the operation, and the operation screen of the user terminal to the generating AI model and executes the content of the operation.

11. The acquisition unit further receives pairs of the explanation content of the process on the user terminal's operation screen and the execution content of the process, The information processing apparatus according to claim 3, wherein the processing unit inputs the received pair, the received document, and a prompt instructing the AI ​​model to execute the process represented by the document, and executes the process.

12. The information processing apparatus according to claim 1, further comprising a creation unit for creating a program for executing processing corresponding to the content of the above operation.

13. The system further includes a module creation unit that creates modules to perform partial processing in programs based on a plurality of programs created by the creation unit, The information processing apparatus according to claim 12, wherein the processing unit inputs the user terminal's operation screen and a new instruction prompt that instructs a process including a new operation on the operation screen using the created module to the generated AI model and executes the instructed process.

14. The acquisition unit further accepts input of variables related to processing that includes new operations, The aforementioned processing unit, The information processing apparatus according to claim 13, which inputs the user terminal operation screen, the variables, and the new instruction prompt to the generating AI model and executes the instructed process.

15. The acquisition unit receives the video image, The information processing apparatus according to claim 1, wherein the preprocessing unit recognizes the divisions between operations in the video, and obtains text representing the operations and an image representing the operation screen at the timing of the operations based on the divisions between operations.

16. The preprocessing unit further includes a preprocessing unit that recognizes the boundaries between each operation in the document, The information processing apparatus according to claim 3, wherein the processing unit inputs the document and a prompt instructing the AI ​​model to execute the process represented by the document, based on the division for each operation, and executes the process.

17. An acquisition unit that receives a video or a series of still images representing a process that includes performing at least one type of operation on the user terminal's operation screen, and explanatory text describing the process that includes the at least one type of operation, A processing unit that inputs the received video or multiple still images, the explanatory text, and a prompt instructing the AI ​​model to create a document using the text or images that describe the process represented by the video, to a generating AI model to create the document, Information processing device including

18. The acquisition unit receives explanatory text describing a process that includes performing at least one type of operation after executing the content of the operation. The processing unit acquires text representing the operation corresponding to the received explanatory text, and an image representing the operation screen at the timing of the operation. The information processing apparatus according to claim 1, which inputs text representing the operation, an image representing the operation screen at the timing of the operation, and the operation screen of a new user terminal into the generating AI model and executes the content of the operation.

19. A video, multiple still images, or operation details representing a process that includes performing at least one type of operation on the user terminal's operation screen. Based on the received video, multiple still images, or operation details, text representing the operation and an image representing the operation screen at the timing of the operation are obtained. The text representing the operation, the image representing the operation screen at the timing of the operation, and the operation screen of the new user terminal are input into the generating AI model to execute the content of the operation. A method of information processing performed by a computer.

20. We accept documents that describe a process involving performing at least one type of operation on the user terminal's operation screen, using text or images. The received document and the user terminal's operation screen are input into the generated AI model, and the process is executed. A method of information processing performed by a computer.

21. The system accepts pairs of the description of a process on the user terminal's operation screen and the execution details of the said process. The received pair, the user terminal's operation screen, and a prompt instructing a process that includes performing at least one type of operation on the operation screen are input to the generating AI model, and the instructed process is executed. A method of information processing performed by a computer.

22. The system receives a video or a series of still images representing a process that includes performing at least one type of operation on the user terminal's operation screen, and explanatory text describing the process that includes the at least one type of operation. The received video or multiple still images, the explanatory text, and a prompt instructing the AI ​​model to create a document using text or images that describe the process represented by the video are input to the generating AI model to create the document. A method of information processing performed by a computer.

23. An information processing program for causing a computer to function as an information processing device according to any one of claims 1 to 18.

Images