Programs, methods, information processing devices, systems

Abstract

There is a challenge in that it does not provide support for operating the user's terminal. [Solution] A program to be executed by a computer having a processor and a memory unit, wherein the processor executes a screen acquisition step of acquiring screen information displayed on the screen of a user's user terminal, a question acquisition step of acquiring question information entered via the user terminal, a creation step of creating a prompt by including the screen information acquired in the screen acquisition step and the question information acquired in the question acquisition step in an instruction to output information that assists in the operation of the user terminal, an input step of inputting the prompt created in the creation step to an artificial intelligence system, and an output acquisition step of acquiring the output content output by the artificial intelligence system in response to the prompt.

Description

Technical Field

[0001] This disclosure relates to a program, a method, an information processing apparatus, and a system.

Background Art

[0002] There is known a system (chatbot system) that outputs response content corresponding to a query such as an online query from a user. Patent Document 1 discloses a technique that enables seamless transfer to a responder when a chatbot automatically responds to a query from a user and is transferred to a human response.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a problem that the operation of the user terminal cannot be supported. Therefore, this disclosure has been made to solve the above problems, and its object is to support the operation of the user terminal.

Means for Solving the Problems

[0005] A program to be executed by a computer having a processor and a memory unit, wherein the processor executes a screen acquisition step of acquiring screen information displayed on the screen of a user's user terminal, a question acquisition step of acquiring question information entered via the user terminal, a creation step of creating a prompt by including the screen information acquired in the screen acquisition step and the question information acquired in the question acquisition step in an instruction to output information that assists in the operation of the user terminal, an input step of inputting the prompt created in the creation step to an artificial intelligence system, and an output acquisition step of acquiring the output content output by the artificial intelligence system in response to the prompt. [Effects of the Invention]

[0006] According to this disclosure, it is possible to support the operation of the user's terminal. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing the functional configuration of System 1. [Figure 2] This block diagram shows the functional configuration of Server 10. [Figure 3] This is a block diagram showing the functional configuration of user terminal 20. [Figure 4] This diagram shows the data structure of user table 1012. [Figure 5] This diagram shows the data structure of question table 1013. [Figure 6] This diagram shows the data structure of answer table 1014. [Figure 7] This is a flowchart showing the operation of the support process. [Figure 8] This diagram shows the functionality of a multimodal large-scale language model. [Figure 9] A block diagram showing the basic hardware configuration of Computer 90. [Modes for carrying out the invention]

[0008] The embodiments of this disclosure will be described below with reference to the drawings. In all the drawings illustrating the embodiments, common components are denoted by the same reference numerals, and repeated explanations are omitted. The following embodiments are not intended to unduly limit the content of this disclosure as described in the claims. Not all components shown in the embodiments are necessarily essential components of this disclosure. Also, each drawing is a schematic diagram and is not necessarily a strict illustration.

[0009] <System 1 Configuration> System 1 in this disclosure is an information processing system that outputs information such as strings, images, videos, audio, and music related to the answer content in response to a question received from a user. System 1 comprises a server 10, a user terminal 20, and an information processing device for generating AI 50, all connected via network N. Figure 1 is a block diagram showing the functional configuration of System 1. Figure 2 is a block diagram showing the functional configuration of server 10. Figure 3 is a block diagram showing the functional configuration of the user terminal 20.

[0010] Each information processing device consists of a computer equipped with an arithmetic unit and a memory device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by said hardware configuration will be described later. For each of the server 10, user terminal 20, and generation AI 50, explanations that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0011] <Server 10 Configuration> Server 10 is an information processing device that provides an information processing service that outputs information such as strings, images, videos, audio, and music related to the answer content in response to a question received from a user. The server 10 includes a storage unit 101 and a control unit 104.

[0012] <Configuration of the storage unit 101 of server 10> The storage unit 101 of server 10 includes an application program 1011, a user table 1012, a question table 1013, and an answer table 1014.

[0013] The user table 1012 is a table that stores and manages information of member users (hereinafter referred to as users) who use the service. By registering for use of the service, the information of the user is stored in a new record of the user table 1012. Thereby, the user can use the service according to the present disclosure. The user table 1012 is a table having columns of user ID, user name, user data, operation log, and browsing log with the user ID as the primary key. FIG. 4 is a diagram showing the data structure of the user table 1012.

[0014] The user ID is an item for storing user identification information for identifying a user. The user identification information is an item for which a unique value is set for each user. The user name is an item for storing the name of the user. The user name may be set to any string such as a nickname instead of the name. The user data includes the unique information of the user himself / herself and the attribute information regarding the characteristics and background of the user. The unique information of the user includes the unique information of the user such as the date of birth (age) and gender of the user. The attribute information of the user includes information such as the educational background (highest educational level, major, graduation year), occupation, work history, interests / concerns, place of residence, and language of the user. The operation log is an item for storing the operation log of the user with respect to the input device 206 of the user terminal 20. Specifically, the operation log includes touch operations, taps, swipes, pinch-in / pinch-out, flicks, clicks, drag-and-drops, wheel operations, scrolls, drags, slides, hovers, other multi-touch gestures, keyboard operations, and other arbitrary input operations on the input device 206 of the user terminal 20, performed by the user on the touch panel, mouse, keyboard, etc. of the user terminal 20. The operation log may include image data and video data of the screen output from the display 2081 of the user terminal 20. In other words, it may include image data and video data showing the results of the user's operation of the input device 206 of the user terminal 20, as displayed on the display 2081, when the user operates the input device 206 of the user terminal 20. The operation log includes browsing history of websites viewed by the user via a web browser, etc., on the user terminal 20. The browsing log is an item that stores information about websites viewed in the browser application of the user terminal 20. Specifically, the browsing log includes text information written in markup languages ​​such as HTML (HyperText Markup Language) and CSS used to describe websites, as well as information such as images and videos.

[0015] Question table 1013 is a table for storing and managing information related to questions (question information). Question table 1013 is a table with Question ID as its primary key, and contains columns for Question ID, Parent Question ID, User ID, Question Text, and Date / Time. Figure 5 shows the data structure of question table 1013.

[0016] The Question ID is an item that stores question identification information to identify a question. The question identification information is an item that has a unique value assigned to each piece of question information. The Parent Question ID is an item that stores parent question identification information to identify the parent question. The User ID is an item that stores user identification information used to identify a user. The question text is an item that stores the string information (text information) related to the question text. The date and time field stores the date and time when a new record was added or updated in question table 1013.

[0017] Answer table 1014 is a table for storing and managing information related to the answers (answer information). Answer table 1014 is a table that has columns for question ID, prompt, answer content, and date and time. Figure 6 shows the data structure of the answer table 1014.

[0018] The Question ID is an item that stores question identification information to identify a question. The prompt is an item that stores string information (text information) related to the prompt (input text) entered into the generating AI50. The response content is an item that stores string information (text information), image information, video information, audio information, music information, etc., related to the output text output from the generating AI 50. In this disclosure, as an example, an example in which text information related to a string is output from the generating AI 50 is described, but it is not limited to this; for example, the configuration may also include outputting image, video, audio, music data, etc., from the generating AI 50. In this disclosure, the response content includes information that supports input operations on the user's user terminal 20. The date and time field stores the date and time when a new record is added or updated in the response table 1016.

[0019] <Configuration of the control unit 104 of server 10> The control unit 104 of the server 10 includes a user registration control unit 1041. The control unit 104 realizes each functional unit by executing the application program 1011 stored in the storage unit 101.

[0020] The user registration control unit 1041 processes information of users who wish to use the services related to this disclosure and stores it in the user table 1012. Information stored in the user table 1012 is obtained when a user opens a web page operated by the service provider from any information processing terminal, enters information into a designated input form, and sends it to the server 10. The user registration control unit 1041 stores the received information in a new record in the user table 1012, and user registration is completed. As a result, users stored in the user table 1012 can use the service. Prior to the registration of user information in the user table 1012 by the user registration control unit 1041, the service provider may perform a prescribed review and restrict whether or not the user can use the service. The user ID can be any string or number that can identify the user, and may be any string or number that the user wishes, or the user registration control unit 1041 may automatically set any string or number.

[0021] <Configuration of User Terminal 20> The user terminal 20 is an information processing device operated by a user of the service. The user terminal 20 may be, for example, a mobile device such as a smartphone or tablet, or a stationary PC (Personal Computer) or laptop PC. It may also be a wearable device such as an HMD (Head Mount Display) or a smartwatch. The user terminal 20 includes a storage unit 201, a control unit 204, an input device 206, and an output device 208.

[0022] <Configuration of the storage unit 201 of the user terminal 20> The storage unit 201 of the user terminal 20 includes a user ID 2011 and an application program 2012.

[0023] User ID 2011 is the user's account ID. The user sends User ID 2011 from user terminal 20 to server 10. Server 10 identifies the user based on User ID 2011 and provides the services related to this disclosure to the user. User ID 2011 includes information such as a session ID that is temporarily assigned by server 10 to identify the user using user terminal 20.

[0024] The application program 2012 may be pre-stored in the memory unit 201, or it may be configured to be downloaded from a web server operated by the service provider via a communication interface. Application Program 2012 includes applications such as web browser applications. Application program 2012 includes an interpreted programming language such as JavaScript (registered trademark) that runs on a web browser application stored on the user terminal 20.

[0025] <Configuration of the control unit 204 of the user terminal 20> The control unit 204 of the user terminal 20 comprises an input control unit 2041 and an output control unit 2042. The control unit 204 realizes each functional unit by executing the application program 2012 stored in the memory unit 201.

[0026] <Configuration of the input device 206 of the user terminal 20> The input device 206 of the user terminal 20 includes a camera 2061, a microphone 2062, a position information sensor 2063, a motion sensor 2064, and a touch device 2065.

[0027] <Configuration of the output device 208 of the user terminal 20> The output device 208 of the user terminal 20 includes a display 2081 and a speaker 2082.

[0028] <Configuration of Generation AI50> Generative AI50 refers to large-scale artificial intelligence models used in the field of natural language processing (NLP). These models learn from large amounts of text data (web pages, books, articles, etc.) to understand patterns in human language and effectively perform natural language generation (NLG) tasks. Generative AI50 is used in many NLP tasks, such as generating responses to specific questions, automatic text generation, text summarization, translation, and sentiment analysis. It can also be used in a variety of applications, including education, entertainment, customer service, and product development. Generative AI50 includes the following types. In this disclosure, a large-scale language model that primarily outputs text information as output information is described as a type of Generative AI50. As shown in Figure 8, the large-scale language model in this disclosure encodes (stringifies) features of text, image, and audio information using a tokenizer, and by inputting these strings into the language model, it is possible to output decoded strings into text, image, and audio information. OpenAI ChatGPT Anthropic Claude Google Gemini Stable Diffusion Midjourney Furthermore, the generation AI50 may be implemented as part of the functions of server 10.

[0029] <System 1 operation> The following describes each process in System 1. Figure 7 is a flowchart showing the operation of the support process.

[0030] <Support Processing> Support processing is a process designed to assist the user in operating the user terminal 20.

[0031] <Overview of support processing> The support process is a series of operations that involves storing an operation log related to the user's input operations on the input device 206 of the user terminal 20, receiving instructions from the user requesting assistance with operations, generating a prompt according to the received instruction, sending the generated prompt to the generation AI 50, receiving the response output from the generation AI 50, and presenting the received response to the user. The operation log includes information about images and videos of the screen displayed on the display 2081 of the user terminal 20. In this disclosure, a configuration in which information is transmitted from the user terminal 20 to the generating AI 50 via the server 10 is disclosed as an example, but the disclosure is not limited to this. For example, information may be transmitted directly from the user terminal 20 to the generating AI 50.

[0032] <Details of support processing> The details of the support process are explained below.

[0033] <Screen acquisition steps> In step S101, the processing unit 1042 of the server 10 executes a screen acquisition step to acquire screen information displayed on the screen of the user's user terminal. Specifically, the user performs input operations on the input device 206 of the user terminal 20. For example, the user performs input operations on the input device 206 of the user terminal 20 and runs a web browser application or the like to browse any website. For example, the user can manipulate objects such as text fields and buttons contained in web forms displayed on the website by operating the input device 206 of the user terminal 20. Objects include text fields, password fields, radio buttons, checkboxes, dropdown lists, date selection fields, time selection fields, file upload fields, text areas, submit buttons, reset buttons, number fields, sliders, switches, and more. In this disclosure, an account opening form on a financial institution's website will be used as an example. The account opening form includes input fields (objects) for entering information such as name, phonetic spelling of name, address, and telephone number. The control unit 204 of the user terminal 20 transmits screen information, including images or videos, displayed on the user terminal 20's display 2081, to the server 10. For example, it transmits screen information, including images or videos, showing the user's input operations on the user terminal 20's input device 206, to the server 10. The control unit 204 of the user terminal 20 transmits the user ID 2011 to the server 10. The control unit 204 of the user terminal 20 may also transmit information such as browsing history stored in the user terminal 20's storage unit 201 (including URLs of websites viewed with the web browser, viewing date and time, and other website screen information such as HTML and images). In addition, it may transmit screen information, including images or videos, showing the user's input operations on websites (web pages) viewed before the website (web page) the user is currently viewing. The control unit 104 of server 10 receives and accepts user ID 2011 and screen information from user terminal 20. The control unit 104 of server 10 stores the received screen information, browsing history, and other information in the operation log and browsing log items of the record identified based on user ID 2011 in user table 1012. The control unit 104 of server 10 stores the acquired screen information, browsing history, etc., in association with user ID 2011.

[0034] In step S101, the screen acquisition step performs the step of acquiring screen information that includes information identifying one or more objects displayed on the screen of the user terminal. The screen acquisition step performs the step of acquiring screen information displayed in the web browser of the user terminal. Specifically, the screen information may include information that identifies objects displayed on the user terminal 20's display 2081. For example, object identification information may include object information such as ID attributes, name attributes, class attributes, type attributes, placeholders, labels, values, ARIA labels, tab indexes, data attributes, style attributes, HTML tag names, text content, and relationships with parent and child elements in the website's HTML (HyperText Markup Language). The screen information may also include information that identifies objects (operation information) of drawing objects in the application running on the user terminal 20. In addition, it may include information that identifies any other object displayed on the user terminal 20's display 2081.

[0035] In this disclosure, the log storage step in step S101 may be configured to be executed independently of the processing from step S102 onward. Furthermore, the log storage step in step S101 may be executed without any explicit input operation by the user. In other words, information such as input operations by the user on the input device 206 of the user terminal 20, screen information, and browsing history may be continuously transmitted to the server 10 and stored in the operation log and browsing log items of the user table 1012 of the server 10. Furthermore, information such as input operations and screen information on the input device 206 of the user terminal 20, as well as browsing history, may be stored in the storage unit 201 of the user terminal 20, and this stored information may be periodically or in response to predetermined input operations by the user and stored in the operation log and browsing log items of the user table 1012 of the server 10.

[0036] <Steps to obtain questions> In step S102, the processing unit 1042 of the server 10 performs a question acquisition step to acquire question information entered via the user terminal. Specifically, the user operates the input device 206 of the user terminal 20 to open an input field for executing support processing. The display of the input field may be provided as a function of an application stored in the memory unit 201 of the user terminal 20, or as a function of a website displayed on the display 2081 of the user terminal 20. Alternatively, the user terminal 20 may be configured to display the input field on the display 2081 of the user terminal 20 based on information received from the control unit 104 of the server 10. In this disclosure, in step S101, the input fields are displayed on the user terminal 20's display 2081 along with the screen information that the server 10 acquires. For example, the user displays the financial institution's website screen on the user terminal 20's display 2081, while input fields for entering question information are also displayed on the user terminal 20's display 2081. The user enters question information into the input field by operating the input device 206 of the user terminal 20. For example, when an input form for opening a financial institution account is displayed on the display 2081 of the user terminal 20, the user enters a string of characters indicating a question about the screen displayed on the display 2081 of the user terminal 20, such as "I cannot press the submit button, so I cannot proceed." The control unit 204 of the user terminal 20 sends the string of characters (question information) entered in the input field to the server 10. The control unit 104 of the server 10 receives and accepts the question information. Please note that the question information does not necessarily have to be in the form of text; it may also include images, audio, and other types of information.

[0037] <Prompt generation step> In step S103, the processing unit 1042 of the server 10 executes a creation step to create a prompt by including the screen information acquired in the screen acquisition step and the question information acquired in the question acquisition step in an instruction to output information that supports the operation of the user terminal. Specifically, the control unit 104 of server 10 searches the user ID field in user table 1012 based on user ID 2011 and obtains the operation log and browsing log fields. Alternatively, the control unit 104 of server 10 may be configured to obtain operation logs and browsing logs from a predetermined period prior (10 seconds, 1 minute, 5 minutes prior) to the time the question information was received in step S102. This allows obtaining the most recent operation logs and browsing logs when the user submits the question information. The control unit 104 of the server 10 creates a prompt by including at least some of the information from the operation log and browsing log in the following support instruction statement (first embodiment). Specifically, the question information received in step S102 is inserted in the ${question statement} part of the support instruction statement (first embodiment), and at least some of the acquired operation log and browsing log is inserted in the ${log information} part. Image data, video data, audio data, etc., included in the operation log and browsing log are encoded into a string in a format such as base64 and inserted into the instruction statement. For example, a string encoded from screen information is inserted in the ${log information} part. In addition, this information may be included as an attachment to the support instruction statement (first embodiment). <Support Instructions (First Embodiment)> Based on the following question and the user's actions, please propose actions to resolve the user's question. #Question ${Question} #Operation details ${log information} #Output format Output images that guide the user through the necessary screen operations to answer their questions.

[0038] In step S103, the creation step performs the step of creating a prompt by including in the instructions a question about one or more objects. The creation step performs the step of creating a prompt by including in the instructions screen information displayed in the web browser and information about the web browser's operation log. Specifically, the control unit 104 of the server 10 inserts operation information, including object information such as ID attribute, name attribute, class attribute, type attribute, placeholder, label, value, ARIA label, tab index, data attribute, style attribute, HTML tag name, text content, and relationships with parent and child elements in the HTML (HyperText Markup Language) of the website, into the ${log information} section of the support instruction statement (second embodiment). For example, information that identifies an object displayed on the display 2081 of the user terminal 20, which was operated on by the user terminal 20 using the input device 206 of the user terminal 20, and information indicating the operation content (operation information) may be inserted. For example, the operation information may include information indicating that the string "Taro Patent" was entered into the name text field (object). The support statement (second embodiment) may include instructions such as, "Please output a program that shows the operation on the object element included in the operation content that is the answer to the user's question," and may also be an instruction statement that outputs a program for performing an operation on an object displayed on the screen.

[0039] <Support Instructions (Second Embodiment)> Based on the following question and the user's actions, please propose actions to resolve the user's question. #Question ${Question} #Operation details ${log information} #Output format Please output the steps for operating on the object elements included in the operation described in the user's question.

[0040] <Generating AI Input Steps> In step S104, the processing unit 1042 of the server 10 performs an input step in which it inputs the prompt created in the creation step to the artificial intelligence system. Specifically, the control unit 104 of the server 10 sends the prompt created in step S103 to the artificial intelligence system 50. The artificial intelligence system 50 accepts the input of the prompt received from the server 10.

[0041] <Generated AI Output Step> In step S105, the processing unit 1042 of the server 10 executes an output acquisition step to acquire the output content output by the artificial intelligence system in response to the prompt. Specifically, the artificial intelligence system 50 outputs a response to the server 10 in response to the prompt input in step S104. The response includes information such as a string, image, or audio indicating the answer to the prompt. The control unit 104 of the server 10 receives and acquires the output from the artificial intelligence system 50 in response to the prompt.

[0042] For example, in response to a prompt input based on a support instruction statement (first embodiment), the artificial intelligence system 50 outputs information such as images or videos, audio, etc., indicating screen operations on the display 2081 of the user terminal 20 to the server 10.

[0043] For example, in response to a prompt input based on a support instruction statement (second embodiment), the artificial intelligence system 50 outputs a string of characters to the server 10 indicating the operation procedure for the drawing object drawn on the display 2081 of the user terminal 20. For example, the system outputs a string to server 10 such as, "The input field for the phonetic spelling of your name is blank. Please enter the phonetic spelling in the input field and then select the submit button." The output of the artificial intelligence system 50 may also include object identification information (ID attribute, name attribute, class attribute, type attribute, etc.) to identify drawing objects such as "phonetic spelling input field" and "button."

[0044] <Support Information Output Step> In step S106, the processing unit 1042 of the server 10 executes a support output step that outputs information on the screen to support the user's operation of the user terminal, based on the output content acquired in the output acquisition step. Specifically, the control unit 104 of the server 10 transmits the output content received from the artificial intelligence system 50 to the user terminal 20. The control unit 204 of the user terminal 20 outputs the received output content from the output device 208 of the user terminal 20. For example, the control unit 204 of the user terminal 20 displays the received image or video on the display 2081 of the user terminal 20 and presents it to the user. Also, the control unit 204 of the user terminal 20 outputs the received audio to the speaker 2082 of the user terminal 20 and presents it to the user. For example, the control unit 204 of the user terminal 20 displays the received string on the display 2081 of the user terminal 20 and presents it to the user. Alternatively, the control unit 204 of the user terminal 20 may read the received string aloud from the speaker 2082 of the user terminal 20.

[0045] In step S106, the support output step performs the step of displaying one or more objects displayed on the user terminal screen, which are identified based on the output content, in an identifiable manner. Specifically, the control unit 204 of the user terminal 20 highlights drawing objects that are identified based on object identification information included in the output from the artificial intelligence system 50, among the drawing objects displayed on the user terminal 20's display 2081, by applying drawing effects. For example, the control unit 204 of the user terminal 20 may draw the color of the drawing object identified based on the object identification information in yellow, red, etc., make it blink, make its border red or thicker, or increase the size of the drawing object. This allows the user to easily see the output of the artificial intelligence system 50 along with the drawing object being manipulated. For example, the user can easily notice that the "Name in katakana input field" is not filled in.

[0046] In step S106, the control unit 204 of the user terminal 20 performs an operation support step to operate the user terminal without accepting any input operations from the user, based on the output content acquired in the output acquisition step. The operation support step performs an operation on one or more objects displayed on the screen of the user terminal, which are identified based on the output content acquired in the output acquisition step. The operation support step performs an operation on one or more objects displayed on the screen of the user terminal based on the script included in the output content acquired in the output acquisition step. Specifically, the control unit 204 of the user terminal 20 may perform operations based on the output content output from the artificial intelligence system 50 without receiving any input operations from the user to the input device 206 of the user terminal 20. Specifically, the control unit 204 of the user terminal 20 performs input operations to the input device 206 of the user terminal 20 based on images, videos, audio, text, etc., included in the output content of the artificial intelligence system 50. As an example, the input device 206 of the user terminal 20 performs processing according to the output content for drawing objects identified based on object identification information included in the output content. For example, the control unit 204 of the user terminal 20 refers to the drawing object for the "Name" field (e.g., Taro Tokkyo) for the drawing object for the "Input field for phonetic spelling of name," identifies the string "Tokkyo Taro," and presses the drawing object corresponding to the "Send" button. As a result, the user can complete the desired operation by only entering the question in step S102, without having to operate the input device 206 of the user terminal 20. Specific input operations on drawing objects based on the output content may be implemented by creating a prompt by inputting the output content into an instruction statement that instructs input operations on the drawing object, and then executing the response content (including scripts such as programs) that is output in response to inputting the prompt into the artificial intelligence system 50. For example, the string "Tokkyo Tarou" may be included in the output content from the AI ​​50 that generates the drawing object in the "Name" field (e.g., Tarou Patent) as input data. The configuration may also be such that the string "Tokkyo Tarou" is entered into the furigana input field by executing the script related to the response content. The scripts included in the answer should contain command statements that define the operations to be performed on each drawing object. For example, this should include scripting languages ​​such as JavaScript used to command events, actions, and other operations on drawing objects.

[0047] <Basic Computer Hardware Configuration> Figure 9 is a block diagram showing the basic hardware configuration of computer 90. Computer 90 includes at least a processor 901, main memory 902, auxiliary memory 903, and a communication interface 991. These are electrically connected to each other by a communication bus 921.

[0048] The processor 901 is hardware for executing the instruction set written in a program. The processor 901 consists of an arithmetic unit, registers, peripheral circuits, etc.

[0049] Main memory 902 is used to temporarily store programs and data processed by programs, etc. For example, it is a volatile memory such as DRAM (Dynamic Random Access Memory).

[0050] Auxiliary storage device 903 refers to a storage device for saving data and programs. Examples include flash memory, HDD (Hard Disc Drive), magneto-optical disk, CD-ROM, DVD-ROM, and semiconductor memory.

[0051] The IF991 communication interface is an interface for inputting and outputting signals for communication with other computers via a network using wired or wireless communication standards. A network consists of various mobile communication systems, such as the internet, LANs, and wireless base stations. For example, a network includes 3G, 4G, and 5G mobile communication systems, LTE (Long Term Evolution), and wireless networks that can connect to the internet via designated access points (e.g., Wi-Fi®). When connecting wirelessly, communication protocols include, for example, Z-Wave®, ZigBee®, and Bluetooth®. When connecting via a wired connection, the network also includes connections made directly via USB (Universal Serial Bus) cables, etc.

[0052] Furthermore, by distributing all or part of each hardware configuration across multiple computers 90 and connecting them to each other via a network, a computer 90 can be virtually realized. Thus, the concept of computer 90 includes not only a computer 90 housed in a single enclosure or case, but also a virtualized computer system.

[0053] <Basic Functional Configuration of Computer 90> The functional configuration of the computer realized by the basic hardware configuration of computer 90 (Figure 9) will be explained. The computer comprises at least one functional unit: a control unit, a memory unit, and a communication unit.

[0054] Furthermore, the functional units of computer 90 can also be realized by distributing all or part of each functional unit across multiple computers 90 interconnected via a network. The concept of computer 90 includes not only a single computer 90 but also a virtualized computer system.

[0055] The control unit is realized when the processor 901 reads various programs stored in the auxiliary storage device 903, loads them into the main memory device 902, and executes processing according to those programs. The control unit can realize various functional units that perform information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0056] The functions realized by the components described herein may be implemented in a circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to realize the functions described herein. A processor includes transistors and other circuits and is considered a circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, and means are hardware programmed to perform or execute the functions described herein. Such hardware may be any hardware disclosed herein, or any hardware known to be programmed to perform or execute the functions described herein. If the hardware is a processor that is considered to be a type of circuitry, then the circuitry, means, or unit is a combination of hardware and software used to constitute the hardware and / or processor.

[0057] The memory unit is implemented by the main memory 902 and the auxiliary memory 903. The memory unit stores data, various programs, and various databases. The processor 901 can also reserve memory areas corresponding to the memory unit in the main memory 902 or the auxiliary memory 903 according to the program. The control unit can also cause the processor 901 to perform operations such as adding, updating, and deleting data stored in the memory unit according to the various programs.

[0058] A database, specifically a relational database, is used to manage and link together tabular data sets called masters, which are structurally defined by rows and columns. In a database, tables are called tables, masters are called masters, the columns of tables are called columns, and the rows of tables are called records. In a relational database, relationships can be established and linked between tables and masters. Typically, each table and master has a primary key column to uniquely identify records, but setting a primary key column is not mandatory. The control unit can instruct the processor 901 to add, delete, or update records in specific tables and masters stored in the memory unit, according to various programs. Furthermore, by storing data, various programs, and various databases in the memory unit, the information processing device and information processing system related to this disclosure can be considered to have been manufactured.

[0059] Furthermore, the databases and masters in this disclosure may include any data structures (lists, dictionaries, associative arrays, objects, etc.) in which information is structurally defined. Data structures also include data that can be considered as data structures by combining data with functions, classes, methods, etc., written in any programming language.

[0060] The communication unit is implemented by the communication IF991. The communication unit provides the functionality to communicate with other computers 90 via the network. The communication unit can receive information transmitted from other computers 90 and input it to the control unit. The control unit can cause the processor 901 to perform information processing on the received information according to various programs. The communication unit can also transmit information output from the control unit to other computers 90.

[0061] <Note> The details described in each of the above embodiments are noted below.

[0062] (Note 1) A program to be executed by a computer having a processor and a memory unit, wherein the processor executes a screen acquisition step (S101) to acquire screen information displayed on the screen of a user's user terminal, a question acquisition step (S102) to acquire question information entered via the user terminal, a creation step (S103) to create a prompt by including the screen information acquired in the screen acquisition step and the question information acquired in the question acquisition step in an instruction to output information that assists in the operation of the user terminal, an input step (S104) to input the prompt created in the creation step to an artificial intelligence system, and an output acquisition step (S105) to acquire the output content output by the artificial intelligence system in response to the prompt. This allows users to obtain information that assists in operating their device based on the screen information displayed on the user's device.

[0063] (Note 2) The question information includes questions about the content displayed on the user's terminal screen, as described in Appendix 1. This allows information to be obtained to assist in operating the user's device based on questions the user asks regarding the content displayed on the user's device screen.

[0064] (Note 3) The program as described in Appendix 1, wherein the screen acquisition step (S101) is a step of acquiring screen information that includes information identifying one or more objects displayed on the screen of the user terminal, and the creation step (S103) is a step of creating a prompt by including a question about one or more objects in the instruction. This allows users to obtain information that assists in operating their device based on one or more objects displayed on the screen of their device. For example, users can obtain specific assistance information such as instructions for operating buttons or text fields.

[0065] (Note 4) The program described in Appendix 1, wherein the screen acquisition step (S101) is a step of acquiring screen information displayed on the user terminal's web browser, and the creation step (S103) is a step of creating a prompt by including the screen information displayed on the web browser and information related to the web browser's operation log in the instructions. This allows users to obtain information to assist in operating their device based on the information displayed in their device's web browser. Furthermore, more suitable assistance information can be obtained based on information related to operation logs, such as browsing history, in the user's device's web browser.

[0066] (Note 5) The program described in Appendix 1, which executes an support output step (S106) that outputs information on the screen to support the user's operation of the user terminal based on the output content acquired in the output acquisition step. This allows users to obtain information that assists in operating their device based on the screen information displayed on the user's device.

[0067] (Note 6) The program described in Appendix 5 is a support output step (S106) which is a step of displaying one or more objects displayed on the user terminal screen in an identifiable manner, based on the output content. This allows users to receive information that assists them in operating their device.

[0068] (Note 7) The program described in Appendix 1, which performs a support operation step (S106) to operate the user terminal without accepting input operations from the user to the user terminal, based on the output content acquired in the output acquisition step. This allows the user to recognize the object being manipulated in an identifiable manner. The user can then operate the user terminal effectively.

[0069] (Note 8) The program described in Appendix 7, wherein the support operation step (S106) is a step of performing an operation on one or more objects displayed on the user terminal screen, which are identified based on the output content acquired in the output acquisition step. This allows users to operate their terminal automatically without having to directly input anything into the terminal.

[0070] (Note 9) The support operation step (S106) is a step in which the program described in Appendix 7 performs an operation on one or more objects displayed on the user terminal screen based on the script included in the output content acquired in the output acquisition step. This allows users to automatically perform operations on one or more objects displayed on their user terminal without having to directly input anything into the terminal.

[0071] (Note 10) A method to be performed on an information processing apparatus comprising a processor and a memory unit, wherein the processor performs all steps performed in any of the inventions described in Appendix 1 to Appendix 9. This allows users to obtain information that assists in operating their device based on the screen information displayed on the user's device.

[0072] (Note 11) An information processing apparatus comprising a processor and a memory unit, wherein the processor performs all steps performed in any of the inventions described in Appendix 1 to Appendix 9. This allows users to obtain information that assists in operating their device based on the screen information displayed on the user's device.

[0073] (Note 12) A system comprising means for performing all steps performed in any of the inventions described in Appendix 1 to Appendix 9. This allows users to obtain information that assists in operating their device based on the screen information displayed on the user's device. [Explanation of symbols]

[0074] 1 System, 10 Servers, 101 Memory Unit, 104 Control Unit, 106 Input Device, 108 Output Device, 20 User Terminals, 201 Memory Unit, 204 Control Unit, 206 Input Device, 208 Output Device, 50 Generating AI, 501 Memory Unit, 504 Control Unit, 506 Input Device, 508 Output Device

Claims

1. A program to be executed by a computer having a processor and a memory unit, The aforementioned processor, A screen acquisition step that obtains screen information displayed on the user's terminal screen, A question acquisition step of acquiring question information entered via the user terminal, A creation step to create a prompt by including the screen information obtained in the screen acquisition step and the question information obtained in the question acquisition step in an instruction to output information that supports the operation of the user terminal, The input step involves inputting the prompt created in the creation step into the artificial intelligence system, An output acquisition step of acquiring the output content output by the artificial intelligence system in response to the prompt, A program that executes this task.

2. The aforementioned question information includes questions regarding the content displayed on the screen of the user terminal. The program according to claim 1.

3. The screen acquisition step is a step of acquiring screen information that includes information identifying one or more objects displayed on the screen of the user terminal, The creation step is to create a prompt by including in the instructions a question about the one or more objects, The program according to claim 1.

4. The screen acquisition step is a step of acquiring the screen information displayed on the web browser of the user terminal, The creation step is to create the prompt by including the screen information displayed in the web browser and information regarding the operation log of the web browser in the instruction. The program according to claim 1.

5. A support output step which outputs information on the screen that supports the user's operation of the user terminal based on the output content acquired in the output acquisition step, Execute The program according to claim 1.

6. The support output step is a step of displaying one or more objects displayed on the user terminal screen, which are identified based on the output content, in an identifiable manner. The program according to claim 5.

7. Based on the output content acquired in the output acquisition step, a support operation step is performed to operate the user terminal without accepting input operations from the user to the user terminal, Execute The program according to claim 1.

8. The support operation step is a step of performing an operation on one or more objects displayed on the screen of the user terminal, which are identified based on the output content acquired in the output acquisition step. The program according to claim 7.

9. The support operation step is a step of performing an operation on one or more objects displayed on the user terminal screen based on the script included in the output content acquired in the output acquisition step. The program according to claim 7.

10. A method to be performed on an information processing apparatus comprising a processor and a storage unit, wherein the processor performs all steps performed in the invention according to any one of claims 1 to 9.

11. An information processing apparatus comprising a processor and a storage unit, wherein the processor performs all steps performed in the invention according to any one of claims 1 to 9.

12. A system comprising means for performing all steps performed in the invention according to any one of claims 1 to 9.

Images