Information processing equipment, terminal equipment, information processing system, program

Abstract

This invention provides an information processing device, terminal device, information processing system, and program that extract information from image data without requiring the prior preparation of logic for extracting information from image data. [Solution] In the information processing system 100, the information processing device 40 is a screen that receives information input from a user into input fields for input items displayed on a terminal device 10, and is capable of receiving instructions to extract information to be entered into the input fields from image data, and includes a registration unit 43 that registers the entered information, a request transmission unit 46b that, when an instruction is received on the screen, sends a request to the generation AI system including image data to be extracted and an instruction to extract information corresponding to the input items, and a transmission unit 41b that transmits the information to the terminal device so that the terminal device can display a screen in which the information corresponding to the input items transmitted from the generation AI system has been entered into the input fields of the corresponding input items on the screen.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a terminal device, an information processing system, and a program.

Background Art

[0002] There is known a technique of analyzing image data using AI and outputting an analysis result. An information processing apparatus equipped with AI can perform output of things shown in image data, classification of image data, anomaly detection, etc.

[0003] Patent Document 1 discloses a technique of extracting information such as a name and a company name from image data of a business card and registering it in a server.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, it was necessary to prepare in advance a logic for extracting a name, a company name, etc. from image data of a business card. For example, when a business card management application recognizes a character string by OCR (Optical Character Reader / Recognition), it is necessary for a developer to program in advance a logic for determining whether the character string is a name or a company name.

[0005] The present disclosure provides a technique for extracting information from image data without preparing in advance a logic for extracting information from the image data.

Means for Solving the Problems

[0006] In view of the above problems, the present invention provides an information processing device that can communicate with a terminal device and a generating AI (Artificial Intelligence) system via a network, comprising: a registration unit that registers information entered on a screen displayed on the terminal device, which accepts information input from a user into input fields for input items, and which is capable of receiving instructions to extract information to be entered into the input fields from image data; a request transmission unit that, when the instruction is received on the screen, transmits to the generating AI system a request including the image data to be extracted and an instruction to extract information corresponding to the input items from the image data; a response receiving unit that receives information corresponding to the input items from the generating AI system as a response to the request; and a transmission unit that transmits the information to the terminal device so that the terminal device can display a screen in which the information corresponding to the input items transmitted from the generating AI system has been entered into the corresponding input fields of the input items on the screen. [Effects of the Invention]

[0007] This invention allows information to be extracted from image data without having to prepare a logic for extracting information from image data in advance. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of an application creation screen displayed on a user's terminal. [Figure 2] This figure shows an example of the input screen of a business card management app displayed on a user's device. [Figure 3] This diagram illustrates the process by which a user enters values ​​into the input fields of a business card management app. [Figure 4] This diagram illustrates the flow of information setting services performed by an information processing system. [Figure 5] This is a diagram showing the system configuration of an example of an information processing system. [Figure 6] This diagram shows an example of the hardware configuration for an application service, a user terminal, and a developer terminal. [Figure 7] This diagram shows an example of the functional configuration of an application service and a user terminal. [Figure 8] This figure shows an example of the information set in the application's input fields. [Figure 9] This figure shows an example of information regarding the input fields of an application. [Figure 10] This is an example sequence diagram illustrating the processing of an information processing system. [Figure 11] Figure 10 shows an example of parameters included in the request message that the application service sends to the AI ​​generation system in step S4. [Figure 12] This diagram explains the prompt that is set in "Prompt" in Figure 11. [Figure 13] Figure 10 shows an example of the format of a response message sent from the AI ​​generation system to the application service in step S5. [Figure 14] This diagram shows an example of an input screen for a business card management app displayed on a user's terminal. [Figure 15] This figure shows an example prompt generated by the request generation unit. [Figure 16] This figure shows an example of a response message from the generation AI system to the request message shown in Figure 15. [Figure 17] This figure shows an example of the input screen of a business card management app with values ​​already set. [Figure 18] This diagram shows an example of an input screen for a book management application displayed on a user's device. [Figure 19] This diagram shows information about the input fields in an example book management application. [Figure 20] This figure shows an example prompt generated by the request generation unit. [Figure 21] This figure shows an example of a response message from the generation AI system to the request message in Figure 20. [Figure 22]This is a diagram showing an input screen of a book management app in a state where values are set in the book management app. [Figure 23] This is a diagram explaining a method for setting values of input items using a function call function. [Figure 24] This is a diagram showing an example of a request message including function arguments. [Figure 25] This is a diagram showing an example of a response message from a generation AI system when the generation AI system has a function call function. [Figure 26] This is a diagram showing an example of an image data selection screen displayed on a part of the input screen of a business card management app or a popup, etc. [Figure 27] This is a diagram showing information about an example of an input item having an input range. [Figure 28] This is a diagram showing an example of a prompt requesting a value of an input item in JSON format. [Figure 29] This is a diagram showing an example of a request message when the generation AI system has a function call function. [Figure 30] This is a diagram showing an example of a prompt using few-shot prompting. [Figure 31] This is a sequence diagram explaining the processing of an information processing system when using few-shot prompting. [Figure 32] This is a diagram showing an example of a request message that an application service sends to a generation AI system. [Figure 33] This is a diagram showing an example of a character string set in the "prompt" of FIG. 32.

Mode for Carrying Out the Invention

[0009] Hereinafter, as an example of a mode for carrying out the present disclosure, an information processing system and a setting method performed by the information processing system will be described.

[0010] <Regarding the application service> Application services are services that assist users in creating applications using low-code or no-code methods. Such services are also known as visual programming. Application services send a web application that assists in application creation to the user's terminal. Users can then create various applications by interacting with the web application running on their terminal.

[0011] Figure 1 shows the application creation screen 200 displayed on the user terminal. The application creation screen 200, as an example, includes a form area 208 and a work area 209. The form area 208 displays a list of forms that can be placed in the application. A form is a display component that makes up a screen, and examples include input forms, selection forms, and file registration forms. Figure 1 shows a text form 201, a numeric form 202, a radio button form 203, a checkbox form 204, and an attachment form 205. These are just examples.

[0012] String form 201 is a form for entering text. Full-width characters, half-width characters, numbers, symbols, etc., can be entered into string form 201.

[0013] Numeric form 202 is a form for entering numerical values. Only numerical values ​​can be entered into Numeric form 202.

[0014] Radio button form 203 is a form for buttons that accept the selection of one option from multiple options. Radio button form 203 has fields for displaying the options.

[0015] Checkbox form 204 is a form for checkboxes that accepts the selection of one or more options from multiple choices. Checkbox form 204 has fields for displaying the options.

[0016] Attachment form 205 is a form that accepts settings for files to be registered in the application. Attachment form 205 may restrict the format of files that can be accepted for registration.

[0017] The workspace 209 is the area where the user places forms. The user can drag and drop forms from the form area 208 into the workspace 209 using the mouse pointer. Alternatively, the user can drag and drop forms into the workspace 209 using their finger or a pen on the touch panel.

[0018] The form placed in work area 209 is referred to as input field 206. Input field 206 has one or more input fields, but for simplicity, it may be simply referred to as input field below. Input field 206 can display a label 207. Label 207 is the name of input field 206. The user enters an appropriate label for input field 206. The user can repeat this process to create any application. For example, by placing a text form 201 in work area 209 and entering labels such as name and company, a business card management application like the one below can be created. Note that the position of an already placed input field 206 can also be changed in work area 209.

[0019] Figure 2 shows the input screen 210 of a business card management application displayed on a user's terminal. This business card management application has fields for name 211, company 212, department 213, job title 214, address 215, phone number 216, email address 217, URL 218, and business card image attachment 219. These are input fields. Users can use applications created in this way for business or personal use. For example, in a business card management application, the user enters the information written on business cards received from customers into each input field. This allows the business card management application to digitize the information. Alternatively, it becomes possible to share business card information within a team.

[0020] Figure 3 illustrates the process of a user entering values ​​into the input fields of a business card management application. Figure 3(a) shows the application list 220 displayed on the user's terminal. The application list 220 is a list of applications created and registered by the user in the application service using the procedure shown in Figure 1. As an example, the application list 220 displays the business card management application 221 and the book management application 222. Here, the user selected the business card management application 221. The application creation button 223 is a button that displays a screen like the one in Figure 1.

[0021] Figure 3(b) shows the record list 224 displayed by the user terminal. The record list 224 displays a table-like list of business card information registered in the business card management application 221. A record is a single row of data when data in a database is arranged in a two-dimensional table. One record represents one piece of business card information. A record 225 has multiple input fields. These input fields are sometimes called fields. A single column is called a column. In Figure 3(b), no business card information is registered, so the record list 224 is blank.

[0022] When the user presses the Add Record button 226, the input screen 210 of the business card management application is displayed. Figure 3(c) shows the input screen 210 of the business card management application. As described above, the input screen 210 of the business card management application has input fields created by the user for the business card management application 221. The user enters values ​​into the input fields and presses the Save button 227. The input content (in this case, business card information) is saved as a record in the application service.

[0023] Figure 3(d) shows the record list 224 displayed by the user terminal. The business card information entered on the input screen 210 of the business card management application in Figure 3(c) is displayed as one record.

[0024] Thus, since input is usually done using a keyboard, it placed a heavy burden on the user. Furthermore, there was a risk of input errors.

[0025] <Processing flow> In this embodiment, the user terminal sends the file (business card image data) entered in the attached file form 205 and a list of application input items to the application service, and the application service requests the generation AI system to generate values ​​to be set in the input items. The generation AI system can perform various natural language processing tasks such as text generation, question answering, text classification, sentiment analysis, information extraction, and text summarization. There are also known generation AI systems such as Copilot® that suggest the next code to write while coding a program. In this embodiment, we will describe a technique for extracting information from image data without the user having to converse (also called chatting) with the generation AI system. Note that the generation AI system or its function may sometimes simply be referred to as AI (Artificial Intelligence).

[0026] The generation AI system analyzes image data and a list of input fields, generates values ​​to be entered into the input fields, and sends them to the application service. The application service sends the generated values ​​to the user terminal. The generation AI system only needs to be provided as a separate system from the system that provides the application-related services, and the application-related services and the generation AI system may be provided by the same provider. The user terminal sets the generated values ​​in the application service. Users can create any application they use for business or other purposes, not just business card management applications, using the application service. Therefore, the information processing system of this embodiment can generate appropriate values ​​from image data for the input fields of any application and set them in the input fields of the application service.

[0027] Figure 4 illustrates the process by which the information processing system 100 sets values ​​in the application service 40. A web application is running on the user terminal 10. The application for which values ​​are to be set in the input fields is assumed to have already been created in the application service 40. The generation AI system 50 analyzes image data such as business cards and a list of input fields for the application to generate values ​​to be set in the input fields.

[0028] (1) The user terminal 10 displays the application's input screen. The user wants to set, for example, a business card image on the input screen, and have the remaining input fields automatically filled with values ​​generated by the generation AI system 50. By pressing a designated button, the user requests the application service 40 to perform AI image analysis input along with the image data.

[0029] (2) The application service 40 identifies a list of input fields and the application name of the application being run on the user terminal 10. The application name is optional, or instead of the application name, the application's description document (the application's description, which has been set in advance by the user) may be obtained. The application service 40 sends the application information (list of input fields, image data, and application name) to the generating AI system 50 and obtains the values ​​to be set for the input fields. That is, the generating AI system 50 analyzes the image data, the list of input fields, and the application name. The generating AI system 50 performs character recognition on the image data and generates feature quantities such as what kind of image data it is, and generates values ​​corresponding to the input fields.

[0030] (3) The application service 40 associates the values ​​obtained from the generation AI system 50 with the input fields and sends them to the user terminal 10. As a result, the user terminal 10 displays an input screen with values ​​set in the input fields, allowing the user to confirm the generated values. When the user presses a button such as register, the user terminal 10 sends the values ​​associated with the input fields to the application service 40, and the application service 40 sets the values ​​in the application record.

[0031] Thus, the information processing system 100 of this embodiment can set values ​​for input items of any application.

[0032] <About Terminology> An application is an application program, a program generated by a computer in response to a specific task. While an OS (Operating System) is general-purpose software that provides basic functions and systems for the computer's operation (file system, communication, display control, etc.), an application provides specific functions on top of the OS. Applications include web applications and native applications. In this embodiment, either type may be developed.

[0033] An API is an application (software) interface, serving as a point of contact for connecting systems and sharing functions and mechanisms. An API defines the specifications for the interface that applications use to exchange information with each other. A computer-to-computer API is a specification for communication between one website and another via HTTP (HTTPS) communication. Through this communication, one website can use the functions of the other website. Computer-to-computer APIs are sometimes referred to as Web APIs.

[0034] An API takes a specific request (e.g., retrieving, updating, deleting, or processing data) and returns a result (e.g., data, update result, deletion result, processing result, etc.) in response. The request is sometimes called a request message, and the result is sometimes called a response message. Calling an API means sending a request according to the API's specifications and obtaining the result. Calling an API may also be described as executing, operating, hitting, or using it.

[0035] A user is an end-user who uses the applications provided by Application Service 40. Users can also develop applications. A developer is a person who configures Application Service 40 for the necessary settings for developing and using applications using no-code or low-code programming.

[0036] An input item is an item that has an input field for entering information. The input field can accept various types of information, including strings, numbers, symbols, image data, audio data, video data, files, and information for selecting options.

[0037] The information corresponding to an input item is the information that should be set in the input field of that input item. This information is contained in image data, for example, and is extracted from the image data by the generating AI system determining whether it corresponds to the input item based on the label, data format, etc. of the input item.

[0038] [First Embodiment] <Example System Configuration> Referring to Figure 5, the system configuration of the information processing system 100 according to this embodiment will be described. Figure 5 is a diagram showing the system configuration of an example of the information processing system 100. The information processing system 100 shown in Figure 5 has a user terminal 10, a developer terminal 60, and an application service 40. The user terminal 10 and the developer terminal 60 are connected to the application service 40 so as to be able to communicate via networks N1 and N2. The information processing system 100 may further have a generation AI system 50. The user terminal 10 and the developer terminal 60 are connected to the generation AI system 50 so as to be able to communicate via networks N1 and N2. The application service 40 may also be able to communicate with the generation AI system 50 via an API (Application Programming Interface).

[0039] The user terminal 10 and developer terminal 60 are located in facilities such as companies or homes and are connected to network N2. Network N2 can be a LAN, Wi-Fi (registered trademark), Wide Area Ethernet (registered trademark), 4G, 5G, 6G, or other mobile phone network. Network N1 is a wide-area network such as the Internet or WAN. The user terminal 10 and developer terminal 60 do not need to be constantly connected to network N2; they only need to be connected when using the generation AI system 50 or application service 40.

[0040] The generative AI system 50 provides a service that allows users to converse with artificial intelligence (AI) in natural language. A well-known example of such a generative AI system 50 is one that utilizes Large Language Models (LLMs). Large Language Models are natural language processing models trained using large amounts of text data. The generative AI system 50 pre-inputs a vast amount of text and learns knowledge from it using deep learning, reinforcement learning, etc. Using this knowledge, the generative AI system 50 provides response messages to chat messages. Chat messages include prompts and image data, which will be described later. A prompt refers primarily to the text data within a chat message.

[0041] The generation AI system 50, which generates text from data based on chat messages, is sometimes referred to as the generation AI. In this embodiment, the response messages provided by the generation AI system 50 are used to generate values ​​for input fields of an application running in the application service 40. The application running in the application service 40 includes a web application that runs on the application service 40 and a native application installed on the user terminal 10. When the native application installed on the user terminal 10 is executed on the user terminal 10, it connects to the application service 40 and executes the functions of the application service 40.

[0042] The generation AI system 50 has the following characteristics: 1. The generating AI system 50 conducts the conversation in a natural flow. 2. The generation AI system 50 can expand on ideas and propose them even in fields where the user lacks expertise. 3. The generation AI system 50 can output accurate program code. Taking advantage of these features, the application service 40 can receive values ​​to be set for the input items from the generating AI system 50 by instructing the generating AI system 50 to provide a list of application input items and image data.

[0043] One of the features of the third system is the function call function (also known as tool_call or function_call), but in this embodiment, it is possible to implement the system either with or without the function call function. However, since the tool_call returned by the generation AI system 50 is accurate (highly reproducible), there is a higher probability of obtaining the input item values ​​in JSON format.

[0044] As a generation AI system 50, systems using LLMs such as GPT-3, GPT-4, Transformer, and BERT (all registered trademarks) are known. In the information processing system 100 of this embodiment, ChatGPT using GPT-3 or GPT-4 can be used as an example, but other systems using LLMs may also be used.

[0045] Application Service 40 is one or more information processing devices that provide applications for users to run. Application Service 40 is also a server device that provides applications for managing information entered by users into input fields. The applications provided by Application Service 40 are, for example, database-type web applications that manage data in a tabular format. Users can arbitrarily create input fields for the applications and customize them to save, read, or process data related to their work. Application Service 40 has multiple applications, such as a business card management application and a book management application.

[0046] In this embodiment, the user sets image data, such as a business card, in the application. The user terminal 10 requests AI input from the application service 40 along with the image data. The application service 40 sends information about the application to the generating AI system 50 and receives a response message (input items and their values) from the generating AI system 50. The application service 40 sends the input items and their values ​​back to the user terminal 10. Therefore, the user can automatically set information that would otherwise need to be manually entered from the image data into the application.

[0047] Examples of application services 40 include cloud services, ASPs (Application Service Providers), SaaS, etc., and may include various services provided over a network. For example, database provision and storage services are examples of such services. Application services 40 may reside on the internet or on-premises.

[0048] Furthermore, the application service 40 may have its functions distributed across multiple information processing devices, or multiple application services 40 with the same function may exist, and the number of information processing devices that distribute video may increase or decrease depending on the processing load.

[0049] In addition, a separate web server may exist in addition to the application service 40, and this web server may communicate with the user terminal 10. In this case, the web server will communicate with the application service 40 instead of the user terminal 10.

[0050] A server is a computer or software that provides information and processing results in response to requests from clients.

[0051] Furthermore, the application service 40 accepts various settings from the developer terminal 60. These settings include user registration with the application service 40 and registration of a web application for creating chat messages. In other words, the administrator performs the task of setting values ​​for the application's input fields using the generation AI system 50 with the application service 40.

[0052] The user terminal 10 or developer terminal 60 is, for example, a terminal device such as a PC (Personal Computer), smartphone, or tablet used by the user or developer. A web browser or native application runs on the user terminal 10 or developer terminal 60. The developer operates the developer terminal 60 to create configuration information for the application. The administrator or user can operate the developer terminal 60 and the user terminal 10 to use various services provided by the generation AI system 50 or application service 40.

[0053] The user terminal 10 or developer terminal 60 can be any information processing device. Examples of such information processing devices include output devices such as electronic whiteboards and digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, mobile phones, smartphones, tablet devices, car navigation systems, game consoles, PDAs (Personal Digital Assistants), digital cameras, and wearable PCs.

[0054] <Example Hardware Configuration> Referring to Figure 6, the hardware configurations of the application service 40, user terminal 10, and developer terminal 60 included in the information processing system 100 will be described. The generation AI system 50 is assumed to be the same as in Figure 6, or to be an information processing device hardware configuration compatible with cloud computing.

[0055] <<Application service 40, user terminal 10, and developer terminal>> Figure 6 shows an example of the hardware configuration of an application service 40, a user terminal 10, and a developer terminal 60. As shown in Figure 6, the application service 40, user terminal 10, and developer terminal 60 are built by a computer 500, which includes a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0056] Of these, the CPU 501 controls the operation of the entire computer 500. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 6.

[0057] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium is a DVD, CD, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0058] <About the features> Next, with reference to Figure 7, the functional configuration of the information processing system 100 will be described. Figure 7 is a diagram showing an example of the functional configuration of the application service 40 and the user terminal 10.

[0059] <<User Terminal>> The user terminal 10 includes a communication unit 11, a display control unit 12, an operation reception unit 13, and an input processing unit 14. Each of these functional units is a function or means realized by the CPU 501 shown in Figure 6 executing instructions contained in one or more programs installed on the user terminal 10. For example, the communication unit 11, the display control unit 12, the operation reception unit 13, and the input processing unit 14 may be realized by a web browser and a web application. The web application is sent from the application service 40 to the user terminal 10. If the user terminal 10 runs a native application, these may be realized by the native application.

[0060] The communication unit 11 sends and receives various types of information to and from the application service 40. The communication unit 11 has a receiving unit 11a and a transmitting unit 11b. In this embodiment, the receiving unit 11a receives screen information such as the application's input screen from the application service 40. The transmitting unit 11b sends image data such as business cards to the application service 40. The receiving unit 11a also receives values ​​from the application service 40 to be set for input items.

[0061] The display control unit 12 interprets screen information from various screens and displays it on the display 506. The operation reception unit 13 receives various user operations on the various screens displayed on the display 506.

[0062] The input processing unit 14 inputs values ​​corresponding to the input items transmitted from the application service 40 into the input fields of the input items. The display control unit 12 displays a screen in which values ​​have been set for the input items. The input processing unit 14 operates by executing the transmission program from the application service 40 in a web browser.

[0063] <<Application Services>> The application service 40 includes a communication unit 41, a screen generation unit 42, a registration unit 43, a specification unit 44, a program transmission unit 45, a message communication unit 46, and an application information storage unit 49. Each functional unit of the application service 40 is a function or means realized by the CPU 501 shown in Figure 6 executing instructions contained in one or more programs installed in the application service 40. The application information storage unit 49 is built on the HD 504 or RAM 503 shown in Figure 6. The application information storage unit 49 does not necessarily have to be present in the application service 40; it only needs to be on a network that the application service 40 can access.

[0064] The communication unit 41 sends and receives various types of information to and from the user terminal 10. The communication unit 41 has a receiving unit 41a and a transmitting unit 41b. The receiving unit 41a receives image data such as business cards from the user terminal 10. The transmitting unit 41b transmits input items and their values ​​to the user terminal 10. The transmitting unit 41b also transmits to the user terminal 10 the web application that the user terminal 10 runs and the screen information that the web application displays.

[0065] The screen generation unit 42 generates screen information for the screen displayed by the user terminal 10. The screen information is a program written in HTML, JSON, XML, a scripting language, and CSS (Cascading Style Sheet), and may be called a web application. The structure of the web page is mainly specified by HTML, the operation of the web application is defined by the scripting language, and the style of the web page is specified by CSS. In some cases, the user terminal 10 may run a native application. A native application is an application that cannot be run unless it is installed on the user terminal 10. In the case of a native application, the user terminal 10 holds the screen configuration, and the information to be displayed is sent in JSON, XML, etc.

[0066] The registration unit 43 manages application information for each application in the application information storage unit 49. The registration unit 43 registers values ​​corresponding to input items transmitted from the user terminal 10 (or the generation AI system 50) in the application information storage unit 49. The application information includes information set in the application's input items and information about the input items that the application has. Therefore, the application information storage unit 49 stores information set in the application's input items and information about the input items (see Figures 8 and 9).

[0067] The identification unit 44 includes an image identification unit 44a, an input item identification unit 44b, and an application identification unit 44c. The image identification unit 44a identifies the image data received from the user terminal 10 along with the AI ​​input request as the image data to be extracted. The input item identification unit 44b receives information identifying the application from the user terminal 10 and identifies the input items of the application based on the information identifying the application. The application identification unit 44c identifies the application name of the application based on the information identifying the application received from the user terminal 10.

[0068] The program transmission unit 45 transmits a program to the user terminal 10 in response to a program request sent from the user terminal 10. This program is a web application, and furthermore, it is a component of the web application, such as JavaScript (registered trademark).

[0069] The message communication unit 46 sends and receives messages with the generation AI system 50. The generation AI system 50 exposes an API, and the message communication unit 46 sends a request message containing a chat message to the generation AI system 50 by calling the API. The message communication unit 46 receives a response message from the generation AI system 50. Thus, the request message is information that includes a chat message, but it is a general term for HTTP communication, and sometimes the chat message is referred to as the request message.

[0070] The message communication unit 46 includes a request generation unit 46a, a request transmission unit 46b, and a response receiving unit 46c. The request generation unit 46a generates a request message that calls an API exposed by the generation AI system 50. The request message requests the generation AI system 50 to generate information. The request message includes a text portion called a prompt and image data (which may be the image data itself or a URL). The request message may also include audio data, etc.

[0071] The request transmission unit 46b sends a request to the generation AI system 50 that includes image data and instructions to extract values ​​corresponding to input items from the image data. This request is included in the request message generated by the request generation unit 46a.

[0072] The response receiving unit 46c receives a response message generated by the generating AI system 50 in response to the instruction sent to the generating AI system 50. This response message contains a value corresponding to the input item.

[0073] <App Information Storage Unit> Figure 8 shows an example of information set in the input fields of an application. This information includes both information manually set by the user and information generated by the AI ​​generation system 50. Figure 8 uses a business card management application as an example to illustrate the information set in the application's input fields. The information set in the input fields is managed on a record basis. In the case of a business card management application, the information for one record is called business card information. In a business card management application, the input fields include name, company, department, job title, etc., and a value is stored for each input field.

[0074] Figure 9 shows an example of information related to application input fields, which is part of the application information. The information related to input fields defines what kind of information is stored in each input field.

[0075] The `label` field is the name (the so-called label) of the input field displayed on the input screen 210 of the business card management app.

[0076] The `name` field is the identifier for the input field that application service 40 uses to manage and identify the input field.

[0077] The `type` field specifies the data format of the input field.

[0078] <Action or process> The overall flow of processing performed by the information processing system 100 will be explained with reference to Figure 10. Figure 10 is a sequence diagram illustrating the processing of the information processing system 100. The business card management application has already been registered in the application service 40. In addition, the business card image data has been set in the application service 40, but it is assumed that no values ​​have been set in the other input fields.

[0079] S1: The user terminal 10 displays the input screen 210 of the business card management application. If the input screen 210 of the business card management application is implemented by a web application, the transmission unit 11b sends a request to the application service 40 for one or more programs to be executed on the user terminal 10. The program transmission unit 45 of the application service 40 sends the web application to the user terminal 10. The web application includes a program. The program displays a screen that accepts input values ​​from the user into input fields and can accept instructions to extract information to be entered into the input fields from image data. When an instruction is received on the screen, the program sends a request to the application service 40 including the image data to be extracted and instructions to extract values ​​corresponding to the input items from the image data. The application service 40 receives the values ​​corresponding to the input items obtained by sending a request to the generation AI system 50, inputs the received values ​​into the input fields of the corresponding input items and displays them, and sends the values ​​entered into the input fields back to the application service 40 for management. The above-mentioned "instructions to extract information to be entered into input fields from image data" can be rephrased as instructions to automatically input information into the input fields.

[0080] The input screen 210 of the business card management app displays the business card image data (thumbnail) in the business card image attachment field 219, as shown in Figure 14 below. The user starts AI image analysis input by pressing the "AI Image Analysis Input" button 228. The operation reception unit 13 of the user terminal 10 receives this operation. AI image analysis input is a series of processes that generate values ​​for input items by analyzing image data and request that these be sent to the user terminal 10.

[0081] S2: In response to this operation, the transmission unit 11b of the user terminal 10 requests AI image analysis input from the application service 40 by specifying the application identification information, the currently displayed record ID (information that identifies the record), and the business card image data set in the business card image attachment field. Since the application currently displayed by the user terminal 10 is identified, the application identification information is also known. To set a value for an application that the user terminal 10 is not currently displaying, for example, the user selects the application. The currently displayed record ID is known because it is the identification information of the record currently displayed by the user terminal 10. If the currently displayed record is not registered with the application service 40, the record ID is undetermined, so the user terminal 10 notifies the application service 40 that it is undetermined, and the application service 40 assigns a new record ID.

[0082] Since the business card image data is already configured in the application service 40, the communication unit 11 does not need to transmit the image data. If the business card image data is not already configured in the application service 40, for example, immediately after the user terminal 10 has taken a picture of the business card, the communication unit 11 will transmit the image data to the application service 40.

[0083] S3: The receiving unit 41a of the application service 40 receives a request for AI image analysis input, and the input item identification unit 44b uses the application information storage unit 49 to identify information about the input items of the application identified by the application identification information, and obtains a list of input items. More specifically, since the label item is appropriate as an input item to be used by the generating AI system 50, the input item identification unit 44b receives a list of label items.

[0084] Furthermore, the application identification unit 44c obtains the application name of the application identified by the application identification information from the application information storage unit 49. The reason for requesting the application name is that the accuracy of the generation AI system 50 in generating appropriate values ​​for the input items is improved by analyzing the application name. Therefore, the application name is not required. The application service 40 may send an application description document to the generation AI system 50 instead of the application name. An application description document might be, for example, "This application is an application for managing business cards."

[0085] Furthermore, if image data is transmitted from the user terminal 10, the image identification unit 44a identifies the image data as the image data to be extracted. If no image data is transmitted from the user terminal 10, the image identification unit 44a obtains the original image data displayed in the business card image attachment field 219, which is identified by the record ID, from the application information storage unit 49.

[0086] The request generation unit 46a of the application service 40 generates a request message using information about the application (business card image data, application name, and list of input items). This request message includes instructions to extract information corresponding to the input items from the image data. An example of this request message is shown in Figure 11.

[0087] S4: The request transmission unit 46b sends a request to the generation AI system 50 to generate a value for an input item along with this request message.

[0088] S5: The generation AI system 50 analyzes the business card image data to generate values ​​for the input fields and determines which input fields those values ​​correspond to. The generation AI system 50 sends a response message (input fields and generated values) to the application service 40.

[0089] S5-2: The response receiving unit 46c of the application service 40 receives the response message (input items and generated value). Upon receipt, the registration unit 43 of the application service 40 may register the generated value in the input item of the application information storage unit 49. If the registration unit 43 registers the generated value in the application information storage unit 49, no further processing is required. Even if the user modifies the generated value, it can be overwritten. In this embodiment, the case in step S10 after the user has confirmed the generated value is described, in which case the generated value is registered in the input item.

[0090] S6: The transmission unit 41b of the application service 40 sends the input items and generated values ​​to the user terminal 10. The receiving unit 11a of the user terminal 10 receives the input items and generated values.

[0091] S7: The input processing unit 14 of the user terminal 10 enters the value corresponding to the input item into the input field of the corresponding input item. The display control unit 12 displays the input screen 210 of the business card management application, where the values ​​have been set in the input items.

[0092] Furthermore, the transmission unit 41b of the application service 40 may not only transmit the input items and generated values, but also transmit screen information to the user terminal 10 for updating the input screen 210 of the business card management application.

[0093] S8: The user views the input screen 210 of the business card management app and checks the values ​​of the input items generated by the generation AI system 50. If necessary, such as if the values ​​are incorrect, the user edits the values ​​on the input screen 210. To save the values ​​of each input item on the input screen 210 of the business card management app, the user performs a save operation (by pressing a button 229 such as a registration button) on the user terminal 10. The operation reception unit 13 accepts the operation.

[0094] S9: The transmission unit 11b of the user terminal 10 sends a save request to the application service 40, specifying the application's identification information and record ID, along with the values ​​of the input items and the image data. If the image data has already been sent, it does not need to be sent again.

[0095] S10: The receiving unit 41a of the application service 40 receives these, and the registration unit 43 identifies the application using the application identification information and identifies the record using the record ID. The registration unit 43 associates the values ​​with the input fields of the identified record and stores them in the application information storage unit 49 (registers them). The transmitting unit 41b of the application service 40 sends an input OK (value registration is complete) message to the user terminal 10. The receiving unit 11a of the user terminal 10 receives the input OK message.

[0096] <Example of a request message from an application service to a generation AI system> Figure 11 shows the parameters included in the request message that the application service 40 sends to the generation AI system 50 in step S4 of Figure 10.

[0097] • messages241 is the API for the generation AI system 50, and the following indicates that it is a chat message.

[0098] Role242 is an API for the Generating AI System 50 and is a classification of the requester of a request message. It includes user, assistant (AI of the Generating AI System 50), and system (instruction for configuration by the AI ​​assistant).

[0099] • content243 is the API for the generation AI system 50, and it is where the dialogue text is set. Since content is an array, it is also possible to specify a prompt and multiple image data. In Figure 11, three parameters 244-246 are described in JSON format. Two of them are image data.

[0100] Parameters 244-246 are the format of the information to be sent to the generating AI system 50. "type" defines the data type. If "type" is "text", the value of "text" is "prompt" 247. A prompt is set in "prompt" 247. An example of a prompt set in "prompt" 247 is shown in Figure 12. If "type" is "image_url", the value of "image_url" is "image"248,249, where "image"248,249 contains either the URL where the image data is stored or the Base64 encoded image. Note that if the application has only one input field for image data, then only one value each of parameters 245 and 246 is needed.

[0101] As shown in Figure 11, a request message combining a prompt and an image is sent to the generation AI system 50.

[0102] Figure 12 is a diagram illustrating the prompt set in "Prompt" 247 in Figure 11. The string in Figure 12 is a template for the request generation unit 46a to generate the prompt. The string has four ${...}. When sending a request message, information about the application is set in ${appName}251, ${labels.join()}252, ${labels.length}253, and ${type}254. That is, the four ${...} are replaced with information about the application. The other strings are boilerplate text and are held in advance by the request generation unit 46a. The prompt's beginning, "Analyze the image" 266, requests that the image data specified by parameter 245 included in the request message be analyzed.

[0103] • ${appName}251 will be set to the app name. The app name is optional, or it can be a description of the application (that explains what the app is about).

[0104] The variable ${labels.join()}252 will contain a list of input fields.

[0105] The value ${labels.length}253 is set to the number of input fields.

[0106] The variable ${type}254 specifies the data format of the input items to be returned in the response message. TypeScript is a statically typed programming language that allows you to declare the data type of variables within the code. In Figure 12, JSON format is specified for ${type}254. In other words, the prompt in Figure 12 instructs the generating AI system 50 to return the input items and their values ​​in JSON format. A specific example of the settings is explained in Figure 15.

[0107] <Example of a response message from a generating AI system to an application service> Figure 13 shows the format of the response message sent from the generating AI system 50 to the application service 40 in step S5 of Figure 10. In other words, Figure 13 is the format, not the response message itself.

[0108] Messages255 indicates that the following is the response message:

[0109] • Role 256 is the classification of the sender that sends the response message, and in this case it is assistant (AI of Generating AI System 50).

[0110] • content257 is the content of the response message. Here, content is response 258 (input items and values) from the generating AI system 50. Details of response 258 from the generating AI system 50 are explained in Figure 16.

[0111] <Example of setting values ​​for input fields using image data> The following examples illustrate how to set values ​​for input fields using image data, using a business card management app and a book management app as examples.

[0112] <<Example of setting information using business card image data>> Referring to Figure 14, etc., an example of setting values ​​in the input fields of the business card management application will be explained. Figure 14 is the input screen 210 of the business card management application as displayed on the user terminal 10. The explanation of Figure 14 will mainly focus on the differences from Figure 2. In Figure 14, the user has entered business card image data into the business card image attachment field 219. The user can manually enter values ​​into each input field on the input screen 210 of the business card management application.

[0113] The business card image attachment field 219 displays a thumbnail of the business card image data. In this state, the user presses the "AI Image Analysis Input" button 228. The "AI Image Analysis Input" button 228 accepts instructions to extract values ​​to be entered into the input fields from the image data. The "AI Image Analysis Input" button 228 may become active (pressable) only when a business card image is entered into the business card image attachment field 219. This business card image does not necessarily have to be registered in the application service 40.

[0114] This section describes information about the business card management app application. First, the app name is "Business Card Management App". Next, according to the information about input items in Figure 9, the list of input items (label items) is name, company, department, job title, address, telephone number, email address, URL, and business card image attachment field. Of these, the business card image attachment field does not require AI image analysis input. Therefore, the request generation unit 46a does not include the business card image attachment field in the prompt. By making only input items with a type of String in the information about input items into the prompt, the request generation unit 46a can exclude input items that do not require value input from the prompt. Therefore, the list of input items is name, company, department, job title, address, telephone number, email address, and URL. Also, the number of input items is 8.

[0115] The input screen 210 of the business card management application is a screen for uploading image data, and the identification unit 44 identifies the uploaded image data as the image from which values ​​will be extracted. The system may be configured so that image data of business cards can be uploaded not only by pressing the "AI Image Analysis Input" button 228, but also by pressing the business card image attachment field 219. The receiving unit 41a of the application service 40 receives information identifying the application selected by the user, and the identification unit 44 identifies the input items based on the information identifying the application (using the information regarding input items in Figure 9). The image identification unit 44a identifies the image data uploaded by the user. The input item identification unit 44b identifies the input items corresponding to the application and record ID. The application identification unit 44c identifies the application name of the application.

[0116] Figure 15 shows the prompt generated by the request generation unit 46a. The following information is set in ${appName}251, ${labels.join()}252, and ${labels.length}253 shown in Figure 12.

[0117] • ${appName}251 is set to "Business Card Management App" 261.

[0118] The label field is set to ${labels.join()}252, which contains the fields "Name, Company, Department, Position, Address, Phone Number, Email Address, URL". In other words, the label field is set for information related to the input field. The reason the name field is not set is that the name field is usually identification information, and it often contains information unrelated to the label (making it difficult for the generating AI system 50 to determine what the input field is).

[0119] • ${labels.length}253 is set to "8"263.

[0120] Furthermore, the ${type}254 shown in Figure 12 is set to the data format 264 for each input item. The data format 264 is an instruction to extract the information described in this data format. "name?: string, company?: string, department?: string, position?: string, address?: string, telephoneNumber?: string, emailAddress?: string, URL?: These are the values ​​for the `name` and `type` fields among the information regarding the input items in Figure 9. The `name` field in data format 264 is arranged in the same order as in 262: "Name, Company, Department, Position, Address, Phone Number, Email Address, URL".

[0121] The reason the `name` field in data format 264 is not the `label` field is that the application service 40 uses the value of the input field returned by the generation AI system 50 for configuration. The application service 40 identifies the input field by the value of the `name` field, not the value of the `label` field. However, the value of the `label` field may be used. In this case, when setting the value to the application service 40, the request generation unit 46a converts the `label` field to the `name` field.

[0122] The "?" at the end of the name field means that if there are input fields for which no value is found in the image data, the generation AI system 50 may omit the input fields. Note that "string" is the value (data type) of the type field.

[0123] The generating AI system 50 understands from the prompt "Analyze the image" 266 at the beginning of the prompt that it will analyze the image data specified by the parameter 245 included in the request message. Next, the generating AI system 50 attempts to generate values ​​for "Name, Company, Department, Position, Address, Phone Number, Email Address, URL" 262 from the image data. Then, the generating AI system 50 determines which input items were able to generate values ​​based on the order of the data format 264 and associates the input items with the generated values.

[0124] Figure 16 shows the response message from the generating AI system 50 to the request message in Figure 15. The generating AI system 50 analyzes the business card image data and obtains values ​​corresponding to "name, company, department, job title, address, phone number, email address, URL" 262. The generating AI system 50 then associates these values ​​with the name field (input field below) contained in the prompt data format 264 and returns them.

[0125] Input field: Value Name: Taro Patent Company: Sample1 Co., Ltd. Department: Sales Department URL: https: / / sample.co.jp Here, the reason why position, address, telephoneNumber, and emailAddress, which are input items included in the prompt data format 264, are not present in the request message is that they are either not included in the business card image data or were not detected by the generating AI system 50.

[0126] The application service 40 sends the input fields and values ​​shown in Figure 16 to the user terminal 10. The user terminal 10 requests the application service 40 to set the values ​​corresponding to the input fields in the business card management application.

[0127] Figure 17 shows the input screen 210 of the business card management application with values ​​set in the application. The input processing unit 14 enters the values ​​corresponding to the input items into the input fields of the corresponding input items. In other words, the values ​​included in the response message are set in the corresponding input items. The values ​​generated by the generation AI system 50 are set in the name field 211, company field 212, department field 213, and URL field, respectively.

[0128] Furthermore, the values ​​entered into the input fields by the input processing unit 14 can be edited manually by the user. Also, when the user presses a button 229 such as a registration button on the input screen 210 of the business card management application, the process of step S8 in Figure 10 is executed.

[0129] In this way, by specifying business card image data, the user can automatically set the values ​​obtained from analyzing the business card image data into the corresponding input fields. Since the user terminal 10 can run any application, it can set appropriate values ​​extracted from the image data into the input fields of any application, not just business card management applications.

[0130] <<Example of setting information using image data of a book cover>> Next, referring to Figure 18, etc., we will explain an example of setting values ​​in the input fields of the book management application. Figure 18 is the input screen 270 of the book management application displayed on the user terminal 10. This book management application has a title field 271, a subtitle field 272, an author field 273, a publisher field 274, a description of the cover appearance 275, a cover image attachment field 276, and a back cover image attachment field 277. These are input fields. Users can use the book management application for business or personal use. For example, a user can enter information about books they have purchased or finished reading into the input fields of the book management application. This allows them to digitize and list the information about books they have purchased or finished reading.

[0131] To set values ​​in the input fields using the generation AI system 50, the user entered image data of the book into the cover image attachment field 276 and the back cover image attachment field 277. The cover image attachment field 276 displays a thumbnail of the image data of the book's cover. The back cover image attachment field 277 displays a thumbnail of the image data of the book's back cover. Either the cover image attachment field 276 or the back cover image attachment field 277 may be filled in.

[0132] In this state, the user presses the "AI Image Analysis Input" button 278. The "AI Image Analysis Input" button 278 may become active (pressable) only when image data is entered into at least one of the cover image attachment field 276 or the back cover image attachment field 277. In addition to pressing the "AI Image Analysis Input" button 278, the system may also be configured to allow uploading of image data of the book's front or back cover by pressing either the cover image attachment field 276 or the back cover image attachment field 277. In this case, only the image data of the pressed field may be sent, or both the front and back cover image data of the book may be uploaded by pressing either the front cover image attachment field 276 or the back cover image attachment field 277. These images do not necessarily have to be registered in the application service 40.

[0133] Figure 19 shows information about the input fields of the book management application. Similar to the information about the input fields of the business card management application (Figure 9), the information about the input fields of the book management application includes label, name, and type fields.

[0134] This section describes information about the book management app application. First, the app name is "Book Management App". Next, according to the information about input items in Figure 19, the list of input items (label items) is title, subtitle, author, publisher, description of the cover appearance, cover image attachment field, and back cover image attachment field. Of these, the cover image attachment field and back cover image attachment field do not require AI image analysis input. Therefore, the request generation unit 46a does not need to include the cover image attachment field and back cover image attachment field in the prompt. By using only input items with a type of String in the information about input items as prompts, the request generation unit 46a can exclude input items that do not require a value from the prompt. Therefore, the list of input items is title, subtitle, author, publisher, and description of the cover appearance. Also, there are 5 input items.

[0135] Figure 20 shows the prompt generated by the request generation unit 46a. The following information is set in ${appName}251, ${labels.join()}252, and ${labels.length}253 shown in Figure 12.

[0136] • ${appName}251 is set to "Book Management App" 281.

[0137] • ${labels.join()}252 has "Title, subtitle, author, publisher, cover appearance description" 282 set.

[0138] • ${labels.length}253 is set to "5" (283).

[0139] Furthermore, the ${type}254 shown in Figure 12 is set to the data format 284 for each input item. "title?: string, subtitle?: string, author?: string, Publisher?: large, cover?: string, g" These are the values ​​of the name and type fields among the information about the input items in Figure 19. The name field in data format 284 is arranged in the same order as "Title, Subtitle, Author, Publisher, Cover Description" 282.

[0140] The reason the `name` field in data format 264 is not the `label` field is that the application service 40 uses the value of the input field returned by the generation AI system 50 for configuration. The application service 40 identifies the input field by the value of the `name` field, not the value of the `label` field. However, the value of the `label` field may be used. In this case, when setting the value to the application service 40, the request generation unit 46a converts the `label` field to the `name` field.

[0141] The "?" and "string" at the end of the name field can be treated the same as in Figure 15.

[0142] The generating AI system 50 understands from the prompt "Analyze the image" 289 at the beginning of the prompt that it will analyze the image data specified by parameter 245 in the request message. Next, the generating AI system 50 attempts to generate the values ​​"Title, subtitle, author, publisher, and cover appearance description" 282 from the image data. Then, the generating AI system 50 determines which input items were able to generate values ​​based on the order of the data format 284 and associates the input items with the generated values.

[0143] Figure 21 shows the response message from the generating AI system 50 to the request message in Figure 20. The generating AI system 50 analyzes the business card image data and obtains values ​​corresponding to "title, subtitle, author, publisher, and cover appearance description" 282. The generating AI system 50 then associates these values ​​with the name field (input field below) contained in the prompt data format 284 and returns them.

[0144] Input field: Value Title: Caterpillar Encyclopedia author : Trademark Hanako publisher:○×publishing Cover: The cover has a green background with an illustration of a purple caterpillar.

[0145] The reason why the subtitle is not present in the request message among the input items queried is that it is not included in the image data of the front or back cover, or that the generation AI system 50 was unable to detect it.

[0146] Note that the value corresponding to "cover," which is "The cover has a green background and an illustration of a purple caterpillar," is not included as text in the image data. This value is what the generating AI system 50 converted into text data to describe the image data. Therefore, the information processing system 100 of this embodiment can automatically set information that is not included as text in the image data in the application service 40.

[0147] The application service 40 sends the input fields and values ​​shown in Figure 21 to the user terminal 10. The user terminal 10 requests the application service 40 to set the input fields and values ​​in the book management application.

[0148] Figure 22 shows the input screen 270 of the book management app with values ​​set in the app. The values ​​included in the response message in Figure 21 are set in the corresponding input fields. The input processing unit 14 enters the values ​​corresponding to the input fields into the corresponding input fields. That is, values ​​are set in the title field 271, the author field 273, the publisher field 274, and the cover appearance description field 275.

[0149] Furthermore, the values ​​entered into the input fields by the input processing unit 14 can be edited manually by the user. Also, when the user presses a button 229 such as a registration button on the input screen 270 of the book management application, the process of step S8 in Figure 10 is executed.

[0150] In this way, by specifying multiple image data of a book, the user can automatically set the values ​​obtained from analyzing these image data into the corresponding input fields. Since the user terminal 10 can run any application, it can set appropriate values ​​extracted from the image data into the input fields of any application, not just the book management application.

[0151] <Main effects> According to the information processing system 100 of this embodiment, the user can specify image data containing the values ​​of input items, and the generating AI system 50 can analyze the image data and automatically set the obtained values ​​in the input items. The user terminal 10 can set appropriate values ​​extracted from the image data in the input items of any application, not just one application.

[0152] [Second Embodiment] In this embodiment, an information processing system 100 that sets values ​​in input items using the function call function provided by the generation AI system 50 will be described.

[0153] In this embodiment, the hardware configuration diagram shown in Figure 6 and the functional block diagram shown in Figure 7, as described in the first embodiment, can be used as references.

[0154] <Request message when function call functionality is available> Some generation AI systems 50 have a function call function. The application service 40 specifies a function and the format of its arguments to the generation AI system 50, and the generation AI system 50 generates the function arguments in the specified format. This function is called the function call function. However, the application service 40 does not have a function to call (it would not cause any problems if there were, but in this embodiment there is no such function so it is called a dummy function), and even though the term "function call function" is used, the generation AI system 50 does not actually call a function to the application service 40. In this embodiment, the application service 40 uses the function call function to specify a function and the format of its arguments to the generation AI system 50 in order to more reliably obtain the values ​​of the input items from the generation AI system 50 in the format specified by the application service 40.

[0155] By using the function call function, the accuracy with which the generating AI system 50 returns values ​​in JSON format can be improved compared to when the application service 40 requests the generating AI system 50 to generate input items and their values ​​in JSON format, and the generating AI system 50 generates the values.

[0156] <Processing flow using function call functionality> Figure 23 illustrates how to set input field values ​​using the function call feature. Figure 23 shows the input screen 210 of the business card management application.

[0157] (1) The user terminal 10 accepts the execution of AI image analysis input from the user. The user terminal 10 sends the request to execute AI image analysis input to the application service 40.

[0158] (2) When the application service 40 receives a request to execute AI image analysis input, it sends a request message that includes the format of the function's arguments. A function is a program interface that performs a specific process with specified arguments and returns a return value as a result. However, in this embodiment, no functions exist in the application service 40. By sending the application service 40 with the format of the function (dummy function)'s arguments in the request message to the generating AI system 50, it is expected that the function call from the generating AI system 50 will include the values ​​of the input items in the format specified by the application service 40.

[0159] The act of the generating AI system 50 requesting the application service 40 to call this dummy function, which does not actually exist, is called a function call (tool_call in this embodiment).

[0160] Furthermore, there is no problem if a function actually exists in the application service 40, and the application service 40 may set a value in the application service 40 by executing the function. Executing a function includes sending input items and values ​​to the user terminal 10 and setting the value corresponding to the input items received from the user terminal 10 in the application service 40.

[0161] The application service 40 does not have to include the format of the function arguments in the same request message as the application information. For example, the application service 40 may include the format of the function arguments in a separate request message from the request message containing the application information and send it to the generating AI system 50.

[0162] (3) Based on the application information contained in the sent request message and the format of the function arguments, the generating AI system 50 sends a response message containing a function call (tool_call) to the application service 40. Here, although it is said that the generating AI system 50 requests a function call from the application service 40, it does not request the execution of the function; rather, the generating AI system 50 merely proposes values ​​for the input items in the specified format.

[0163] In other words, the generating AI system 50 includes the application input items and values ​​sent from the application service 40 as function arguments in the response message. These values ​​are generated by analyzing information about the application.

[0164] (4) The application service 40 receives a response message from the generation AI system 50, retrieves the input items and values ​​included in the function call (tool_call) contained in the response message, and sends the input items and values ​​to the user terminal 10.

[0165] (5) The user terminal 10 receives the input items and values, and when the user confirms the values ​​and presses a button 229 for registration or the like, the user terminal 10 requests the application service 40 to set the values ​​for the input items. The application service 40 stores the values ​​corresponding to the received input items in the application information storage unit 49.

[0166] <Example of a request message to the AI ​​generation system when the AI ​​generation system has a function call function> Figure 24 shows an example of a request message including function arguments. The request message in Figure 24 is assumed to be for a business card management application. Messages 291, role 292, and content 293 are the same as in Figure 11. Parameter 294 indicates that the input item "type" is "image_url", and the value of "image_url" is "image" 320. "image" 320 can contain, for example, a URL where business card image data is stored or a Base64 encoded image.

[0167] Furthermore, the parameter 244 (prompt) in Figure 11 has been replaced with tool295.

[0168] • tools295 is the API for the Generative AI System 50, and below it means specifying the format of the function arguments.

[0169] • "type": "function" 296 indicates that the object's type is a function.

[0170] Function297 indicates that it is a description of a function. Name298 is the function name, and description299 is the function's features. Both are pre-held by the request generation unit 46a. Parameters301 is a description of the function's arguments, and "type":"object"302 indicates that the arguments are described in object format. Function297, name298, description299, and parameters301 are all APIs of the generation AI system 50.

[0171] properties303 describes a list of information related to the input fields of the business card management application in a nested JSON structure. In other words, properties303 requests the generating AI system 50 to return the function arguments in JSON format.

[0172] • name304 specifies how to return a value for the input field "name". The input field "name" is obtained from the name field of the information about the input field. Therefore, name304 specifies that the "type" of the input field "name" is "string". "description" specifies that the input field "name" should return "name". This "name" means that the generation AI system 50 should return what it has determined to be a name after analyzing the image data.

[0173] The same description applies to the subsequent input fields: company305, department306, position307, address308, telephoneNumber309, emailAddress310, and url311.

[0174] Thus, tools295 contains a list of input items among the application information. Image data is included in parameter 245 in Figure 11, as in the first embodiment. The request message in Figure 24 does not include the application name, but it may include it. On the other hand, name298 or description299 serve the role of the application name, and these may be considered as the application name.

[0175] The generating AI system 50 understands, via name298 or description299, that it will analyze the image data specified by parameter 245 included in the request message. Next, the generating AI system 50 attempts to generate the values ​​specified by name304, company305, department306, position307, address308, telephoneNumber309, emailAddress310, and url311 (name, company, department, job title, address, telephone number, email address, URL) from the image data. Finally, the generating AI system 50 returns the generated values ​​in JSON format.

[0176] <Example of a response message from a generating AI system to an application service> Figure 25 shows an example of a response message from the generating AI system 50 when the generating AI system 50 has a function call function. Messages 321, role 322, and content 323 are the same as in Figure 13. In the response message in Figure 25, the generating AI system 50 requests a function call from the application service 40.

[0177] • tool_calls324 indicates that the following is a function call: the generating AI system 50 is requesting the application service 40 to call a dummy function that does not actually exist. However, the function may actually exist.

[0178] • The value "type": "function" indicates that the object's type is a function.

[0179] • function326 indicates that it is a description of a function.

[0180] name327 is a function name.

[0181] Arguments328 refers to the function arguments. The arguments include the following input items and values. That is, the generating AI system 50 analyzes the business card image data and generates a JSON format specified in properties303 in Figure 24, associating the name item (input item) with its value.

[0182] Input field: Value Name: Taro Patent Company: Sample1 Co., Ltd. Department: Sales Department URL: https: / / sample1.co.jp It can be seen that these input items and values match the information included in the response message of FIG. 16 in the first embodiment. The application service 40 transmits the input items and values of FIG. 25 to the user terminal 10. The user terminal 10 requests the application service 40 to set the input items and values in the business card management application. As a result, as shown in FIG. 17, the values of the input items are set in the input screen 210 of the business card management application.

[0183] <Main effects> According to this embodiment, in addition to the effects of the first embodiment, the accuracy of the generation AI system 50 returning values in JSON format can be improved. Since the application service 40 can obtain the values of the input items in JSON format, the application service 40 can surely obtain the values of the input items.

[0184] [Third Embodiment] In this embodiment, a modification common to the first embodiment and the second embodiment will be described.

[0185] <Selection of image data used for AI image analysis input> One record of an application may have a plurality of image data. For example, in the case of a business card management application, there may be a business card image attachment field and a face image attachment field. Since the image data in the face image attachment field is the face image of the customer, it does not contain the value of the input item. In this case, if the generation AI system 50 analyzes the face image of the customer, it will increase the cost in terms of time and processing load. Also, when the generation AI system 50 is on a pay-per-use basis, extra costs will occur.

[0186] Therefore, it is effective to enable the user to select the image data to be used for AI image analysis input on the input screen 210 of the business card management application.

[0187] Figure 26 shows the image data selection screen 330, which is displayed on the user terminal 10 as part of the input screen 210 of the business card management application or as a pop-up. When displayed as a pop-up, it appears when the "AI Image Analysis Input" button 228 is pressed. The image data selection screen 330 has a message 351 that reads, "Please select the attachment file form to be used as input for AI image recognition." If the business card management application has a business card image attachment field and a face image attachment field, the image data selection screen 330 has a checkbox 332 for selecting the business card image attachment field and a checkbox 333 for selecting the face image attachment field. The user wants the AI ​​system 50 to analyze only the image data in the business card image attachment field. Therefore, the user selects checkbox 332 (business card image attachment field).

[0188] By doing this, the request message created by the request generation unit 46a in step S4 of Figure 10 will only contain image data from the business card image attachment field with checkbox 332 checked. Therefore, the content of the prompt will be the same as in Figure 15.

[0189] <Request message specifying the range of values ​​to be entered> Some application input fields may have input ranges. For example, input fields that use strings as data may have a maximum and minimum number of characters that can be entered. Similarly, input fields that use numbers may have a maximum and minimum value that can be entered.

[0190] Figure 27 shows information about input fields that have input ranges. Compared to Figure 9, Figure 27 has a constraints field. The constraints field defines the input range for the value of the input field. For example, for an input field with the label field set to "Name", the input range is set to a minimum number of characters (minLength) of 1 and a maximum number of characters (maxLength) of 64.

[0191] When generating a prompt, the request generation unit 46a also includes information about the input range in the prompt. This prevents the value generated by the generation AI system 50 from being outside the application's input range.

[0192] Figure 28 is an example of a prompt that requests input field values ​​in JSON format without using function calls. Figure 28 illustrates the difference from Figure 15. The prompt in Figure 28 has additional text data 265 that reads, "The input range for the name is a minimum of 1 character and a maximum of 64 characters. If it exceeds the maximum number of characters, please truncate the end." The "1" and "64" in this text are modified by the constraints item in the input field information in Figure 27.

[0193] In other words, as shown below, the template in Figure 12 has ${……} set to correspond to the maximum and minimum number of characters. "The input range for names is a minimum of ${minLength} characters and a maximum of ${maxLength} characters. If the name exceeds the maximum number of characters, please truncate the end." The request generation unit 46a replaces ${minLength} with "1" and ${maxLength} with "64". Therefore, all text data 265 except for "1" and "64" are standard text.

[0194] The generation AI system 50 analyzes the text data 265 included in the prompt and generates a value for "Name" such that the value generated does not fall outside the input range.

[0195] Figure 29 shows an example of a request message when the generating AI system 50 has a function call function. The explanation of Figure 29 mainly describes the differences from Figure 24. In Figure 29, text data 341 has been added to name 304.

[0196] Text data 341 states, "The input range for names is a minimum of 1 character and a maximum of 64 characters. If the number of characters exceeds the maximum, please truncate the end." Text data 341 specifies that there is an input range for "Name" and how to handle cases where the input range is exceeded.

[0197] The generation AI system 50 analyzes the description of the function's arguments (that there is an input range, and how to handle cases where the input range is exceeded) and generates values ​​so that the value of the input item, name, does not fall outside the input range.

[0198] In addition to the input range, the data format for dates (YYYY year MM month DD day, YYYY / MM / DD), time data format (hhmmss), telephone number data format (with or without hyphens, etc.), fax number data format (with or without hyphens, etc.), postal code data format (with or without hyphens, etc.), address data format (with or without hyphens below the street number, etc.), and email address data format (containing only one @ symbol) may also be specified.

[0199] <<Main Effects>> According to this embodiment, in addition to the effects of the first and second embodiments, it is possible to prevent the values ​​generated by the generation AI system 50 from falling outside the input range of the application.

[0200] <Few-shotプロンプティング> The generation AI system 50 employs a technique called few-shot prompting, which improves the accuracy of value generation by providing several output examples in the prompt. In this embodiment as well, the accuracy of generating values ​​for the input items can be improved by performing few-shot prompting.

[0201] Figure 30 shows a prompt using few-shot prompting. Since few-shot prompting is a technique that includes one or more example outputs in the prompt, the request generation unit 46a includes the information already registered in the application service 40 (Figure 8) in the prompt. Note that Figure 30 mainly explains the differences from Figure 15.

[0202] Message 351 states, "For reference, we are providing two past inputs as sample1 and sample2 in a TypeScript implementation," notifying the user that the following information is registered in the application and that they should be used as reference.

[0203] The variable `const sample1` (with the code 352) indicates the first input. `const sample1` contains the values ​​for one record of information already registered in the application, mapped to the `name` field, which represents information about the input item.

[0204] The variable `const sample2` (with the code 353) indicates that it is the second input. `const sample2` contains the value for one record of information already registered in the application, associated with the `name` field, which represents information about the input item.

[0205] Figure 31 is a sequence diagram illustrating the processing of the information processing system 100 when using few-shot prompting. Figure 31 illustrates the differences from Figure 10. In Figure 31, the processing in steps S3 and S5 differs from that in Figure 10.

[0206] S3: The receiving unit 41a of the application service 40 receives the request for AI image analysis input. The method for identifying the list of input items, the application name, and image data (if not sent from the user terminal 10) can be the same as in Figure 10. Furthermore, the application identification unit 44c identifies one or more records registered in the application running on the user terminal 10 (for example, a business card management application) from the application information identified by the application identification information.

[0207] The request generation unit 46a of the application service 40 generates a request message using information about the application (business card image data, application name, list of input fields) and registered records. This request message includes instructions to extract information corresponding to the input fields from the image data.

[0208] S4: The request transmission unit 46b sends a request to the generation AI system 50 to generate a value for an input item along with this request message.

[0209] S5: The generation AI system 50 analyzes the image data to generate values ​​from the image data and determines which input field the generated values ​​correspond to by referring to one or more records. The generation AI system 50 sends a response message (input field and generated value) to the application service 40.

[0210] The subsequent steps can be the same as in Figure 10.

[0211] By performing few-shot prompting, the generation AI system 50 can more easily determine the value to associate with the input item, thereby improving the accuracy of value generation. Therefore, the generation AI system 50 can more easily generate values ​​that correspond to the input item.

[0212] <<Main Effects>> According to this embodiment, in addition to the effects of the first and second embodiments, the accuracy of generating the values ​​of the input items can be improved by performing few-shot prompting.

[0213] <When analyzing files other than image data> The generation AI system 50 can analyze files in formats other than text data and image data, such as document data, video data, or audio data. In this embodiment as well, in order to generate values ​​for input items, the user terminal 10 can send document data, video data, and audio data to the generation AI system 50 via the application service 40, similar to image data.

[0214] Figure 32 shows the request message that the application service 40 sends to the generation AI system 50. The explanation of Figure 32 will describe the differences from Figure 11. Messages 361, role 362, and content 363 are the same as in Figure 11. Each parameter has a set of "type" and its value. Parameter 364 now specifies "file_url" as the "type". When "type" is "file_url", the value of "url" is "file" 366. "file" 366 contains the URL where the file is saved or a Base64 encoded image.

[0215] Figure 33 shows an example of the string set for "Prompt" 365 in Figure 32. Figure 33 mainly explains the differences from Figure 12. In Figure 33, the first character of Figure 12 has changed from "Image" to "File" 365. The generation AI system 50 understands that it recognizes "File" 365 when analyzing the prompt, and by analyzing "File" included in the request message, it determines to generate the value for the input field of the business card management application.

[0216] In this way, the request generation unit 46a can change the format of the data to be analyzed by the generation AI system 50 by changing the prompt description. For example, in the case of video data, the generation AI system 50 can generate values ​​for input items even if the business card is captured in video. In the case of document data, the generation AI system 50 can generate values ​​for input items even if names, etc., are included in prose or forms. In the case of audio data, the generation AI system 50 can generate values ​​for input items even if names, etc., are included in conversations.

[0217] Furthermore, since multiple files can be specified in a prompt, the request generation unit 46a may include two or more of the image data, document data, video data, and audio data in a single prompt and send it to the generation AI system 50. In this case, "file" 365 is changed to "recognize images, documents, videos, and audio," etc.

[0218] <<Main Effects>> According to this embodiment, in addition to the effects of the first and second embodiments, the generation AI system 50 can analyze a text data or image data file and generate values ​​for input items.

[0219] [Other application examples] Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0220] For example, in this embodiment, the application service 40 transmits image data to the generation AI system 50, but the image data may be stored on a predetermined server. In this case, the application service 40 transmits information to the generation AI system 50 specifying the image data on the server. The generation AI system 50 retrieves the image data from the server and generates values ​​for the input items.

[0221] Furthermore, in this embodiment, the values ​​of the input items generated by the generation AI system 50 are in JSON format, but the values ​​of the input items may be in other formats such as XML or CSV.

[0222] Furthermore, in this embodiment, the user terminal 10 sets the generated value in the input field of the application managed by the application service 40. However, the user terminal 10 may also set the generated value in the input field of a native application running on its own device. For example, if the user terminal 10 is running a spreadsheet application, it may set the generated value in a cell of the spreadsheet application.

[0223] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the application service 40 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0224] Furthermore, the application service 40 can be configured to share the processing steps disclosed in this embodiment, such as those shown in Figure 10, in various combinations. For example, a process executed by a predetermined unit may be executed by multiple information processing devices of the application service 40. Also, the application service 40 may be consolidated into a single server device or divided among multiple devices.

[0225] Furthermore, the configuration examples in Figure 7 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the application service 40. The present invention is not limited by the way the processing units are divided or their names. The processing of the application service 40 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit contains even more processing.

[0226] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0227] Embodiments of the present invention bring about significant improvements in the capabilities and functionality of computers. These improvements enable users to utilize computers that provide more efficient and robust interaction with tables, which are methods for storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of a more efficient, powerful, and robust user interface. Such a user interface provides better interaction between humans and machines.

[0228] [Aspect] [Aspect 1] An information processing device that can communicate with a terminal device and a generating AI (Artificial Intelligence) system via a network, A screen displayed on the terminal device that accepts information input from the user into input fields for input items, and a registration unit that registers the information entered on a screen capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, a request transmission unit transmits a request to the generating AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A response receiving unit that receives information corresponding to the input items from the generation AI system as a response to the aforementioned request, A transmission unit transmits information corresponding to the input items sent from the generation AI system to the terminal device in order for the terminal device to display a screen in which the information is entered into the corresponding input field of the input item on the screen. An information processing device having [Aspect 2] The request transmission unit further transmits information to the generating AI system for calling a function that enables the registration unit to function. The response receiving unit receives information specifying the function and information corresponding to the input items from the generation AI system. Upon execution of the aforementioned function, the registration unit registers the information entered on the screen. The information processing apparatus according to Embodiment 1. [Aspect 3] The request transmission unit further transmits information to the generating AI system for calling a non-existent dummy function. The response receiving unit receives information specifying the dummy function and information corresponding to the input items from the generation AI system. The registration unit registers the information entered on the screen. The information processing apparatus according to Embodiment 1. [Aspect 4] The transmission unit transmits information corresponding to the input field of the input item to the terminal device, or The information corresponding to the aforementioned input item is transmitted to the terminal device, along with the information entered into the corresponding input field of the aforementioned input item on the screen. An information processing device according to any one of the three embodiments. [Aspect 5] The aforementioned screen is a screen for accepting the upload of the image data. The system further includes an identification unit that identifies the uploaded image data as image data to be extracted. An information processing device according to any one of embodiments 1 to 4. [Aspect 6] The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, It has a receiving unit that receives information identifying the application selected by the user, The identifying unit identifies the input items based on information that identifies the application. The information processing apparatus described in Embodiment 5. [Aspect 7] The system further includes a request generation unit that generates the aforementioned request. The information processing apparatus according to embodiment 3. [Aspect 8] The request transmission unit transmits to the generation AI system the data format of the input item and an instruction to extract the information described in the data format of the input item. An information processing device according to any one of embodiments 1 to 7. [Aspect 9] The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The request transmission unit transmits the request, including the list of input items and the name of the application, to the generation AI system. The information processing apparatus according to embodiment 8. [Aspect 10] The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The aforementioned application is an application created by accepting settings for the aforementioned input items from the user. An information processing device according to any one of the embodiments 1 to 9. [Aspect 11] The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The specified part is, An image identification unit that identifies the image data uploaded by the user, An input item identification unit that identifies the input items corresponding to the application, An application identification unit that identifies the application name of the application, The information processing apparatus according to aspect 5, further having [Aspect 12] The transmission unit transmits one or more programs to be executed on the terminal device to the terminal device, The terminal device can execute a web browser, The input processing unit that inputs the information corresponding to the input item received by the terminal device into the input field of the corresponding input item operates by executing the program transmitted from the information processing apparatus on the web browser. The information processing apparatus according to aspect 7.

Explanation of Signs

[0229] 10 User terminal 40 Application service 50 Generation AI system 100 Information processing system

Prior Art Documents

Patent Documents

[0230]

Patent Document 1

Claims

1. An information processing device that can communicate with a terminal device and a generating AI (Artificial Intelligence) system via a network, A screen displayed on the terminal device that accepts information input from the user into input fields for input items, and a registration unit that registers the information entered on a screen capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, the request transmission unit transmits a request to the generating AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A response receiving unit that receives information corresponding to the input items from the generation AI system as a response to the aforementioned request, A transmission unit transmits information corresponding to the input items sent from the generation AI system to the terminal device in order for the terminal device to display a screen in which the information corresponding to the input items on the screen is entered into the corresponding input fields of the input items on the screen. An information processing device having

2. The request transmission unit further transmits information to the generation AI system for calling a function that enables the registration unit to function. The response receiving unit receives information specifying the function and information corresponding to the input items from the generation AI system. Upon execution of the aforementioned function, the registration unit registers the information entered on the screen. The information processing apparatus according to claim 1.

3. The request transmission unit further transmits information to the generating AI system for calling a non-existent dummy function. The response receiving unit receives information specifying the dummy function and information corresponding to the input items from the generation AI system. The registration unit registers the information entered on the screen. The information processing apparatus according to claim 1.

4. The transmission unit transmits information corresponding to the input field of the input item to the terminal device, or The information corresponding to the aforementioned input item is transmitted to the terminal device, along with the information entered into the corresponding input field of the aforementioned input item on the screen. The information processing apparatus according to any one of claims 1 to 3.

5. The aforementioned screen is a screen for accepting the upload of the image data. The system further includes an identification unit that identifies the uploaded image data as image data to be extracted. The information processing apparatus according to claim 1.

6. The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, It has a receiving unit that receives information identifying the application selected by the user, The identifying unit identifies the input items based on information that identifies the application. The information processing apparatus according to claim 5.

7. The system further includes a request generation unit that generates the aforementioned request. The information processing apparatus according to claim 3.

8. The request transmission unit transmits to the generation AI system the data format of the input item and an instruction to extract the information described in the data format of the input item. The information processing apparatus according to claim 1.

9. The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The request transmission unit transmits the request, including the list of input items and the name of the application, to the generation AI system. The information processing apparatus according to claim 8.

10. The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The aforementioned application is an application created by accepting settings for the aforementioned input items from the user. The information processing apparatus according to claim 1.

11. The information processing device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The specified part is, An image identification unit that identifies the image data uploaded by the user, An input item identification unit that identifies the input items corresponding to the application, An application identification unit that identifies the application name of the application, The information processing apparatus according to claim 5, further comprising the above.

12. The transmitting unit transmits one or more programs to be executed on the terminal device to the terminal device. The aforementioned terminal device can run a web browser, The input processing unit that inputs information corresponding to the input item received by the terminal device into the input field of the corresponding input item operates by executing the program transmitted from the information processing device in the web browser. The information processing apparatus according to claim 7.

13. An information processing device capable of communicating with a generative AI (Artificial Intelligence) system, and a terminal device capable of communicating via a network, A display control unit that displays a screen that accepts input values ​​from the user into input fields for input items, and that can accept instructions to extract information to be entered into the input fields from image data, When an instruction is received on the aforementioned screen, the transmission unit sends a request to the information processing device that includes the image data to be extracted and an instruction to extract values ​​corresponding to input items from the image data. The system includes an input processing unit which receives a value corresponding to the input item obtained by the information processing unit transmitting the request to the generation AI system, and inputs the received value into the input field of the corresponding input item. The transmission unit transmits the value entered in the input field to the information processing device in order to manage it in the information processing device. Terminal device.

14. An information processing system that manages information entered by a user into input fields, and that can communicate with a terminal device and a generating AI (Artificial Intelligence) system via a network, The aforementioned information processing device is A screen displayed on the terminal device that accepts input of information into the input fields, and a registration unit that registers the input information on a screen capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, the request transmission unit transmits a request to the generating AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A response receiving unit that receives information corresponding to the input items from the generation AI system as a response to the aforementioned request, A transmission unit transmits information corresponding to the input items sent from the generation AI system to the terminal device in order for the terminal device to display a screen in which the information corresponding to the input items on the screen is entered into the corresponding input fields of the input items on the screen. An information processing system having [a certain feature].

15. An information processing device that can communicate with a terminal device and a generating AI (Artificial Intelligence) system via a network, A screen displayed on the terminal device that accepts information input from the user into input fields for input items, and a registration unit that registers the information entered on a screen capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, the request transmission unit transmits a request to the generating AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A response receiving unit that receives information corresponding to the input items from the generation AI system as a response to the aforementioned request, A transmission unit transmits information corresponding to the input items sent from the generation AI system to the terminal device in order for the terminal device to display a screen in which the information corresponding to the input items on the screen has been entered into the corresponding input fields of the input items on the screen. A program designed to function as such.

Images