Information processing equipment, server equipment, information processing system, program

Abstract

Extracting information from image data without having to prepare a pre-defined logic for extracting information from the image data. [Solution] The present invention relates to an information processing device comprising: a display control unit 12 that displays a screen for receiving information input from a user into an input field and which is capable of receiving instructions to extract information to be entered into the input field from image data; a request transmission unit 15 that, when the instruction is received on the screen, sends a request to a generation AI system including the image data to be extracted and instructions to extract information corresponding to an input item from the image data; a receiving unit 11a that receives information corresponding to an input item transmitted from the generation AI system; an input processing unit 16 that inputs the information corresponding to an input item into the corresponding input item; and a transmission unit 11b that transmits the information corresponding to the input item entered into the input field to the server device for the server device to manage the information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a server apparatus, an information processing system, and a program.

Background Art

[0002] There is known a technique for analyzing image data using AI and outputting an analysis result. An information processing apparatus equipped with AI can perform output of things shown in image data, classification of image data, anomaly detection, etc.

[0003] Patent Document 1 discloses a technique for extracting information such as a name and a company name from business card image data and registering it in a server.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the conventional technology, it was necessary to prepare in advance a logic for extracting a name, a company name, etc. from business card image data. For example, when a business card management application recognizes a character string by OCR (Optical Character Reader / Recognition), it is necessary for a developer to program in advance a logic for determining whether the character string is a name or a company name.

[0005] The present disclosure provides a technique for extracting information from image data without having to prepare in advance a logic for extracting information from the image data.

Means for Solving the Problems

[0006] The present invention relates to a server device that manages information entered by a user into input fields for input items, and an information processing device that can communicate with a generation AI (Artificial Intelligence) system via a network, comprising: a display control unit that displays a screen for receiving information from a user into the input fields and which can receive instructions to extract information to be entered into the input fields from image data; a request transmission unit that, when the instruction is received on the screen, transmits a request to the generation AI system including the image data to be extracted and an instruction to extract information corresponding to the input items from the image data; a receiving unit that receives information corresponding to the input items transmitted from the generation AI system; an input processing unit that inputs the received information corresponding to the input items into the corresponding input fields of the input items; and a transmission unit that transmits the information corresponding to the input items entered into the input fields to the server device for the server device to manage the information. [Effects of the Invention]

[0007] This invention allows information to be extracted from image data without having to prepare a logic for extracting information from image data in advance. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of an application creation screen displayed on a user's terminal. [Figure 2] This figure shows an example of the input screen of a business card management app displayed on a user's device. [Figure 3] This diagram illustrates the process by which a user enters values ​​into the input fields of a business card management app. [Figure 4] This diagram illustrates the flow of information setting services performed by an information processing system. [Figure 5] This is a diagram showing the system configuration of an example of an information processing system. [Figure 6] This diagram shows an example hardware configuration for an application service, a user terminal, and a developer terminal. [Figure 7] This diagram shows an example of the functional configuration of an application service and a user terminal. [Figure 8] This figure shows an example of the information set in the application's input fields. [Figure 9] This figure shows an example of information regarding the input fields of an application. [Figure 10] This is an example sequence diagram illustrating the processing of an information processing system. [Figure 11] This figure shows an example of parameters included in the request message that the user terminal sends to the AI ​​generation system in step S8 of Figure 10. [Figure 12] This diagram explains the prompt that is set in "Prompt" in Figure 11. [Figure 13] Figure 10 shows an example of the format of a response message sent from the AI ​​generation system to the user terminal in step S9. [Figure 14] This diagram shows an example of an input screen for a business card management app displayed on a user's terminal. [Figure 15] This figure shows an example prompt generated by the request generation unit. [Figure 16] This figure shows an example of a response message from the generation AI system to the request message shown in Figure 15. [Figure 17] This figure shows an example of the input screen of a business card management app with values ​​already set. [Figure 18] This diagram shows an example of an input screen for a book management application displayed on a user's device. [Figure 19] This diagram shows information about the input fields in an example book management application. [Figure 20] This figure shows an example prompt generated by the request generation unit. [Figure 21] This figure shows an example of a response message from the generation AI system to the request message in Figure 20. [Figure 22]It is a diagram showing an input screen of a book management app in a state where values are set in the book management app. [Figure 23] It is a diagram explaining a method of setting values of input items using a function call function. [Figure 24] It is a diagram showing an example of a request message including function arguments. [Figure 25] It is a diagram showing an example of a response message from a generation AI system when the generation AI system has a function call function. [Figure 26] It is a diagram showing an example of an image data selection screen displayed on a part of the input screen of a business card management app or a popup, etc. [Figure 27] It is a diagram showing information about an example of an input item having an input range. [Figure 28] It is a diagram showing an example of a prompt requesting a value of an input item in JSON format. [Figure 29] It is a diagram showing an example of a request message when the generation AI system has a function call function. [Figure 30] It is a diagram showing an example of a prompt using few-shot prompting. [Figure 31] It is a sequence diagram explaining the processing of an information processing system when using few-shot prompting. [Figure 32] It is a diagram showing an example of a request message that a user terminal sends to a generation AI system. [Figure 33] It is a diagram showing an example of a character string set in the "prompt" of FIG. 32.

Modes for Carrying Out the Invention

[0009] Hereinafter, as an example of a mode for carrying out the present disclosure, an information processing system and a setting method performed by the information processing system will be described.

[0010] <Regarding Application Services> Application services are services that assist users in creating applications using low-code or no-code methods. Such services are also known as visual programming. Application services send a web application that assists in application creation to the user's terminal. Users can then create various applications by interacting with the web application running on their terminal.

[0011] Figure 1 shows the application creation screen 200 displayed on the user terminal. The application creation screen 200, as an example, includes a form area 208 and a work area 209. The form area 208 displays a list of forms that can be placed in the application. A form is a display component that makes up a screen, and examples include input forms, selection forms, and file registration forms. Figure 1 shows a text form 201, a numeric form 202, a radio button form 203, a checkbox form 204, and an attachment form 205. These are just examples.

[0012] String form 201 is a form for entering text. Full-width characters, half-width characters, numbers, symbols, etc., can be entered into string form 201.

[0013] Numeric form 202 is a form for entering numerical values. Only numerical values ​​can be entered into Numeric form 202.

[0014] Radio button form 203 is a form for buttons that accept the selection of one option from multiple options. Radio button form 203 has fields for displaying the options.

[0015] Checkbox form 204 is a form for checkboxes that accepts the selection of one or more options from multiple choices. Checkbox form 204 has fields for displaying the options.

[0016] Attachment form 205 is a form that accepts settings for files to be registered in the application. Attachment form 205 may restrict the format of files that can be accepted for registration.

[0017] The workspace 209 is the area where the user places forms. The user can drag and drop forms from the form area 208 into the workspace 209 using the mouse pointer. Alternatively, the user can drag and drop forms into the workspace 209 using their finger or a pen on the touch panel.

[0018] The form placed in work area 209 is referred to as input field 206. Input field 206 has one or more input fields, but for simplicity, it may be simply referred to as input field below. Input field 206 can display a label 207. Label 207 is the name of input field 206. The user enters an appropriate label for input field 206. The user can repeat this process to create any application. For example, by placing a text form 201 in work area 209 and entering labels such as name and company, a business card management application like the one below can be created. Note that the position of an already placed input field 206 can also be changed in work area 209.

[0019] Figure 2 shows the input screen 210 of a business card management application displayed on a user's terminal. This business card management application has fields for name 211, company 212, department 213, job title 214, address 215, phone number 216, email address 217, URL 218, and business card image attachment 219. These are input fields. Users can use applications created in this way for business or personal use. For example, in a business card management application, the user enters the information written on business cards received from customers into each input field. This allows the business card management application to digitize the information. Alternatively, it becomes possible to share business card information within a team.

[0020] Figure 3 illustrates the process of a user entering values ​​into the input fields of a business card management application. Figure 3(a) shows the application list 220 displayed on the user's terminal. The application list 220 is a list of applications created and registered by the user in the application service using the procedure shown in Figure 1. As an example, the application list 220 displays the business card management application 221 and the book management application 222. Here, the user selected the business card management application 221. The application creation button 223 is a button that displays a screen like the one in Figure 1.

[0021] Figure 3(b) shows the record list 224 displayed by the user terminal. The record list 224 displays a table-like list of business card information registered in the business card management application 221. A record is a single row of data when data in a database is arranged in a two-dimensional table. One record represents one piece of business card information. A record 225 has multiple input fields. These input fields are sometimes called fields. A single column is called a column. In Figure 3(b), no business card information is registered, so the record list 224 is blank.

[0022] When the user presses the Add Record button 226, the input screen 210 of the business card management application is displayed. Figure 3(c) shows the input screen 210 of the business card management application. As described above, the input screen 210 of the business card management application has input fields created by the user for the business card management application 221. The user enters values ​​into the input fields and presses the Save button 227. The input content (in this case, business card information) is saved as a record in the application service.

[0023] Figure 3(d) shows the record list 224 displayed by the user terminal. The business card information entered on the input screen 210 of the business card management application in Figure 3(c) is displayed as one record.

[0024] Thus, since input is usually done using a keyboard, it placed a heavy burden on the user. Furthermore, there was a risk of input errors.

[0025] <Processing flow> Therefore, in this embodiment, the user terminal sends the file (business card image data) entered in the attached file form 205 and a list of application input items to the generation AI system and requests the generation of values ​​to be set in the input items. The generation AI system can perform various natural language processing tasks such as text generation, question answering, text classification, sentiment analysis, information extraction, and text summarization. In addition, generation AI systems such as Copilot®, which suggest the next code to be written while coding a program, are also known. In this embodiment, a technique for extracting information from image data without the user having to converse (also called chat) with the generation AI system will be described. Note that the generation AI system or its function may sometimes be simply referred to as AI (Artificial Intelligence).

[0026] The generation AI system analyzes image data and a list of input fields, generates values ​​to be entered into the input fields, and sends them to the user terminal. The generation AI system only needs to be provided as a separate system from the system that provides the application-related services; however, the application-related services and the generation AI system may be provided by the same provider. The user terminal sets the generated values ​​in the application service. Users can create any application they use for business or other purposes, not just a business card management application, using the application service. Therefore, the information processing system of this embodiment can generate appropriate values ​​from image data for the input fields of any application and set them in the input fields of the application service.

[0027] Figure 4 illustrates the process by which the information processing system 100 sets values ​​in the application service 40. A web application is running on the user terminal 10. The application for which values ​​are to be set in the input fields is assumed to have already been created in the application service 40. The generation AI system 50 analyzes image data such as business cards and a list of input fields for the application to generate values ​​to be set in the input fields.

[0028] (1) First, the user terminal 10, which is operated by the user, retrieves the configured image data from the application service 40. The image data is, for example, image data of a business card and contains the values ​​set in the input fields of the application service 40. If the user terminal 10 already has the image data, it does not need to retrieve it from the application service 40.

[0029] (2) The user terminal 10 also obtains the application name from the application service 40 to set a value in the input field. The application name is optional, or instead of the application name, the user may obtain the application description document (the application description, which has been set in advance by the user).

[0030] (3) The user terminal 10 also obtains a list of input items provided by the application from the application service 40. Note that (1) to (3) are not in any particular order, and the user terminal 10 may obtain the information from (1) to (3) simultaneously.

[0031] (4) The user terminal 10 sends the application information obtained in (1) to (3) (list of input items, image data, and application name) to the generating AI system 50 and obtains the values ​​to be set in the input items. That is, the generating AI system 50 analyzes the image data, the list of input items, and the application name. The generating AI system 50 performs character recognition on the training data and generates feature quantities such as what kind of image data it is, and generates values ​​corresponding to the input items.

[0032] (5) The user terminal 10 associates the values ​​obtained from the generation AI system 50 with the input items and sends them to the application service 40, thereby setting the values ​​in the input items of the application service 40.

[0033] Thus, the information processing system 100 of this embodiment can set values ​​for input items of any application.

[0034] <About Terminology> An application is an application program, a program generated by a computer in response to a specific task. While an OS (Operating System) is general-purpose software that provides basic functions and systems for the computer's operation (file system, communication, display control, etc.), an application provides specific functions on top of the OS. Applications include web applications and native applications. In this embodiment, either type may be developed.

[0035] An API is an application (software) interface, serving as a point of contact for connecting systems and sharing functions and mechanisms. An API defines the specifications for the interface that applications use to exchange information with each other. A computer-to-computer API is a specification for communication between one website and another via HTTP (HTTPS) communication. Through this communication, one website can use the functions of the other website. Computer-to-computer APIs are sometimes referred to as Web APIs.

[0036] An API takes a specific request (e.g., retrieving, updating, deleting, or processing data) and returns a result (e.g., data, update result, deletion result, processing result, etc.) in response. The request is sometimes called a request message, and the result is sometimes called a response message. Calling an API means sending a request according to the API's specifications and obtaining the result. Calling an API may also be described as executing, operating, hitting, or using it.

[0037] A user is an end-user who uses the applications provided by Application Service 40. Users can also develop applications. A developer is a person who configures Application Service 40 for the necessary settings for developing and using applications using no-code or low-code programming.

[0038] An input item is an item that has an input field for entering information. The input field can accept various types of information, including strings, numbers, symbols, image data, audio data, video data, files, and information for selecting options.

[0039] The information corresponding to an input item is the information that should be set in the input field of that input item. This information is contained in image data, for example, and is extracted from the image data by the generating AI system determining whether it corresponds to the input item based on the label, data format, etc. of the input item.

[0040] [First Embodiment] <Example System Configuration> Referring to Figure 5, the system configuration of the information processing system 100 according to this embodiment will be described. Figure 5 is a diagram showing the system configuration of an example of the information processing system 100. The information processing system 100 shown in Figure 5 has a user terminal 10, a developer terminal 60, and an application service 40. The user terminal 10 and the developer terminal 60 are connected to the application service 40 so as to be able to communicate via networks N1 and N2. The information processing system 100 may further have a generation AI system 50. The user terminal 10 and the developer terminal 60 are connected to the generation AI system 50 so as to be able to communicate via networks N1 and N2. The application service 40 may also be able to communicate with the generation AI system 50 via an API (Application Programming Interface).

[0041] The user terminal 10 and developer terminal 60 are located in facilities such as companies or homes and are connected to network N2. Network N2 can be a LAN, Wi-Fi (registered trademark), Wide Area Ethernet (registered trademark), 4G, 5G, 6G, or other mobile phone network. Network N1 is a wide-area network such as the Internet or WAN. The user terminal 10 and developer terminal 60 do not need to be constantly connected to network N2; they only need to be connected when using the generation AI system 50 or application service 40.

[0042] The generative AI system 50 provides a service that allows users to converse with artificial intelligence (AI) in natural language. A well-known example of such a generative AI system 50 is one that utilizes Large Language Models (LLMs). Large Language Models are natural language processing models trained using large amounts of text data. The generative AI system 50 pre-inputs a vast amount of text and learns knowledge from it using deep learning, reinforcement learning, etc. Using this knowledge, the generative AI system 50 provides response messages to chat messages. Chat messages include prompts and image data, which will be described later. A prompt refers primarily to the text data within a chat message.

[0043] The generation AI system 50, which generates text from data based on chat messages, is sometimes referred to as the generation AI. In this embodiment, the response messages provided by the generation AI system 50 are used to generate values ​​for input fields of an application running in the application service 40. The application running in the application service 40 includes a web application that runs on the application service 40 and a native application installed on the user terminal 10. When the native application installed on the user terminal 10 is executed on the user terminal 10, it connects to the application service 40 and executes the functions of the application service 40.

[0044] The generation AI system 50 has the following characteristics: 1. The generating AI system 50 conducts the conversation in a natural flow. 2. The generation AI system 50 can expand on ideas and propose them even in fields where the user lacks expertise. 3. The generation AI system 50 can output accurate program code. Taking advantage of these features, users can instruct the AI ​​generation system 50 to generate a list of application input items and image data, and receive values ​​to set in the input items from the AI ​​generation system 50.

[0045] One of the features of the third system is the function call function (also known as tool_call or function_call), but in this embodiment, it is possible to implement the system either with or without the function call function. However, since the tool_call returned by the generation AI system 50 is accurate (highly reproducible), there is a higher probability of obtaining the input item values ​​in JSON format.

[0046] As a generation AI system 50, systems using LLMs such as GPT-3, GPT-4, Transformer, and BERT (all registered trademarks) are known. In the information processing system 100 of this embodiment, ChatGPT using GPT-3 or GPT-4 can be used as an example, but other systems using LLMs may also be used.

[0047] Application Service 40 is one or more information processing devices that provide applications for users to run. Application Service 40 is also a server device that provides an application for managing information entered by users into input fields. The applications provided by Application Service 40 are, for example, database-type web applications that manage data in a tabular format. Users can create their own input fields for the application and customize it to save, read, or process data related to their work.

[0048] In this embodiment, the user terminal 10 obtains information about the application from the application service 40. The user terminal 10 sends the information about the application to the generation AI system 50 and receives a response message (input items and their values) from the generation AI system 50. The user terminal 10 sends the input items and their values ​​back to the application service 40. Therefore, the user can automatically set the information that would otherwise be manually entered from the image data into the application.

[0049] Examples of application services 40 include cloud services, ASPs (Application Service Providers), SaaS, etc., and may include various services provided over a network. For example, database provision and storage services are examples of such services. Application services 40 may reside on the internet or on-premises.

[0050] Furthermore, the application service 40 may have its functions distributed across multiple information processing devices, or there may be multiple application services 40 having the same function, and the number of information processing devices that distribute video may increase or decrease depending on the processing load.

[0051] In addition, a separate web server may exist in addition to the application service 40, and this web server may communicate with the user terminal 10. In this case, the user terminal 10 still communicates with the generation AI system 50, but the web server communicates with the application service 40 instead of the user terminal 10.

[0052] A server is a computer or software that provides information and processing results in response to requests from clients.

[0053] Furthermore, the application service 40 accepts various settings from the developer terminal 60. These settings include user registration with the application service 40 and registration of a web application for creating chat messages. In other words, the administrator performs the task of setting values ​​for the application's input fields using the generation AI system 50 with the application service 40.

[0054] The user terminal 10 or developer terminal 60 is, for example, a terminal device (an example of an information processing device) such as a PC (Personal Computer), smartphone, or tablet used by the user or developer. A web browser or native application runs on the user terminal 10 or developer terminal 60. The developer operates the developer terminal 60 to create configuration information for the application. The administrator or user can operate the developer terminal 60 and the user terminal 10 to use various services provided by the generation AI system 50 or application service 40.

[0055] The user terminal 10 or developer terminal 60 can be any information processing device. Examples of such information processing devices include output devices such as electronic whiteboards and digital signage, HUD (Head Up Display) devices, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, mobile phones, smartphones, tablet devices, car navigation systems, game consoles, PDAs (Personal Digital Assistants), digital cameras, and wearable PCs.

[0056] <Example Hardware Configuration> Referring to Figure 6, the hardware configurations of the application service 40, user terminal 10, and developer terminal 60 included in the information processing system 100 will be described. The generation AI system 50 is assumed to be the same as in Figure 6, or to be an information processing device hardware configuration compatible with cloud computing.

[0057] <<Application service 40, user terminal 10, and developer terminal>> Figure 6 shows an example of the hardware configuration of an application service 40, a user terminal 10, and a developer terminal 60. As shown in Figure 6, the application service 40, user terminal 10, and developer terminal 60 are built by a computer 500, which includes a CPU 501, ROM 502, RAM 503, HD (Hard Disk) 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, optical drive 514, and media I / F 516.

[0058] Of these, the CPU 501 controls the operation of the entire computer 500. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501. The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network I / F 509 is an interface for data communication using the network N2. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 6.

[0059] The keyboard 511 is a type of input means equipped with multiple keys used for inputting characters, numbers, or various instructions. The pointing device 512 is a type of input means used for selecting and executing various instructions, selecting processing targets, moving the cursor, etc. The optical drive 514 controls the reading or writing of various data to the optical storage medium 513, which is an example of a removable recording medium. The optical storage medium is a DVD, CD, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0060] <About the features> Next, with reference to Figure 7, the functional configuration of the information processing system 100 will be described. Figure 7 is a diagram showing an example of the functional configuration of the application service 40 and the user terminal 10.

[0061] <<User Terminal>> The user terminal 10 includes a communication unit 11, a display control unit 12, an operation reception unit 13, a request generation unit 14, a request transmission unit 15, an input processing unit 16, and a specification unit 17. Each of these functional units is a function or means realized by the CPU 501 shown in Figure 6 executing instructions contained in one or more programs installed on the user terminal 10. For example, the communication unit 11, the display control unit 12, the operation reception unit 13, the request generation unit 14, the request transmission unit 15, the input processing unit 16, and the specification unit 17 may be realized by a web browser and a web application. The web application is sent from the application service 40 to the user terminal 10. If the user terminal 10 runs a native application, these may be realized by the native application.

[0062] The communication unit 11 sends and receives various types of information between the application service 40 and the generation AI system 50. The communication unit 11 has a receiving unit 11a and a transmitting unit 11b. In this embodiment, the receiving unit 11a receives information about the application from the application service 40. The transmitting unit 11b transmits information about the application to the generation AI system 50. The receiving unit 11a receives values ​​to be set in input fields from the generation AI system 50. The generation AI system 50 exposes an API, and the transmitting unit 11b sends a request message including a chat message to the generation AI system 50 by calling the API. Thus, the request message is information including a chat message, but this is a general term for HTTP communication, and sometimes the chat message is referred to as the request message.

[0063] The display control unit 12 interprets screen information from various screens and displays it on the display 506. The operation reception unit 13 receives various user operations on the various screens displayed on the display 506.

[0064] The request generation unit 14 generates a request message that calls the API exposed by the generation AI system 50. The request message requests the generation AI system 50 to generate information. The request message includes a text portion called a prompt and image data (which may be the image itself or a URL). The request message may also include other data such as audio data.

[0065] The request transmission unit 15 sends a request to the generation AI system 50 that includes image data and instructions to extract values ​​corresponding to input items from the image data. This request is a request message generated by the request generation unit 14.

[0066] The input processing unit 16 inputs values ​​corresponding to the input items transmitted from the generation AI system 50 into the input fields of the input items.

[0067] The identification unit 17 identifies the image data uploaded to the generation AI system 50 as the image data to be extracted. The identification unit 17 further receives information from the generation AI system 50 that identifies the application selected by the user, and identifies the input items based on the information that identifies the application.

[0068] The identification unit 17 includes an image identification unit 17a, an input item identification unit 17b, and an application identification unit 17c. The image identification unit 17a identifies image data uploaded by the user. The input item identification unit 17b identifies input items corresponding to the application. The application identification unit 17c identifies the application name of the application.

[0069] <<Application Services>> The application service 40 includes a communication unit 41, a screen generation unit 42, a registration unit 43, a program transmission unit 44, and an application information storage unit 49. Each functional unit of the application service 40 is a function or means realized by the CPU 501 shown in Figure 6 executing instructions contained in one or more programs installed in the application service 40. The application information storage unit 49 is built on the HD 504 or RAM 503 shown in Figure 6. The application information storage unit 49 does not necessarily have to be present in the application service 40; it only needs to be on a network that the application service 40 can access.

[0070] The communication unit 41 sends and receives various information with the user terminal 10. The communication unit 41 sends application-related information to the user terminal 10 in response to requests from the user terminal 10. The communication unit 41 also sends the web application that the user terminal 10 runs and the screen information that the web application displays to the user terminal 10.

[0071] The screen generation unit 42 generates screen information for the screen displayed by the user terminal 10. The screen information is a program written in HTML, JSON, XML, a scripting language, and CSS (Cascading Style Sheet), and may be called a web application. The structure of the web page is mainly specified by HTML, the operation of the web application is defined by the scripting language, and the style of the web page is specified by CSS. In some cases, the user terminal 10 may run a native application. A native application is an application that cannot be run unless it is installed on the user terminal 10. In the case of a native application, the user terminal 10 holds the screen configuration, and the information to be displayed is sent in JSON, XML, etc.

[0072] The registration unit 43 manages application information for each application in the application information storage unit 49. The registration unit 43 registers values ​​corresponding to input items sent from the user terminal 10 in the application information storage unit 49. The application information includes information set in the application's input items and information about the input items that the application possesses.

[0073] Therefore, the application information storage unit 49 stores information set in the application's input fields and information related to those input fields (see Figures 8 and 9).

[0074] The program transmission unit 44 transmits a program to the user terminal 10 in response to a program request sent from the user terminal 10. This program is a web application, and furthermore, it is a web application that includes, for example, JavaScript (registered trademark).

[0075] Figure 8 shows an example of information set in the input fields of an application. This information includes both information manually set by the user and information generated by the AI ​​generation system 50. Figure 8 uses a business card management application as an example to illustrate the information set in the application's input fields. The information set in the input fields is managed on a record basis. In the case of a business card management application, the information for one record is called business card information. In a business card management application, the input fields include name, company, department, job title, etc., and a value is stored for each input field.

[0076] Figure 9 shows an example of information regarding application input fields. This information defines what kind of information is stored in each input field.

[0077] The `label` field is the name (the so-called label) of the input field displayed on the input screen 210 of the business card management app.

[0078] The `name` field is the identifier for the input field that application service 40 uses to manage and identify the input field.

[0079] The `type` field specifies the data format of the input field.

[0080] <Action or process> Next, we will explain the overall flow of processing performed by the information processing system 100 with reference to Figure 10. Figure 10 is a sequence diagram illustrating the processing of the information processing system 100. The business card management application has already been registered in the application service 40. In addition, the business card image data has been set in the application service 40, but it is assumed that no values ​​have been set in the other input fields.

[0081] S1: The user terminal 10 displays the input screen 210 of the business card management application. If the input screen 210 of the business card management application is implemented by a web application, the transmission unit 11b sends a request for one or more programs to be executed on the user terminal 10 to the application service 40. The program transmission unit 44 of the application service 40 sends the web application to the user terminal 10. The web application includes a program. The program displays a screen that accepts input values ​​from the user into input fields and can accept instructions to extract information to be entered into the input fields from image data. When an instruction is received on the screen, the program sends a request to the generation AI system 50 including the image data to be extracted and an instruction to extract values ​​corresponding to the input items from the image data. The program receives values ​​corresponding to the input items sent from the generation AI system 50, enters the received values ​​into the input fields of the corresponding input items, and sends the values ​​entered into the input fields to the application service 40 for management.

[0082] The input screen 210 of the business card management application displays the business card image data (thumbnail) in the business card image attachment field 219, as shown in Figure 14 below. The user starts AI image analysis input by pressing the "AI Image Analysis Input" button 228. The operation reception unit 13 of the user terminal 10 receives this operation. AI image analysis input is a series of processes that generate values ​​for input items by analyzing image data and set them in the application service 40.

[0083] S2: In response to this operation, the communication unit 11 of the user terminal 10 requests the application service 40 to provide the image data set in the business card image attachment field, specifying the application identification information and the currently displayed record ID (information that identifies the record).

[0084] S3: The communication unit 41 of the application service 40 receives the request, and the registration unit 43 retrieves the original image data displayed in the business card image attachment field 219, which is identified by the record ID, from the application information identified by the application identification information. The communication unit 41 of the application service 40 sends the image data set in the business card image attachment field 219 to the user terminal 10.

[0085] The receiving unit 11a of the user terminal 10 receives the original image data displayed in the business card image attachment field 219. If the business card image data has not yet been set in the application service 40, the user terminal 10 may use the business card image data stored in the HD 504 and does not need to obtain it from the application service 40.

[0086] S4: Next, the communication unit 11 of the user terminal 10 requests the application name from the application service 40 by specifying the application identification information. Since the application currently displayed by the user terminal 10 is identified, the application identification information is also known. If a value is to be set for an application that is not currently displayed by the user terminal 10, for example, the user selects the application. The reason for requesting the application name is that the accuracy of the generation AI system 50 in generating appropriate values ​​for the input items is improved by analyzing the application name. Therefore, the application name is not required. Alternatively, the user terminal 10 may request a description of the application instead of the application name. The description of the application may be, for example, "This application is an application for managing business cards."

[0087] S5: The communication unit 41 of the application service 40 receives the request, and the registration unit 43 obtains the application name from the application information identified by the application identification information. The communication unit 41 of the application service 40 sends the application name to the user terminal 10. The receiving unit 11a of the user terminal 10 receives the application name.

[0088] S6: Next, the communication unit 11 of the user terminal 10 requests a list of input items from the application service 40, specifying the application's identification information. More specifically, since the label item is appropriate as an input item used by the generation AI system 50, the receiving unit 11a receives a list of label items.

[0089] S7: The communication unit 41 of the application service 40 receives the request, and the registration unit 43 obtains a list of input items from the information about the input items identified by the application identification information. The communication unit 41 of the application service 40 sends the list of input items to the user terminal 10. The receiving unit 11a of the user terminal 10 receives the list of input items.

[0090] S8: Next, the request generation unit 14 of the user terminal 10 generates a request message (business card image data, application name, list of input items) using information about the application. This request message includes instructions to extract information corresponding to the input items from the image data. The request transmission unit 15 sends this request message along with a request to generate values ​​for the input items to the generation AI system 50. An example of this request message is shown in Figure 11.

[0091] S9: The generation AI system 50 analyzes the image data of the business card to generate values ​​for the input fields and determines which input fields those values ​​correspond to. The generation AI system 50 sends a response message (input fields and generated values) to the user terminal 10. The receiving unit 11a of the user terminal 10 receives the input fields and generated values.

[0092] S10: Next, the communication unit 11 of the user terminal 10 generates a request message requesting that the input items and generated values ​​be set in the application. The transmission unit 11b sends this request message to the application service 40.

[0093] S11: The communication unit 41 of the application service 40 receives a request message, and the registration unit 43 stores (registers) the value in the application information storage unit 49, associating it with the input item. The communication unit 41 of the application service 40 sends an input OK (value registration complete) message to the user terminal 10. The receiving unit 11a of the user terminal 10 receives the input OK message.

[0094] S12: The transmitting unit 11b of the user terminal 10 sends a screen update request to the application service 40, and the receiving unit 11a receives the latest application information, causing the input processing unit 16 to input values ​​corresponding to the input items into the input fields of the corresponding input items. The display control unit 12 displays the input screen 210 of the business card management application with values ​​set in the input items.

[0095] <Example of a request message from user terminal 10 to generation AI system 50> Figure 11 shows the parameters included in the request message that the user terminal 10 sends to the generation AI system 50 in step S8 of Figure 10.

[0096] • messages241 is the API for the generation AI system 50, and the following indicates that it is a chat message.

[0097] Role242 is an API for the Generating AI System 50 and is a classification of the requester of a request message. It includes user, assistant (AI of the Generating AI System 50), and system (instruction for configuration by the AI ​​assistant).

[0098] • content243 is the API for the generation AI system 50, and it is where the dialogue text is set. Since content is an array, it is also possible to specify a prompt and multiple image data. In Figure 11, three parameters 244-246 are described in JSON format. Two of them are image data.

[0099] Parameters 244-246 are the format of the information to be sent to the generating AI system 50. "type" defines the data type. If "type" is "text", the value of "text" is "prompt" 247. A prompt is set in "prompt" 247. An example of a prompt set in "prompt" 247 is shown in Figure 12. If "type" is "image_url", the value of "image_url" is "image"248,249, where "image"248,249 contains either the URL where the image data is stored or the Base64 encoded image. Note that if the application has only one input field for image data, then only one value each of parameters 245 and 246 is needed.

[0100] As shown in Figure 11, a request message combining a prompt and an image is sent to the generation AI system 50.

[0101] Figure 12 is a diagram illustrating the prompt set in "Prompt" 247 in Figure 11. The string in Figure 12 is a template for the request generation unit 14 to generate the prompt. The string has four ${...}. When sending a request message, information about the application is set in ${appName}251, ${labels.join()}252, ${labels.length}253, and ${type}254. That is, the four ${...} are replaced with information about the application. The other strings are boilerplate text and are held in advance by the request generation unit 14. The prompt's beginning, "Analyze the image" 266, requests that the image data specified by parameter 245 included in the request message be analyzed.

[0102] • ${appName}251 will be set to the app name. The app name is optional, or it can be a description of the application (that explains what the app is about).

[0103] The variable ${labels.join()}252 will contain a list of input fields.

[0104] The value ${labels.length}253 is set to the number of input fields.

[0105] The variable ${type}254 specifies the data format of the input items to be returned in the response message. TypeScript is a statically typed programming language that allows you to declare the data type of variables within the code. In Figure 12, JSON format is specified for ${type}254. In other words, the prompt in Figure 12 instructs the generating AI system 50 to return the input items and their values ​​in JSON format. A specific example of the settings is explained in Figure 15.

[0106] <Example of a response message from the generating AI system 50 to the user terminal 10> Figure 13 shows the format of the response message sent from the generating AI system 50 to the user terminal 10 in step S9 of Figure 10. In other words, Figure 13 is the format, not the response message itself.

[0107] Messages255 indicates that the following is the response message:

[0108] • Role 256 is the classification of the sender that sends the response message, and in this case it is assistant (AI of Generating AI System 50).

[0109] • content257 is the content of the response message. Here, content is response 258 (input items and values) from the generating AI system 50. Details of response 258 from the generating AI system 50 are explained in Figure 16.

[0110] <Example of setting values ​​for input fields using image data> The following examples illustrate how to set values ​​for input fields using image data, using a business card management app and a book management app as examples.

[0111] <<Example of setting information using business card image data>> Referring to Figure 14, etc., an example of setting values ​​in the input fields of the business card management application will be explained. Figure 14 is the input screen 210 of the business card management application as displayed on the user terminal 10. The explanation of Figure 14 will mainly focus on the differences from Figure 2. In Figure 14, the user has entered business card image data into the business card image attachment field 219. The user can manually enter values ​​into each input field on the input screen 210 of the business card management application.

[0112] The business card image attachment field 219 displays a thumbnail of the business card image data. In this state, the user presses the "AI Image Analysis Input" button 228. The "AI Image Analysis Input" button 228 accepts instructions to extract values ​​to be entered into the input fields from the image data. The "AI Image Analysis Input" button 228 may become active (pressable) only when a business card image has been entered into the business card image attachment field 219.

[0113] This section describes information about the business card management app application. First, the app name is "Business Card Management App". Next, according to the information about input items in Figure 9, the list of input items (label items) is name, company, department, job title, address, telephone number, email address, URL, and business card image attachment field. Of these, the business card image attachment field does not require AI image analysis input. Therefore, the request generation unit 14 does not include the business card image attachment field in the prompt. By making only input items with a type of String in the information about input items into the prompt, the request generation unit 14 can exclude input items that do not require value input from the prompt. Therefore, the list of input items is name, company, department, job title, address, telephone number, email address, and URL. Also, there are 8 input items.

[0114] The input screen 210 of the business card management application is a screen for uploading image data, and the identification unit 17 identifies the uploaded image data as the image from which values ​​will be extracted. The receiving unit 11a receives information identifying the application selected by the user, and the identification unit 17 identifies the input items based on the information identifying the application (using the information regarding input items in Figure 9). The image identification unit 17a identifies the image data uploaded by the user. The input item identification unit 17b identifies the input items corresponding to the application. The application identification unit 17c identifies the application name of the application.

[0115] Figure 15 shows the prompt generated by the request generation unit 14. The following information is set in ${appName}251, ${labels.join()}252, and ${labels.length}253 shown in Figure 12.

[0116] • ${appName}251 is set to "Business Card Management App" 261.

[0117] The label field is set to ${labels.join()}252, which contains the fields "Name, Company, Department, Position, Address, Phone Number, Email Address, URL". In other words, the label field is set for information related to the input field. The reason the name field is not set is that the name field is usually identification information, and it often contains information unrelated to the label (making it difficult for the generating AI system 50 to determine what the input field is).

[0118] • ${labels.length}253 is set to "8"263.

[0119] Furthermore, the ${type}254 shown in Figure 12 is set to the data format 264 for each input item. The data format 264 is an instruction to extract the information described in this data format. "name?: string, company?: string, department?: string, position?: string, address?: string, telephoneNumber?: string, emailAddress?: string, URL?: `false` These are the values ​​for the `name` and `type` fields among the information regarding the input items in Figure 9. The `name` field in data format 264 is arranged in the same order as in 262: "Name, Company, Department, Position, Address, Phone Number, Email Address, URL".

[0120] The reason the `name` field in data format 264 is not the `label` field is that the user terminal 10 sends the value of the input field returned by the generation AI system 50 to the application service 40. The application service 40 identifies the input field by the value of the `name` field, not the value of the `label` field. However, the value of the `label` field may also be used. In this case, when setting the value to the application service 40, the request generation unit 14 converts the `label` field to the `name` field.

[0121] The "?" at the end of the name field means that if there are input fields for which no value is found in the image data, the generation AI system 50 may omit the input fields. Note that "string" is the value (data type) of the type field.

[0122] The generating AI system 50 understands from the prompt "Analyze the image" 266 at the beginning of the prompt that it will analyze the image data specified by the parameter 245 included in the request message. Next, the generating AI system 50 attempts to generate values ​​for "Name, Company, Department, Position, Address, Phone Number, Email Address, URL" 262 from the image data. Then, the generating AI system 50 determines which input items were able to generate values ​​based on the order of the data format 264 and associates the input items with the generated values.

[0123] Figure 16 shows the response message from the generating AI system 50 to the request message in Figure 15. The generating AI system 50 analyzes the business card image data and obtains values ​​corresponding to "name, company, department, job title, address, phone number, email address, URL" 262. The generating AI system 50 then associates these values ​​with the name field (input field below) contained in the prompt data format 264 and returns them.

[0124] Input field: Value Name: Taro Patent Company: Sample1 Co., Ltd. Department: Sales Department URL: https: / / sample.co.jp Here, the reason why position, address, telephoneNumber, and emailAddress, which are input items included in the prompt data format 264, are not present in the request message is that they are not included in the business card image data or that the generating AI system 50 was unable to detect them.

[0125] The user terminal 10 uses the response message shown in Figure 16 to request the application service 40 to set a value in the business card management application.

[0126] Figure 17 shows the input screen 210 of the business card management application with values ​​set in the application. The input processing unit 16 enters the values ​​corresponding to the input items into the input fields of the corresponding input items. In other words, the values ​​included in the response message are set in the corresponding input items. The values ​​generated by the generation AI system 50 are set in the name field 211, company field 212, department field 213, and URL field, respectively.

[0127] Furthermore, the values ​​entered into the input fields by the input processing unit 16 can be edited manually by the user. Also, when the user presses a button such as "Register" on the input screen 210 of the business card management application, the process of step S10 in Figure 10 is executed.

[0128] In this way, by specifying business card image data, the user can automatically set the values ​​obtained from analyzing the business card image data into the corresponding input fields. Since the user terminal 10 obtains information about the application from the application service 40, it can set appropriate values ​​extracted from the image data into the input fields of any application, not just the business card management application.

[0129] <<Example of setting information using image data of a book cover>> Next, referring to Figure 18, etc., we will explain an example of setting values ​​in the input fields of the book management application. Figure 18 is the input screen 270 of the book management application displayed on the user terminal 10. This book management application has a title field 271, a subtitle field 272, an author field 273, a publisher field 274, a description of the cover appearance 275, a cover image attachment field 276, and a back cover image attachment field 277. These are input fields. Users can use the book management application for business or personal use. For example, a user can enter information about books they have purchased or finished reading into the input fields of the book management application. This allows them to digitize and list the information about books they have purchased or finished reading.

[0130] To set values ​​in the input fields using the generation AI system 50, the user entered image data of the book into the cover image attachment field 276 and the back cover image attachment field 277. The cover image attachment field 276 displays a thumbnail of the image data of the book's cover. The back cover image attachment field 277 displays a thumbnail of the image data of the book's back cover. Either the cover image attachment field 276 or the back cover image attachment field 277 may be filled in.

[0131] In this state, the user presses the "AI Image Analysis Input" button 278. Note that the "AI Image Analysis Input" button 278 may become active (pressable) only when image data has been entered into at least one of the cover image attachment field 276 or the back cover image attachment field 277.

[0132] Figure 19 shows information about the input fields of the book management application. Similar to the information about the input fields of the business card management application (Figure 9), the information about the input fields of the book management application includes label, name, and type fields.

[0133] This section describes information about the book management app application. First, the app name is "Book Management App". Next, according to the information about input items in Figure 19, the list of input items (label items) is title, subtitle, author, publisher, description of the cover appearance, cover image attachment field, and back cover image attachment field. Of these, the cover image attachment field and back cover image attachment field do not require AI image analysis input. Therefore, the request generation unit 14 does not need to include the cover image attachment field and back cover image attachment field in the prompt. By using only input items with a type of String in the information about input items as prompts, the request generation unit 14 can exclude input items that do not require a value from the prompt. Therefore, the list of input items is title, subtitle, author, publisher, and description of the cover appearance. Also, there are 5 input items.

[0134] Figure 20 shows the prompt generated by the request generation unit 14. The following information is set in ${appName}251, ${labels.join()}252, and ${labels.length}253 shown in Figure 12.

[0135] • ${appName}251 is set to "Book Management App" 281.

[0136] • ${labels.join()}252 has "Title, subtitle, author, publisher, cover appearance description" 282 set.

[0137] • ${labels.length}253 is set to "5" (283).

[0138] Furthermore, the ${type}254 shown in Figure 12 is set to the data format 284 for each input item. "title?: string, subtitle?: string, author?: string, Publisher?: large, cover?: string, g" These are the values ​​of the name and type fields among the information about the input items in Figure 19. The name field in data format 284 is arranged in the same order as "Title, Subtitle, Author, Publisher, Cover Description" 282.

[0139] The reason the `name` field in data format 264 is not the `label` field is that the user terminal 10 sends the value of the input field returned by the generation AI system 50 to the application service 40. The application service 40 identifies the input field by the value of the `name` field, not the value of the `label` field. However, the value of the `label` field may also be used. In this case, when setting the value to the application service 40, the request generation unit 14 converts the `label` field to the `name` field.

[0140] The "?" and "string" at the end of the name field can be treated the same as in Figure 15.

[0141] The generating AI system 50 understands from the prompt "Analyze the image" 289 at the beginning of the prompt that it will analyze the image data specified by parameter 245 in the request message. Next, the generating AI system 50 attempts to generate the values ​​"Title, subtitle, author, publisher, and cover appearance description" 282 from the image data. Then, the generating AI system 50 determines which input items were able to generate values ​​based on the order of the data format 284 and associates the input items with the generated values.

[0142] Figure 21 shows the response message from the generating AI system 50 to the request message in Figure 20. The generating AI system 50 analyzes the business card image data and obtains values ​​corresponding to "title, subtitle, author, publisher, and cover appearance description" 282. The generating AI system 50 then associates these values ​​with the name field (input field below) contained in the prompt data format 284 and returns them.

[0143] Input field: Value Title: Caterpillar Encyclopedia author : Trademark Hanako publisher:○×publishing Cover: The cover has a green background with an illustration of a purple caterpillar.

[0144] The reason why the subtitle is not present in the request message among the input items queried is that it is not included in the image data of the front or back cover, or that the generation AI system 50 was unable to detect it.

[0145] Note that the value corresponding to "cover," which is "The cover has a green background and an illustration of a purple caterpillar," is not included as text in the image data. This value is what the generating AI system 50 converted into text data to describe the image data. Therefore, the information processing system 100 of this embodiment can automatically set information that is not included as text in the image data in the application service 40.

[0146] The user terminal 10 uses the response message shown in Figure 21 to request the application service 40 to set a value in the book management application.

[0147] Figure 22 shows the input screen 270 of the book management app with values ​​set in the app. The values ​​included in the response message in Figure 21 are set in the corresponding input fields. The input processing unit 16 enters the values ​​corresponding to the input fields into the corresponding input fields. That is, values ​​are set in the title field 271, the author field 273, the publisher field 274, and the cover appearance description field 275.

[0148] Furthermore, the values ​​entered into the input fields by the input processing unit 16 can be edited manually by the user. Also, when the user presses a button such as "Register" on the input screen 270 of the book management application, the process in step S10 of Figure 10 is executed.

[0149] In this way, by specifying multiple image data of a book, the user can automatically set the values ​​obtained from analyzing these image data into the corresponding input fields. Since the user terminal 10 obtains information about the application from the application service 40, it can set appropriate values ​​extracted from the image data into the input fields of any application, not just the book management application.

[0150] <Main effects> According to the information processing system 100 of this embodiment, the user can specify image data containing the values ​​of the input items, and the generating AI system 50 can analyze the image data and automatically set the obtained values ​​in the input items. Since the user terminal 10 obtains information about the application from the application service 40, it can set appropriate values ​​extracted from the image data in the input items of any application, not just one application.

[0151] [Second Embodiment] In this embodiment, an information processing system 100 that sets values ​​in input items using the function call function provided by the generation AI system 50 will be described.

[0152] In this embodiment, the hardware configuration diagram shown in Figure 6 and the functional block diagram shown in Figure 7, which were described in the first embodiment, can be used as references.

[0153] <Request message when function call functionality is available> Some generation AI systems 50 have a function call function. The user terminal 10 specifies a function and the format of its arguments to the generation AI system 50, and the generation AI system 50 generates the function arguments in the specified format. This function is called the function call function. However, there is no function to be called on the user terminal 10 or the application service 40 (even if there were, it would not cause any problems, but in this embodiment there is no such function so it is called a dummy function), and even though the term "function call function" is used, the generation AI system 50 does not actually call a function on the user terminal 10 or the application service 40. In this embodiment, the user terminal 10 specifies a function and the format of its arguments to the generation AI system 50 using the function call function in order to more reliably obtain the values ​​of the input items from the generation AI system 50 in the format specified by the user terminal 10.

[0154] By using the function call function, the accuracy with which the generating AI system 50 returns values ​​in JSON format can be improved compared to when the user terminal 10 requests the generating AI system 50 to generate input items and their values ​​in JSON format, and the generating AI system 50 generates the values.

[0155] <Processing flow using function call functionality> Figure 23 illustrates how to set the values ​​of input items using the function call function. It assumes that the user terminal 10 generates a request message to the AI ​​generation system 50.

[0156] (1) The user terminal 10 sends a request message including the format of the function's arguments. A function is a program interface that performs a specific process based on specified arguments and returns a return value as a result. However, in this embodiment, no functions exist in the user terminal 10 or the application service 40. By the user terminal 10 including the format of the function (dummy function)'s arguments in the request message and sending it to the generation AI system 50, it is expected that the function call from the generation AI system 50 will include the values ​​of the input items in the format specified by the user terminal 10.

[0157] The act of the generating AI system 50 requesting the user terminal 10 to call this dummy function, which does not actually exist, is called a function call (tool_call in this embodiment).

[0158] Furthermore, it is not a problem if the function actually exists on the user terminal 10 or the application service 40, and the user terminal 10 may set a value in the application service 40 by executing the function.

[0159] The user terminal 10 does not have to include the format of the function arguments in the same request message as the application information. For example, the user terminal 10 may send the format of the function arguments to the generating AI system 50 in a separate request message from the request message containing the application information.

[0160] (2) Based on the application information contained in the sent request message and the format of the function arguments, the generating AI system 50 sends a response message containing a function call (tool_call) to the user terminal 10. Here, although it is said that the generating AI system 50 requests a function call from the user terminal 10, it does not request the execution of the function; rather, the generating AI system 50 merely proposes values ​​for the input items in the specified format.

[0161] In other words, the generating AI system 50 includes the application input items and values ​​sent from the user terminal 10 as function arguments in the response message. These values ​​are generated by analyzing information about the application.

[0162] (3) The user terminal 10 obtains the input items and values ​​included in the function call (tool_call) contained in the response message sent from the generation AI system 50, and requests the application service 40 to set the values ​​in the input items. More specifically, the user terminal 10 generates a request message to call the API of the application service 40 and sends the generated request message to the application service 40. In this embodiment, the user terminal 10 sets the values ​​of the input items in the application service 40 in response to the function call (it does not execute the called function).

[0163] <Example of a request message to the generation AI system 50 when the generation AI system 50 has a function call function> Figure 24 shows an example of a request message including function arguments. The request message in Figure 24 is assumed to be for a business card management application. Messages 291, role 292, and content 293 are the same as in Figure 11. Parameter 294 indicates that the input item "type" is "image_url", and the value of "image_url" is "image" 320. "image" 320 can contain, for example, a URL where business card image data is stored or a Base64 encoded image.

[0164] Furthermore, the parameter 244 (prompt) in Figure 11 has been replaced with tool295.

[0165] • tools295 is the API for the Generative AI System 50, and below it means specifying the format of the function arguments.

[0166] • "type": "function" 296 indicates that the object's type is a function.

[0167] Function297 indicates that it is a description of a function. Name298 is the function name, and description299 is the function's features. Both are pre-held by the request generation unit 14. Parameters301 is a description of the function's arguments, and "type":"object"302 indicates that the arguments are described in object format. Function297, name298, description299, and parameters301 are all APIs of the generation AI system 50.

[0168] properties303 describes a list of information related to the input fields of the business card management application in a nested JSON structure. In other words, properties303 requests the generating AI system 50 to return the function arguments in JSON format.

[0169] • name304 specifies how to return a value for the input field "name". The input field "name" is obtained from the name field of the information about the input field. Therefore, name304 specifies that the "type" of the input field "name" is "string". "description" specifies that the input field "name" should return "name". This "name" means that the generation AI system 50 should return what it has determined to be a name after analyzing the image data.

[0170] The same description applies to the subsequent input fields: company305, department306, position307, address308, telephoneNumber309, emailAddress310, and url311.

[0171] Thus, tools295 contains a list of input items among the application information. Image data is included in parameter 245 in Figure 11, as in the first embodiment. The request message in Figure 24 does not include the application name, but it may include it. On the other hand, name298 or description299 serve the role of the application name, and these may be considered as the application name.

[0172] The generating AI system 50 understands, via name298 or description299, that it will analyze the image data specified by parameter 245 included in the request message. Next, the generating AI system 50 attempts to generate the values ​​specified by name304, company305, department306, position307, address308, telephoneNumber309, emailAddress310, and url311 (name, company, department, job title, address, telephone number, email address, URL) from the image data. Finally, the generating AI system 50 returns the generated values ​​in JSON format.

[0173] <Example of a response message from the generating AI system 50 to the user terminal 10> Figure 25 shows an example of a response message from the generating AI system 50 when the generating AI system 50 has a function call function. Messages 321, role 322, and content 323 are the same as in Figure 13. In the response message in Figure 25, the generating AI system 50 requests a function call from the user terminal 10.

[0174] • tool_calls324 indicates that the following is a function call. That is, the generating AI system 50 is requesting a call to a dummy function that does not actually exist on the user terminal 10. However, the function may actually exist.

[0175] • The value "type": "function" indicates that the object's type is a function.

[0176] ·function326 indicates a description about a function.

[0177] ·name327 is the function name.

[0178] ·arguments328 means the arguments of the function. The following input items and values are included in the arguments. That is, the generation AI system 50 analyzes the image data of the business card and generates it in the JSON format specified by properties303 in FIG. 24, associating the name item (input item) with the value.

[0179] Input item: Value name: Patent Taro company: Sample1 Co., Ltd. department: Sales Department url: https: / / sample1.co.jp" It can be seen that these input items and values match the information included in the response message in FIG. 16 of the first embodiment. The user terminal 10 uses the response message in FIG. 25 to request the application service 40 to set the values to the business card management app. As a result, as shown in FIG. 17, the values of the input items are set on the input screen 210 of the business card management app.

[0180] <Main effects> According to this embodiment, in addition to the effects of the first embodiment, the accuracy of the generation AI system 50 returning values in the JSON format can be improved. Since the user terminal 10 can obtain the values of the input items in the JSON format, the application service 40 can surely set the values of the input items.

[0181] [Third Embodiment] In this embodiment, a modification example common to the first embodiment and the second embodiment will be described.

[0182] [Selection of Image Data Used for AI Image Analysis Input] An application record may contain multiple image data. For example, a business card management application might have fields for attaching a business card image and a face image. The image data in the face image attachment field is a customer's face image, so it does not contain the values ​​of the input fields. In this case, if the AI ​​generation system 50 analyzes the customer's face image, it will increase costs in terms of both time and processing load. Furthermore, if the AI ​​generation system 50 is billed on a pay-per-use basis, extra costs will be incurred.

[0183] Therefore, it is effective to allow the user to select the image data to be used for AI image analysis input on the input screen 210 of the business card management app.

[0184] Figure 26 shows the image data selection screen 330, which is displayed on the user terminal 10 as part of the input screen 210 of the business card management application or as a pop-up. When displayed as a pop-up, it appears when the "AI Image Analysis Input" button 228 is pressed. The image data selection screen 330 has a message 351 that reads, "Please select the attachment file form to be used as input for AI image recognition." If the business card management application has a business card image attachment field and a face image attachment field, the image data selection screen 330 has a checkbox 332 for selecting the business card image attachment field and a checkbox 333 for selecting the face image attachment field. The user wants the AI ​​system 50 to analyze only the image data in the business card image attachment field. Therefore, the user selects checkbox 332 (business card image attachment field).

[0185] By doing this, the request message created by the request generation unit 14 in step S8 of Figure 10 will only contain the image data from the business card image attachment field with checkbox 332 checked. Therefore, the content of the prompt will be the same as in Figure 15.

[0186] <Request message specifying the range of values ​​to be entered> Some application input fields may have input ranges. For example, input fields that use strings as data may have a maximum and minimum number of characters that can be entered. Similarly, input fields that use numbers may have a maximum and minimum value that can be entered.

[0187] Figure 27 shows information about input fields that have input ranges. Compared to Figure 9, Figure 27 has a constraints field. The constraints field defines the input range for the value of the input field. For example, for an input field with the label field set to "Name", the input range is set to a minimum number of characters (minLength) of 1 and a maximum number of characters (maxLength) of 64.

[0188] When generating a prompt, the request generation unit 14 also includes information about the input range in the prompt. This prevents the value generated by the generation AI system 50 from being outside the application's input range.

[0189] Figure 28 is an example of a prompt that requests input field values ​​in JSON format without using function calls. Figure 28 illustrates the difference from Figure 15. The prompt in Figure 28 has additional text data 265 that reads, "The input range for the name is a minimum of 1 character and a maximum of 64 characters. If it exceeds the maximum number of characters, please truncate the end." The "1" and "64" in this text are modified by the constraints item in the input field information in Figure 27.

[0190] In other words, as shown below, the template in Figure 12 has ${……} set to correspond to the maximum and minimum number of characters. "The input range for names is a minimum of ${minLength} characters and a maximum of ${maxLength} characters. If the name exceeds the maximum number of characters, please truncate the end." The request generation unit 14 replaces ${minLength} with "1" and ${maxLength} with "64". Therefore, all text data 265 except for "1" and "64" are standard text.

[0191] The generation AI system 50 analyzes the text data 265 included in the prompt and generates a value for "Name" such that the value generated does not fall outside the input range.

[0192] Figure 29 shows an example of a request message when the generating AI system 50 has a function call function. The explanation of Figure 29 mainly describes the differences from Figure 24. In Figure 29, text data 341 has been added to name 304.

[0193] Text data 341 states, "The input range for names is a minimum of 1 character and a maximum of 64 characters. If the number of characters exceeds the maximum, please truncate the end." Text data 341 specifies that there is an input range for "Name" and how to handle cases where the input range is exceeded.

[0194] The generation AI system 50 analyzes the description of the function's arguments (that there is an input range, and how to handle cases where the input range is exceeded) and generates values ​​so that the value of the input item, name, does not fall outside the input range.

[0195] In addition to the input range, the data format for dates (YYYY year MM month DD day, YYYY / MM / DD), time data format (hhmmss), telephone number data format (with or without hyphens, etc.), fax number data format (with or without hyphens, etc.), postal code data format (with or without hyphens, etc.), address data format (with or without hyphens below the street number, etc.), and email address data format (containing only one @ symbol) may also be specified.

[0196] <<Main Effects>> According to this embodiment, in addition to the effects of the first and second embodiments, it is possible to prevent the values ​​generated by the generation AI system 50 from falling outside the input range of the application.

[0197] <Few-shotプロンプティング> The generation AI system 50 employs a technique called few-shot prompting, which improves the accuracy of value generation by providing several output examples in the prompt. In this embodiment as well, the accuracy of generating values ​​for the input items can be improved by performing few-shot prompting.

[0198] Figure 30 shows a prompt using few-shot prompting. Since few-shot prompting is a technique that includes one or more example outputs in the prompt, the request generation unit 14 includes the information already registered in the application service 40 (Figure 8) in the prompt. Note that Figure 30 mainly explains the differences from Figure 15.

[0199] Message 351 states, "For reference, we are providing two past inputs as sample1 and sample2 in a TypeScript implementation," notifying the user that the following information is registered in the application and that they should be used as reference.

[0200] The variable `const sample1` (with the code 352) indicates the first input. `const sample1` contains the values ​​for one record of information already registered in the application, mapped to the `name` field, which represents information about the input item.

[0201] The variable `const sample2` (with the code 353) indicates that it is the second input. `const sample2` contains the value for one record of information already registered in the application, associated with the `name` field, which represents information about the input item.

[0202] Figure 31 is a sequence diagram illustrating the processing of the information processing system 100 when using few-shot prompting. Figure 31 illustrates the differences from Figure 10. In Figure 31, steps S21 and S22 are added. Also, some of the processing in steps S8 and S9 is changed.

[0203] S21: The request generation unit 14 of the user terminal 10 specifies the application's identification information and requests the application service 40 for one or more records of information previously registered in the application.

[0204] S22: The communication unit 41 of the application service 40 receives a request. The registration unit 43 retrieves one or more samples of information registered in the application from the application information storage unit 49, and the communication unit 41 sends one or more records to the user terminal 10.

[0205] S8: The request generation unit 14 of the user terminal 10 generates a request message (business card image data, application name, list of input items, one or more records), and the request transmission unit 15 sends the request message to the generation AI system 50.

[0206] S9: The generation AI system 50 analyzes the image data of the business card to generate a value from the image data and adjusts the number of characters in the value by referring to one or more records. The generation AI system 50 determines which input field the generated value corresponds to. The generation AI system 50 sends a response message (the value associated with the input field) to the user terminal 10. The receiving unit 11a of the user terminal 10 receives the value associated with the input field.

[0207] The subsequent steps can be the same as in Figure 10.

[0208] By performing few-shot prompting, the generation AI system 50 can more easily determine the value to associate with the input item, thereby improving the accuracy of value generation. Consequently, the user terminal 10 can more easily obtain the value corresponding to the input item.

[0209] <<Main Effects>> According to this embodiment, in addition to the effects of the first and second embodiments, the accuracy of generating the values ​​of the input items can be improved by performing few-shot prompting.

[0210] <When analyzing files other than image data> The generation AI system 50 can analyze files in formats other than text data and image data, such as document data, video data, or audio data. In this embodiment as well, in generating values ​​for input items, the user terminal 10 can send document data, video data, and audio data to the generation AI system 50, just as it does image data.

[0211] Figure 32 shows the request message that the user terminal 10 sends to the generation AI system 50. The explanation of Figure 32 will describe the differences from Figure 11. Messages 361, role 362, and content 363 are the same as in Figure 11. Each parameter has a set of "type" and its value. Parameter 364 now specifies "file_url" as the "type". When "type" is "file_url", the value of "url" is "file" 366. "file" 366 contains either the URL where the file is saved or a Base64 encoded image.

[0212] Figure 33 shows an example of the string set for "Prompt" 365 in Figure 32. Figure 33 mainly explains the differences from Figure 12. In Figure 33, the first character of Figure 12 has changed from "Image" to "File" 365. The generation AI system 50 understands that it recognizes "File" 365 when analyzing the prompt, and by analyzing "File" included in the request message, it determines to generate the value for the input field of the business card management application.

[0213] In this way, the request generation unit 14 can change the format of the data to be analyzed by the generation AI system 50 by changing the prompt description. For example, in the case of video data, the generation AI system 50 can generate values ​​for input items even if the business card is captured in video. In the case of document data, the generation AI system 50 can generate values ​​for input items even if names, etc., are included in prose or forms. In the case of audio data, the generation AI system 50 can generate values ​​for input items even if names, etc., are included in conversations.

[0214] Furthermore, since multiple files can be specified in a prompt, the request generation unit 14 may include two or more of the image data, document data, video data, and audio data in a single prompt and send it to the generation AI system 50. In this case, "file" 365 will be changed to "recognize images, documents, videos, and audio," etc.

[0215] <<Main Effects>> According to this embodiment, in addition to the effects of the first and second embodiments, the generation AI system 50 can analyze a text data or image data file and generate values ​​for input items.

[0216] [Other application examples] Although the best mode for carrying out the present invention has been described above using examples, the present invention is not limited in any way to these examples, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0217] For example, in this embodiment, the user terminal 10 transmits image data to the generation AI system 50, but the image data may be stored on a predetermined server. In this case, the user terminal 10 transmits information to the generation AI system 50 specifying the image data on the server. The generation AI system 50 retrieves the image data from the server and generates values ​​for the input items.

[0218] Furthermore, in this embodiment, the values ​​of the input items generated by the generation AI system 50 are in JSON format, but the values ​​of the input items may be in other formats such as XML or CSV.

[0219] Furthermore, in this embodiment, the user terminal 10 sets the generated value in the input field of the application managed by the application service 40. However, the user terminal 10 may also set the generated value in the input field of a native application running on its own device. For example, if the user terminal 10 is running a spreadsheet application, it may set the generated value in a cell of the spreadsheet application.

[0220] Furthermore, the apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the application service 40 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and perform the processing disclosed herein.

[0221] Furthermore, the application service 40 can be configured to share the processing steps disclosed in this embodiment, such as those shown in Figure 10, in various combinations. For example, a process executed by a predetermined unit may be executed by multiple information processing devices of the application service 40. Also, the application service 40 may be consolidated into a single server device or divided among multiple devices.

[0222] Furthermore, the configuration examples in Figure 7 and other figures are divided according to their main functions to facilitate understanding of the processing performed by the application service 40. The present invention is not limited by the way the processing units are divided or their names. The processing of the application service 40 can be further divided into many more processing units depending on the processing content. It can also be divided so that one processing unit contains even more processing.

[0223] Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0224] Embodiments of the present invention bring about significant improvements in the capabilities and functionality of computers. These improvements enable users to utilize computers that provide more efficient and robust interaction with tables, which are methods for storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of a more efficient, powerful, and robust user interface. Such a user interface provides better interaction between humans and machines.

[0225] <Mode> [Aspect 1] A server device that manages information entered by a user into input fields, and an information processing device that can communicate with a generation AI (Artificial Intelligence) system via a network, A display control unit that displays a screen for receiving information input from a user into the input fields, and which is capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, a request transmission unit transmits a request to the generating AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A receiving unit that receives information corresponding to the input items transmitted from the generation AI system, An input processing unit that inputs information corresponding to the received input item into the input field of the corresponding input item, A transmission unit that transmits information to the server device in order for the server device to manage the information corresponding to the input items entered in the input field, An information processing device having [Aspect 2] The request transmission unit further transmits information to the generating AI system for calling a function that enables the input processing unit to function. The receiving unit receives information specifying the function and information corresponding to the input items from the generating AI system. Upon execution of the aforementioned function, the input processing unit inputs the information corresponding to the received input item into the corresponding input field of the input item. The information processing apparatus according to Embodiment 1. [Aspect 3] The request transmission unit further transmits information to the generating AI system for calling a non-existent dummy function. The receiving unit receives information specifying the dummy function and information corresponding to the input items from the generation AI system. The input processing unit inputs the information corresponding to the received input item into the input field of the corresponding input item. The information processing apparatus according to Embodiment 1. [Aspect 4] The display control unit displays the screen in which the input processing unit has entered information corresponding to the input item into the input field of the input item. An information processing device according to any one of the three embodiments. [Aspect 5] The aforementioned screen is a screen for accepting the upload of the image data. It further comprises an identification unit that identifies the uploaded image data as image data to be extracted, An information processing device according to any one of embodiments 1 to 4. [Aspect 6] The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The receiving unit further receives information identifying the application selected by the user, and the identification unit identifies the input item based on the information identifying the application. The information processing apparatus according to embodiment 5. [Aspect 7] The system further includes a request generation unit that generates the aforementioned request. The information processing apparatus according to embodiment 3. [Aspect 8] The request transmission unit transmits to the generation AI system the data format of the input item and an instruction to extract the information described in the data format of the input item. An information processing device according to any one of embodiments 1 to 7. [Aspect 9] The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The request transmission unit transmits the request, including the list of input items and the name of the application, to the generation AI system. An information processing device according to any one of embodiments 1 to 8. [Aspect 10] The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The aforementioned application is an application created by accepting settings for the aforementioned input items from the user. The information processing apparatus according to embodiment 5, further comprising the above. [Aspect 11] The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The specified part is, An image identification unit that identifies the image data uploaded by the user, An input item identification unit that identifies the input items corresponding to the application, An application identification unit that identifies the application name of the application, The information processing apparatus according to embodiment 5, further comprising the above. [Aspect 12] The transmission unit transmits a request for one or more programs to be executed by the information processing device to the server device. The aforementioned information processing device can run a web browser. The receiving unit, the transmitting unit, the request generation unit, the request transmission unit, and the input processing unit operate by executing the program sent from the server device in the web browser. The information processing apparatus according to embodiment 7. [Explanation of Symbols]

[0226] 10 user terminals 40 Application Services 50 Generative AI Systems 100 Information Processing Systems [Prior art documents] [Patent Documents]

[0227] [Patent Document 1] Japanese Patent Publication No. 2020-198022

Claims

1. A server device that manages information entered by a user into input fields, and an information processing device that can communicate with a generation AI (Artificial Intelligence) system via a network, A display control unit that displays a screen for receiving information input from a user into the input fields, and which is capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, the request transmission unit transmits a request to the generating AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A receiving unit that receives information corresponding to the input items transmitted from the generation AI system, An input processing unit that inputs information corresponding to the received input item into the input field of the corresponding input item, A transmission unit that transmits information to the server device in order for the server device to manage the information corresponding to the input items entered in the input field, An information processing device having

2. The request transmission unit further transmits information to the generation AI system for calling a function that enables the input processing unit to function. The receiving unit receives information specifying the function and information corresponding to the input items from the generation AI system. Upon execution of the aforementioned function, the input processing unit inputs the information corresponding to the received input item into the corresponding input field of the input item. The information processing apparatus according to claim 1.

3. The request transmission unit further transmits information to the generating AI system for calling a non-existent dummy function. The receiving unit receives information specifying the dummy function and information corresponding to the input items from the generation AI system. The input processing unit inputs the information corresponding to the received input item into the input field of the corresponding input item. The information processing apparatus according to claim 1.

4. The display control unit displays the screen in which the input processing unit has entered information corresponding to the input item into the input field of the input item. The information processing apparatus according to any one of claims 1 to 3.

5. The aforementioned screen is a screen for accepting the upload of the image data. It further comprises an identification unit that identifies the uploaded image data as image data to be extracted, The information processing apparatus according to claim 1.

6. The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The receiving unit further receives information identifying the application selected by the user, and the identification unit identifies the input item based on the information identifying the application. The information processing apparatus according to claim 5.

7. The system further includes a request generation unit that generates the aforementioned request. The information processing apparatus according to claim 3.

8. The request transmission unit transmits to the generation AI system the data format of the input item and an instruction to extract the information described in the data format of the input item. The information processing apparatus according to claim 1.

9. The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The request transmission unit transmits the request, including the list of input items and the name of the application, to the generation AI system. The information processing apparatus according to claim 8.

10. The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The aforementioned application is an application created by accepting settings for the aforementioned input items from the user. The information processing apparatus according to claim 1.

11. The server device is a server device that provides an application for managing information entered by a user into the input fields of the input items, The specified part is, An image identification unit that identifies the image data uploaded by the user, An input item identification unit that identifies the input items corresponding to the application, An application identification unit that identifies the application name of the application, The information processing apparatus according to claim 5, further comprising the above.

12. The transmission unit transmits a request for one or more programs to be executed by the information processing device to the server device. The aforementioned information processing device can run a web browser, The receiving unit, the transmitting unit, the request generation unit, the request transmission unit, and the input processing unit operate by executing the program sent from the server device in the web browser. The information processing apparatus according to claim 7.

13. A server device that manages information entered by a user into input fields, A program transmission unit that transmits a program in response to a program request sent from an information processing device that communicates with the server device, It has a registration unit that registers information transmitted from the aforementioned information processing device, The program displays a screen that accepts information input from the user into the input fields, and that can accept instructions to extract the information to be entered into the input fields from image data. When the program receives such instructions on the screen, it sends a request to the generating AI system that includes the image data to be extracted and instructions to extract information corresponding to the input items from the image data. This program receives information corresponding to the input items transmitted from the generation AI system, inputs the received information into the input fields of the corresponding input items, and transmits the information entered into the input fields to the server device for management by the server device. Server device.

14. An information processing system comprising a server device that manages information entered by a user into input fields, and an information processing device that can communicate with a generating AI (Artificial Intelligence) system via a network, The aforementioned information processing device is A display control unit that displays a screen for receiving information input from a user into the input fields, and which is capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, a request transmission unit transmits a request to the generation AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A receiving unit that receives information corresponding to the input items transmitted from the generation AI system, An input processing unit that inputs information corresponding to the received input item into the input field of the corresponding input item, A transmission unit that transmits information to the server device in order for the server device to manage the information corresponding to the input items entered in the input field, An information processing system having [a certain feature].

15. A server device that manages information entered by users into input fields, and an information processing device that can communicate with a generation AI (Artificial Intelligence) system via a network, A display control unit that displays a screen for receiving information input from a user into the input fields, and which is capable of receiving instructions to extract the information to be entered into the input fields from image data, When the aforementioned instruction is received on the aforementioned screen, a request transmission unit transmits a request to the generation AI system that includes the image data to be extracted and an instruction to extract information corresponding to the input items from the image data. A receiving unit that receives information corresponding to the input items transmitted from the generation AI system, An input processing unit that inputs information corresponding to the received input item into the input field of the corresponding input item, A transmission unit transmits information to the server device in order for the server device to manage the information corresponding to the input items entered in the input field. A program designed to function as such.

Images