Information processing device and method for creating AI prompts

Abstract

Regarding AI conversation technology, this will improve the accuracy of AI responses that meet user expectations. [Solution] An information processing device equipped with an interface for conversation between a user and an AI, comprising a database for storing user preference information, and when creating a prompt for the AI ​​in response to an input instruction sentence to the interface, it creates an additional instruction sentence by referring to the preference information in the database based on the words in the input instruction sentence, sends a prompt with the additional instruction sentence attached to the instruction sentence to the AI, and obtains a response sentence generated by the AI ​​that reflects the input instruction sentence and the additional instruction sentence.

Description

Technical Field

[0001] The present disclosure relates to a technique for performing conversations such as questions and answers for artificial intelligence (AI) such as large language models (LLMs).

Background Art

[0002] An information processing device transmits (inputs) a question (which can also be called a prompt, instruction, etc.) to an AI such as an LLM based on a user input, and receives (outputs) an answer (response, etc.) from the AI.

[0003] As a prior art example, there is Patent No. 7329585 (Patent Document 1). Patent Document 1 describes that it provides a persona chatbot control method or the like that can maintain a dialogue entity having a chatbot persona even if the dialogue with the user continues. Patent Document 1 describes that it includes a step of adding a user utterance to a prompt including an instruction related to an explanation of the chatbot character.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In prior art examples such as Patent Document 1, only the past conversation history is referred to, so as a way of answering by the AI, it can act as a character, but it cannot answer unknown topics.

[0006] In prior art examples such as Patent Document 1, in order to obtain an expected answer, it is necessary to describe a long explanation (conversation history). However, in that case, it is likely that expected results can be obtained faster by searching on a general web.

[0007] The purpose of this disclosure is to provide technology that can improve the accuracy of AI responses that meet user expectations, regarding the AI ​​conversation technology described above. [Means for solving the problem]

[0008] A typical embodiment of this disclosure has the configuration shown below. One embodiment is an information processing device equipped with an interface for conversation between a user and an AI, comprising a database for storing the user's preference information, and when creating a prompt for the AI ​​in response to an input instruction sentence to the interface, it creates an additional instruction sentence by referring to the preference information in the database based on the words in the input instruction sentence, sends the prompt with the additional instruction sentence added to the input instruction sentence to the AI, and obtains a response sentence from the AI ​​that reflects the input instruction sentence and the additional instruction sentence. [Effects of the Invention]

[0009] According to a representative embodiment of this disclosure, the AI ​​conversation technology described above can achieve improvements in the accuracy of AI responses that meet user expectations. Other issues, configurations, and effects will be shown in the embodiments for carrying out the invention. [Brief explanation of the drawing]

[0010] [Figure 1] An overview of each embodiment is shown. [Figure 2] The system configuration of Example 1 is shown. [Figure 3] This shows an example of the server configuration in Example 1. [Figure 4] This shows an example of the configuration of the information processing device in Example 1. [Figure 5] This shows an example configuration of the application (additional instruction creation program) of the information processing device in Example 1. [Figure 6A] An example of the software and screen configuration in Example 1 (First Example) is shown. [Figure 6B]Shows an example of the software and screen configuration in Example 1 (Second Example). [Figure 6C] Shows an example of the software and screen configuration in Example 1 (Third Example). [Figure 7A] Shows an example of the search screen in Example 1 (First Example). [Figure 7B] Shows an example of the search screen in Example 1 (Second Example). [Figure 7C] Shows an example of the search screen in Example 1 (Third Example). [Figure 8] Shows an example of the Q&A text DB in Example 1. [Figure 9] Shows an example of other preference information in Example 1. [Figure 10] Shows the flow of creating the Q&A text DB in Example 1. [Figure 11] Shows the flow of the AI conversation in Example 1. [Figure 12] Shows an example of the Q&A text (preference information) in Example 1. [Figure 13] Shows an example of the prompt and response text in Example 1. [Figure 14] Shows examples of instruction texts, etc. in a modified example of Example 1. [Figure 15] Shows an example of a screen for displaying additional instruction texts in a modified example of Example 1. [Figure 16] Shows examples of cases where only positive information is used and cases where only negative information is used in a modified example of Example 1. [Figure 17] Shows the system configuration of Example 2. [Figure 18] Shows an example of the prompt and response text in Example 2. [Figure 19] Shows the system configuration of Example 3. [Figure 20] Shows the flow of creating the photographed image DB in Example 3. [Figure 21] Shows an example of the photographed image DB in Example 3. [Figure 22] Shows the flow of the AI conversation in Example 3. [Figure 23] Examples of prompts and responses in Example 3 are shown below. [Figure 24] The system configuration of Example 4 is shown. [Figure 25] The flow of the AI ​​conversation in Example 4 is shown. [Figure 26] Examples of prompts and responses in Example 4 are shown below. [Figure 27] An example screen showing a modified version is provided. [Modes for carrying out the invention]

[0011] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, the same parts are generally denoted by the same reference numerals, and repeated descriptions are omitted. In the drawings, the representation of components may not show their actual location, size, shape, extent, etc., in order to facilitate understanding of the invention.

[0012] In explanations, when describing program-based processing, the focus may sometimes be on the program, functions, or processing units. However, the core hardware component is the processor, or a controller, device, computer, or system composed of such a processor. The computer, using its processor, executes processing according to the program read into memory, utilizing resources such as memory and communication interfaces as appropriate. This realizes the specified functions and processing units. The processor is composed of semiconductor devices such as CPUs / MPUs and GPUs. Processing is not limited to software program processing; it can also be implemented using dedicated circuits. FPGAs, ASICs, CPLDs, etc., can be used as dedicated circuits.

[0013] The program may be pre-installed as data on the target computer, or it may be distributed as data to the target computer from the program source. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium, such as a memory card or disk. The program may consist of multiple modules. The computer system may consist of multiple devices. The computer system may consist of a client-server system, a cloud computing system, an IoT system, etc. Various types of data and information are composed of structures such as tables and lists, but are not limited to these. Representations such as identification information, identifiers, IDs, names, and numbers are interchangeable.

[0014] [Challenges and prerequisite technologies, etc.] The accuracy of LLM depends on the size of the corpus used for pre-training and the size of the Transformer model. For example, GPT-3 has been reported to be able to answer unfamiliar questions (fields or topics) that it has not directly trained on, by modifying the input text. This is called emergent ability.

[0015] Furthermore, various prompt engineering techniques have been proposed to improve the accuracy of LLM responses by devising prompts (instructions) for the AI. One example is few-shot prompting. Also known as in-context learning, few-shot prompting is an approach to solving unknown problems by providing multiple examples of problems and answers.

[0016] This example involves devising a way to create prompts (instructional statements).

[0017] Stable Diffusion is another well-known image generation AI. In the learning phase, Stable Diffusion receives pairs of images (e.g., an image of a dog) and text (e.g., text describing the dog in the image) as input and reconstructs the image as output. The diffusion model in Stable Diffusion stores the latent representation of the diffusion model corresponding to the input text. On the other hand, in the inference phase (when generating images), only text is input to the diffusion model. The latent representation of the image output by the model will be in line with the input text, and it can generate an image that matches the input text.

[0018] In this embodiment, an image generation AI is used to generate images that meet the user's expectations.

[0019] Another well-known pose estimation technique is OpenPose. By deploying (installing) the OpenPose model into an AI engine, it is possible to detect the body parts and posture of objects (such as people) in an image.

[0020] Furthermore, YOLO and other models are known as object detection models. By deploying (installing) the YOLO model into an AI engine, it is possible to detect objects (object types such as animals and vehicles) in an image.

[0021] On a PC, a screenshot of the displayed screen is saved as an image within the PC. The processor (especially the NPU: Neural Processing Unit) analyzes the image and indexes its contents. From this indexed information, past information such as files used by the user, URLs (web pages) accessed, and emails exchanged can be searched.

[0022] In contrast, this embodiment utilizes the user's pre-search history (search results and selections from those search results) as a database. In this embodiment, the AI ​​prompts are designed using search results and selections that reflect the user's preferences (in other words, interests and tastes), as well as images that have been taken and saved.

[0023] [Overview and relationships of each example] The following describes several embodiments in order. Figure 1 is a table summarizing the overview and relationships of each embodiment. The multiple embodiments share the common feature of using search results and selection results that reflect the user's preferences, as well as images that have been taken and saved, in other words, using user preference information to devise prompts (additional instructions, etc.) for the AI.

[0024] Example 1 uses LLM as the generating AI model (language / image) and Q&A text as the database (source database, preference database) that reflects user preferences.

[0025] In Example 1, the Q&A consists of a Q (question), which is the search term / keyword used during the initial search, and an A (answer), which is the result of selecting from the search results, particularly from the information items in the search results. This information reflects the user's preferences and interests. This system (information processing device) creates a database of such Q&A texts for each user. When a user asks a question to the AI ​​(LLM), this system extracts information (preference information) related to the question from this database, uses the extracted information to create an additional instruction sentence, and adds it to the question prompt. The AI ​​(LLM) generates an answer considering the information in the additional instruction sentence. This makes it easier to obtain an answer that reflects the user's preferences and meets their expectations.

[0026] Furthermore, the database selection results are specifically categorized as positive information (positive) for information items selected by the user and negative information (negative) for information items not selected by the user. By using this positive / negative information, it becomes possible to answer unknown questions (fields, topics), similar to few-shot prompting.

[0027] Example 2 uses an image generation model as the generation AI model and Q&A text as the database source (preference database). Similar to Example 1, this system creates a preference database using Q&A text in advance. The user inputs text (a sentence expressing the image the user requests) to the AI. At that time, this system extracts relevant information from the preference database (Q&A text), creates additional instruction sentences, and adds them to the text (prompt) to the AI. The AI ​​responds and outputs an image generated according to that text.

[0028] Example 3 uses an image generation model as the generation AI model and captures images (captured image database) as the database source (preference database). This system creates a database of images (captured image database) that the user has captured and saved in advance. The images that the user has captured and saved reflect the user's preferences. This system creates the captured image database by analyzing the images that the user has captured and saved using AI (such as extracting feature points). The user inputs text (a sentence that expresses the image the user wants) to the AI. At that time, this system extracts relevant information from the captured image database, creates an additional instruction sentence, and adds it to the text (prompt) to the AI. The AI ​​responds and outputs an image that it has generated according to that text.

[0029] Example 4 uses LLM as the generating AI model and a captured image database as the source database (preference database). This system creates a database of images captured and saved by the user in advance. The user inputs a question to the AI ​​(LLM). At that time, this system extracts relevant information from the captured image database, creates additional instruction sentences, and adds them to the question (prompt) to the AI. The AI ​​responds and outputs a response sentence that it has generated according to the question.

[0030] As described above, all of the embodiments share the commonality of using information that reflects the user's preferences (preference information) to craft prompts (additional instructions) for the AI, making it easier to obtain the user's expected response (answer text or image) from the AI.

[0031] [Solution, etc.] This embodiment is an information processing device equipped with a conversational interface between a user and an AI, and includes a function to create prompts for the AI. This embodiment databases the user's web search keywords and selection results. This embodiment extracts example questions and answers from the database and adds them as supplementary instructions to the prompts for the AI ​​(LLM, etc.). This enables the AI ​​(LLM, etc.) to answer questions it has not learned. This technology is effective for inference in local fields where training data is scarce.

[0032] In this embodiment, a database is created containing the user's search keywords related to the question posed to the AI ​​and the selected search results. This information, including the selected search results, is then added to the question (prompt) as supplementary instructions (in other words, supplementary instructional information) when a question is posed to the AI. Furthermore, in addition to reflecting the search results (information items, e.g., words) selected by the user as positive information, the search results (information items, e.g., words) not selected by the user may also be reflected as negative information. This allows the information from the supplementary instructions to be reflected in the AI's response, improving the accuracy of the response. In other words, even in areas where the AI ​​has not learned much, it becomes easier to obtain the answer the user expects.

[0033] The information processing device in this embodiment is, for example, a computer or terminal owned by the user, but is not limited to that. The information processing device in this embodiment is equipped with an application for interacting and conversing with AI (LLM, etc.) and provides the user with an interface for that purpose. The information processing device in this embodiment may also be referred to as an AI conversation device, an AI prompt creation device, an AI response output device, etc.

[0034] <Example 1> This document describes the system, information processing device, and AI prompt creation method of Example 1.

[0035] [System Configuration] Figure 2 shows an example of the system configuration of Embodiment 1. This system consists of an information processing device 1 used by the user and a server (AI server) 2 equipped with AI 2a, connected via a communication network 4, etc. In other words, this system is an AI conversation system, etc. Furthermore, general search servers 7 and various content servers are also connected to the communication network 4, such as the Internet.

[0036] The information processing device 1 is, for example, a mobile terminal such as a smartphone, but is not limited to this, and may also be a PC, a wearable terminal such as a head-mounted display (HMD) / smart glasses, etc. The information processing device 1 connects to an access point 3A, etc. via wireless communication and can communicate with a server 2, etc. via a communication network 4. The information processing device 1 is equipped with an application 1a for conversing with AI2a (in other words, an additional instruction creation program, etc.). In addition to the function of realizing the user interface for conversation with AI2a, this application 1a includes a control program (additional instruction creation program, etc.) for realizing the characteristic functions of this embodiment (i.e., the function of creating a preference database and additional instruction statements, etc.). Note that the application and the control program may be configured separately.

[0037] Server 2 is equipped with an LLM as AI2a. Server 2 generates an answer 6 (in other words, a response, etc.) in response to a question 5 (in other words, a prompt, request, etc.) sent from the information processing device 1 and sends it to the information processing device 1.

[0038] The information processing device 1 may, in advance, send search keywords to the search server 7 when a user is performing a search, and receive search result information from the search server 7.

[0039] In this embodiment, the functionality of the additional instruction creation program (application 1a) is basically implemented within the information processing device 1, but it is not limited to this and may be implemented in the form of a client-server system or the like.

[0040] [server] Figure 3 shows an example configuration of Server 2. Server 2 is equipped with a processor 201, an AI engine 202, memory 203, a communication interface 204, a display 205, etc., which are interconnected by an architecture such as a bus.

[0041] The processor 201 controls the entire server 2 and its various components. The processor 201 has, for example, a CPU. The processor 201 loads programs and data into memory 203. This enables various functions. Memory 203 (which may also be an external storage device) stores programs such as the OS, middleware, and applications, as well as various data used and processed by the processor 201.

[0042] All components are connected to processor 201 via a parallel or serial interface. Initial setup during power-up and control during use are possible from processor 201.

[0043] In this embodiment, Server 2 is equipped with an AI engine 202. The AI ​​engine 202 can be configured with a GPU or NPU (Neural Network Unit) that excels at parallel processing of multiply-accumulate operations. In this embodiment, the environment of Server 2 is assumed to have sufficient power, and the explanation describes the use of a GPU / NPU, but it is not limited to this.

[0044] For LLM2a on Server 2, a corpus (a dataset for learning natural language) is obtained from the internet and used to train the AI ​​engine 202. The training can be supervised or unsupervised. In this embodiment, after pre-training with unsupervised learning, fine-tuning (supervised learning and reinforcement learning based on human feedback) is performed to ensure that socially appropriate answers are produced. There are also models that incorporate reinforcement learning based on human feedback. An example of a base model is a model called Transformer. Examples of training for these models are published, for example, in Reference 1.

[0045] [Reference 1] Long Ouyang, et. al. “Training language models to follow instructions with human feedback”, https: / / arxiv.org / pdf / 2203.02155.pdf

[0046] For LLM2a, after training is complete, the trained model is installed (deployed) into the neural network (NN) inside Server 2 (AI Engine 202), and inference is performed. Note that the training server and the inference server may be the same or different.

[0047] In inference using LLM2a, the system generates and outputs a response to an instruction (prompt) input from an external source (information processing device 1).

[0048] [Information Processing Device] Figure 4 shows an example configuration of the information processing device 1 (e.g., a smartphone). The information processing device 1 includes a processor 101, an AI engine 102, memory 103, a communication interface 104, a display 105, a microphone 106, a speaker 107, an out-camera 108a, an in-camera 108b, a distance sensor 109, a position and orientation sensor 110, a geomagnetic sensor 111, a positioning sensor 112, an RTC (real-time clock) 113, etc., which are interconnected by an architecture such as a bus. The information processing device 1 also includes other components, such as a battery and physical buttons, although these are not shown in the figure.

[0049] All components are connected to the processor 101 via a parallel or serial interface, allowing for initial power-on setup and control during use from the processor 101.

[0050] The processor 101 controls the entire information processing device 1 and its various parts. The processor 101 has, for example, a CPU. The processor 101 loads programs and data into the memory 103. This enables various functions. The memory 103 (which may also be an external storage device) stores programs such as the OS, middleware, and applications, as well as various data used and processed by the processor 101.

[0051] The communication interface 104 implements a wireless LAN communication interface, a telephone network / base station communication interface, a short-range wireless communication interface, etc., including an antenna and circuitry. Suitable wireless communication interfaces include Bluetooth®, IrDA®, Zigbee®, HomeRF®, and Wi-Fi®. Suitable base station communication interfaces include LTE® and WiMAX®.

[0052] The display 105 includes a touch panel 105a, displays information on the screen, and accepts input such as touch operations on the screen. The display 105 and others are examples of output devices, and the microphone 106 and others are examples of input devices. The rear camera 108a and front camera 108b capture images (still images and videos).

[0053] The position and orientation sensor 110 is used to measure the position and orientation of the information processing device 1. Orientation refers to the three-dimensional rotational position of the information processing device 1 in the real space in which it exists. Specific examples of the configuration of the position and orientation sensor 110 include a position and orientation camera (PP camera) 110a, a position and orientation distance measuring sensor (PP distance measuring sensor) 110b, an acceleration sensor 110c, and a gyro sensor 110d. PP stands for Position AND Posture. The PP camera 110a may be the same as the rear camera 108a. Also, the PP distance measuring sensor 110b may be the same as the distance measuring sensor 109.

[0054] The positioning sensor 112 is used to measure the indoor / outdoor location of the information processing device 1. Specific examples include indoor use of beacon signals or location markers, and outdoor use of GPS / GNSS receivers that utilize signals from satellites.

[0055] The information processing device 1 in this embodiment may be equipped with an AI engine 102. The AI ​​engine 102 can be configured as a GPU or NPU that excels at parallel processing of multiply-accumulate operations.

[0056] In Example 1, the information processing device 1 stores an additional instruction generation program (application 1a) and a Q&A text database (preference database 1b) as programs and data in the memory 103.

[0057] [Program for generating additional instructions] Figure 5 shows the configuration of functions provided by the additional instruction generation program (application 1a) in the information processing device 1. The additional instruction generation program is a control program that implements the search / database creation function 501 and the additional instruction generation function 502.

[0058] The search and database creation function 501 extracts the search keywords (search information) used by the user during a search, the search results information, and the selection information from the search results, and registers this information (preference information) in the Q&A text database (preference database) 503.

[0059] The supplementary instruction creation function 502 is a function that, when a user asks a question to the AI, extracts information related to the question (preference information) from the Q&A text DB (preference DB) 503, creates supplementary instructions, and adds them to the question prompt.

[0060] The Q&A text DB503 is a database (preference DB1b in Figure 1) that holds preference information obtained through the above functions.

[0061] In Example 1 (Figure 5), while external general search sites and search interfaces may be used for searching, the information processing device 1 may also provide its own search function and search interface (as part of application 1a) (described later).

[0062] Furthermore, in Example 1 (Figure 4), the information processing device 1 is equipped with an AI engine 102. This AI engine 102 has a different function from the AI ​​engine 202 of the server 2 and is used as an aid for creating additional instruction sentences. Specifically, the AI ​​engine 102 is equipped with a trained model that can generate summarized text from preference information (search information, selection results, etc.) through text summarization processing, and generate sentences from search keywords of preference information, etc. By utilizing the functions of this AI engine 102, prompts containing suitable additional instruction sentences can be quickly created.

[0063] The AI ​​engine 102's pre-trained model may be one trained on server 2, or one trained on a server on a different network.

[0064] As an alternative configuration, the information processing device 1 may not be equipped with the AI ​​engine 102, and instead may be configured to access the AI ​​engine of an external server when needed and receive the results of similar processing.

[0065] Processor 101 creates and maintains a Q&A text database (preference database) 503 from the user's search keywords and search results / selections through the processing of the "additional instruction creation program" (application 1a) corresponding to the above functions. When the user inputs an instruction (question) to the AI, processor 101 extracts preference information such as keywords and context related to that instruction from the Q&A text database 503. Using the extracted preference information, processor 101 generates an additional instruction and combines it with the instruction based on the user input to create a prompt (question 5) for input to AI 2a on server 2.

[0066] In this case, more specifically, it may be possible to generate and send a single integrated prompt, or to send two sets of instructions—the user input instruction and the additional instruction—sequentially.

[0067] The AI ​​engine 102 on the information processing device 1 can also be configured with a GPU or NPU. However, since smartphones often operate on a limited battery, power consumption must be carefully considered. This embodiment describes an example of using an AI engine 102 that includes an NPU, which consumes less power.

[0068] Furthermore, while this embodiment describes an example of performing only inference using a pre-trained model, it is not limited to this. For example, by performing transfer learning, fine tuning, or distillation on an existing pre-trained model, optimization can be carried out according to the usage environment. Moreover, by using a high-performance AI engine, it is possible to perform not only inference but also corpus learning (creation of pre-trained models) on a smartphone.

[0069] [Example Implementation Configuration] Figure 6A, etc., shows three examples of software and screen configurations when the "additional instruction statement creation program" (application 1a) is implemented on the smartphone, which is the information processing device 1.

[0070] Figure 6A shows the software configuration (protocol stack) at the top and the screen configuration at the bottom. The OS has a browser and an "additional instruction generation program" (application 1a), which are separate applications / programs. The browser is the basis of the search and AI conversation interface. In the configuration of Figure 6A, the OS retrieves information from server 2 based on user instructions received from the "additional instruction generation program" and displays it on the screen. Browser screen 601 displays the GUI for search and AI conversation, as well as question and answer sentences. This is an example where the additional instruction generation screen 602 is displayed when application 1a is launched. Users can transition between browser screen 601 and other screens as appropriate through user operation.

[0071] Figure 6B shows a configuration in which an "additional instruction generation program" is incorporated into a part of the browser. The content of the additional instruction generation screen 602 generated by the additional instruction generation program is displayed on a part of the browser screen 601, and the browser responds to user instructions.

[0072] Figure 6C shows a configuration in which an "additional instruction generation program" is placed between the OS and the browser. This program intercepts the communication between the browser and the OS, and stores search keywords and search results based on user operations in a preference database. The screen 602 generated by the additional instruction generation program is integrated as a single unit with the browser screen 601.

[0073] The following explanation will be based on the configuration shown in Figure 6A, but is not limited to this configuration.

[0074] The browser is the interface for searching, or in other words, for Q&A. The supplementary instruction generation program is a program that performs prior learning about the user's preferences (creation of a Q&A text database) and creates supplementary instructions when questions are asked to the AI. The user input instructions for the questions to the AI ​​are displayed on the browser screen. The supplementary instruction generation program automatically creates supplementary instructions from the user input instructions, using the information in the Q&A text database.

[0075] [Search screen] Figure 7A shows an example configuration of the search screen (Q&A detection screen) in Example 1. The information processing device 1 provides a dedicated search interface on the browser screen for collecting preference information and creating a preference database. This search interface is provided by application 1a of the information processing device 1 (an additional instruction creation program, particularly the search and database creation function 501 in Figure 5). The data and information related to this search interface may be prepared in advance within the information processing device 1, and it is not necessary to access an external search site (search server 7 in Figure 2), etc.

[0076] Figure 7A shows a case where a dedicated search interface screen is provided by application 1a for learning and creating Q&A text DB 503. In the example of Figure 7A, the browser screen has a search keyword field 701 and a search results field 702. The user enters a desired search keyword corresponding to their preferences and interests in the search keyword field 701 and performs a search. In this example, the search keyword "ramen" is entered. Application 1a of the information processing device 1 performs a search using the user-entered search keyword, obtains the search results information, and displays it in the search results field 702. This search may also be a custom search using pre-prepared category classification data by application 1a. In this example, application 1a of the information processing device 1 aggregates the search results through its own search process and displays them in the search results field 702. Multiple information items that are search results are displayed side by side in the search results field 702. In this example, the information items include "soy sauce ramen," "miso ramen," "salt ramen," "tonkotsu ramen," "tsukemen," and "abura soba." The words in these information items are words that correspond to subcategories and related categories, with the search keyword "ramen" being the higher-level category.

[0077] The user views the search results field 702 and selects the desired information items that match their preferences and interests. For example, suppose the user selects two information items that they like: "Soy Sauce Ramen" and "Pork Bone Ramen." The information processing device 1 adds the selected one or more information items to the Q&A text DB (preference DB) 503 in Figure 5 as preference information (in other words, selected information, etc.), in a format associated with the search keywords. In this example, two information items are selected and associated, but it is also possible to select only one or three or more.

[0078] Furthermore, the information processing device 1, at that time, registers the information items selected by the user from the search results (in this example, the six items shown) as positive information and the information items not selected as negative information, associating them with each other. In addition, if the order of selection by the user is to be given meaning, the order of selection and priority information may also be registered in association with each other. For example, the first selection and first priority might be "soy sauce ramen," and the second selection and second priority might be "pork bone ramen." Also, if there are many information items in the search results, they may be divided into pages or other displays as appropriate.

[0079] Figure 7B shows an example of a search interface screen when using a general search site (search server 7 in Figure 2) as a modified example. The information processing device 1 accesses the external search server 7 on the browser screen and displays the search site page. The user enters search keywords on the search site page and performs a search. A list of search results, such as URLs and descriptions, is displayed in priority order. The user selects the desired information item (URL) from the search results. At this time, the additional instruction generation program of the information processing device 1 grasps the information item selected by the user (the content and keywords of the corresponding URL), creates preference information, and similarly adds it to the Q&A text DB (preference DB) 503. For example, if the user selects URL1, the information processing device 1 saves URL1, the description and content of URL1, or words resulting from the analysis of the content of URL1 as preference information, especially positive information, in association with the search keywords. The information processing device 1 saves information on URLs that were not selected from the search results as negative information. Alternatively, information on URLs that were not selected may not be saved.

[0080] Furthermore, if there are many search results, it would be very time-consuming to treat all unselected information items (URLs) as negative information due to the large amount of data. Therefore, it is acceptable to treat only a predetermined number of unselected items from the top of the search results as negative information. In other words, the maximum number of items to be treated as negative and the maximum number of items to be treated as positive may be set in advance.

[0081] Furthermore, the display of multiple information items (e.g., words) in the search results may simply be a list of multiple information items (without any particular ranking), or the multiple information items may be displayed in a ranked order. For example, based on the user's search history, multiple information items may be displayed in an order that is presumed to be of interest to the user (see, for example, Figure 7C). The user can select their preferred words from the search results and register them in the database as positive. Words that are not selected are registered as negative.

[0082] Among the multiple information items in the search results, all selected words may be registered as positive, or up to a predetermined maximum number may be registered. Similarly, among the multiple information items in the search results, all unselected words may be registered as negative, or up to a predetermined maximum number may be registered. Furthermore, when registering multiple words as positive or negative, they may be registered as a simple list without ranking, or they may be registered with a predetermined ranking. If the search results have rankings according to the user's preferences, up to a predetermined number of words (from the highest or lowest rank up to a predetermined number) may be registered as positive or negative in the order of the rankings. If preference information has rankings, the ranking (in other words, priority) can be included in the additional instruction to reflect it in the AI's response. The method for creating and registering preference information through selection from search results is not limited to the above example; any predetermined algorithm may be used.

[0083] In the example in Figure 7C, the search results are displayed in order from top to bottom according to a predetermined ranking. Multiple words selected by the user and registered in the Q&A text database are registered with numbers according to a predetermined ranking corresponding to the selection operation. For example, if the user selects "Tonkotsu Ramen" first and "Soy Sauce Ramen" second, then in the positive information of the registered preference information, rank number = 1 will be registered as "Tonkotsu Ramen" and rank number = 2 as "Soy Sauce Ramen". In this example, the maximum number of words that can be registered as positive words is 2, and the maximum number of words that can be registered as negative words is also 2. For negative information, up to the maximum number (2) of words from the four items (words 3 to 6) that the user did not select will be automatically selected according to their ranking in the search results, for example, rank number = 1 will be registered as "Miso Ramen" and rank number = 2 as "Salt Ramen".

[0084] [Q&A Text Database (Preference Database)] Figure 8 shows an example of the configuration of the Q&A text DB (preference DB) 503 in Example 1. For example, if there are search results and selection results as shown in Figure 7A, the following information will be registered in the preference DB: User ID, date and time, search keyword: "Ramen", Positive: 1. "Tonkotsu Ramen", 2. "Soy Sauce Ramen", Negative: 1. "Miso Ramen", 2. "Salt Ramen", 3. "Tsukemen", 4. "Abura Soba".

[0085] Figure 9 shows another example of the search screen and preference information. In the example in Figure 9, the search keyword is "fishing," and "sea bream fishing" and "flounder fishing" are selected from the search results. The following information is registered in the preference database: User ID, date and time, search keyword: "fishing," positive: 1. "sea bream fishing," 2. "flounder fishing," negative: 1. "yellowtail fishing," 2. "cherry salmon fishing," 3. "black bass fishing," 4. "sweetfish fishing." In this example, the category is the type of fish targeted for fishing. However, there are various other categories (perspectives, etc.), such as fishing location, fishing method, fishing equipment, etc.

[0086] The following are variations of the search screen. Figures 7A and 9 show cases where the user selects their preferred information items (positive), but it is not limited to this. It is also possible to have the user select information items they do not like (negative) from multiple information items, such as search results. In this case, the selected information items are registered as negative information in the Q&A text database.

[0087] Furthermore, when the information processing device 1 stores preference information in the preference database, it may reduce the number of characters and data size to be stored by extracting main keywords from the search results and selection results. For example, in Figure 9, for positive and negative information, the word "fishing," which is repeated in each information item, may be omitted, and only words such as "sea bream" (in this example, a subcategory of the type of fish targeted for fishing) may be registered.

[0088] The above example shows a case where, for a given search keyword, more detailed subcategories and related categories (in other words, classifications, etc.) are output as search results / options, and the user selects an item or word that suits their preferences from among these options. The information processing device 1 (additional instruction creation program) may output various categories and other information as search results for a search keyword, not limited to the above example. For example, Figure 7A shows a case where, for the search keyword "ramen," categories such as "soy sauce ramen" are output. Other categories could include the location of the restaurant, how the ramen is made, etc. In the case of "fishing" in Figure 9, not only the type of fish but also fishing locations, fishing methods, fishing equipment, etc., could be listed. Users can also specify and limit categories as search keywords, for example, "ramen" AND "restaurant location."

[0089] Furthermore, the screen may provide an interface that allows users to select other category information. For example, a "Show other category information" button may be provided on the screen to display search results based on other category information. Alternatively, various category items may be displayed initially, allowing users to select from them. The program for generating additional instructions may also create and maintain a database of categories and classifications in advance.

[0090] In Example 1, the information processing device 1 (additional instruction creation program) creates user preference information (especially selection results from search results) and stores the preference DB 503 in the memory resources (memory 103 in Figure 3) within the information processing device 1. However, the system is not limited to this; an external server or the like may create the user preference information, or an external server or the like may maintain a DB equivalent to the preference DB 503. For example, a general search service provider, a general information provider or matching service provider (e.g., a site providing gourmet information), or a general store (e.g., a ramen shop) may have a DB that stores user preference information (equivalent information such as usage history). The information processing device 1 references and obtains the user preference information from the external server or the like and uses it to create additional instruction statements.

[0091] [Creating a preference database] Figure 10 shows the processing flow, or sequence diagram, when the information processing device 1 (additional instruction creation program) of Example 1 creates and updates the Q&A text DB (preference DB) 503. In this example, the case where a search is performed using the server 7 (general search server) shown in Figure 1 is shown. In step S101, the information processing device 1 displays a search screen based on user operation. In step S102, the user enters search keywords and executes the search. In step S103, the information processing device 1 executes a search on the server 7 using the search keywords. In step S104, the search keywords, etc., are sent to the server 7. In step S105, the server 7 performs the search processing from the search keywords. The server 7 extracts candidate URLs from the search DB, creates search result information including a list of candidate URLs, and sends it to the information processing device 1 in step S106. In step S107, the information processing device 1 displays the search results on the screen.

[0092] In step S108, the user selects a desired information item (URL) from the search results. In step S109, the information processing device 1 accesses the content server, etc., of the selected URL, and in step S110, it receives and retrieves the content (e.g., a web page) of the URL from the content server, etc. In step S111, the information processing device 1 displays the content of the URL on the screen. In step S112, the user may also view desired information items within the content as appropriate. In step S113, based on the user's selection operations in steps S108 to S112, the information processing device 1 associates the search keywords with the selected URL, the content of the URL, keywords within the content, etc., and registers and saves them as user preference information in the Q&A text DB (preference DB) 503. The information processing device 1 registers the information items selected by the user as positive information.

[0093] The information processing device 1 may analyze keywords contained in the URL content, obtain analysis information from the server 7, or further understand user preferences based on selections made by the user from the detailed information items contained in the URL content.

[0094] Figure 10 shows the case where the search server 7 is used, corresponding to the case in Figure 7B. However, the search process (in other words, the process of understanding the user's preferences) may also be completed by application 1a within the information processing device 1. In that case, the processes shown in steps S104 to S112 in Figure 10 are simplified to processes performed by the additional instruction creation program as shown in Figure 7A.

[0095] [AI Conversation] Figure 11 shows the processing flow for creating prompts and obtaining response texts in a conversation with an AI, based on Figure 10. In step S201, the information processing device 1 displays the AI ​​conversation screen. The AI ​​conversation screen has input fields for questions to the AI ​​and display fields for responses from the AI. In step S202, the user enters a question (instruction) into the input field on the screen. In this embodiment, the question to the AI ​​is text (in other words, a string of characters), but it is not limited to this; the user may input the question as voice input, or the voice may be converted to text, or the prompt to the AI ​​may be composed of voice data.

[0096] In step S203, the processor (instruction statement creation program) of the information processing device 1 refers to the Q&A text DB (preference DB) 503 based on the text information of the input question, extracts preference information (such as information items of the selection result) related to the question, and creates an additional instruction statement using the extracted preference information. The original user-input question may be described as the first instruction statement and the additional instruction statement as the second instruction statement. In a specific example, if the user's question statement contains the word "ramen," positive information such as the word "tonkotsu ramen" associated with the word "ramen" can be extracted from the selection result in Figure 7A and the preference DB in Figure 8. An additional instruction statement (specific examples will be described later) can be created using such preference information.

[0097] Next, in step S204, the processor creates a prompt for the AI ​​(a prompt related to a single request / question to server 2) by adding an additional instruction (second instruction) to the user's question (first instruction). Here, the prompt for the AI ​​is a dataset consisting of the first instruction and the second instruction in sequence, but is not limited to this. In this embodiment, the prompt for the AI ​​is text information (in other words, a string), but is not limited to this.

[0098] In step S205, the information processing device 1 sends a prompt to the AI ​​(question 5 in Figure 2) as a request to the server (AI server) 2. In step S206, the AI ​​(LLM) 2a of the server 2 generates a response sentence in response to the received prompt (user input instruction sentence + additional instruction sentence). In step S207, the server 2 sends the generated response sentence (response 6 in Figure 2) to the information processing device 1 as a response. In this embodiment, the response sentence is text information (in other words, a string), but is not limited to this. In step S208, the information processing device 1 displays the received response sentence on the screen. In this embodiment, the output of the response sentence is a screen display, but is not limited to this. The response sentence may also be output as voice by speech conversion. In step S209, the user receives the response sentence by visually confirming it, etc. After that, the user can repeat the question and answer exchange with the AI ​​in the same manner.

[0099] AI(LLM)2a on Server 2 generates a response that reflects the positive and negative information contained in the question prompt. AI2a generates the response with high priority given to information related to positive words and low priority given to information related to negative words. This improves the accuracy of the AI's response so that it matches the user's preferences.

[0100] As a variation, in step S204, a command (prompt) may be created by integrating and synthesizing the user input command and the additional command into a single sentence. A separate AI (AI engine) may be used for this integration process.

[0101] [Q&A Text (Preference Information)] Figure 12 shows an example of Q&A text (preference information) as the content of Q&A text DB503 (Figure 8). Figure 12 corresponds to an alternative configuration example to Figure 8. In the table in Figure 12, preference information is stored row by row. The table has the following column items: "Genre," "Keyword," "Positive," and "Negative." "Genre" corresponds to the higher-level category (in other words, the field to which it belongs) of the "Keyword" category. "Positive" and "Negative" are the information items (words) that were selected and not selected from the detailed categories (subcategories) of the "Keyword" category. For example, under the genre (higher-level category) of "Favorite Food," there are various values ​​such as "Ramen," "Curry," and "Sushi."

[0102] Genre information (in other words, higher-level category information) may be prepared by the information processing device 1 (additional instruction creation program), or it may be reused from information provided by external search sites, etc. Based on this information, the information processing device 1 can associate genres (higher-level categories) with the categories of search keywords and register them as preference information as shown in Figure 12. In addition, genre information may be displayed on screens such as Figure 7A, allowing the user to select a genre.

[0103] In the Q&A text DB 503, preference information may be stored categorized by genre. In other words, user preference information may be organized and stored by genre, category, etc. The classification of genres, etc., may be configured in advance in application 1a (additional instruction creation program) of the information processing device 1. Even for events where the classification is not clear, the preference DB can be registered by associating it with related words, etc.

[0104] One example of using genre information is when creating and sending a prompt with additional instructions attached to the user's question, where genre information may be included instead of, or in addition to, search keywords and positive / negative information of the selection results. For example, there may be a limit on the number of characters in the prompt sent to Server 2, or charges may be based on the number of characters (tokens) in the prompt. If it is desired to reduce the number of characters (tokens) in the prompt, the information processing device 1 may shorten the question, or it may include only genre information related to the question as additional instructions (additional instruction information) in the prompt. This reduces the number of characters (tokens) in the prompt and thus reduces the amount charged.

[0105] [Reducing the number of characters] Furthermore, if the user input instruction (first instruction) contains a large number of characters (for example, exceeding the set number of characters), the information processing device 1 may reduce the number of characters (tokens) using the following method.

[0106] (1) Use the AI ​​engine (AI engine 202 of server 2 or AI engine 102 of information processing device 1) to summarize the user's instructions. Use the obtained summary to create a prompt.

[0107] (2) Using the AI ​​engine (same as above), extract only the keywords from the user's instructions. Create a prompt using the extracted keywords.

[0108] Furthermore, the following methods may be used for additional instructions.

[0109] (1) For each user, all the information stored in the Q&A text DB503 (preference information, in other words, user profile information) may be sent to the server 2 as additional instruction text (additional instruction information).

[0110] (2) Using the AI ​​engine (same as above), the genre of the user's instruction can be detected, and as described above, the data may be limited to only the genre related to the instruction as an additional instruction. For example, if "ramen" is extracted from the user's instruction, the genre "favorite food": "ramen" will be used as the additional instruction. Also, if, for example, values ​​other than "ramen" (foods) are registered in the Q&A text DB503 for the genre "favorite food", all of those values ​​belonging to "favorite food" may be added as additional instructions.

[0111] (3) You may limit it to only relevant keywords. An example of this is shown in Figure 13 below.

[0112] [Prompt example] In the basic configuration of this embodiment 1, the information processing device 1 sends a prompt (question 5) to the server 2, which consists of a user-input instruction (first instruction) and an additional instruction (second instruction). In other words, the information processing device 1 sends the user-input instruction (first instruction) and the additional instruction (second instruction) to the server 2 in succession and almost simultaneously. The AI ​​2a of the server 2 generates an answer 6 in response to these prompts. However, the user-input instruction and the additional instruction may be integrated and combined into a single instruction (prompt) before being sent to the server 2. For example, the AI ​​engine 102 may generate a single, conversational instruction from the user-input instruction and the additional instruction.

[0113] Figure 13 shows a concrete example of an instruction sentence (prompt to the AI). (A) shows a user-input instruction sentence (first instruction sentence) 1301 and an automatically created and added instruction sentence (second instruction sentence) 1302 as prompts. Instruction sentence 1301 is, for example, "Please recommend a good ramen restaurant in Yokohama." The extracted word "ramen" is underlined in the illustration.

[0114] The additional instruction 1302 is created by the additional instruction creation program of the information processing device 1, which extracts preference information related to the keyword "ramen" in instruction 1301 from the Q&A text DB (preference DB) 503. As an example, the preference information from row #1 in Figure 12 is used for the additional instruction 1302.

[0115] (B) is the response from AI(LLM)2a to prompt (A). The AI ​​engine 202 on server 2 creates a response like (B) in response to prompt (A). AI(LLM)2a creates the response by considering the information from supplementary instruction 1302 along with instruction 1301. The response might look like this: "You like tonkotsu ramen and soy sauce ramen, don't you? Yokohama ramen is characterized by its rich and flavorful tonkotsu soy sauce-based soup, a fusion of Kyushu tonkotsu ramen and soy sauce ramen. The noodles are thick, firm, and straight, and they go perfectly with the rich soup. I recommend M Shoten in Hodogaya. It's always crowded, so I recommend making a reservation." This response reflects the user's preferences, especially positive information (elements that are presumed to be preferred), such as "tonkotsu ramen" and "soy sauce ramen," and does not include negative information (elements that are presumed to be disliked), such as "miso ramen."

[0116] An example of limiting the search to only relevant keywords would be to create a single, integrated prompt such as "Tonkotsu ramen, Shoyu ramen, Restaurant, Yokohama".

[0117] Regarding the processing of the AI ​​conversation described above, the processor of the information processing device 1 may use the AI ​​engine 102 to extract keywords from the user's instruction sentence and extract additional instruction sentences from the Q&A text DB 503 using a word-based search process based on the extracted keywords. For example, the processor may create an additional instruction sentence by continuing the positive and negative information extracted from the Q&A text DB 503 on a word-based basis for the keyword "ramen" that was found by searching the instruction sentence (example of additional instruction sentence 1302).

[0118] Alternatively, the processor may use the AI ​​engine 102 to create additional instructions that include keywords for positive or negative information. For example, a prompt might be created that incorporates positive information and integrates the user's instruction 1301 and the additional instruction 1302 into one sentence, such as, "I like tonkotsu ramen and soy sauce ramen. Please recommend a good ramen restaurant in Yokohama." (Positive words are explicitly included). An example of a sentence incorporating negative information would be, "Please recommend a good ramen restaurant in Yokohama. I don't like miso ramen, salt ramen, tsukemen, or abura soba." (Negative words are explicitly included). It is also possible to create a sentence that explicitly reflects both positive and negative information.

[0119] Note that while Figure 13 shows an example where additional instructions are generated by focusing on the keyword "ramen" in the user-inputted instructions, the system is not limited to this. Additional instructions can be generated similarly from other keywords included in the user-inputted instructions. For example, if the preference database contains preference information related to the keyword "Yokohama" (e.g., positive: detailed location information such as "Yokohama Station" and "Totsuka Ward"), additional instructions can be generated using that preference information.

[0120] As an alternative, the additional instruction generation program of the information processing device 1 may perform processing by referring to the AI ​​engine 102 (Figure 4). For example, the AI ​​engine 102 may perform the process of creating colloquial additional instructions based on keywords (preference information) extracted from the Q&A text DB, or on additional instructions (word permutations) that have been created based on those keywords.

[0121] Figure 14 shows an example of a modified version in which the AI ​​engine 102 creates a conversational supplementary instruction based on keywords (preference information) extracted from the Q&A text database. It is based on the example of supplementary instruction 1302 (word permutation) in Figure 13. A conversational supplementary instruction 1302b created based on supplementary instruction 1302 would look like this: "My favorite food is ramen. I especially like tonkotsu ramen and soy sauce ramen. I don't like miso ramen, salt ramen, tsukemen, or abura soba."

[0122] Furthermore, the AI ​​engine 102 may create a single integrated prompt from the user input instruction 1301 and the additional instruction 1302 or additional instruction 1302b. In the example in Figure 14, the integrated prompt 1303 would be something like, "My favorite food is ramen. I especially like tonkotsu ramen and soy sauce ramen. I don't like miso ramen, etc. Please recommend a good ramen restaurant in Yokohama." In this example, the sentence containing preference information is added before the user input question, but this is not the only example.

[0123] In the basic configuration of Example 1, the created additional instructions are processed in the background and do not need to be displayed on the screen. In other words, there is no need to show the additional instructions to the user. Figure 13 shows a case where the additional instructions are displayed in parallel with the user-inputted instructions for illustrative purposes, but it is not necessary to display the additional instructions. In the modified example, the created additional instructions are displayed on the screen (a browser screen such as Figure 6A or the screen of the additional instruction creation application) as shown in Figure 13. In this case, the user can see and confirm the content of the additional instructions on the screen.

[0124] Furthermore, in a further modification, the user may be able to edit the content of the additional instructions displayed on the screen. Figure 15 shows an example screen in a modification. An additional instructions field is displayed in the browser screen and has OK, Edit, Cancel buttons, etc. The user checks the additional instructions on the screen along with the user-inputted instructions. If the user wants to use the content of the additional instructions, they press the OK button; if they do not want to use the content of the additional instructions, they press the Cancel button. If the Cancel button is pressed, the question (prompt) will be displayed with only the original user-inputted instructions, without the additional instructions. The user can also edit / modify the content of the additional instructions by pressing the Edit button. The user can then edit the content (text) in the additional instructions field. For example, the user can add or delete positive information items, or add or delete negative information items. After confirming the additional instructions by marking them as OK, if the user wants to use this content as a prompt to ask a question to the AI, they can press, for example, the Execute Question button. This will send the prompt to Server 2. If the user does not want to ask a question, they can cancel with the Cancel button.

[0125] The above method can also be considered a method in which the user can directly edit the content of the additional instruction. When editing an additional instruction, the information processing device 1 automatically creates the additional instruction and displays it on the screen. The screen accepts editing of the displayed additional instruction, allowing the user to change information such as positive or negative. After editing, the changes can be confirmed by pressing a confirmation button or similar. The additional instruction may also be saved with a name.

[0126] Furthermore, user instructions, supplementary instructions, and their datasets that are frequently used by the user may be stored as historical information or templates on the information processing device 1 or server 2 side for each user ID, allowing for reuse by selecting from the history. For example, a list of user instructions, supplementary instructions, or datasets previously used by the user may be displayed as options on the screen, allowing the user to select from the options and execute them directly as a question. This can reduce effort and communication volume.

[0127] As a variation, instead of being limited to the method of creating the Q&A text DB 503 through the aforementioned search screen (Figure 7A, etc.), a method may be used in which the user directly sets their preference information in advance. In advance, the user directly specifies words such as things they like (positive) and things they dislike (negative) on the user settings screen and sets them as preference information (in other words, profile information). The contents of the preference information settings can be changed at any time. The information processing device 1 refers to this preference information during AI conversation and creates additional instruction sentences. The above method can also be considered as a method in which the user can directly edit the contents (preference information) of the Q&A text DB 503. When editing, for example, the user can input information such as keywords, positive, negative, etc. for each genre in a table like the one in Figure 12 on the screen.

[0128] [Differentiation] The following are also possible variations of Example 1.

[0129] In AI2a, the LLM may be a multimodal LLM that accepts images, audio, etc.

[0130] The same functionality may be achieved using AI (corresponding AI engine) on the information processing device 1 side, rather than on the server 2 side. In other words, the information processing device 1 may have a local LLM and generate answers using the local LLM. The AI ​​model used on the information processing device 1 side (the AI ​​deployed to the AI ​​engine 102) may be the pre-trained model created on the server 2 side, or it may be converted before use. For example, to ensure suitable operation on the AI ​​engine 102 of the information processing device 1, which is a smartphone, conversion from floating-point to fixed-point or from 24 bits to 8 bits may be performed. Furthermore, in this case, methods such as transfer learning, fine tuning, and distillation may be used to optimize and reduce the size of the model.

[0131] Example 1 describes an example of obtaining and understanding user preferences, such as what they like (positive) and what they dislike (negative), from web search results (especially selection results), but it is not limited to this. In addition to web information, it can be similarly applied to apps that can obtain user preferences from images taken or viewed by photo apps, as well as from content that users repeatedly input or select from UI menus, such as emails and games. The additional instruction creation program of the information processing device 1 can monitor and understand the user's operations and history in the target app, create preference information, and store it in a database.

[0132] Furthermore, while Example 1 described an example where both positive and negative information is reflected in the additional instruction statement, it is not limited to this. As a variation, a configuration using only one of either positive or negative information is also possible.

[0133] Figure 16 shows examples of prompts in modified forms. (A) is an example where the additional instruction uses only positive information. (B) is an example where the additional instruction uses only negative information. Alternatively, the system or user may be able to select and set whether to use positive or negative information, or both. For example, a screen like the one in Figure 15 may have buttons such as "Use positive information only" and "Use negative information only" to allow selection.

[0134] In cases like (A), where only positive information is provided, the probability of responses prioritizing information from the designated positive category increases. In cases like (B), where only negative information is provided, the probability of responses prioritizing exclusion of information from the designated negative category increases. In this case, since no positive category is specified, the responses will reflect information from other categories (categories that do not fall under negative), not just "tonkotsu ramen," which has the advantage of potentially leading to the discovery of new ramen varieties.

[0135] As described above, according to Example 1, by creating additional instruction sentences using user preference information regarding AI conversations, it is possible to improve the accuracy of the AI ​​responses that meet the user's expectations.

[0136] Example 1 describes an example where LLM is deployed (installed) on the AI ​​engine 202 on the server 2 side. Next, Example 2 shows an example of deploying (installing) a model that supports image generation AI.

[0137] <Example 2> The system and information processing device of Example 2 will be described. The basic configuration in Example 2 is the same as and common to Example 1, and the following will mainly describe the components in Example 2 that differ from Example 1. As mentioned above (Figure 1), Example 2 uses an image generation AI and Q&A text.

[0138] [System Configuration] Figure 17 shows an example of the system configuration in Example 2. The difference in system configuration compared to Figure 2 is that AI2b on Server 2 is an image generation AI, not an LLM. This system is an AI conversation system, or in other words, an image generation system that uses AI conversation. Similar to Example 1, a Q&A text DB503 (preference DB1b) reflecting the user's preferences is created in advance through a search screen. A specific example is shown in Figure 9 above. The additional instruction creation program (application 1c) of the information processing device 1 asks questions to the image generation AI2b on Server 2 and obtains images generated by the image generation AI2b as answers. When asking a question (request), the additional instruction creation program (application 1c) creates an additional instruction that reflects the user's preference information.

[0139] For example, the information processing device 1, which is a smartphone, has a function (camera function) to take photographic images using the rear camera 108a shown in Figure 4.

[0140] Example 2 shows a case where AI2b (AI engine 202) on server 2 is configured to perform image generation AI, such as a neural network (NN), in addition to language processing. In this example, we describe the application of Staple Diffusion (Reference 2) as the diffusion model in image generation AI2b, but it is not limited to this. The latest NN models can be applied.

[0141] Reference 2: Robin Rombach, et. al. “High-Resolution Image Synthesis with Latent Diffusion Models”, https: / / arxiv.org / pdf / 2112.10752

[0142] Image generation AI models learn by studying pairs of images and their corresponding texts, thereby memorizing the features of the neural networks (coefficients between the neural networks, layer configuration) based on the text. A trained image generation AI model can generate images from text even without an input image, by outputting the features of the neural networks based on the text and converting them into an image.

[0143] The overall processing flow in Example 2 is the same as in Figures 10 and 11. The structure of the preference information in the Q&A text database is also the same as in Figure 12, etc.

[0144] [Prompts and responses to image generation AI] Figure 18 shows an example of prompts and responses to the image generation AI 2b in Example 2. (A) shows the prompt created by the information processing device 1 (application 1c), and (B) shows the response text (including the image) generated by the image generation AI 2b.

[0145] User input instruction 1801 is, for example, "Please draw a picture of a fish." Additional instruction 1802, which is automatically generated by the additional instruction program, is, for example, "Hobby: Fishing, Positive: Red sea bream, Flounder, Negative: Yellowtail, Cherry salmon, Black bass, Sweetfish."

[0146] The response text is, for example, "Your hobby is fishing, and you like sea bream and flounder. How about the following images of fish you've caught?" and includes images 1811 and 1812 (schematic diagrams) generated by the image generation AI2b. Image 1811 is an image of a sea bream, and image 1812 is an image of a flounder. In this specific example, the images generated by the image generation AI2b are photographic / realistic images corresponding to the training source images. However, the training source images and the images to be generated may be non-realistic images such as illustrations.

[0147] In this way, the information processing device 1 creates an additional instruction sentence 1802 that reflects the user's preference information, especially positive and negative information. The AI2b of the server 2 generates a response sentence and an image that reflects the user's preference information from the additional instruction sentence 1802. In the example in Figure 18, even though the user input instruction sentence 1801 only specifies "fish," images 1811 and 1812 are generated that reflect the positive words "sea bream" and "flounder."

[0148] As described above, according to Example 2, even when generating images through conversation with AI, it is possible to improve the accuracy of the AI's response that meets the user's expectations.

[0149] [Differentiation] Example 2 demonstrates a case where preference information is obtained from images captured using the photo-taking function (camera function). However, the information processing device 1 is not limited to this, and it is possible to obtain preference information from any images collected, selected, and saved by the user.

[0150] In each embodiment using the Q&A text database, user preferences are broadly categorized into two values: positive and negative. However, this is not the only way to do so. Preferences may be evaluated using a scale of three or more values. For example, a user may rate their preference for a subject on a screen using a five-star scale (one star being the lowest, five stars being the highest, etc.). This system reflects such evaluation values ​​(preference information) in the additional instruction text. In the additional instruction text, words may be written using evaluation values ​​of three or more levels, or a threshold may be used to ultimately determine whether the evaluation values ​​are positive or negative.

[0151] <Example 3> The system and information processing device of Example 3 will now be described. As mentioned above (Figure 1), Example 3 uses an image generation AI and a captured image database.

[0152] Example 3 shows a case where a prompt (additional instruction) is generated from a photographic image taken by an information processing device 1, which is a smartphone, in order to awaken the emergent capabilities of the AI ​​on the server 2. An example of application during image capture is described, but it can be similarly applied when playing back images stored in the information processing device 1 or when viewing images on the web.

[0153] [System Configuration] Figure 19 shows the system configuration in Embodiment 3. The information processing device 1 is equipped with an application 1e that corresponds to the photo-taking function, and this application 1e includes a program for creating additional instruction statements. The information processing device 1 also has a captured image DB 1d that stores the photographic images taken by application 1e along with related information. The server 2 is equipped with an image generation AI 2b, similar to Embodiment 2.

[0154] In Example 3, an object detection model is deployed (installed) on the AI ​​engine 102 of the information processing device 1.

[0155] While various object detection models are available, this example uses YOLO-World (Reference 3).

[0156] Reference 3: Tianheng Cheng, et. al. “YOLO-World: Real-Time Open-Vocabulary Object Detection”, https: / / arxiv.org / pdf / 2401.17270

[0157] An object detection model can detect what objects are in a photograph (object type), as well as their location and size. Furthermore, by fine-tuning the model using an annotation dataset (data with correct object type labels) applied to the image, it becomes possible to determine the object type. For example, by fine-tuning a photograph using data labeled with fish species, it becomes possible to determine the type of fish. The processor then saves the detection results of the object detection model to a database.

[0158] Next, the attitude detection model will be deployed (installed) to the AI ​​engine 102 on the information processing device 1. Although various models can be used for attitude detection, in this example, OpenPose (Reference 4) will be used.

[0159] Reference 4: Zhe Cao, et. al. “OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, https: / / arxiv.org / pdf / 1812.08008

[0160] The posture detection model can detect the posture of people and other objects in a photograph (standing, sitting, arms raised, facing in a certain direction, etc.). The processor then stores these detection results in a database.

[0161] This embodiment 3 shows a case where the AI ​​engine 102 of the information processing device 1 performs pose detection after object detection, but it is not limited to this. Pose detection may be performed first, or a model may be used in which object detection and pose detection can be performed almost simultaneously.

[0162] [Creating a database of captured images] Figure 20 shows the processing flow for creating the captured image DB1d in Example 3.

[0163] In step S301, the information processing device 1 may display a screen provided by application 1e. In step S302, the user takes a desired photograph based on an operation input to the information processing device 1 (for example, pressing the capture button). The information processing device 1 creates a photographic image of the captured photograph. In step S303, the user performs a save operation if they wish to save the photograph (photographic image) they have taken, or a delete operation if they do not wish to save it. In step S304, the information processing device 1 saves the data of the photographic image designated for saving to the DB (photographic image DB1d). Alternatively, the photographic image may be automatically saved to the DB (photographic image DB1d) even without user operation. The above photographic shooting and saving process is repeated as needed. The user may delete the photographic image data in the photographgraphic image DB1d (which may also be a folder in the file system, etc.) as needed.

[0164] In this embodiment, one or more photographic images stored in the captured image DB1d are data that reflects the user's preferences and are automatically used as targets for analyzing the user's preferences, specifically as targets for analysis by an object detection model or the like.

[0165] In step S305, the object detection model is deployed (installed) to the AI ​​engine 102 of the information processing device 1. In step S306, the AI ​​engine 102 (object detection model) analyzes the features of the photographic images in the captured image DB 1d to detect objects (object type, position, size, etc.) in the photographs. In step S307, the processor of the information processing device 1 saves the analysis results (object detection information) from step S306 in the captured image DB 1d.

[0166] In step S308, the posture detection model is deployed (installed) to the AI ​​engine 102 of the information processing device 1. In step S309, the AI ​​engine 102 (posture detection model) analyzes the features of the photographic images in the captured image DB1d to detect the posture of the objects in the photographs. In step S310, the processor of the information processing device 1 saves the analysis results (posture detection information) from the posture detection in step S309 into the captured image DB1d.

[0167] Although the processes in steps S305-S307 and steps S308-S310 are shown separately in the diagram, they may be combined into a single process.

[0168] Figure 20 shows the case where both the object detection model and the pose detection model are used, and the case where the object detection model is processed first, followed by the pose detection model. However, this is not the only possible order. The order in which these models are processed does not matter, and they can also be processed simultaneously.

[0169] In this embodiment, the user's action of taking a photograph of a certain content and saving it to the captured image DB1d represents the user's preferences, and the content of that photograph represents positive preferences. Therefore, the image information in the captured image DB1d can be considered as user preference information / preference DB, and similarly, additional instruction statements can be created using this preference information.

[0170] The following methods are also possible for selecting target images as preference information to be used as the basis for creating additional instruction statements. The information processing device 1 automatically saves the data of the photographic images taken by the user to the captured image DB1d or a designated memory or folder. The information processing device 1 (application 1e) does not use all the photographic images saved in the folder as target images, but rather uses those selected by the user as target images. The information processing device 1 (application 1e) displays the photographic images saved in the folder on a dedicated screen, and the user makes a selection and registration regarding positive / negative from these photographic images. The image is similar to selecting positive / negative photographic images instead of operating on a search screen like in Figure 7A. The information processing device 1 (application 1e) creates preference information according to the selection results from that operation. For example, based on the analysis of the content of the photographic image selected as positive, a positive evaluation value is assigned to the object depicted in that photographic image.

[0171] [Image Database] Figure 21 shows an example of the contents of the captured image DB1d. The captured image DB1d stores detection results regarding detected objects and their characteristics in photographic images taken by the user. The AI ​​engine 102 detects the type and characteristics of objects, their posture, background conditions, etc. from the photographic images, and the processor stores this detection information in the captured image DB1d. The captured image DB1d in Figure 21 includes photographic image data 2101 and captured image information (analysis results) 2102. The photographic image data 2101 shows, for example, the case of having image 1 and image 2. Image 1 is a photograph of a sea bream. Image 2 is a photograph of a person on a boat at sea holding a flounder that they caught.

[0172] In the captured image information (analysis results) 2102, the table contains text descriptions of the analysis results for object detection and posture detection for the photographic image data 2101. For example, for image 1, the detected object is a "sea bream," and the characteristics of the "sea bream" are "a red fish with a large head, a high body, and one eye on each side." For image 2, the detected objects are a "flounder," a "person," and the "background." The characteristics of the "flounder" are "a flat, brown fish with its head on the left and two eyes on its back." The characteristics of the "person" are "sitting on a boat, holding a fish they caught facing left and showing it to the camera. They are holding the fish's head in their right hand and its tail in their left." The characteristics of the "background" are "the sea and an island."

[0173] [Prompt creation and image generation] Figure 22 shows the processing flow for prompt creation and image generation in Example 3. In step S401, application 1e of the information processing device 1 displays the AI ​​conversation screen. In step S402, the user inputs an instruction sentence on the screen to cause the AI ​​to generate an image. In step S403, the processor (additional instruction sentence creation program) of the information processing device 1 extracts relevant information (text representing objects and features) from the captured image DB 1d regarding the user input instruction sentence (first instruction sentence), and creates an additional instruction sentence based on the extracted information. The additional instruction sentence to be created (second instruction sentence) will contain information that specifies the user's preferences (content of the captured photograph) regarding the image to be generated by the AI.

[0174] In step S404, the information processing device 1 adds an additional instruction to the user input instruction to create a prompt for the AI. In step S405, the information processing device 1 sends the created prompt to the server 2. In step S406, the AI ​​(image generation AI) 2b on the server 2 generates a response text and image in response to the prompt and sends it to the information processing device 1 as a response in step S407. In step S408, the information processing device 1 outputs the response text and image to the user (e.g., displays it on the screen). In step S409, the user receives the response text and image by viewing it, etc.

[0175] Figure 23 shows examples of prompts and responses. In prompt (A), the user input instruction 2301 is, "Create an image of a sea bream being caught." The additional instruction 2302, which is automatically generated based on the information in the captured image DB1d, is, "Sea bream: A red fish with a large head, a high body, and one eye on each side. Person: Sitting on a boat, holding the fish they caught facing left and showing it to the viewer. Holding the fish's head in their right hand and its tail in their left. Background: Sea and island." For example, based on the keywords ("sea bream," "caught") contained in instruction 2301, the information in the table in Figure 21 can be extracted from the captured image DB1d, and the additional instruction 2302 can be generated using the extracted information.

[0176] The response generated by the image generation AI2b in (B) is "I have created an image of a sea bream being caught," and the generated image 2303 is attached. Image 2303 is an image of a person on a boat at sea, holding a sea bream that they have caught. Image 2303 reflects the content (features) of the attached instruction, and is an image in which the flounder in Image 2 has been changed to the sea bream in Image 1.

[0177] As shown in the example above, in Embodiment 3, the processor (additional instruction creation program) of the information processing device 1 extracts features of the user's captured image from the captured image DB1d, creates a prompt by adding it as an additional instruction to the question, and sends it to the server 2. The image generation AI2b (AI engine 202) of the server 2 generates an image that reflects the features of the additional instruction from that prompt. This makes it possible to generate an image that reflects the user's preferences even if it is not an image that the AI ​​model of the server 2 has learned.

[0178] <Example 4> Let's describe Example 4. In Example 4, as described above (Figure 1), the AI ​​uses a language model (LLM), and the database source (preference information) is captured images.

[0179] [System Configuration] Figure 24 shows the system configuration of Example 4. Server 2, like in Example 1, is equipped with LLM as AI2a, and LLM is deployed on AI engine 202. Information processing device 1, like in Example 3, is equipped with captured image DB1d.

[0180] The aforementioned Embodiment 3 shows a case in which the system has a function to create additional instructions that enable the generation of images related to an image based on a captured image DB1d created from the feature information of an image captured and saved by the user. In contrast, Embodiment 4 shows a case in which the system has a function to create additional instructions that enable the generation of language-based responses based on a captured image DB1d similar to that of Embodiment 3.

[0181] In Example 4, the creation of the captured image DB1d is the same as in Example 3. In the information processing device 1, the user takes a photograph and saves the photograph to the captured image DB1d. The information processing device 1 (application 1f) analyzes the photograph to extract features and registers the analysis information in the captured image DB1d.

[0182] [Creating prompts and generating responses] Figure 25 shows the processing flow for prompt creation and response text generation in Example 4. In step S501, application 1f of the information processing device 1 displays the AI ​​conversation screen. In step S502, the user inputs an instruction. In step S503, the processor (application 1f) of the information processing device 1 extracts relevant information from the captured image DB 1d regarding the user input question and creates an additional instruction. In step S504, the processor (application 1f) of the information processing device 1 adds the additional instruction to the question to create a prompt. In step S505, the information processing device 1 sends the prompt to the server 2. In step S506, the AI ​​(LLM) 2a of the server 2 generates a text-based response to the prompt. In step S507, the server 2 sends the response to the information processing device 1. In step S508, the information processing device 1 outputs the response. In step S509, the user receives the response by viewing the response on the screen, etc.

[0183] Figure 26 shows an example of a prompt and response. In prompt (A), the user input instruction 2601 is, for example, "Please tell me a delicious fish dish." The application 1f of the information processing device 1 extracts, for example, the keyword "fish" from the user input instruction 2601. Based on this keyword "fish," the processor extracts words such as "sea bream" and "flounder" from the captured image DB 1d (for example, the same as in Figure 21). The AI ​​engine 102 of the information processing device 1 uses the extracted words to create a sentence, which becomes the additional instruction 2602. For example, the additional instruction 2602 is "I caught a sea bream and a flounder."

[0184] (B) The response from AI(LLM)2a is, for example, "You caught some fish! I'll introduce some recommended dishes using sea bream and flounder." 1. Sea Bream Dish: Sashimi Ingredients: ... Instructions: ... 2. Flounder dish: Meunière Ingredients: ... Instructions: ... It would be something like, "Fresh fish is best prepared simply. Please try it."

[0185] When suggesting fish dishes, AI2a on Server 2 takes into account from the additional instructions that sea bream and flounder were caught, and recognizes that the fish are fresh because they were caught, and reflects this in its response. For example, the response recommends simple dishes such as sashimi and meunière.

[0186] As shown in the example above, in Example 4, the AI ​​becomes more likely to provide the user's expected response based on image information that reflects the user's preferences.

[0187] [Differentiation] In Examples 3 and 4, the captured image DB1d was considered to contain images captured and saved by the user, but this is not limited to this. The user may select positive or negative images on the screen for the images in the captured image DB1d. Figure 27 shows an example screen in a modified example. A list of images from the captured image DB is displayed on the screen, and the user selects their preferred image from the list. The information processing device 1 assigns information to the selected image as positive and the unselected images as negative. This enables a function similar to the creation of additional instruction texts using positive and negative information in the Q&A text DB in Example 1.

[0188] Although embodiments of this disclosure have been described in detail above, the invention is not limited to the embodiments described above and can be modified in various ways without departing from the gist of the invention. Each embodiment can be modified by adding, deleting, or replacing components, except for essential components. Unless otherwise specified, each component may be singular or plural. Combinations of each embodiment and its variations are also possible. [Explanation of symbols]

[0189] 1... Information processing device, 2... Server, 1a... Application (additional instruction creation program), 2a... AI, 5... Prompt, 6... Answer, 501... Search / DB creation function, 502... Additional instruction creation function, 503... Q&A text DB (preference DB).

Claims

1. An information processing device equipped with an interface for conversation between a user and an AI, Control unit and The system includes a database that stores the user's preference information, The control unit, When creating a prompt for the AI ​​in response to an input instruction to the interface, the system controls the AI ​​to create an additional instruction based on the words in the input instruction, by referring to the preference information in the database. The system controls the AI ​​to send the prompt, which is the input instruction with the additional instruction added to it. The system controls the AI ​​to obtain a response sentence that is generated by reflecting the input instruction sentence and the additional instruction sentence. Information processing device.

2. In the information processing apparatus according to claim 1, Regarding the database that stores the aforementioned preference information, The system displays search results based on user-entered search keywords on a search screen, allows the user to select desired information items from the search results, and stores the selection results as preference information in the database. Information processing device.

3. In the information processing apparatus according to claim 2, The information items selected by the user from the aforementioned search results are registered as positive information. The aforementioned additional instruction statement includes the positive information. Information processing device.

4. In the information processing apparatus according to claim 2, From the aforementioned search results, the information items that the user did not select are registered as negative information. The negative information is included in the aforementioned additional instruction statement. Information processing device.

5. In the information processing apparatus according to claim 2, The information items selected by the user from the aforementioned search results are registered as positive information. From the aforementioned search results, the information items that the user did not select are registered as negative information. The aforementioned additional instruction statement includes the positive information and the negative information. Information processing device.

6. In the information processing apparatus according to claim 1, Regarding the database that stores the aforementioned preference information, The system analyzes images taken, saved, or selected by the user to extract features and stores them in the database as preference information. Information processing device.

7. In the information processing apparatus according to claim 1, The aforementioned AI is a Large-Scale Language Model (LLM). Information processing device.

8. In the information processing apparatus according to claim 1, The aforementioned AI is an image generation AI, The AI ​​acquires the response text and image generated by reflecting the input instruction text and the additional instruction text. Information processing device.

9. In the information processing apparatus according to claim 1, To create a sentence as the aforementioned additional instruction: Information processing device.

10. In the information processing apparatus according to claim 1, The input instruction and the additional instruction are used to create the prompt, which is a single integrated sentence. Information processing device.

11. In the information processing apparatus according to claim 1, The additional instruction text is displayed on the screen, and the user is allowed to confirm and edit the additional instruction text. Information processing device.

12. In the information processing apparatus according to claim 2, The database storing the aforementioned preference information also registers genre information, which is a higher-level category related to the user's search keywords, in association with it. The aforementioned additional instruction statement also includes the genre information. Information processing device.

13. In the information processing apparatus according to claim 5, The additional instruction statement sets the maximum number of words to be included as positive information and the maximum number of words to be included as negative information. Information processing device.

14. A method for creating an AI prompt, which is executed by an information processing device equipped with an interface for conversation between a user and an AI, The steps include creating a database to store the user's preference information, When creating a prompt for the AI ​​in response to an input instruction to the interface, the steps include: creating an additional instruction based on the words in the input instruction, referring to the preference information in the database, and sending the prompt with the additional instruction added to the input instruction to the AI; The steps include obtaining a response sentence generated by the AI, which reflects the input instruction sentence and the additional instruction sentence, A method for creating an AI prompt, comprising the following:

Images