Information processing device, information processing program, information processing system, and database generation method

Abstract

This technology provides the ability to obtain appropriate answers tailored to user requests from large-scale language models. [Solution] An exemplary information processing device is an information processing device that generates prompts to be input to a large-scale language model, and uses a time-series vector database which collects multiple vector data representing features including time information of captured images to perform a similarity search to search for data similar to the user's input, and generates the prompt based on the user's input and image data associated with the vector data extracted from the time-series vector database by the similarity search.

Description

Technical Field

[0001] The present invention relates to a technique for generating a prompt to be input into a large language model (LLM).

Background Art

[0002] Conventionally, large language models, which are natural language processing models trained using a large amount of text data, are known. A large language model takes as input a sentence called a prompt and outputs a sentence. Such a large language model is applied, for example, to a system that performs question-and-answer. When a question sentence (prompt) is input, an answer sentence is output.

[0003] As a technique for improving the accuracy of answers given by large language models, information is added to the question sentence to generate a prompt (see, for example, Patent Document 1). Patent Document 1 discloses a sentence generation method for generating a prompt for input into a large language model by adding reference information to the input question sentence. In this sentence generation method, the reference information is obtained from a sentence database prepared separately from the large language model according to the characteristics of the input question sentence.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] For example, a driver might ask a question about the scenery around the vehicle they see while driving. In such cases, if the system is configured to extract reference information to add to the prompt from a text database, it may be difficult to extract reference information that is appropriate to the question. For instance, if a driver asks, "Tell me the information about the shop I just passed," it is difficult to extract appropriate reference information from the text database. If appropriate reference information cannot be obtained, the user is less likely to receive a satisfactory answer from the large-scale language model.

[0006] In recent years, multimodal large-scale language models have emerged that can process data other than text data, such as image data. Considering this, it is conceivable to add image data captured during driving as supplementary information. However, in the example of "the store we just passed," the exact timeframe of "just now" is unclear. Therefore, it becomes necessary to use video footage of a certain duration, such as up to 5 minutes before the question was asked, as supplementary information. In this case, sending large amounts of image data to the large-scale language model would create problems in terms of communication costs and processing speed. In other words, using image data as supplementary information requires innovative approaches not seen in conventional methods.

[0007] In view of the above, the present invention aims to provide a technology that can obtain appropriate answers in response to user requests from a large-scale language model. [Means for solving the problem]

[0008] An exemplary information processing device of the present invention is an information processing device that generates prompts to be input to a large-scale language model, and uses a time-series vector database which collects multiple vector data representing features including time information of captured images to perform a similarity search to search for data similar to the user's input, and generates the prompt based on the user's input and image data associated with the vector data extracted from the time-series vector database by the similarity search. [Effects of the Invention]

[0009] According to an exemplary information processing device of the present invention, prompts input to a large-scale language model can be generated by taking into account not only user input that may contain ambiguous expressions, but also image data related to the user input. This allows for the appropriate communication of user requests to the large-scale language model and the acquisition of appropriate responses from the model. Furthermore, according to the exemplary information processing device, image data related to user input can be appropriately extracted in order to utilize similarity search using vector data. As a result, prompts that prompt the large-scale language model to provide responses appropriate to the user can be generated. Additionally, the image data provided to the large-scale language model can be narrowed down to data for a limited time period by utilizing similarity search using a time-series vector database that includes time information. This prevents the amount of image data provided to the large-scale language model from becoming unnecessarily large, thus suppressing, for example, increased communication costs. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing an overview of the information processing system [Figure 2] A simplified schematic diagram showing the vector space obtained as a result of embedding. [Figure 3] A diagram illustrating prompts to input into a large-scale language model and the responses from the large-scale language model. [Figure 4] Block diagram showing the general configuration of the dialogue device. [Figure 5] A diagram illustrating object detection processing for captured images. [Figure 6] This figure shows an example of data after converting labels acquired by the scene recognition unit to strong labels. [Figure 7] A diagram illustrating the correction process for data after strong label conversion. [Figure 8]A flowchart illustrating the flow of the time-series vector database generation process performed by an interactive device (information processing device). [Figure 9] A flowchart illustrating the flow of prompt generation processing performed by an interactive device (information processing device). [Figure 10] Figure 6 shows the result of converting the shooting time information into relative time information based on the time of user input (speech). [Figure 11] Block diagram showing an overview of the information processing system in the first modified example. [Figure 12] Block diagram showing an overview of the information processing system in the second modified example. [Modes for carrying out the invention]

[0011] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In this specification, information processing refers to the processing performed by the device.

[0012] <1. Information Processing Systems> Figure 1 is a block diagram illustrating an overview of an information processing system 100 according to an embodiment of the present invention. As shown in Figure 1, the information processing system 100 comprises a server 1 and an interaction device 2. The server 1 and the interaction device 2 are each provided to be connectable to a communication network (not shown), such as the Internet or a telephone network. The server 1 and the interaction device 2 can communicate with each other via the communication network.

[0013] Server 1 is, in particular, a cloud server. As shown in FIG. 1, server 1 includes a large language model (LLM) 1a. The large language model 1a is software provided to enable natural language processing according to a model trained using a large amount of text data. In particular, the large language model 1a is a multimodal LLM. That is, the large language model 1a can handle data such as images in addition to text. For example, the large language model 1a is input with prompts such as command sentences and question sentences by the dialogue device 2, and responds to the dialogue device 2 with responses such as answer sentences in response to the input of the prompt. Also, for example, the large language model 1a is input with a prompt accompanied by an image in a command sentence or the like by the dialogue device 2, and responds to the dialogue device 2 with responses such as answer sentences in response to the input of the prompt.

[0014] The dialogue device 2 has a function of extracting commands, questions, etc. from the user U1 by voice recognition and performing processing according to the commands, etc. from the user U1 in cooperation with the large language model 1a. The dialogue device 2 generates a prompt to be input to the large language model 1a. In the present embodiment, the dialogue device 2 is an in-vehicle device mounted on a vehicle (not shown) such as an automobile. The dialogue device 2 may be configured to be included in, for example, a navigation device or a display audio. In the present embodiment, the user U1 is assumed to be an occupant of the vehicle on which the dialogue device 2 is mounted. The user U1 is, in particular, assumed to be a driver.

[0015] The dialogue device 2 includes a dialogue processing unit 2a, an embedding model 2b, a vector database (vector DB) 2c, and a preprocessing unit 2d.

[0016] The dialogue processing unit 2a performs processing related to dialogue in the dialogue device 2. The dialogue processing unit 2a performs, for example, dialogue processing with the user U1, communication processing with the large language model 1a, and generation processing of a prompt to be input to the large language model 1a.

[0017] Embedding Model 2b is a machine learning model that performs embedding on data such as words and sentences (text data). Embedding refers to the mathematical representation of data that encapsulates the meaning of words and sentences, and specifically refers to converting data such as words and sentences into a vector representation. A vector is an array of real numbers arranged in a sequence, such as [0.50, -0.10, 0.25, 0.55, ...]. For example, a vector with 100 numbers is called a "100-dimensional" vector.

[0018] In the vector space obtained by embedding various types of data, vectors with highly related meanings are placed in close proximity to each other. Figure 2 is a simplified schematic diagram showing the vector space obtained as a result of embedding. In Figure 2, the vector space is shown in two dimensions for ease of understanding, but in reality, the vector space has a much larger dimension. In the example shown in Figure 2, "I want to watch baseball" and "I want to watch soccer," which are semantically related in the sense of watching sports, are placed in close proximity in the vector space. On the other hand, the sentence "I want to walk my dog," which is semantically unrelated to watching sports, is placed far away from "I want to watch baseball" and "I want to watch soccer" in the vector space.

[0019] Vector Database 2c is a database containing data (vector data) that has been converted into a vector representation format by the embedding model 2b. Vector Database 2c consists of, for example, vector data and the original data associated with that vector data. The original data is the data before it is converted into a vector, and is, for example, text data.

[0020] Vector Database 2c is a collection of data that may be useful when generating prompts to input into the large-scale language model 1a. More specifically, Vector Database 2c is a collection of data used for similarity searches (vector searches). Similarity searches are a search method that calculates similarity by comparing vector data that have been converted into vector representations, and finds similar (related) information. For example, vector data whose positions in the vector space are within a predetermined range are treated as similar (related) information.

[0021] In this embodiment, the vector database 2c comprises a first vector database 2ca and a second vector database 2cb. The first vector database 2ca and the second vector database 2cb differ in their database generation methods. Details of the first vector database 2ca and the second vector database 2cb will be described later. In this specification, the second vector database 2cb may be referred to as the time-series vector database 2cb.

[0022] The preprocessing unit 2d performs preprocessing to generate the second vector database (time-series vector database) 2cb. The preprocessing unit 2d receives image data from the drive recorder DR1 (specifically, camera 200) of the vehicle equipped with the dialogue device 2. The preprocessing unit 2d uses the input image data to perform preprocessing to generate the second vector database 2cb. The second vector database 2cb is generated using the data obtained through preprocessing. Details of the processing performed by the preprocessing unit 2d will be described later.

[0023] In this embodiment, the camera 200 is included in the drive recorder DR1, but this is merely an example. The camera 200 that acquires image data to be input to the dialogue device 2 only needs to be capable of photographing the area around the vehicle and does not need to be included in the drive recorder. It is preferable that the camera 200 photographs a scene similar to what the vehicle occupants see, and more preferably a scene similar to what the driver sees. That is, it is preferable that the camera 200 is positioned to photograph the view looking forward from inside the vehicle. In this embodiment, the camera 200 included in the drive recorder DR1 is configured to photograph the area ahead from inside the vehicle.

[0024] Furthermore, in this embodiment, the pre-processing unit 2d is included in the dialogue device 2, but the pre-processing unit 2d may be configured as a separate in-vehicle device from the dialogue device 2. In such a configuration, the in-vehicle device constituting the pre-processing unit 2d only needs to be provided to enable data transmission and reception with the dialogue device 2.

[0025] Here, we will explain the information processing flow in the information processing system 100 shown in Figure 1, in which the large-scale language model 1a responds to user U1 based on input such as questions and commands from user U1. Note that the information processing shown in Figure 1 is realized by computer-based arithmetic processing (computer processing).

[0026] As a first prerequisite, a process is performed to generate the first vector database 2ca. This process is completed when user U1, who is in the vehicle, begins using the dialogue device 2. The process of generating the first vector database 2ca includes the processes indicated by the symbols (a1) and (a2) in Figure 1.

[0027] The process indicated by the symbol (a1) is the process of inputting knowledge data D1 into the embedding model 2b. The embedding model 2b, upon receiving knowledge data D1, performs the process of converting the input data into vector data. Knowledge data D1 is text data such as words and sentences. Knowledge data D1 is data that may be used as useful information when generating prompts. For example, when the large-scale language model 1a is asked to answer a question entered by user U1, knowledge data D1 is data that provides information that can be used as a reference for the answer (reference information). Multiple pieces of knowledge data D1 are input into the embedding model 2b, and each of the multiple pieces of knowledge data D1 is converted into vector data.

[0028] The process indicated by the symbol (a2) is the process of combining each vector data generated by the embedding model 2b in response to each knowledge data D1 input to create the first vector database 2ca. Each vector data stored in the first vector database 2ca is stored in the first vector database 2ca in a state linked to (as a set of) the original data before vector transformation.

[0029] As a second prerequisite, a process is performed to generate the second vector database 2cb. The process of generating the second vector database 2cb starts when the drive recorder DR1 installed in the vehicle begins recording with the camera 200, and continues as long as recording by the camera 200 continues. The process of generating the second vector database 2cb includes the processes indicated by the symbols (b1), (b2), and (b3) in Figure 1.

[0030] The process indicated by (b1) is the process of inputting the image data (image data) of the captured image taken by the drive recorder DR1 (camera 200) into the interactive device 2. The image data input into the interactive device 2 is stored in the memory of the interactive device 2 so as to be stored in association with the vector data stored in the second vector database 2cb. Note that the image data only needs to be stored in association with the vector data stored in the second vector database 2cb, so it may be included in the second vector database 2cb or not.

[0031] Furthermore, the image data input to the dialogue device 2 is processed by the preprocessing unit 2d. The preprocessing unit 2d performs a process to recognize a driving scene based on the input image data and associates the recognized driving scene with the image data as a feature of the captured image. The process of recognizing a driving scene may be, for example, an object detection process that detects objects in the captured image. The object detection process may be performed using a known object detection algorithm. Examples of known object detection algorithms include SSD, YOLO, or Faster R-CNN. The type of object to be detected in object detection may be appropriately determined during the design phase depending on what kind of driving scene is to be detected. When object detection is used as the process for recognizing a driving scene, the type of object detected by the object detection process is associated with the image data as a feature of the captured image.

[0032] The process for recognizing the driving scene does not have to be object detection; for example, it could be speech recognition, or a combination of object detection and speech recognition. If speech recognition is used as the process for recognizing the driving scene, for example, the content of the conversation of the vehicle occupants during the same time period as the time the captured image was taken may be associated with the image data as a feature of the captured image. The content of the conversation of the vehicle occupants can be acquired, for example, using the microphone included in the dashcam.

[0033] The process indicated by the symbol (b2) is the process of inputting feature data (for example, the data shown in Figure 6 below) of the captured image acquired by the preprocessing unit 2d, including information about the time of capture, into the embedding model 2b. The embedding model 2b, upon receiving the feature data of the captured image, performs a process to convert the input data into vector data. The first vector database 2ca is basically processed when the vehicle is not moving. On the other hand, the second vector database 2cb is basically processed when the vehicle is moving. For this reason, the timing at which the knowledge data D1 and the feature data of the captured image are input to the embedding model 2b are usually different.

[0034] The process indicated by the symbol (b3) is the process of generating a second vector database 2cb by combining the vector data generated by the embedding model 2b according to the characteristics of each captured image obtained in time series. The second vector database 2cb is updated moment by moment as images are acquired. Each vector data stored in the second vector database 2cb is stored in the second vector database 2cb while being linked to the image data of each captured image. As long as the image data and vector data are linked, the image data itself does not need to be stored in the second vector database 2cb.

[0035] Assuming that the first and second prerequisites described above have been fulfilled, that is, that the first vector database 2ca and the second vector database 2cb have been generated, user U1 has made input such as a question (process (c1), which will be described below). User U1's input may be, for example, a command or question using voice input. However, user U1's input may not be voice input, for example, text input using a character key. An example of user U1's input is, "How much did it cost at that gas station we just passed?"

[0036] When user U1 inputs to the dialogue device 2, for example, the dialogue processing unit 2a converts user U1's input voice (voice data) into text data. The text data obtained through this conversion (hereinafter referred to as the question text for the sake of explanation) is input to the embedding model 2b (process (c2)). The embedding model 2b converts the question text into a vector and outputs the vector data of the question text (process (c3)). It is possible that the input of the question text and the input of the data indicating the features of the captured image are performed at the same time to the embedding model 2b. In such cases, the processing of the question text input may be given priority.

[0037] Next, the dialogue processing unit 2a performs a similarity search (vector search) using the vector data of the question text and the vector database 2c (process (c4)). In this embodiment, as described above, the vector database 2c includes a first vector database 2ca and a second vector database 2cb. For this reason, a similarity search is performed using these two vector databases 2ca and 2cb. In this invention, the first vector database 2ca is not essential, but it is preferable to provide the first vector database 2ca.

[0038] Through similarity search, data with a highly relevant meaning to the question text is extracted from vector database 2c (processing (c5)). The data extracted by similarity search may be singular or plural. The data extracted by similarity search may be extracted from the first vector database 2ca or from the second vector database 2cb. Furthermore, the data extracted by similarity search may be extracted from both the first vector database 2ca and the second vector database 2cb, or not extracted from either.

[0039] In detail, data associated with vector data extracted from the first vector database 2ca and the second vector database 2cb via similarity search is extracted as reference information. The data associated with vector data extracted from the first vector database 2ca via similarity search (data considered as reference information) is the original data before conversion to vector data, and is text data such as words and sentences. The data associated with vector data extracted from the second vector database 2cb via similarity search (data considered as reference information) is image data stored in association with the vector data.

[0040] Let's say the question entered by user U1 is "How much did the gas station I saw earlier cost?" In this case, "was there earlier" is a phrase that identifies past information and is related to time information. That is, reference information related to "was there earlier" can be extracted from the second vector database (time-series vector database) 2cb, which contains time-series information.

[0041] The dialogue processing unit 2a generates a prompt based on the question entered by user U1 and the extracted reference information, and inputs the generated prompt into the large-scale language model 1a (process (c6)). The generated prompt may simply be a list of the question and reference information, or it may be a processed form in which the question has been modified according to the reference information.

[0042] When a prompt is input, the large-scale language model 1a generates a response sentence according to the content of the input prompt and outputs it to the dialogue device 2 (process (c7)). The response sentence generated by the large-scale language model 1a is derived by adding reference information to the user U1's question sentence. Upon receiving the response sentence, the dialogue device 2 outputs the obtained response sentence to user U1 using, for example, voice or screen display.

[0043] Figure 3 illustrates the prompt to be input to the large-scale language model 1a and the response from the large-scale language model 1a. In the example shown in Figure 3, the question is assumed to be "How much did the gas station we just passed cost?". In this example, it is also assumed that two image data of a gas station were extracted as reference information through a similarity search using the vector database 2c.

[0044] In Figure 3, the prompt is not the user U1's question exactly as it was asked, but rather a modified version incorporating supplementary information. The text portion of the prompt in Figure 3, such as "Please answer the user's question based on the following information. The following two images are of gas stations from 5 minutes ago to the present time. Question: How much does gas cost at a gas station?", can be generated by selecting an appropriate template from a pre-prepared (stored) set of templates according to the question content.

[0045] Furthermore, in the example shown in Figure 3, two image data extracted as reference information are included in the prompt. The image data included in the prompt may be, for example, the captured image obtained from camera 200, or it may be processed, such as by changing the data size. In addition, in the example shown in Figure 3, time information ("3 minutes 00 seconds ago," etc.) is added to the image data to make it easier to distinguish between multiple image data. This makes it easier for the large-scale language model 1a to generate a response sentence.

[0046] Furthermore, the prompt may include additional information related to the extracted image data. For example, the location of a gas station can be estimated using the vehicle's location sensor (e.g., GPS sensor) and driving history based on the time information of the extracted image data. In such cases, the prompt may also include map information showing the surrounding area of ​​the gas station as reference information.

[0047] The prompt contains image data suitable for answering user U1's question. Therefore, the large-scale language model 1a can obtain an appropriate answer to user U1's question (in the example shown in Figure 3, gas station price information) by performing image processing on the acquired image data. Alternatively, the large-scale language model 1a may perform a search using a website based on the image data included in the prompt and obtain gas station price information as a result of that search.

[0048] As shown in Figure 3, the large-scale language model 1a uses the acquired gas station price information to generate a response sentence such as, "The gas station I passed 3 minutes ago was 148 yen / L, and the gas station I passed 4 minutes ago was 140 yen / L." The large-scale language model 1a then sends (replies to) the generated response sentence to the dialogue device 2.

[0049] As described above, in this embodiment, by performing a similarity search using a time-series vector database 2cb that stores feature information including time information of images taken in the past, it is possible to extract some images that should be included in the prompt from among the images taken in the past. Here, the images that should be included in the prompt are images that enable the extraction of appropriate answers to questions from user U1. According to this embodiment, it is not necessary to transmit image data taken in the past to the large-scale language model 1a over a long period of time. For example, this can reduce communication costs and prevent the time to respond to questions from becoming longer. The configuration of this embodiment, which has such effects, will be described in more detail below.

[0050] <2. Interaction Device> The configuration of the dialogue device 2 provided in the information processing system 100 will now be described. In this embodiment, the dialogue device 2 includes an information processing device 20 (see Figure 4 below) that generates prompts to be input to the large-scale language model 1a. The information processing device 20 is composed of the above-mentioned dialogue processing unit 2a, embedding model 2b, vector database 2c, and preprocessing unit 2d. The information processing device 20 only needs to have the function of generating prompts to be input to the large-scale language model 1a. For this reason, at least one of the embedding model 2b, vector database 2c, and preprocessing unit 2d does not need to be included in the information processing device 20.

[0051] Figure 4 is a block diagram illustrating the schematic configuration of the dialogue device 2 according to an embodiment of the present invention. Note that Figure 4 shows the components necessary to explain the features of the dialogue device 2 according to this embodiment, and descriptions of general components are omitted. As shown in Figure 4, the dialogue device 2 comprises a controller 21 and a memory 22. The dialogue device 2 further comprises a communication unit 23, a microphone 24, and a speaker 25. The information processing device 20 included in the dialogue device 2 is composed of the controller 21, the memory 22, and the communication unit 23.

[0052] The controller 21 is configured to include an arithmetic circuit that performs arithmetic processing. More specifically, the controller 21 includes a processor that performs arithmetic processing, etc. The processor is configured to include, for example, a CPU (Central Processing Unit). The controller 21 may consist of one processor or multiple processors. If it consists of multiple processors, those processors should be connected to each other so that they can communicate with one another.

[0053] Memory 22 is comprised of volatile memory and non-volatile memory. Volatile memory is specifically RAM (Random Access Memory). Non-volatile memory is specifically ROM (Read Only Memory). Non-volatile memory may also include flash memory or a hard disk drive. Non-volatile memory stores programs (computer programs) and data that can be read by the computer. These programs include information processing programs that cause the computer (controller 21) to execute a process to generate prompts to be input into the large-scale language model 1a.

[0054] In this embodiment, the memory 22 stores model information 221 of the above-mentioned embedding model 2b (see Figure 1). The model information 221 includes the structure and parameters of the embedding model 2b, as well as the code instructions for executing the embedding model 2b.

[0055] Furthermore, memory 22 stores a vector database 2c (see Figures 1 and 4) which includes a first vector database 2ca and a second vector database (time-series vector database) 2cb. In other words, the dialogue device 2 (information processing device 20) has memory 22 that stores the first vector database 2ca and the second vector database (time-series vector database) 2cb. The first vector database 2ca is a collection of vector data obtained by collecting multiple vector data in which knowledge data D1 is represented by vectors according to its characteristics. The second vector database (time-series vector database) 2cb is a collection of vector data obtained by collecting multiple vector data representing features including time information of images captured by the camera 200 (see Figure 1).

[0056] The communication unit 23 is configured as a communication interface having an interface circuit for connecting to a communication network (not shown) such as the Internet. The microphone 24 is provided as a means for inputting the voice of user U1 to the dialogue device 2. The speaker 25 is provided as a means for outputting voice responses, etc., from the large-scale language model 1a to user U1. The dialogue device 2 may also include a display device (not shown) with a touch panel that functions as a means for inputting commands, etc., from user U1 and as a means for outputting responses from the large-scale language model 1a.

[0057] In detail, the controller 21 includes, as its functions, an interaction unit 211, an embedding unit 212, a search unit 213, a prompt generation unit 214, a scene recognition unit 215, and a strong label conversion unit 216. The functions of the controller 21 are realized by the processor executing arithmetic processing according to a program stored in memory 22. The program that realizes the functions of the controller 21 may consist of a single program or multiple programs.

[0058] The program stored in memory 22 may be provided, for example, on a computer-readable non-volatile recording medium. The non-volatile recording medium may be, for example, an optical recording medium (e.g., an optical disc), a magneto-optical recording medium (e.g., a magneto-optical disc), a USB memory, or an SD card, in addition to the non-volatile memory described above. As another example, the program stored in memory 22 may be provided from a program provision server via a communication line such as the Internet (a configuration provided by so-called download).

[0059] Furthermore, in this embodiment, the functions of the controller 21 are realized by software, i.e., by the arithmetic circuit (processor) executing arithmetic processing according to the program, but this is an example, and it may be realized by other methods. At least some of the functions of the controller 21 may be realized using, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). In other words, at least some of the functions of the controller 21 may be realized by hardware using a dedicated IC or the like. Also, at least some of the functions of the controller 21 may be realized by using both software and hardware.

[0060] Furthermore, each functional unit 211 to 216 is a conceptual component. The function performed by one component may be distributed among multiple components. Alternatively, the functions of multiple components may be integrated into a single component.

[0061] Furthermore, in this embodiment, the dialogue unit 211, the search unit 213, and the prompt generation unit 214 correspond to the dialogue processing unit 2a (see Figure 1) described above. The embedding unit 212 corresponds to the embedding model 2b described above. The scene recognition unit 215 and the strong label conversion unit 216 correspond to the preprocessing unit 2d (see Figure 1) described above.

[0062] The dialogue unit 211 receives input such as commands and questions from user U1. More specifically, the dialogue unit 211 acquires the voice spoken by user U1 as voice data via microphone 24 and converts the acquired voice data into text data using known speech recognition technology. Speech recognition may be performed using, for example, an AI model for speech recognition. In this embodiment, user input is the user's voice input. More specifically, voice input is input in which questions or commands are made by voice. By using voice input, user U1 can ask questions to the large-scale language model 1a in a safe manner, even while driving a vehicle. However, user input may be other than voice input, for example, input using an input device such as a touch panel.

[0063] Furthermore, the dialogue unit 211 acquires a response sentence (text data) from the large-scale language model 1a. The dialogue unit 211 converts the acquired response sentence into speech data using known speech synthesis technology (text-to-speech technology) and outputs it to the user U1 via the speaker 25. The generation of speech using speech synthesis technology may be performed, for example, using an AI model for speech generation. In other words, in this embodiment, the output of the response from the large-scale language model 1a in the dialogue device 2 is speech output. By using speech output, the user U1 can understand the response from the large-scale language model 1a in a safe state, even while driving a vehicle. However, the output of the response may be other than speech output, for example, it may be screen output using a display device.

[0064] The embedding unit 212 performs embedding processing using the embedding model 2b (see Figure 1) described above. More specifically, the embedding processing is a vector data generation process in which data is input to the embedding model 2b and vector data is generated. In this embodiment, as described above, the embedding processing by the embedding unit 212 is performed in multiple situations.

[0065] The embedding unit 212 uses the embedding model 2b to generate vector data in which knowledge data D1 is represented by vectors corresponding to its features. The embedding unit 212 also uses the embedding model 2b to generate vector data in which images captured by the camera 200 (see Figure 1) are represented by vectors corresponding to their features (including capture time information). Furthermore, the embedding unit 212 generates vector data in which user input (questions, etc.) is represented by vectors corresponding to its features.

[0066] Specifically, the controller 21 (interaction device 2, information processing device 20) performs vector data generation processing to generate vector data that constitutes the first vector database 2ca. The controller 21 (interaction device 2, information processing device 20) also performs vector data generation processing to generate vector data that constitutes the second vector database (time-series vector database) 2cb.

[0067] The vector data generation process (embedding process) may be configured to be performed on a separate device (e.g., a server device) from the dialogue device 2 and the information processing device 20. However, by configuring the information processing device 20 and the dialogue device 2, which are configured as in-vehicle devices, to have the function of performing the vector data generation process, as in this embodiment, the information processing system 100 can be constructed with less use of communication with the outside of the vehicle.

[0068] The handling of vector data generated by the embedding unit 212 differs depending on the scenario described above. Vector data obtained by vectorizing knowledge data D1 is stored in the first vector database 2ca. Vector data obtained by vectorizing the features of captured images is stored in the second vector database (time-series vector database) 2cb. Vector data obtained by vectorizing user input (questions, etc.) is used for similarity search (vector search) using the vector database 2c.

[0069] The search unit 213 performs a similarity search using the vector database 2c to find data similar to the user's input. In this embodiment, in detail, the similarity search is performed using the first vector database 2ca and the second vector database 2cb. That is, the controller 21 (interaction device 2, information processing device 20) performs a similarity search using the first vector database 2ca to find data similar to the user's input. The controller 21 (interaction device 2, information processing device 20) also performs a similarity search using the second vector database (time-series vector database) 2cb to find data similar to the user's input. More specifically, the similarity search is performed using the first vector database 2ca, the second vector database (time-series vector database) 2cb, and vector data obtained by vectorizing the user's input.

[0070] In similarity search (vector search), vector data semantically similar (related) to the vector data obtained by vectorizing user U1's question text etc. using embedding model 2b is extracted from vector databases 2ca and 2cb. The number of extracted vector data may be singular or plural. It is also possible that no vector data will be extracted.

[0071] If similar vector data is extracted from the first vector database 2ca, the original data (text data) before it was converted to that vector data will be obtained as reference information. If similar vector data is extracted from the second vector database 2cb, the image data associated with that vector data will be obtained as reference information.

[0072] The prompt generation unit 214 generates a prompt based on the user's input and the reference information obtained through the similarity search. The reference information obtained through the similarity search may be text data, image data, or both, as described above.

[0073] For example, if the reference information is text data, the prompt generation unit 214 generates a prompt based on the text data such as the question entered by user U1 and the text data acquired as reference information. That is, the controller 21 (dialogue device 2, information processing device 20) generates a prompt based on the user's input and the text data associated with the vector data extracted from the first vector database 2ca by similarity search. The generated prompt may be, for example, a list of the user's question and the text data acquired as reference information, or a configuration in which the question is processed according to the reference information.

[0074] Furthermore, for example, if the reference information is image data, the prompt generation unit 214 generates a prompt based on the text data such as the question entered by user U1 and the image data acquired as reference information. That is, the controller 21 (dialogue device 2, information processing device 20) generates a prompt based on the user's input and image data associated with vector data extracted from the time-series vector database 2cb by similarity search. The generated prompt may be, for example, a list of the user's question and the image data acquired as reference information, or a configuration in which the question is processed according to the reference information. The image data used to generate the prompt may be still image data or moving image data. With this configuration, image data can be flexibly provided to the large-scale language model 1a according to the data obtained by similarity search.

[0075] According to the configuration of this embodiment, prompts input to the large-scale language model 1a can be generated by taking into account not only user input that may contain ambiguous expressions, but also reference information related to the user input. In particular, in this embodiment, image data can be provided as reference information. As a result, user requests can be appropriately conveyed to the large-scale language model 1a, and appropriate responses can be obtained from the large-scale language model 1a.

[0076] Furthermore, according to the configuration of this embodiment, in order to utilize similarity search using vector data, reference information such as image data related to the user's input can be appropriately extracted. As a result, prompts that cause the large-scale language model 1a to provide an appropriate response for the user can be generated.

[0077] Furthermore, according to the configuration of this embodiment, the image data provided to the large-scale language model 1a can be narrowed down to data for a limited time period by using similarity search with a time-series vector database. As a result, it is possible to suppress the amount of image data provided to the large-scale language model 1a from becoming unnecessarily large, and thus suppress increases in communication costs, for example.

[0078] The scene recognition unit 215 processes data (image data) of the captured image taken by the camera 200 (see Figure 1) to recognize a driving scene, and associates the recognized driving scene with the image data as a feature of the captured image. In this embodiment, the scene recognition unit 215 performs object detection processing on the captured image (more specifically, the image data), considers the result obtained from the object detection processing as the recognition result of the driving scene, and associates it with the image data as a feature of the captured image.

[0079] In other words, the controller 21 (dialogue device 2, information processing device 20) performs object detection processing to acquire features of the captured image. The scene recognition unit 215 may be configured to be executed on a separate device (such as a server) from the dialogue device 2 and the information processing device 20. However, by configuring the information processing device 20 and the dialogue device 2, which are configured as in-vehicle devices, to have the function of performing object detection processing, as in this embodiment, the information processing system 100 can be constructed with less use of communication with the outside of the vehicle.

[0080] Figure 5 is a diagram illustrating the object detection process for captured images 300. Captured images 300, captured by camera 200 (see Figure 1), are input to the scene recognition unit 215 at regular time intervals (for example, every 5 seconds). The scene recognition unit 215 performs object detection processing sequentially each time captured images 300 are input.

[0081] As shown in Figure 5, in the object detection process, if an object of a predetermined type (usually multiple types) (target object) is present in the captured image 300, the position and type of that target object are detected. In Figure 5, the rectangular frame 301 (so-called bounding box) indicates the position information of the detected target object, and the text 302 above the rectangular frame 301 indicates the type information of the detected object. Hereafter, the type information of the detected target object will be referred to as a label. In the example shown in Figure 5, multiple types of target objects are present in the captured image 300, and the labels "gas station," "sign," and "traffic light" are acquired. Note that the number of labels acquired in each captured image 300 can be zero, singular, or multiple.

[0082] The strong label conversion unit 216 performs a process to convert the label indicating the type of object obtained by the object detection process into a strong label. Here, a strong label refers to a label that includes not only the type of object but also information about the time when the object was detected. In other words, the process of converting a label into a strong label is a process of adding time information to each label obtained by the object detection process. The time information is, in detail, the time information of the captured image in which the label was detected. As can be seen from the above explanation, the controller 21 (interaction device 2, information processing device 20) performs a time information addition process that adds the time information of the captured image to the label indicating the type of object obtained by the object detection process. The time information addition process referred to here can be called a strong label conversion process.

[0083] By converting labels to strong labels, it becomes possible to easily extract images containing a specific type of object from a collection of images acquired continuously at regular time intervals from camera 200, based on strong label information. In this embodiment, a time-series vector database 2cb is generated, which is a collection of time-series vector data, by converting labels representing the features of the captured images into strong labels and then performing vector conversion. In other words, the time-series vector database 2cb is configured in which vector data is collected in a form that has time information. For this reason, in similarity searches, it is possible to extract image data (still image data or video image data) from a vast amount of video image data captured in the past, limited to only the data for the required time period.

[0084] Figure 6 shows an example of data after strong label conversion of labels acquired by the scene recognition unit 215. In Figure 6, "Image ID" is identification information that identifies the data of the captured image taken by the camera 200. An Image ID is assigned to each captured image (a single still image).

[0085] Furthermore, the "recognition time" in Figure 6 is the time when the driving scene that was to be recognized occurred, and more specifically, it is the time the captured image (still image) taken by camera 200 was captured. The capture time is given in standard time format. In the example shown in Figure 6, the system is configured to capture images every 5 seconds, but the time interval between captures can be set as appropriate. The "label" in Figure 6 is information that indicates the characteristics of the driving scene, and more specifically, it is the type of object detected by the object detection process.

[0086] The data after strong label conversion of each captured image (strong label conversion data) is structured so that the capture time information and label information are linked to each image ID. In the example shown in Figure 6, the parts where the label information is blank indicate that no label was detected. In other words, the parts where the label information is blank indicate that the target object was not detected by the object detection process.

[0087] The data shown in Figure 6 is obtained when a vehicle equipped with the dialogue device 2 and camera 200 performs a driving sequence as shown in the following driving image. (Image of the vehicle in motion) Time 10:02:00: While driving, I saw a gas station, and in the distance there were signs and traffic lights. Time 10:02:10: After driving a short distance, only signs and traffic lights were visible. Time 10:02:20: After passing the traffic light, there were no landmarks to guide me. Time 10:03:30: As we continued driving, we began to see the parking lot.

[0088] The strongly labeled data obtained for each captured image is converted into vector data by the embedding model 2b and stored in the time-series vector database 2cb. Specifically, the image ID is not converted into vector data, while the recognition time information and label information are converted into vector data. The image data itself associated with the image ID may be stored in the time-series vector database 2cb or in a separate database for image data storage. Furthermore, for captured images from which label information could not be obtained, the strongly labeled conversion process and the embedding process (vector data generation process) may not be performed.

[0089] Furthermore, the strongly labeled data obtained for each captured image may be embedded immediately after the data is obtained. However, this is merely an example, and the system may be configured to perform the embedding process after a certain period of time (e.g., 30 seconds) of data has accumulated. In such a configuration, correction processing may be performed according to the relationship between the strongly labeled data of multiple captured images, and the embedding process may be performed on the corrected data.

[0090] Figure 7 is a diagram illustrating the correction process for data after strong label conversion. Figure 7 shows data obtained assuming the same driving scene as in Figure 6. In the example shown in Figure 7, unlike in Figure 6, the label information associated with image ID "01026" does not include "sign". On the other hand, "sign" is included in the label information of images IDs "01025" and "01027", which are taken in a different time sequence. Based on the results of other images taken in a different time sequence, it is presumed that "sign" should also be included in the label information for image ID "01026", and that an error occurred in the object detection process. In such cases, a correction may be performed to add "sign" to the label information associated with image ID "01026".

[0091] <3. Time-series vector database generation process> Next, the process for generating the time-series vector database 2cb, which is executed in the dialogue device 2 (information processing device 20) configured as described above, will be explained. Note that the process for generating the time-series vector database 2cb may be executed by a device other than the dialogue device, or it may be executed through cooperation between the dialogue device and another device.

[0092] Figure 8 is a flowchart illustrating the flow of the time-series vector database 2cb generation process executed by the dialogue device 2 (information processing device 20). This flowchart shows the technical details of the computer program that enables the computer to implement the database generation method.

[0093] The process shown in Figure 8 becomes executable when the power to the dialogue device 2 is turned on and it transitions to a state where it can perform the time-series vector database 2cb generation process. The process shown in Figure 8 starts when this executable state is reached and the camera 200 mounted on the drive recorder DR1 begins to periodically transmit captured image data to the dialogue device 2.

[0094] In step S1, the controller 21 (scene recognition unit 215) acquires captured images (more specifically, data) from the camera 200. In other words, the database generation method includes a step of acquiring data from captured images taken by the camera 200. Once the controller 21 has acquired the captured image data, the process proceeds to the next step S2.

[0095] In step S2, the controller 21 (scene recognition unit 215) performs a scene recognition process to recognize the driving scene using the acquired image data. As described above, the scene recognition process is performed to acquire features of the captured image, and in this embodiment, it is an object detection process. The object detection process acquires labels that represent the features of the captured image. Once the labels are acquired, the process proceeds to the next step S3.

[0096] In step S3, the controller 21 (strong label conversion unit 216) performs a strong label conversion process to convert the acquired labels into strong labels. In detail, the strong label conversion process is the process of adding shooting time information to the acquired labels, as described above. The data obtained by the strong label conversion process is data that shows the characteristics of the captured image, including shooting time information and label information, as explained using Figure 6 above. In other words, the database generation method includes a step of processing to acquire the characteristics of the captured image, including shooting time information. Once the strong label conversion process is completed, the process proceeds to the next step S4.

[0097] In step S4, the controller 21 (embedding unit 212) performs an embedding process to convert the data obtained from the strong label conversion process into vector data. The data to be vectorized is, in detail, the capture time information and label information that represent the features of the captured image. The vector data obtained from the vector conversion process is data in which the features, including the time information of the captured image, are represented as vectors. In other words, the database generation method includes a step of generating vector data that represents the features of the captured image as vectors. Once the embedding process is completed, the process proceeds to the next step S5.

[0098] In step S5, the controller 21 (embedding unit 212) processes the vector data acquired in step S4 and stores it in the database. Once the process of storing the vector data in the database is complete, the process proceeds to the next step S6.

[0099] In step S6, the controller 21 (scene recognition unit 215) determines whether there is a reason to terminate the process of generating the time-series vector database 2cb. Reasons for termination include, for example, a power-off command for the dialogue device 2 or the occurrence of a malfunction. Other reasons for termination include the power being turned off or a malfunction occurring in the drive recorder DR1 equipped with the camera 200. If there is a reason for termination (Yes in step S6), the process of generating the time-series vector database 2cb shown in Figure 8 is terminated. If there is no reason for termination (No in step S6), the process returns to step S1, and the process from step S1 onward is repeated.

[0100] Unless a reason for termination is determined in step S6, captured images taken at regular time intervals are processed sequentially, and vector data representing the features of the captured images as vectors are sequentially stored in the database. A time-series vector database 2cb is generated by collecting vector data for each sequentially captured image in a time series. In other words, the database generation method includes the step of collecting vector data for each sequentially captured image to create a time-series vector database.

[0101] According to the database generation method of this embodiment, vector data representing the features of captured images as vectors will be included in the database, thus providing a database suitable for similarity searches that search for similar data by comparing vectors. Furthermore, according to the database generation method of this embodiment, since the vector data representing the features of images will include information on the time of capture, when extracting data similar to user U1's questions or commands from the database, it becomes possible to narrow down the data extraction to data within the relevant time period.

[0102] Furthermore, according to the method for generating the time-series vector database 2cb in this embodiment, the amount of data stored in the time-series vector database 2cb will continue to increase. To address this, each data stored in the time-series vector database 2cb may be configured to be deleted after a certain period of time (e.g., 8 hours) has elapsed since it was stored. This makes it possible to suppress an increase in the size of the memory 22 that stores the time-series vector database 2cb.

[0103] <4. Prompt generation process> Next, we will describe the prompt generation process executed in the dialogue device 2 (information processing device 20) configured as described above.

[0104] Figure 9 is a flowchart illustrating the flow of the prompt generation process executed by the dialogue device 2 (information processing device 20). This flowchart shows the technical details of the computer program that enables the computer to implement the prompt generation method. The process shown in Figure 9 becomes executable when the dialogue device 2 is powered on and transitions to a state where it can interact with user U1.

[0105] In step S11, the controller 21 (interaction unit 211) acquires input from user U1. User U1's input here refers to questions and commands spoken by user U1. Specifically, the controller 21 acquires the questions and commands spoken by user U1 as audio data via the microphone 24 and converts the audio data into text data. In other words, the controller 21 acquires the user's input in the form of text data. The controller 21 temporarily stores the acquired text data as the input statement of user U1 in the RAM or other memory provided by memory 22. Once the input statement of user U1 is acquired, the process proceeds to the next step S12.

[0106] In step S12, the controller 21 (embedding unit 212) performs embedding on the input sentence of user U1 that it has acquired. This gives vector data of the input sentence of user U1. The acquired vector data of the input sentence is temporarily held in RAM or the like in memory 22. Once the embedding process of user U1's input sentence is complete, the process proceeds to the next step S13.

[0107] In step S13, the controller 21 performs a time information conversion process to convert the time information of the captured image taken by the camera 200 into relative time information based on the time of input (utterance) by user U1, and includes it in the time series vector database 2cb. As a result, the time series vector database 2cb will contain relative time information (vector-converted information). It is preferable that, after this process, the capture time information (vector-converted information) that was originally included in the time series vector database 2cb is also included in the time series vector database 2cb, just as it was before the process.

[0108] In detail, the controller 21 (search unit 213) converts the shooting time information of the original data before vectorization into relative time information based on the time of input (utterance) by user U1. The original data before vectorization is stored, for example, in the time-series vector database 2cb. Figure 10 shows the result of converting the shooting time information (time information before vectorization) from Figure 6 into relative time information based on the time of input (utterance) by user U1. In the example shown in Figure 10, it is assumed that user U1's input (utterance) occurred at 10:05:00. For this reason, for example, the shooting time of 10:02:00 is 3 minutes 00 seconds earlier in relative time, and the shooting time of 10:03:30 is 1 minute 30 seconds earlier in relative time.

[0109] The controller 21 (embedding unit 212) embeds the obtained relative time information and adds the vectorized vector data obtained by the embedding process to the time-series vector database 2cb.

[0110] For example, suppose user U1's question is, "How much did the gas station I passed earlier cost?" In this case, if the time information of the photograph included in the time-series vector database 2cb is in standard time format, it is difficult to identify data similar to the time expression "earlier." However, by converting it to a relative time based on the time the question was asked, as described above, it becomes possible to extract data similar to "earlier" through similarity search using vector data. In other words, by performing time information conversion processing as in this embodiment, it becomes easier to appropriately handle various time expressions from user U1.

[0111] Once the time information conversion process in step S13 is completed, the process proceeds to the next step, S14.

[0112] In step S14, the controller 21 (search unit 213) performs a similarity search using the first vector database 2ca and the second vector database (time-series vector database) 2cb. The similarity search is a process that searches for data similar to the input sentence of user U1. In the similarity search, the input sentence of user U1 is used in the form of vector data obtained in step S12. In the similarity search, for example, a predetermined number of data are extracted in order of the degree of semantic similarity to the input sentence of user U1. As another example, in the similarity search, data whose degree of similarity to the input sentence of user U1 is above a predetermined threshold is extracted. When similar vector data is extracted, information associated with that vector data is obtained as reference information. As described above, when similar vector data is obtained from the first vector database 2ca, the reference information is the original data before it is converted into vector data, and is text data such as words and sentences. When similar vector data is obtained from the second vector database 2cb, the reference information is image data stored in association with the vector data. Once the similarity search process is complete, the process proceeds to the next step, S15.

[0113] In step S15, the controller 21 (prompt generation unit 214) generates a prompt based on the question entered by user U1 and the extracted reference information. The prompt generation process is as described above and has been explained in detail using Figure 3, so the explanation is omitted here.

[0114] <5. Variation> Figure 11 is a block diagram showing an overview of the first modified information processing system 100A. In the above-described embodiment, the dialogue device 2 or information processing device 20, which is configured as an in-vehicle device, is equipped with an embedding model 2b. Alternatively, as shown in Figure 11, the embedding model 2b may be provided in a server device 3 that is configured to communicate with the dialogue device 2A or an information processing device included in the dialogue device 2A (hereinafter referred to as dialogue device 2A, etc.). In this case, the data to be subjected to embedding processing (vector data generation processing) is sent to the server device 3. Then, the embedding processing is executed in the server device 3, and the vector data obtained by the embedding processing is transmitted to the dialogue device 2A, etc.

[0115] In the configuration shown in Figure 11, for example, if the dialogue device 2A etc. is an in-vehicle device, the embedding model 2b may be considered not to be included in the dialogue device 2A etc. On the other hand, the dialogue device 2A etc. does not necessarily have to consist solely of in-vehicle devices. Under this interpretation, the server device 3, which is provided to communicate with the dialogue device 2A etc., may be considered part of the dialogue device 2A etc. In this case, even in the configuration shown in Figure 11, the dialogue device 2A etc. may be considered to include the embedding model 2b.

[0116] Figure 12 is a block diagram showing an overview of the second modified information processing system 100B. As shown in Figure 12, the server device 3A, which is provided to communicate with the dialogue device 2B or the information processing device included in the dialogue device 2B (hereinafter referred to as the dialogue device 2B, etc.), may be configured to be equipped with an embedding model 2b and a vector database 2c. In such a configuration, not only embedding processing but also similarity search will be performed on the server device 3A.

[0117] For example, in the configuration shown in Figure 12, the dialogue device 2B, etc., will obtain similar vector data extracted from the time-series vector database 2cb as vector data similar to the input of user U1 from the server device 3A. The dialogue device 2B, etc., will then generate a prompt based on the input of user U1 and the image data associated with the similar vector data obtained from the server device 3A. In this configuration, it is preferable that the image data is stored in the memory 22 (see Figure 4) of the dialogue device 2B, etc., in order to reduce the communication burden.

[0118] In the configuration shown in Figure 12, for example, from the standpoint that the dialogue device 2B etc. is an in-vehicle device, the embedding model 2b and vector database 2c may be considered not to be included in the dialogue device 2B etc. On the other hand, the dialogue device 2B etc. does not necessarily have to consist solely of in-vehicle devices. Under this interpretation, a server device 3A that is provided to communicate with the dialogue device 2B etc. may be considered part of the dialogue device 2B etc. In this case, even in the configuration shown in Figure 12, the dialogue device 2B etc. may be considered to include the embedding model 2b and vector database 2c.

[0119] <6. Points to note> The various technical features disclosed in the embodiments for carrying out the invention as described herein can be modified in various ways without departing from the spirit of the technical creation. Furthermore, the multiple embodiments and modifications disclosed in the embodiments for carrying out the invention as described herein may be combined to the extent possible. [Explanation of symbols]

[0120] 1. Server 1a. Large-scale language models 2, 2A, 2B... Interaction devices 2b...Embedding Model 2cb...Time-series vector database 20. Information Processing Devices 21. Controller 22...memory 100, 100A, 100B... Information Processing Systems 200...camera 300...Photographed images

Claims

1. An information processing device that generates prompts to be input to a large-scale language model, Using a time-series vector database containing multiple vector data representing features of captured images, including time information, a similarity search is performed to find data similar to the user's input. An information processing device that generates the prompt based on the user input and image data associated with vector data extracted from the time-series vector database by the similarity search.

2. The information processing apparatus according to claim 1, which performs a vector data generation process to generate vector data that constitutes the time-series vector database.

3. The information processing apparatus according to claim 1 or 2, which performs object detection processing to acquire the features of the captured image.

4. The information processing apparatus according to claim 3, which performs a time information assignment process to assign the shooting time information to a label indicating the type of object obtained by the object detection process.

5. The information processing apparatus according to claim 1, which, when performing the similarity search, performs a time information conversion process to convert the shooting time information into relative time information based on the time input by the user and includes it in the time series vector database.

6. The information processing apparatus according to claim 1, wherein the image data is still image data or moving image data.

7. The information processing apparatus according to claim 1, wherein the user input is the user's voice input.

8. An information processing program that causes a computer to perform the process of generating prompts to be input into a large-scale language model, The aforementioned computer, This involves performing a similarity search using a time-series vector database that collects multiple vector data representing features of captured images, including time information, to find data similar to the user's input. The prompt is generated based on the user input and the image data associated with the vector data extracted from the time-series vector database by the similarity search. An information processing program that functions as a means to perform an action.

9. A server equipped with a large-scale language model, A dialogue device that generates prompts to be input to the aforementioned large-scale language model, Equipped with, The dialogue device is Controller and A memory that stores a time-series vector database containing multiple vector data representing features of captured images, including time information, It has, The aforementioned controller, Using the aforementioned time-series vector database, a similarity search is performed to find data similar to the user's input. An information processing system that generates the prompt based on the user input and image data associated with vector data extracted from the time-series vector database by the similarity search.

10. An information processing device that generates prompts to be input to a large-scale language model, From a time-series vector database containing multiple vector data representing features of captured images, including time information, similar vector data extracted from the user's input is obtained. An information processing device that generates the prompt based on the user input and the image data associated with the similar vector data.

11. The data of the image captured by the camera is acquired, The aforementioned features of the captured image are obtained, including information about the time of capture. Vector data is generated that represents the features of the captured image as vectors. A database generation method that collects the vector data for each of the sequentially captured images to form a time-series vector database.

Images