Information processing apparatus and program

Abstract

To provide an information processor and a program capable of reflecting information on environments and situations in use of a user who uses a generated AI on an answer sentence.SOLUTION: An information processing apparatus according to an embodiment includes an acquisition unit that acquires a query text indicating a content of a query input by a user, a first conversion unit that converts a state indicated by sensor data obtained from a sensor that senses a surrounding environment of the user into a label text that qualitatively represents the state, a first generation unit that generates a prompt based on the query text and the label text, a second generation unit that generates a response to the content of the query by inputting the prompt to a generative AI, and a storage control unit that stores data in which the query text, the response, and the sensor data and / or the label text are associated with each other in a storage device.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device and a program. [Background technology]

[0002] In recent years, natural language processing systems that utilize generative AI (Artificial Intelligence) such as large language models (LLMs) that can generate natural-sounding sentences have emerged. Furthermore, a method called Retrieval Augmented Generation (RAG) is used to improve the accuracy of answers generated by generative AI. RAG involves obtaining supplemental information from queries previously input to the generative AI, adding the supplemental information to the query, and then inputting it to the generative AI.

[0003] Users of the AI ​​system may input queries under various circumstances. In such cases, the answer desired by the user may differ depending on environmental changes such as temperature.

[0004] However, while conventional technology can add supplemental information to a query, the supplemental information is limited to previously entered query text itself, the time period when it was entered, frequency, etc. Therefore, conventional technology cannot reflect the environment or situation of the user using the AI ​​in the answer text, and there is room for improvement in terms of improving the accuracy of the answer text generated by the AI. Summary of the Invention [Problem to be solved by the invention]

[0005] The problem that the present invention aims to solve is to provide an information processing device and program that can reflect information about the environment and situation at the time of use of the user using the generation AI in the answer sentence. [Means for solving the problem]

[0006] An information processing device according to an embodiment includes an acquisition means for acquiring a query statement indicating the content of an inquiry entered by a user, a first conversion means for converting a state indicated by sensor data obtained from a sensor that senses the user's surrounding environment into label text that qualitatively expresses the state, a first generation means for generating a prompt based on the query statement and the label text, a second generation means for generating an answer statement to the content of the inquiry by inputting the prompt into a generation AI, and a storage control means for storing data that associates the query statement, the answer statement, the sensor data, and / or the label text in a storage device. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing an example of a schematic configuration of a concierge system according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the interface device according to the embodiment. [Figure 3] FIG. 3 is a block diagram illustrating an example of a hardware configuration of the text generator according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a data configuration of the label dictionary according to the embodiment. [Figure 5] FIG. 5 is a diagram showing an example of the data configuration of the question record DB stored in the text generator according to the embodiment. [Figure 6] FIG. 6 is a block diagram showing an example of a functional configuration of the interface device and the text generator according to the embodiment. [Figure 7] FIG. 7 is a sequence diagram illustrating an example of a control process of the concierge system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of an information processing device and a program will be described with reference to the drawings. Note that the present invention is not limited to the following embodiment.

[0009] FIG. 1 is a schematic diagram showing an example of the overall configuration of a concierge system S according to an embodiment. As shown in FIG. 1, the concierge system S includes an interface device 1 and a text generation device 2. The concierge system S is installed in a store such as a supermarket or a department store, and provides customers with functions such as guiding them to product placement locations and suggesting recommended products. The interface device 1 and the text generation device 2 are connected to each other via wire or wirelessly so that they can communicate with each other.

[0010] The interface device 1 is, for example, a communication robot installed in a store. The interface device 1 exchanges various types of information between the interface device 1 and a user who uses the concierge system S. In this embodiment, the user is, for example, a customer of the store.

[0011] Specifically, when the interface device 1 detects a user using the concierge system S, it acquires sensor data relating to the user or the environment around the store using various sensors 111 (see FIG. 2). Furthermore, when the interface device 1 receives voice data input from the user via a microphone 109 (see FIG. 2), it transmits the sensor data and the voice data to the text generator 2. Furthermore, when the interface device 1 receives voice data from the text generator 2, it outputs the received voice data via a speaker 108 (see FIG. 2).

[0012] In this embodiment, the interface device 1 is a communication robot, but is not limited to this. For example, the interface device 1 may be a mobile terminal loaned to a customer from a store, a tablet terminal installed in a cart, or a portable terminal such as a smartphone owned by a user.

[0013] The text generator 2 is an example of an information processing device in this embodiment. The text generator 2 converts voice data transmitted from the interface device 1 into text data and generates a query sentence. The text generator 2 also converts sensor data transmitted from the interface device 1 into label text, which will be described later. The text generator 2 then generates an answer sentence based on the query sentence and the label text, and outputs the generated answer sentence. An answer sentence is text that includes a response to the item (inquiry content, etc.) described in the query sentence. Specifically, the text generator 2 generates an answer sentence that takes into account environmental information about the environment around the user (or store), based on various sensor data acquired by the sensor 111.

[0014] In this embodiment, the text generator 2 is described as being realized by a single device, but it may be realized by multiple devices. Also, the interface device 1 and the text generator 2 may be integrated into one device.

[0015] Next, a description will be given of the hardware configuration of the interface device 1. Fig. 2 is a block diagram showing an example of the hardware configuration of the interface device 1 according to this embodiment.

[0016] As shown in FIG. 2, the interface device 1 includes a CPU (Central Processing Unit) 101, which is an example of a processor, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a memory unit 104, a display unit 105, an operation unit 106, an imaging unit 107, a speaker 108, a microphone 109, a device interface 110, a sensor 111, and a communication unit 112.

[0017] The CPU 101 is an example of a processor, and performs overall control of each unit of the interface device 1. The ROM 102 stores various programs. The RAM 103 is a workspace where programs and various data are developed.

[0018] The memory unit 104 is a non-volatile memory such as a hard disk drive (HDD) or flash memory that retains stored information even when the power is turned off. The memory unit 104 includes a control program unit 1041.

[0019] The control program section 1041 stores a control program for controlling the interface device 1. The CPU 101, the ROM 102, the RAM 103, and the memory section 104 are connected to one another via a bus 113. The CPU 101, the ROM 102, and the RAM 103 constitute a control section 100 having a computer configuration. That is, the control section 100 executes a control process for the interface device 1, which will be described later, by the CPU 101 operating in accordance with the control program section 1041 stored in the ROM 102 or the memory section 104 and expanded in the RAM 103.

[0020] The control unit 100 is connected to a display unit 105 , an operation unit 106 , an imaging unit 107 , a speaker 108 , a microphone 109 , a device interface 110 , and a communication unit 112 via a bus 113 .

[0021] The display unit 105 is a display device such as an LCD (Liquid Crystal Display), etc. The display unit 105 displays various information under the control of the CPU 101.

[0022] The operation unit 106 receives various inputs from the user. The operation unit 106 is, for example, a touch panel laminated on the display surface of the display unit 105. The operation unit 106 may also be an input device such as a keyboard or a pointing device.

[0023] The imaging unit 107 is an imaging device having an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), etc. The imaging unit 107 detects the user operating the interface device 1 and captures an image of the user.

[0024] The speaker 108 is an example of an audio output device, and outputs audio data input from the CPU 101 as audio.

[0025] The microphone 109 is an example of a voice input device. The microphone 109 acquires voice uttered by the user as voice data and outputs the acquired voice data to the CPU 101.

[0026] The device interface 110 acquires sensor data from the sensor 111. If the sensor 111 outputs an analog value, the device interface 110 includes a signal processing circuit and an A / D converter. If the sensor 111 has a communication function and transmits a measurement value to the interface device 1 as digital data, the device interface 110 includes a communication interface that can communicate with the sensor 111 via wire or wirelessly. The sensor data acquired by the device interface 110 is transmitted to the control unit 100.

[0027] The sensor 111 is a sensor device that senses the surrounding environment. Here, the surrounding environment in this embodiment refers to the surrounding environment of the user to whom the answer sentence is provided. Note that the surroundings of the user may include not only the vicinity of the user, but also the range of the area where the user and the store the user is currently visiting (i.e., the store where the concierge system S is installed) are located. For example, the surrounding environment refers to the temperature, humidity, wind speed, weather, etc. of the area including the store where the concierge system S is installed. Note that the surrounding environment may also include other elements.

[0028] In this embodiment, the sensor 111 is installed, for example, at the entrance of a store or at a sales floor, and measures data related to the environment around the installation location of the sensor 111. The sensor 111 is, for example, a temperature sensor, a humidity sensor, a barometric pressure sensor, an illuminance sensor, a human presence sensor, an ultrasonic sensor, or other various sensors. Note that the sensor 111 is not limited to the sensors described above. The various sensors 111 transmit measurement results to the device interface 110 as sensor data. The sensor data is, for example, numerical data indicating the measurement results of the sensor 111, such as temperature and humidity. Note that the sensor data output from the various sensors 111 may be analog data or digital data. Hereinafter, the sensor data related to the surrounding environment measured by the various sensors 111 will also be referred to as "environmental information."

[0029] The sensor 111 may be installed outside the store to measure data relating to the environment around the store where the concierge system S is installed.

[0030] The communication unit 112 is a communication interface such as a LAN I / F (Interface) and is connected to the network Na. The communication unit 112 transmits and receives various information to and from, for example, the text generator 2 via the network Na. Furthermore, the communication unit 112 can be connected to a network such as the Internet or other information processing devices under the control of the control unit 100. Note that if the sensor 111 has a communication function, the communication unit 112 may also function as the device interface 110.

[0031] The communication unit 112 may acquire environmental information indicating the environment around the store where the concierge system S is installed from a server (not shown) via a network such as the Internet. For example, the communication unit 112 may acquire data such as the temperature, humidity, and probability of precipitation in the area around the store where the concierge system S is installed from a server that manages data related to weather forecasts.

[0032] Next, a description will be given of the hardware configuration of the text generator 2. Fig. 3 is a block diagram showing an example of the hardware configuration of the text generator 2 according to the embodiment.

[0033] As shown in FIG. 3, the text generator 2 includes a CPU 201, which is an example of a processor, a ROM 202, a RAM 203, a memory unit 204, a communication unit 205, and the like.

[0034] The CPU 201 comprehensively controls each unit of the text generator 2. The ROM 202 stores various programs. The RAM 203 is a workspace where programs and various data are developed.

[0035] The memory unit 204 is a non-volatile memory such as a HDD or flash memory that retains stored information even when the power is turned off. The memory unit 204 includes a control program unit 2041, a label dictionary 2042, a text generation LLM 2043, and a question record DB 2044.

[0036] Control program section 2041 stores a control program for controlling text generator 2. CPU 201, ROM 202, RAM 203, and memory section 204 are connected to one another via bus 206. CPU 201, ROM 202, and RAM 203 constitute control section 200, which is a computer configuration. That is, control section 200 executes control processing for text generator 2, which will be described later, by CPU 201 operating in accordance with control program section 2041 stored in ROM 202 or memory section 204 and expanded in RAM 203.

[0037] The label dictionary 2042 is a database in which label text corresponding to each class of sensor data is registered for each type of sensor data. Fig. 4 is a diagram showing an example of the data configuration of the label dictionary 2042 according to the embodiment. As shown in Fig. 4, the label dictionary 2042 stores the type of sensor data, the class, and the label text in association with each other.

[0038] The type of sensor data may correspond to sensor data measured by one sensor 111, or may be a combination of sensor data measured by multiple sensors 111. In the example shown in Fig. 4, two types of sensor data are registered in the label dictionary 2042: temperature measured by a temperature sensor, and a combination of temperature measured by a temperature sensor and humidity measured by a humidity sensor. Note that the types of sensor data are not limited to the example shown in Fig. 4.

[0039] A class is a class in which sensor data is classified according to the value of the acquired sensor data. Here, classifying sensor data means associating input sensor data with one of multiple classes for each type of sensor data. Possible classification methods include a method of classifying numerical data acquired as sensor data using a threshold value, or a method of classifying numerical data acquired as sensor data using a machine learning trained model.

[0040] A different label text is associated with each class into which sensor data is classified. The label text is text containing an expression that qualitatively describes the state indicated by the sensor data. For example, in the example shown in FIG. 4, the temperature value measured by the temperature sensor is classified into one of classes 1 to 3, and the label texts "cool," "comfortable climate," and "warm" are associated with classes 1 to 3, respectively. In addition, the combination of the temperature measured by the temperature sensor and the humidity measured by the humidity sensor is classified into one of classes 1 to 6, and the label texts "freezing cold," "chilly," "comfortable climate," "humid," "dry heat," and "scorching hot" are associated with classes 1 to 6, respectively. In other words, the label text is text that describes environmental information corresponding to the sensor data value classified into each class.

[0041] Furthermore, the label text itself does not contain any numerical values, as the purpose of the label text is to convert numerical data into text.

[0042] 4, classes and label texts are associated one-to-one, but multiple label texts may be associated with one class. Also, in FIG. 4, label texts for different types of sensor data are registered in one table, but the label dictionary 2042 may be configured with multiple tables in which label texts for each type of sensor data are registered.

[0043] Returning to FIG. 3, the text generation LLM 2043 is a generation AI that generates text, such as a large language model (LLM). The text generation LLM 2043 receives a prompt input including a query sentence, and generates a response sentence corresponding to the query sentence. Note that, although an LLM is used as the generation AI in this embodiment, the generation AI is not limited to an LLM as long as it is capable of generating text.

[0044] The text generation LLM2043 is built using well-known deep learning technology and is functionally capable of outputting a response based on a prompt containing conditions such as an inquiry, etc. Here, the conditions may be, for example, conditions for providing product location information or suggesting recommended products.

[0045] The text generation LLM 2043 of this embodiment generates text based on label text that reflects environmental information and a query from the user, that is, generates a response sentence that takes environmental information into account in response to a prompt input.

[0046] The text generation LLM2043 may be fine-tuned to suit the needs of the store using the concierge system S. Fine-tuning may involve changing the content of responses to input prompts or changing the wording of output sentences. For example, the text generation LLM2043 used in this embodiment may have learned specific wording, such as the tone of voice or endings of characters in the store using the concierge system S.

[0047] The question record DB2044 is a database that manages records related to interactions between the interface device 1 and a user. Fig. 5 is a diagram showing an example of the data configuration of the question record DB2044 stored in the text generation device 2 according to an embodiment. As shown in Fig. 5, the question record DB2044 stores a question ID, a query statement, a response statement to the query statement, a class of sensor data related to the query statement, label text corresponding to the class, and a value of the sensor data, in association with each other. Hereinafter, the above-mentioned data set stored in the question record DB2044 will also be referred to as question record data.

[0048] The question ID is identification information that can uniquely identify a query statement based on the voice data input to the interface device 1 by the user.

[0049] The inquiry sentence is a text converted based on voice data including the inquiry content input by the user to the interface device 1. For example, in the example shown in Fig. 5, the inquiry sentence "What are today's recommended products?" corresponds to question ID "0001", and the inquiry sentence "Do you have product A in stock?" corresponds to question ID "0002".

[0050] The answer text is text indicating the answer content generated by the text generation LLM2043 in response to the prompt input. For example, in the example shown in Figure 5, the answer text "How about some cool and delicious ice cream?" corresponds to question ID "0001," and the answer text "Product A is out of stock. Instead, how about product B, which is perfect for the current weather?" corresponds to question ID "0002."

[0051] The sensor data classes and the label texts corresponding to the classes correspond to the classes and label texts in the label dictionary 2042 shown in Fig. 4. In Fig. 5, the temperature class is listed as an example of a sensor data class, and the temperature label text is listed as an example of a label text corresponding to the class. For example, in the example shown in Fig. 5, the sensor data classes "3" and "warm" are associated with question ID "0001," and the sensor data classes "2" and "comfortable climate" are associated with question ID "0002."

[0052] Here, the label text and class stored in the question record DB 2044 are the label text used when generating the prompt to be input to the text generation LLM 2043 and the class corresponding to that label text.

[0053] The sensor data value is a numerical value of the sensor data measured by the sensor 111. In Fig. 5, sensor data of temperature is shown as an example of the sensor data value.

[0054] The data stored in the question record DB 2044 is not limited to the above examples. For example, the data may include daily or monthly sales history of products that appear in inquiries and answers. In this way, by storing the sales history of products that appear in inquiries and answers, it becomes possible to analyze whether sales of products suggested to users of the concierge system S have changed.

[0055] Returning to FIG. 3 , the control unit 200 is connected to the communication unit 205 via a bus 206. The communication unit 205 is a communication interface such as a LAN I / F, and is connected to a network Na. The communication unit 205 transmits and receives various information to and from, for example, the interface device 1 via the network Na. Furthermore, under the control of the control unit 200, the communication unit 205 can be connected to a network such as the Internet or other information processing devices.

[0056] Next, a description will be given of the functional configuration of the interface device 1 and the text generator 2. Fig. 6 is a block diagram showing an example of the functional configuration of the interface device 1 and the text generator 2 according to the embodiment.

[0057] As shown in FIG. 6, the control unit 100 of the interface device 1 includes an input reception processing unit 1001, a sensor data acquisition unit 1002, a communication processing unit 1003, and an output control unit 1004 as functional components.

[0058] Specifically, the control unit 100 (CPU 101) of the interface device 1 realizes the above-mentioned functional configuration by executing the control program unit 1041 stored in the memory unit 104. Note that in this embodiment, the above-mentioned functional configuration is a software configuration realized by cooperation between the processor and program of the interface device 1, but it is not limited to this, and a hardware configuration in which part or all of the functional configuration is realized by a dedicated circuit or the like may also be used. Furthermore, the functional configuration of the interface device 1 is not limited to this.

[0059] The input reception processing unit 1001 outputs voice guidance when it detects a user. Specifically, when the input reception processing unit 1001 detects a user using the interface device 1 via the imaging unit 107, it outputs voice guidance via the speaker 108. Here, the voice guidance is audio information that prompts the user to operate the interface device 1. For example, if the interface device 1 is installed in a product sales floor, when the imaging unit 107 detects that a user has approached the interface device 1, the input reception processing unit 1001 outputs voice guidance such as "Welcome. Do you have the product you are looking for?" via the speaker 108.

[0060] The method for detecting the user is not particularly limited, and the user may be detected via a sensor 111 (for example, a human sensor, etc.). Furthermore, the input reception processing unit 1001 may output guidance to the user on the display unit 105 instead of outputting voice guidance.

[0061] Furthermore, the input reception processing unit 1001 receives input of voice data from the user. Specifically, the input reception processing unit 1001 receives input of voice data from the user via the microphone 109. For example, when the user utters a question (inquiry) such as "What are today's recommended products?" in response to the voice guidance, the input reception processing unit 1001 receives voice data representing the inquiry via the microphone 109.

[0062] The method of accepting questions from users is not particularly limited, and the input acceptance processing unit 1001 may display an input field on the display unit 105 where characters can be entered by touch operation, allowing the user to enter questions via the operation unit 106.

[0063] The sensor data acquisition unit 1002 acquires sensor data from various sensors 111. Specifically, the sensor data acquisition unit 1002 acquires sensor data from the sensors 111 via the device interface 110. Note that the sensor data acquisition unit 1002 may acquire sensor data from multiple sensors 111.

[0064] The communication processing unit 1003 transmits various data to the text generator 2. Specifically, the communication processing unit 1003 transmits the voice data received by the input reception processing unit 1001 and the sensor data acquired by the sensor data acquisition unit 1002 to the text generator 2.

[0065] The output control unit 1004 outputs voice data based on the answer sentence generated by the text generation LLM 2043. Specifically, when the output control unit 1004 receives voice data from the text generator 2, it outputs the voice data via the speaker 108.

[0066] Note that the output control unit 1004 may receive an answer sentence generated by the text generation LLM 2043 and display the answer sentence on the display unit 105, instead of receiving voice data and outputting the voice data via the speaker 108.

[0067] On the other hand, the control unit 200 of the text generator 2 has, as its functional configuration, a sensor data conversion processing unit 2001, a character conversion processing unit 2002, a prompt generation unit 2003, a text generation unit 2004, a voice data conversion processing unit 2005, a communication processing unit 2006, a memory control unit 2007, and an analysis processing unit 2008.

[0068] Specifically, the control unit 200 (CPU 201) of the text generator 2 realizes the above-mentioned functional configuration by executing the control program unit 2041 stored in the memory unit 204. In this embodiment, the above-mentioned functional configuration is a software configuration realized by cooperation between the processor and program of the text generator 2, but is not limited to this, and a hardware configuration in which part or all of the functional configuration is realized by a dedicated circuit or the like may also be used. Furthermore, the functional configuration of the text generator 2 is not limited to this.

[0069] The sensor data conversion processing unit 2001 of the text generator 2 is an example of a first conversion means. The sensor data conversion processing unit 2001 acquires sensor data transmitted from the interface device 1. Furthermore, the sensor data conversion processing unit 2001 converts the acquired sensor data into label text that qualitatively expresses the state indicated by the acquired sensor data, based on the label dictionary 2042.

[0070] Specifically, when the sensor data conversion processing unit 2001 receives sensor data from the interface device 1, it classifies the sensor data into one of the corresponding classes for each type of sensor data. For example, if the temperature measured by the temperature sensor is less than 15°C, it is classified into class 1, if it is 15°C or more but less than 25°C, it is classified into class 2, and if it is 25°C or more, it is classified into class 3. Then, the sensor data conversion processing unit 2001 refers to the label dictionary 2042 and obtains label text corresponding to the class into which the sensor data is classified for each type of sensor data.

[0071] The sensor data conversion processing unit 2001 may input various types of sensor data to a trained model that has been trained to classify the sensor data into classes corresponding to the sensor data, and obtain the classification results obtained by the trained model. The sensor data conversion processing unit 2001 may input various types of sensor data to a trained model that has been trained to determine thresholds for the sensor data, i.e., boundary values ​​for classifying the sensor data, and classify the sensor data based on the thresholds determined by the trained model. The label text registered in the label dictionary 2042 may also be output in advance by a trained model that has learned the relationship between temperature and the hot / cold sensation experienced by humans. The number of classes into which each piece of sensor data is classified is not limited to the above example.

[0072] The character conversion processing unit 2002 is an example of an acquisition unit and a second conversion unit. The character conversion processing unit 2002 acquires an inquiry sentence indicating the content of an inquiry entered by a user. Specifically, the character conversion processing unit 2002 acquires voice data transmitted from the interface device 1, that is, voice data including the content of the inquiry made by the user. The character conversion processing unit 2002 also converts the voice data including the content of the inquiry spoken by the user into text.

[0073] The character conversion processing unit 2002 converts the voice data received from the interface device 1 into text, and uses the converted text as a query statement describing the content of the user's inquiry. Note that the conversion from voice data to a query statement may use a learning model using a known voice recognition technology, or may use an algorithm that does not use machine learning.

[0074] The prompt generator 2003 is an example of a first generator. The prompt generator 2003 generates a prompt based on the label text and a query statement including the content of the user's query.

[0075] Specifically, the prompt generation unit 2003 generates a prompt by combining label text converted from sensor data by the sensor data conversion processing unit 2001 and a query sentence converted from voice data by the character conversion processing unit 2002.

[0076] For example, suppose that the sensor data value acquired from the temperature sensor is classified into class "3" by the sensor data conversion processing unit 2001. In the label dictionary 2042 shown in FIG. 4, the label text associated with the temperature class "3" is "warm." Also, suppose that the combination of the sensor data value acquired from the temperature sensor and the sensor data value acquired from the humidity sensor is classified into class "3" by the sensor data conversion processing unit 2001. In the label dictionary 2042 shown in FIG. 4, the label text associated with the class "3" of the combination of temperature and humidity is "comfortable climate."

[0077] Under the above conditions, the prompt generation unit 2003 combines the label text corresponding to each class classified by the sensor data conversion processing unit 2001 with the query sentence to generate a prompt such as, "It's warm and comfortable today. What are your recommended products for today?" By generating a prompt incorporating label text corresponding to sensor data in this way, it is expected that the text generation LLM 2043 will provide an answer appropriate to the user's current situation.

[0078] In other words, the prompt generation unit 2003 converts the sensor data, which is numerical data, into label text, which is text data containing quantitative expressions, before incorporating it into the prompt. This is because LLMs generally have difficulty handling numerical values ​​such as sensor data, and even if the numerical values ​​of the sensor data are incorporated into the prompt as is, there are cases in which an answer that appropriately corresponds to the meaning indicated by the numerical values ​​cannot be obtained.

[0079] The text generator 2004 is an example of a second generator. The text generator 2004 generates text including a response to the query by inputting the prompt generated by the prompt generator 2003 to the text generation LLM 2043.

[0080] For example, the text generation unit 2004 inputs a prompt such as "It's warm and comfortable today. What's your recommended product today?" to the text generation LLM 2043. Upon receiving the above prompt, the text generation LLM 2043 outputs a response that reflects the environment represented by the label text included in the input prompt, such as "How about some cool and delicious ice cream?"

[0081] In the above example, the answer sentence of the text generation LLM2043 does not simply answer "recommended products," but answers "recommended products" that are in line with the environmental conditions expressed by the label texts "warm" and "comfortable climate." Note that these answer sentences are only examples, and the content and wording of the answer sentences are not limited to these.

[0082] The voice data conversion processing unit 2005 is an example of a third conversion means. The voice data conversion processing unit 2005 converts text (answer sentence) including a reply to the inquiry sentence generated by the text generation unit 2004 into voice data.

[0083] The communication processing unit 2006 is an example of a providing means. The communication processing unit 2006 transmits the voice data converted from the answer sentence by the voice data conversion processing unit 2005 to the interface device 1.

[0084] The storage control unit 2007 is an example of a storage control means. The storage control unit 2007 stores data in which a query sentence, a response sentence, and sensor data and / or label text are associated with each other in a storage device. Specifically, the storage control unit 2007 stores the above-mentioned question record data in the question record DB 2044.

[0085] More specifically, each time the storage control unit 2007 receives a set of voice data and sensor data from the interface device 1, it stores a query statement generated from the voice data, a response statement to a prompt generated based on the query statement, and the value, class, and label text of the sensor data in association with each other.

[0086] In addition, when a user who has received the voice data sent to the interface device 1 utters a question (hereinafter also referred to as a follow-up question) in response to the received voice data, the storage control unit 2007 may convert the voice data including the follow-up question into text using the character conversion processing unit 2002, and store the text in the question record DB 2044 in association with the question record data corresponding to the answer sentence.

[0087] Furthermore, when a user repeatedly asks a follow-up question, the storage control unit 2007 may add a common identifier to a series of follow-up questions and store them in the question record DB 2044 in a state that makes it possible to identify that they are data relating to the same user.

[0088] The analysis processing unit 2008 performs statistical analysis based on various data stored in the question record DB 2044. Specifically, the analysis processing unit 2008 uses a known statistical analysis technique to statistically analyze the accuracy and validity of the answer sentences generated by the text generation LLM 2043 based on the question record data stored in the question record DB 2044.

[0089] The analysis processing unit 2008 may perform various analysis processes such as statistical analysis on a database that stores, for example, product sales results, in addition to the question record DB 2044. The analysis target and processing method of the analysis processing unit 2008 may be changed for each store that manages the question record DB 2044.

[0090] Next, the control process of the concierge system S will be described with reference to FIG.

[0091] 7 is a sequence diagram showing an example of control processing of the concierge system S according to the embodiment. The sequence diagram shown in FIG. 7 shows a processing example in which the text generator 2 receives sensor data and voice data transmitted from the interface device 1, then inputs a prompt generated from label text converted based on the sensor data and a query sentence converted based on the voice data to the text generator LLM 2043, and acquires a response sentence generated by the text generator LLM 2043. The sequence diagram shows a processing example in which the voice data converted based on the acquired response sentence is then transmitted to the interface device 1.

[0092] First, when a user using the interface device 1 is detected via the imaging unit 107, the input reception processing unit 1001 of the interface device 1 outputs voice guidance via the speaker 108 (step S101). Next, the input reception processing unit 1001 receives input of voice data from the user via the microphone 109 (step S102).

[0093] Next, the sensor data acquiring unit 1002 of the interface device 1 acquires sensor data from the sensor 111 via the device interface 110 (step S103). Next, the communication processing unit 1003 of the interface device 1 transmits the voice data accepted by the input acceptance processing unit 1001 and the sensor data acquired by the sensor data acquiring unit 1002 to the text generator 2 (step S104).

[0094] In the text generator 2, when the voice data and the sensor data are received from the interface device 1, the sensor data conversion processing unit 2001 classifies the sensor data into one of the corresponding classes for each type of sensor data. Next, the sensor data conversion processing unit 2001 refers to the label dictionary 2042 and acquires label text corresponding to the class into which the sensor data is classified for each type of sensor data (step S105).

[0095] Furthermore, the character conversion processing unit 2002 of the text generator 2 converts the voice data including the inquiry content received from the interface device 1 into text (query sentence) and acquires it (step S106). Next, the prompt generation unit 2003 of the text generator 2 generates a prompt by combining the label text converted from the sensor data by the sensor data conversion processing unit 2001 with the query sentence converted from the voice data by the character conversion processing unit 2002 (step S107).

[0096] Next, the text generator 2004 of the text generator 2 generates text (answer sentence) including an answer to the query sentence by inputting the prompt generated by the prompt generator 2003 into the text generation LLM 2043 (step S108). Next, the voice data conversion processor 2005 of the text generator 2 converts the answer sentence generated by the text generator 2004 into voice data (step S109). Next, the communication processor 2006 of the text generator 2 transmits the voice data converted by the voice data conversion processor 2005 to the interface device 1 (step S110).

[0097] When the output control unit 1004 of the interface device 1 receives the voice data from the text generator 2, it outputs the voice data via the speaker 108 (step S111).

[0098] Meanwhile, the storage control unit 2007 of the text generation device 2 associates the answer sentence generated in step S108 with the query sentence acquired in step S106, and stores the class of the sensor data of the various sensors 111 transmitted from the interface device 1, the label text corresponding to the class, and the value of the sensor data in the question record DB 2044 (step S112).

[0099] As described above, the text generator 2 of this embodiment converts voice data including the content of a user's inquiry into a query sentence, and inputs a prompt generated based on the query sentence and label text corresponding to sensor data related to the environment around the user (or store) into the text generation LLM 2043, thereby obtaining a response sentence from the text generation LLM 2043. This makes it possible to obtain a response sentence to the query sentence that reflects environmental information based on sensor data obtained from the various sensors 111. Therefore, information related to the environment and situation at the time of use by the user can be reflected in the response sentence.

[0100] Furthermore, according to the text generator 2 of this embodiment, the answer sentence, the class of sensor data from the various sensors 111, the label text corresponding to the class, and the value of the sensor data are stored in association with the query sentence in the question record DB 2044. This makes it possible to accumulate data for analyzing what kind of information the user wants under what kind of environment, what the user tends to want to know, etc., when the user uses the concierge system S.

[0101] The above-described embodiment can be modified as needed by partially changing the configuration or functions of each of the above-described devices. Therefore, several modifications of the above-described embodiment will be described below as other embodiments. The following mainly focuses on differences from the above-described embodiment, and detailed descriptions of commonalities with the content already described will be omitted. The modifications described below may be implemented individually or in appropriate combination.

[0102] (Variation 1) In the above-described embodiment, the label dictionary 2042 stores data for converting quantitative measurement values ​​indicated by the sensor data into qualitative values, but it is not limited to this and may be configured to store other data.

[0103] For example, the label dictionary 2042 may store data that allows classification of features expressed by a captured image of a user (hereinafter also referred to as an image) into information that represents predetermined attributes of the user. For example, the label dictionary 2042 may store the estimated age or gender of the user as label text.

[0104] In this case, when acquiring sensor data from the various sensors 111, the sensor data acquisition unit 1002 of the interface device 1 acquires, via the imaging unit 107, an image that allows identification of the user's features, such as the user's face and clothing.

[0105] Next, the communication processing unit 1003 of the interface device 1 transmits the image as part of the sensor data to the text generator 2. Furthermore, the sensor data conversion processing unit 2001 of the text generator 2 extracts information representing user attributes such as gender and age by extracting user features from the received image using known image processing technology or image recognition technology. Then, the sensor data conversion processing unit 2001 refers to the label dictionary 2042 and converts the extracted user attributes into label text.

[0106] In this way, a prompt that reflects the attributes of the user who uses the concierge system S can be generated, and therefore an answer sentence that reflects the user's attributes can be obtained.

[0107] (Variation 2) In the above embodiment, the query sentences stored in the question record DB 2044 are generated by converting voice data containing the content of the user's query into text by the character conversion processing unit 2002. However, the query sentences stored in the question record DB 2044 are not limited to this, and may be prompts, i.e., the above-mentioned text to which label text has been added.

[0108] The programs executed by the concierge system S of the embodiment and the modified example may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by the concierge system S of the embodiment and the modified example may be provided or distributed via a network such as the Internet.

[0109] The programs executed by each device in the above-described embodiments are provided in a state where they are pre-installed in a ROM, a storage unit, etc. The programs executed by each device in the above-described embodiments may be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a digital versatile disk (DVD).

[0110] These novel embodiments and their modifications may be embodied in various other forms, and various omissions, substitutions, changes, and combinations may be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the inventions, and are also included in the inventions described in the claims and their equivalents. [Explanation of symbols]

[0111] S Concierge System 1 Interface device 2. Text Generator 1001 Input reception processing unit 1002 Sensor data acquisition unit 1003 Communication processing unit 1004 Output control section 2001 Sensor data conversion processing unit 2002 Character conversion processing unit 2003 Prompt Generation Unit 2004 Text Generation Department 2005 Audio data conversion processing unit 2006 Communication Processing Unit 2007 Memory Control Unit 2008 Analysis Processing Unit [Prior art documents] [Patent documents]

[0112] [Patent Document 1] Patent No. 7252690

Claims

1. an acquisition means for acquiring an inquiry statement indicating the content of an inquiry entered by a user; a first conversion means for converting a state indicated by sensor data obtained from a sensor that senses the user's surrounding environment into a label text that qualitatively expresses the state; a first generating means for generating a prompt based on the query statement and the label text; a second generating means for generating a response sentence to the inquiry content by inputting the prompt to a generating AI; a storage control means for storing data in a storage device that associates the query sentence, the response sentence, the sensor data, and / or the label text; An information processing device comprising:

2. The system further includes a second conversion means for converting voice data including the inquiry content uttered by the user into text, the acquiring means acquires the text converted by the second converting means as the query sentence; The information processing device according to claim 1 .

3. a third conversion means for converting the answer sentence into voice data; providing means for providing the user with the voice data converted by the third conversion means; The information processing device according to claim 1 , further comprising:

4. The first conversion means converts the characteristics of the user shown in an image obtained from an imaging device that captures an image of the user into label text that qualitatively represents the characteristics of the user. The information processing device according to claim 1 .

5. the first conversion means converts the sensor data for each type based on a dictionary in which the label text corresponding to each of a plurality of classes set according to the type of sensor data is registered; The information processing device according to claim 1 .

6. The computer of the information processing device, an acquisition means for acquiring an inquiry statement indicating the content of an inquiry entered by a user; a first conversion means for converting a state indicated by sensor data obtained from a sensor that senses the user's surrounding environment into a label text that qualitatively expresses the state; a first generating means for generating a prompt based on the query statement and the label text; a second generating means for generating a response sentence to the inquiry content by inputting the prompt to a generating AI; a storage control means for storing data in a storage device that associates the query sentence, the response sentence, the sensor data, and / or the label text; A program to function as a

Images