Information processing device, terminal device, information processing program, information processing system, and information processing method

Abstract

The present invention provides an information processing device and the like capable of better using sensor information. An information processing device (210) is provided with an information conversion unit (23) for converting information output from a sensor (13) having acquired information related to an external environment of a user, into text information in a natural language. The information processing device (210) processes the information output from the sensor (13) by using the text information.

Description

Information processing device, terminal device, information processing program, information processing system, and information processing method

[0001] The present disclosure relates to an information processing device, a terminal device, an information processing program, an information processing system, and an information processing method.

[0002] In recent years, various proposals have been made regarding information processing techniques that utilize language models (for example, Patent Document 1).

[0003] Japanese Patent Application Publication No. 2023-73095

[0004] An object of one aspect of the present disclosure is to realize an information processing device or the like that can make better use of sensor information.

[0005] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure includes an information conversion unit that converts information output from a sensor that acquires information about the user's external environment into text information in natural language, and processes the information output from the sensor using the text information.

[0006] In order to solve the above problem, an information processing program according to one embodiment of the present disclosure causes a computer to execute the steps of converting information output from a sensor that acquires information about the user's external environment into text information in natural language, and processing the information output from the sensor using the text information.

[0007] In order to solve the above problem, an information processing system according to one embodiment of the present disclosure includes an information conversion unit that converts information output from a sensor that acquires information about the user's external environment into text information in natural language, and processes the information output from the sensor using the text information.

[0008] In order to solve the above problem, an information processing method according to one embodiment of the present disclosure includes a step of converting information output from a sensor that acquires information about a user's external environment into text information in natural language, and a step of processing the information output from the sensor using the text information.

[0009] According to one aspect of the present disclosure, sensor information can be better utilized.

[0010] Fig. 1 is a block diagram illustrating a configuration of an information processing system according to the present disclosure; Fig. 2 is a flowchart illustrating an example of overall processing (information processing method) in the information processing system according to the present disclosure; Fig. 3 is a flowchart illustrating another example of overall processing (information processing method) in the information processing system according to the present disclosure; Fig. 4 is a block diagram illustrating a configuration of another information processing system according to the present disclosure; Fig. 5 is a perspective view illustrating a terminal device according to the present disclosure.

[0011] First Embodiment Hereinafter, one embodiment of the present disclosure will be described in detail.

[0012] FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system 100 according to a first embodiment. The information processing system 100 is an AI (Artificial Intelligence) conversation system. As shown in FIG. 1 , the information processing system 100 includes a microphone 11, a camera 12, a sensor 13, a fingerprint sensor 14, an output device 15, and a storage unit 16. The information processing system 100 also includes a voice input unit 21, an image input unit 22, an information conversion unit 23, an authentication unit 25, a context analysis unit 26, a Large Language Model (LLM) determination unit 27, a simple response generation unit 28, a response control unit 29, a response output unit 30, a conversation history management unit 31, and an experience information generation unit 32. The information processing system 100 also includes a first LLM 51 and a second LLM 52. The information processing system 100 may further include a location sensor (not shown) to acquire experience information (described later). The information processing system 100 may also include a home appliance control unit 53.

[0013] The microphone 11 is a voice input device that accepts voice input from the user to the information processing system 100. The camera 12 is an imaging device that captures an image in accordance with a user's operation or in accordance with predetermined setting conditions.

[0014] The sensor 13 acquires information about the external environment of the user. As an example, the sensor 13 is installed in the user's living environment.

[0015] "Information about the user's external environment" refers to information about the user's surroundings (e.g., the user's residential environment, etc.), excluding information about the user himself / herself. Information about the user's external environment includes, for example, information about the physical environment (temperature, humidity, light, sound, air pressure, etc.), the geographical environment (the user's location or position, etc.), or the surrounding situation (the movements of other people, obstacles, etc.). The user's external environment includes the surrounding environment in which the user is currently located and the environment of the place where the user is usually located. Specifically, the user's external environment may include the environment of the user's residence or the environment of the user's car. The user's external environment may also include the work environment, in the sense that it includes an environment that the user recognizes as their own location and does not cause inconvenience to others.

[0016] The sensor 13 is, for example, an unlocking sensor attached to the front door, which can identify authorized users (family members).

[0017] Another example of the sensor 13 is a motion sensor installed at the entrance (front door) of the residence. The motion sensor can detect a person at the entrance (front door) of the residence. The motion sensor may be a voiceprint sensor based on microphone input, or a motion recognition sensor based on video analysis of camera input. The microphone and camera used in the motion sensor may be separate from the microphone 11 and camera 12 described above, and may be fixedly installed in a specific residential environment.

[0018] Another example of the sensor 13 is a gas sensor or a temperature sensor installed on a gas stove in a kitchen, in a living room, or in a bedroom. The gas sensor can measure the ambient air environment. The temperature sensor can measure the ambient air temperature.

[0019] The sensor 13 is not limited to the examples given here, and may be configured by combining a plurality of sensors.

[0020] The fingerprint sensor 14 is a fingerprint detection device that detects the user's fingerprint.

[0021] The output device 15 is a device that transmits information to the user. Information is transmitted to the user in response to a voice input by the user. The output device 15 is, for example, an image display device and a speaker. The image display device is a device that displays responses to the user's conversation, displays images captured by the camera 12, and notifies the user by display. The speaker is an audio output device that outputs audio responses to the user's conversation and notifies the user by voice. The output device 15 may also include a vibration device for emphasizing notifications to the user, a transmission device for emergency contact, etc.

[0022] The storage unit 16 stores information necessary for controlling the information processing system 100. The information processing system 100 may be communicably connected to an external storage device as the storage unit 16. That is, the storage unit 16 may be provided outside the information processing system 100.

[0023] The voice input unit 21 functions as a language input unit that accepts voice input (language input) from the user via the microphone 11. The voice input unit 21 converts the input voice into text data. Alternatively, the user may input language by text input. When inputting language by text input, a text input unit that accepts text as language input via a touch panel or a text input device such as a smartphone that is communicably connected to the context analysis unit 26 may be provided. In the following description, it is assumed that the user inputs language by voice.

[0024] The image input unit 22 acquires an image from the camera 12 and converts it into embedded data (text data). This conversion is performed to convert the image into a data format that can be understood by the first LLM 51 or the second LLM 52. That is, the image from the camera 12 may be converted into text data and incorporated into a conversation response request sent to the first LLM 51 or the second LLM 52. The image input unit 22 converts the image into text data that indicates a natural language that can be understood by the first LLM 51 or the second LLM 52.

[0025] There are several known methods for converting image data into text data. In the field of AI, the following known methods can be appropriately selected or combined for use.

[0026] For example, the conversion method may be Base64 encoding. Base64 encoding is a method for converting binary data into ASCII text and is widely used when handling binary data such as image files in text format. Base64 encoding is often used when embedding images as data URIs in applications, and can also be used in the information processing system 100.

[0027] The conversion method may also be hexadecimal encoding. Hexadecimal encoding is a method of converting binary data into a hexadecimal string. Hexadecimal encoding is generally used more often when visualizing binary data for debugging or data analysis than for images, and is rarely used in the information processing system 100. However, if the image provided is a CG image or, in particular, a mechanical design image, it is easy to capture the characteristics of the image and may be used depending on the application.

[0028] Alternatively, the conversion method may be URL encoding. URL encoding is often used in applications such as embedding images directly into HTML, and is widely used because it is easy to redisplay the converted image. In the information processing system 100, it is preferable that the LLM be an easy-to-use format, so it is not particularly selected as a conversion method for the chat system. However, URL encoding may be used in applications where it is important to redisplay the input image on an image display device.

[0029] Although it is not a direct conversion method, there is also a method called JSON encoding. This method encodes binary data using Base64 and stores the result in JSON format. Currently available open LLMs and closed LLMs that support many image modes support this method, so it can be used effectively in the information processing system 100.

[0030] In any case, the conversion method can be selected based on the learning method of the LLM to be used, as long as it is in a format that can be interpreted by the LLM that ultimately generates the response. Various encoding methods are well known, and conversion can be performed at the input stage of various LLMs. Therefore, the method can be selected taking into account resources, response time, and other qualities. The important thing is that these conversion methods convert image data into text data, thereby providing a method for the context analysis unit 26 and storage unit 16 to handle image input in the same way as normal conversational input.

[0031] The information conversion unit 23 converts information (sensor information) output from the sensor 13, which acquires information about the user's external environment, into text information in natural language. "Natural language" refers to a language that humans use on a daily basis, and includes various languages ​​such as Japanese, English, French, and German. The information conversion unit 23 outputs the converted text information to the context analysis unit 26.

[0032] Next, we will explain how to convert the information output from the sensor 13 (sensor information) into text information in natural language. Normally, the output from various sensors is simply an analog voltage or digital value, so the meaning of the output cannot be directly understood. This is also true in a conversation system using LLM, where the information must be converted into text information that the system can understand.

[0033] For example, converting the output of an infrared sensor into a temperature value that humans can understand is called "calibration" or "data conversion." These processes are procedures that convert the sensor's output into an accurate physical quantity (in this case, temperature). This process involves correcting or adjusting the sensor's output voltage or digital signal to convert it into an actual temperature.

[0034] In the information processing system 100, the data conversion portion is included in the processing by the information conversion unit 23. On the other hand, the portion corresponding to calibration is performed at the stage of creating the information processing system 100 or at the stage of first starting up the information processing system 100, and is stored as basic data for the information conversion unit 23. However, it goes without saying that the processing by the information conversion unit 23 of the information processing system 100 may include a calibration function.

[0035] Here, we will provide a basic explanation of everything from calibration to data conversion. Of course, specific operations may differ depending on the type of sensor. However, since the relationship between the sensor and physical quantity is well-known, it can be handled using well-known methods.

[0036] Accurately establishing the relationship between a sensor's output and the actual physical quantity (temperature) essentially involves the following processes: ・Measurement of a reference point: Measure the sensor's output in a known temperature environment to obtain a reference point. ・Determining the relationship: Determine the relationship between the measured output and the actual temperature (e.g., linear or non-linear). ・Adjustment: Adjust the sensor's output to match the actual temperature. This may mean changing the sensor's hardware or software settings. ・Verification: Verify that the sensor's output is accurate after calibration. This may involve re-measuring in a different temperature environment.

[0037] Data conversion uses calibration data to convert the sensor's output (e.g., analog voltage or digital value) into a human-understandable format (e.g., a temperature number). In other words, data conversion is the mapping of the sensor's output to a meaningful physical quantity. This mapping involves calculations and corrections based on the sensor's characteristics.

[0038] For example, in converting temperature from an infrared sensor, the infrared sensor measures the intensity of infrared light emitted from an object and outputs that data as a voltage or digital signal. Converting this output to temperature requires the following steps: ・Understanding the sensor's characteristics: Check the sensor's datasheet to understand the relationship between output and temperature. For example, check how the output voltage corresponds to temperature over a specific range. ・Obtaining reference data: Measure the sensor's output in a known temperature environment to obtain reference data. For example, record the output at a known temperature point, such as the freezing point or boiling point. ・Creating a conversion formula: Create a conversion formula to calculate temperature from the sensor's output. This can be done using techniques such as linear interpolation, nonlinear interpolation, or fitting a calibration curve. Then, using the sensor's output and conversion formula, you can convert the data into data such as "Temperature: 36.5°C."

[0039] In this way, by incorporating a calibration and data conversion system, it is possible to convert information output from the sensor 13 (sensor information) into text information in natural language. For example, the output of the sensor 13 (unlocking sensor) installed in the electronic lock at the front door is converted into text information such as "unlocked from outside by legitimate means." The information conversion unit 23 converts the sensor information into text information in natural language. The information processing device according to the present disclosure then processes the information output from the sensor 13 using the text information. Specifically, the context analysis unit 26 and the LLM determination unit 27, which are located downstream of the context analysis unit 26, etc., process the information output from the sensor 13 using the text information. This allows the sensor information to be treated in the same way as normal conversational input, thereby enabling better use of the sensor information.

[0040] As described above, the voice input unit 21 converts input voice into text data. The image input unit 22 acquires an image from the camera 12 and converts it into embedded data (text data). Therefore, the voice input unit 21 and the image input unit 22 may be understood to function as the information conversion unit 23.

[0041] In a typical conversation system, a conversation may be started triggered by a voice input. In contrast, the information processing system 100 may start a conversation triggered by an input from the image input unit 22 and / or the information conversion unit 23. When a conversation is started by some input, the information processing system 100 also considers other inputs, if any, as conversation inputs. However, the information processing system 100 may start a conversation with any input alone. Specifically, the information processing system 100 may start a conversation without waiting for a voice input from the user when the sensor output changes by a predetermined amount or more. The information processing system 100 may start a conversation triggered by detecting that the front door has been unlocked or by a change in room temperature.

[0042] The authentication unit 25 performs personal authentication of the user based on information from the fingerprint sensor 14. The authentication unit 25 functions as a sensor information input unit that acquires personal authentication information as sensor information from the fingerprint sensor 14. The authentication unit 25 compares the fingerprint acquired from the fingerprint sensor 14 with a pre-registered fingerprint of the user, and determines whether the acquired fingerprint is that of the registered user. If the acquired fingerprint is that of the registered user, the authentication unit 25 verifies the user's personal ID or the organization to which the user belongs, and grants access rights to necessary information.

[0043] The authentication unit 25 outputs the determination result to the context analysis unit 26. The context analysis unit 26 may change the personal information used for context analysis depending on whether the user is a registered user or a guest user.

[0044] 1 illustrates a fingerprint sensor 14 as the personal authentication sensor, but other personal authentication sensors such as an iris authentication sensor, a voiceprint authentication sensor, or a vein sensor may also be used. Personal authentication may also be performed using an image of the user's face or voice. For example, the authentication unit 25 may perform personal authentication of the user through iris authentication or voiceprint authentication. Of course, authentication may also be performed using multiple sensors.

[0045] The context analysis unit 26 performs a context analysis of a general conversation indicated by the user's language input. At this time, the context analysis unit 26 may perform the context analysis using a past conversation history and / or personal information of the user.

[0046] As described above, the information conversion unit 23 converts the sensor information into text information in natural language. The information conversion unit 23 outputs the text information to the context analysis unit 26. The context analysis unit 26 uses the text information to generate a request for generating a response (conversation response request / response request).

[0047] The context analysis unit 26 may perform context analysis of the language input based on sensor information converted into text information in natural language. The context analysis unit 26 may perform context analysis based on sensor information converted into text information in natural language, voice data converted into text data, image data converted into text data, conversation history, experience information, and / or a combination thereof.

[0048] The context analysis by the context analysis unit 26 may be, for example, an analysis using a small-scale language model that extracts keywords from the user's language input based on past history and organizes correlations between the keywords. Alternatively, the context analysis may include a process of determining attributes of the language input by analyzing keywords extracted from the user's language input using information from a database.

[0049] The attributes of the language input are simple tags corresponding to the content of the language input, such as a question, a greeting, an impression, a request for analysis, a request, or a knowledge field. The LLM determination unit 27 can select a response generation unit based on these tags.

[0050] If the user's authentication information (personal information) is present, the context analysis unit 26 may perform context analysis by using the user's own conversation history and, if necessary, the conversation history of a group (such as a family or business group). If the context does not depend on personal information, the previous history can be referenced. The conversation history stores the conversation user ID or conversation group ID and the conversation text itself, and the context analysis unit 26 may perform context analysis by using this information.

[0051] Furthermore, before generating a conversation response request, the context analysis unit 26 may determine whether to generate the request based on the user's authority. For example, if the question input by the user's voice is one for which the user does not have the authority to obtain an answer, the context analysis unit 26 may determine not to generate a conversation response request. Alternatively, the context analysis unit 26 may generate a request for generating a response such as "I cannot answer (with a reason depending on the situation)."

[0052] The context analysis unit 26 functions as a request generation unit that generates a conversation response request based on the sensor information converted into text information in natural language and / or the user's language input. The conversation response request may include the sensor information converted into text information in natural language and / or the user's language input, as well as the results of the context analysis, as well as image data converted into embedded data, conversation history or experience information, or a combination of these. The context analysis unit 26 outputs the generated conversation response request to the LLM determination unit 27.

[0053] The LLM determination unit 27 determines to which of the plurality of response generation units the conversation response request should be sent based on the content of the conversation response request. The response generation units will be described later. The LLM determination unit 27 sends the conversation response request to at least one of the plurality of response generation units. The LLM determination unit 27 may send the conversation response request to a plurality of response generation units. The context analysis unit 26 and the LLM determination unit 27 may be implemented as a single block.

[0054] The conversational response request generated by the context analysis unit 26 includes, for example, the user's voice input information such as "hello" and a tag called "greetings." On the other hand, the simple response generation unit 28 includes the "greetings" attribute as one of its attributes. The LLM determination unit 27 may select a response generation unit by matching the attribute tag associated with each response generation unit with the tag included in the conversational response request.

[0055] The first LLM 51, the second LLM 52, and the simple response generator 28 function as a response generator that generates a response to a conversational response request generated from sensor information converted into text information in natural language and / or a user's language input. The LLM determination unit 27 determines to which of the first LLM 51, the second LLM 52, and the simple response generator 28 the conversational response request should be sent based on the content of the conversational response request. In this disclosure, when there is no need to limit the type of response generator, the LLMs and the simple response generator are collectively referred to simply as the response generator. The number of LLMs is not limited to two and may be three or more. The LLM determination unit's determination allows multiple LLMs to be switched as needed to select the optimal LLM.

[0056] The simple response generator 28 may be a response generator that generates simple responses and is provided in a terminal that includes the LLM determination unit 27. Meanwhile, the first LLM 51 and the second LLM 52 may be response generators that generate complex responses and are provided in a server or the like external to the terminal that includes the LLM determination unit 27. Furthermore, the first LLM 51 and the second LLM 52 may be LLMs that differ in the generation of responses to conversational response requests. Specifically, the first LLM 51 and the second LLM 52 may be language models that differ in the type and amount of trained data, the maximum input size that can be processed, the response time, and / or the response accuracy.

[0057] The LLM determination unit 27 selects a response generation unit suitable for generating a response to the conversation response request according to the content of the conversation response request, and transmits the conversation response request. Furthermore, the LLM determination unit 27 may transmit a conversation response request including confidential information that should not be sent to an external server to the simple response generation unit 28 without transmitting the request to the external server.

[0058] Furthermore, for example, if the first LLM 51 is located on an internal server, the LLM determination unit 27 may select the first LLM 51 as the destination of a conversation response request containing confidential information that should not be sent to an external server. Each LLM may be tagged with an attribute indicating its location (affiliated organization) and / or its handling authority for confidential information. The LLM determination unit 27 may refer to the attribute tag of each response generator in addition to the content of the response request to select a response generator suitable for generating a response.

[0059] The response control unit 29 functions as a response control unit that receives a response generated in response to a conversation response request and controls the output of the response. Specifically, when there are multiple responses, the response control unit 29 may control the output order of the responses. Furthermore, when multiple responses can be integrated or summarized, the response control unit 29 may integrate or summarize the multiple responses. Furthermore, the response control unit 29 may format the response generated by the response generation unit so that it becomes a natural conversation in response to the user's voice input, or may generate a response message. Furthermore, the response control unit 29 may output the generated response message to the conversation history management unit 31. The response control unit 29 outputs the generated response message to the response output unit 30.

[0060] For example, when there is a response from the simple response generation unit 28 and a response from the LLM, the response control unit 29 may prioritize and output the response from the simple response generation unit 28 first. Also, for example, when the response from the LLM contains a tag that restricts speech or a tag such as a character that is not suitable for pronunciation or an attribute, the response control unit 29 may extract and format the speechable portion and output it to the response output unit 30.

[0061] In this way, the response control unit 29 controls the output of the response message based on the response generated by the response generation unit 29. Therefore, in the present disclosure, the response control unit 29 is also referred to as an output control unit.

[0062] Furthermore, if the response generated by the response generation unit does not require speech (for example, if the response consists only of home appliance control content, which will be described later), the response control unit 29 may not output the response to the response output unit 30. Furthermore, even if the response control unit 29 does not output the response to the response output unit 30, it may output the response to the conversation history management unit 31. Furthermore, the response control unit 29 may additionally output an instruction of "(do not speak)" to the response output unit 30.

[0063] The response output unit 30 outputs the response message generated by the response control unit 29 via the output device 15. Furthermore, if the context analysis unit 26 does not generate a conversational response request, or if the response message includes an instruction to "(do not speak)," the response output unit 30 may output a message indicating that it will not respond to the user's voice input.

[0064] The conversation history management unit 31 records and manages the authentication information as additional information in the conversation history in addition to the conversation text including the user's language input and the response to the language input. The additional information is managed by appropriate tagging 8. For example, if the additional information has not changed from the previous time or if the context analysis unit 26 does not require the additional information, the conversation history management unit 31 does not need to record the additional information in the conversation history.

[0065] For example, for information that anyone can access, the conversation history management unit 31 does not need to record authentication information in the conversation history. By appropriately managing the additional information in this way, the conversation history management unit 31 can appropriately manage the memory resources used for the conversation history without wasting them.

[0066] Note that past conversation history data becomes large as the usage time and frequency of the information processing system 100 increases. Therefore, storing all conversation history data is undesirable from the perspective of increasing memory resources and data processing time. Of course, memory capacity and memory access speeds are still improving year by year, and it is highly likely that in the future it will be possible to record virtually all conversations, if not all. However, at the time a system is implemented, available resources are limited, and it is desirable to be able to efficiently manage conversation history using limited resources. Therefore, the conversation history management unit 31 may have a function to maintain data stored in the storage unit at an appropriate size. There are several methods for maintaining an appropriate data size for conversation history, and any of these methods can be applied to the information processing system 100.

[0067] The simplest method is to set a limit on the data size of the conversation history in advance, and delete the oldest data when the limit is exceeded. This method is easy to implement and is effective in reducing the size. However, this method has the problem that because old information is automatically deleted, it is difficult to maintain consistency in the conversation, especially with old information. Therefore, this method is suitable for applications where consistency in the conversation is sufficient over a relatively short period of time, such as one day's worth of data.

[0068] Another method is to set an upper limit on the data size of the conversation history and use a separate LLM (which can be the same model) to summarize it and reduce it to a predetermined amount of data. This method removes meaningless conversation history through summarization, increasing the ratio of valid conversations in the record, and therefore makes it possible to keep important conversations, even if they are old, for a relatively long time.

[0069] Furthermore, a suitable method for the information processing system 100 is to use Retrieval-Augmented Generation (RAG). RAG is an approach that combines information retrieval and generative models to generate more accurate answers to user questions. Below, we will briefly explain the basic operation of RAG, from creating embedded data to the search method.

[0070] 1. Creating Embedded Data: First, the conversation history is divided into segments of a predetermined size or a predetermined period, such as one day's worth of data, and each segment is summarized. The summarized data is then converted into vectors (embedded data) using a known embedding model, such as a pre-trained model like BERT, RoBERTa, or Sentence-BERT. An appropriate index may then be constructed. Furthermore, image data can be separated during the summaries and indexes, and stored, for example, on a cloud or on other inexpensive, high-capacity media where the image data can be referenced by the index. This improves memory utilization efficiency and enables longer-term storage. Furthermore, by including a brief description of the separated images in the summaries, older images can be recalled from the conversation as needed.

[0071] 2. Processing of User Questions: A user's question is converted into vector data using the same model as the history data. Using methods such as inner product calculations, the top three most similar historical data are extracted and sent to the context analysis unit and answer generation unit, the LLM. Furthermore, if there is no data showing a similarity level above a predetermined threshold, it can be prevented from being sent. This prevents the generation of an incorrect answer due to being influenced by less relevant information. Therefore, no matter how large the history data, context analysis and answer generation can utilize historical data that is closely related to the question within a practical data size.

[0072] Furthermore, the conversation history management unit may delete the vectorized data itself in order to manage the size of the history data itself. As mentioned above, the deletion method may be to delete the oldest data first, or to appropriately re-summarize the data. Even in this case, important data can be retained for a much longer period of time than if non-vectorized information were deleted.

[0073] This method using RAG allows the contents of conversation history data to be retained relatively accurately and for a long period of time, making it particularly suitable for use in the information processing system 100. However, in order for RAG to function effectively, a certain limit on the number of histories and memory capacity is required. Therefore, a method using RAG or another method may be selected depending on the application and available resources. There are also several well-known methods for appropriately managing memory, and any of these may be used. Furthermore, multiple methods for managing history may be used in combination.

[0074] In either method, the conversation history is organized at an appropriate timing during breaks in the conversation. However, in order to maintain the consistency of the most recent conversation, it is preferable that the most recent conversation history, for example, a conversation history of about 10 turns, is neither compressed by summarization nor deleted.

[0075] The conversation history is recorded in the storage unit 16. The conversation history management unit 31 can access the storage unit 16 as necessary and pass the conversation history to the context analysis unit 26. Furthermore, the conversation history management unit 31 may include a description of the image generated by the response generation unit in the conversation history.

[0076] The experience information generation unit 32 generates experience information in which vital information of the user at a certain point in time is associated with information indicating the user's situation when the vital information at the certain point in time was acquired. The information indicating the user's situation may include location information, images, and time. The conversation history management unit 31 records and manages the experience information in the storage unit 16, including it in the conversation history.

[0077] The experience information generation unit 32 may store the generated experience information directly in the storage unit 16, and the experience information generation unit 32 may manage the experience information. In other words, the experience information generation unit 32 may function as an experience information management unit.

[0078] The experience information is, for example, information indicating when (time), where (location information), and what (image) the user saw. One piece of experience information may include information identifying a subject in an image or a description of the image.

[0079] The home appliance control unit 53 generates commands to control the home appliances based on instructions from the response generation unit that are based on the sensor information converted into text information in natural language, as well as the user's voice input (language input), conversation history, experience information, etc. The home appliances may be any home appliances, such as an air conditioner, a gas stove, a television, or lighting.

[0080] The home appliance control unit 53 extracts related tags from natural language including tagging such as "home appliance control, air conditioner, ON," and generates a command for controlling the home appliance. The home appliance control unit 53 may generate the command from the tag information based on a rule using a database, or may generate the command using a simple language model.

[0081] The generated command may be a call command for an API for controlling the ITO home appliance, or may be a code for a control remote control. The information processing system 100 may include an infrared output device and / or a Wi-Fi output device that transmits the code for the control remote control.

[0082] If the response generator (e.g., the first LLM 51) has attributes suitable for controlling a home appliance, the command may be generated by the first LLM 51. Also, a simple home appliance control command may be generated by the simple response generator 28.

[0083] That is, the home appliance control unit 53 may be a part of the response generation unit. Although not shown, the home appliance control unit 53 may receive a control instruction shaped by the response control unit 29 (output control unit) instead of receiving the output of the response generation unit.

[0084] For example, suppose that in a past conversation history, the user instructed, "It's hot today, so please turn on the air conditioner" when the temperature was 27°C. The information processing system 100 refers to the conversation history and, when the sensor 13 (temperature sensor) detects a temperature of 27°C, asks the user, "Do you want to turn on the air conditioner?" In response to this question, the user replies, "Turn on the air conditioner." The home appliance control unit 53 then generates a command to turn on the air conditioner according to the user's voice input (language input), and turns on the air conditioner.

[0085] Two specific examples will be described in more detail below.

[0086] (Specific Example 1) It is assumed that a person is in a room and is talking to the information processing system 100, the room temperature rises, and the sensor 13 (temperature sensor) detects 27°C.

[0087] The information conversion unit 23 converts the information output from the sensor 13 (temperature sensor) into text information in natural language, and outputs the text information to the context analysis unit 26 .

[0088] When there is no input from the voice input unit, the context analysis unit 26 starts context analysis using the information "Temperature: 27°C" from the information conversion unit 23. The context analysis unit 26 searches the conversation history management unit 31 for a temperature of 27°C and finds a conversation such as "It's hot today, so please turn on the air conditioner." The context analysis unit 26 has also confirmed that there is no information that says "hot" for a temperature of 26°C. The context analysis unit 26 generates a conversation response request "Temperature 27°C, tag (air conditioner control), past conversation history (history IDs related to 27°C and 26°C)" and outputs it to the LLM determination unit 27.

[0089] The LLM determination unit 27 selects a response generation unit (for example, the first LLM 51 ) suitable for controlling the home appliance, and outputs the conversation response request and the past conversation history to the first LLM 51 .

[0090] The first LLM 51 refers to the input history and determines that 27°C is not definitive, and generates a response such as, "It's getting a little hot, isn't it? Shall I turn on the air conditioner?" This response is presented to the user via the output device 15.

[0091] Next, when the user inputs "Turn on the air conditioner" via the voice input unit 21, the context analysis unit 26 combines this input with the previous conversational response request to generate a new conversational response request. The first LLM 51, selected as before, determines that the instruction is definitive and generates a response including an air conditioner control instruction, such as "(Air conditioner control: Cooling ON) Got it? I'll cool the room."

[0092] The home appliance control unit 53 turns on the air conditioner in accordance with this air conditioner control instruction. The output device 15 outputs "I understand. I will cool the room," which has been formatted by the response output unit 30. In other words, "(air conditioner control: cooling ON)" is an air conditioner control instruction, and since this air conditioner control instruction does not need to be presented to the user, it is deleted. This process corresponds to "formatting."

[0093] (Specific Example 2) It is assumed that a person is in a room and is talking to the information processing system 100, the room temperature rises, and the sensor 13 (temperature sensor) detects 27°C.

[0094] The information conversion unit 23 converts the information output from the sensor 13 (temperature sensor) into text information in natural language, and outputs the text information to the context analysis unit 26 .

[0095] When there is no input from the voice input unit, the context analysis unit 26 starts context analysis using the information "temperature: 27°C" from the information conversion unit 23. Furthermore, the context analysis unit 26 searches the conversation history management unit 31 for a temperature of 27°C and finds multiple conversations about turning on the air conditioner when the temperature is 27°C. The context analysis unit 26 generates a conversation response request "temperature 27°C, tag (air conditioner control), past conversation history (history ID related to 27°C)" and outputs them to the LLM determination unit 27.

[0096] The LLM determination unit 27 selects a response generation unit (for example, the first LLM 51 ) suitable for controlling the home appliance, and outputs the conversation response request and the past conversation history to the first LLM 51 .

[0097] The first LLM 51 refers to the past history that has been input and determines that 27°C is definitely hot enough for the residents to feel, and generates a response that includes an air conditioner control instruction, such as "(Air conditioner control: Cooling ON) It's hot today. Let's turn on the air conditioner."

[0098] Next, the home appliance control unit 53 turns on the air conditioner in accordance with this air conditioner control instruction. The output device 15 outputs "It's hot today, isn't it? I'll turn on the air conditioner." formatted by the response output unit 30. Alternatively, when the response of the first LLM 51 does not include speech information, such as "(air conditioner control: cooling ON)," the home appliance control unit 53 may automatically turn on the air conditioner without any output through the output device 15.

[0099] In this way, the information processing system 100 repeatedly engages in conversation with the user, associates the sensor output with the conversation history, and stores and references it. This allows the information processing system 100 to provide a conversation system capable of spontaneously and proactively controlling home appliances to automatically realize a situation that is favorable to the user.

[0100] As described above, the home appliance control unit 53 controls the home appliance based on the instruction generated by the response generation unit. That is, a conversational response request not only causes the response generation unit to generate a conversational response (in other words, an "utterance to the user"), but also may cause the response generation unit to generate a response for controlling the home appliance. Therefore, the response generated by the response generation unit includes a conversational response and a response for controlling the home appliance. Therefore, in the present disclosure, a "conversational response request" is also referred to as a "response request." Furthermore, as described above, the home appliance control unit 53 controls the output of a command for controlling the home appliance based on the response generated by the response generation unit, and therefore, in the present disclosure, the home appliance control unit 53 is also referred to as an output control unit.

[0101] The information processing system 100 may have an API (Application Program Interface) calling function for controlling home appliances (e.g., IoT home appliances). The information processing system 100 may have a function for generating remote control commands for controlling the home appliances. The information processing system 100 may have an infrared transmitter for outputting the remote control commands.

[0102] 2 is a flowchart showing an example of the overall processing (information processing method) in the information processing system 100. In the information processing system 100, the voice input unit 21 accepts voice input from a user (S11). When the voice input unit 21 accepts the voice input, the context analysis unit 26 acquires the user's conversation history (S12) and analyzes the context of the input voice using the conversation history (S13). The context analysis unit 26 generates a conversation response request based on the context analysis result (S14).

[0103] Based on the content of the conversation response request, the LLM determination unit 27 selects a response generation unit and transmits the conversation response request (S15). The response generation unit selected by the LLM determination unit 27 generates a response to the conversation response request (S16). Based on the content of the response generated by the response generation unit, the response control unit 29 generates a response message to be output to the user (S17). The conversation history management unit 31 updates the user's conversation history with the response message generated by the response control unit 29 (S18). The response output unit 30 outputs the response message to the user via the output device 15 (S19). Note that the conversation history update (S18) may be performed after the response message output (S19).

[0104] (Another Example of Processing Flow) FIG. 3 is a flowchart showing another example of the overall processing (information processing method) in the information processing system 100. In the information processing system 100, the sensor 13 acquires information (sensor information) related to the user's external environment (S21). Next, the information conversion unit 23 converts the sensor information into text information in natural language (S22) and outputs the text information to the context analysis unit 26. Next, the context analysis unit 26 acquires the user's conversation history (S23) and analyzes the context of the input text information using the conversation history (S24). Thereafter, the context analysis unit 26 generates a request (conversational response request) requesting the generation of a response using the text information (S25). Next, based on the content of the conversational response request, the LLM determination unit 27 selects a response generation unit and transmits the conversational response request (S26). The response generation unit selected by the LLM determination unit 27 generates a response to the conversational response request (S27). Based on the content of the response generated by the response generation unit, the response control unit 29 generates a response message to be output to the user (S28). The conversation history management unit 31 updates the user's conversation history with the response message generated by the response control unit 29 (S29). The response output unit 30 outputs the response message to the user via the output device 15 (S30). Note that the updating of the conversation history (S29) may be performed after the output of the response message (S30).

[0105] (Example) Hereinafter, an example in which the information processing system 100 is installed in the user's residential environment will be described.

[0106] As an example, when a family member returns home, the information conversion unit 23 acquires sensor information from a camera capturing an image of the entrance, a microphone capturing the sound (voice) of the person at the entrance, and / or an unlocking sensor attached to the entrance door. The information conversion unit 23 converts the sensor information into text information in natural language and outputs the converted text information to the context analysis unit 26. As a result, the information processing system 100 begins a conversation based on the text information, saying kind words like "Welcome home."

[0107] This process will be explained in more detail below. First, assume that a family member has returned home and is about to enter through the front door. At this time, sensor 13 (front door camera) detects that a person is in front of the front door. Sensor 13 (unlocking sensor) also detects that the front door has been unlocked by a legitimate operation from outside. Information conversion unit 23 converts each sensor information into text information in natural language, and outputs the converted text information to context analysis unit 26.

[0108] Based on the input text information, the context analysis unit 26 determines that a family member (at least not a suspicious person) has returned home. Furthermore, the context analysis unit 26 confirms, based on past conversation history, that the user often utters "I'm home" in similar cases. Therefore, the context analysis unit 26 generates a conversational response request "(greeting) I'm home" and outputs it to the LLM determination unit 27. Because the greeting tag is attached, the LLM determination unit 27 selects the simple response generation unit 28 and transmits the conversational response request "(greeting) I'm home" to the simple response generation unit 28. The simple response generation unit 28 generates "Welcome home" as a response to the conversational response request "(greeting) I'm home." "Welcome home" is output via the response control unit 29, the response output unit 30, and the output device 15.

[0109] In this way, when the user returns home, the information processing system 100 can grasp the situation of the user's return home and start a friendly conversation with the user without the user even having to say anything to him or her.

[0110] The context analysis unit 26 may transmit a conversation response request including the past conversation history to the LLM determination unit 27. By referring to the past conversation history, the information processing system 100 can provide a conversation that is more attuned to the user. Furthermore, the more conversation history is accumulated, the more attuned the conversation can be to the user.

[0111] As shown in this example, the conversation may be initiated by the information processing system 100 rather than by a family member. That is, the information processing system 100 can detect a change in sensor information and generate a conversational response even when there is no user voice input. Alternatively, the information processing system 100 may detect a change in sensor information, generate a pseudo-user voice input, and generate a conversational response. Furthermore, when the information processing system 100 starts a conversation, when it determines that a family member has returned home, it may set a utterance from the family member such as "I'm home," and generate a conversation.

[0112] That is, suppose that the context analysis unit 26 receives text information (e.g., "A family member has returned {entrance, someone is there, door, successfully unlocked}") based on sensor information capturing a family member returning home from the information conversion unit 23. At this time, the context analysis unit 26 may convert the text information into a linguistic input such as "User, I'm home." Then, the context analysis unit 26 may generate a conversation response request based on the linguistic input. As a result, the information processing system 100 may utter "Welcome home" and start a conversation with the user.

[0113] In this way, the context analysis unit 26 may perform processing by replacing sensor information (information that detects that the user has returned home) with user behavior (utterance such as "I'm home"). If a conversational response request is generated based on the sensor information itself, it may take time for the response generation unit to generate a response. In contrast, if a conversational response request is generated after replacing the sensor information with user behavior, a simple response can be generated, the time required for the response generation unit to generate a response can be reduced, and a conversation can be held in a timely manner.

[0114] The above configuration can also be implemented using simple sensors. For example, by defining a rule that converts the touch of an authorized person (family member) on an electronic lock into "I'm home," the above configuration can be realized simply by the electronic lock responding. In this way, by using purpose-specific sensors, the information processing system 100 can be incorporated into a low-resource conversation system.

[0115] As another example, suppose that the sensor 13 detects a person entering the premises of a house. If it is determined that the person is not a family member, the utterance content is determined to be "Someone has come. I don't know anyone I've seen before." The response control unit 29 outputs the utterance content to the family member in the house via the response output unit 30 and the output device 15.

[0116] As another example, suppose the sensor 13 detects the presence of a person other than a household member at the entrance, and further detects that the visitor does nothing or performs a general action such as ringing the doorbell. Furthermore, suppose the sensor 13 also detects that no household member is present in the home. In this case, the information processing system 100 does not need to generate a particular conversation, so the sensor 13 stores the fact that a stranger has visited and the image of the visitor captured at that time in the conversation history. In this process, no particular speech is required. However, if a part of the output device 15 (e.g., a speaker) is installed at the entrance, the information processing system 100 may explicitly generate an instruction to stop speaking.

[0117] As another example, suppose the sensor 13 detects the presence of a non-resident at the front door and further detects that a visitor is attempting to open the front door or is accessing the front door lock by unauthorized means. Furthermore, suppose the sensor 13 also detects that no resident is present in the home. In this case, the information processing system 100 may store the sensor history and video associated with the visitor's actions in the conversation history.

[0118] The information processing system 100 may generate a response request to generate a message to warn the visitor, or may instruct the visitor not to speak. Although not shown, the information processing system 100 may issue a notification to the outside using a communication function provided in the information processing system 100, or may operate a warning appliance (such as a loud bell or alarm). The information processing system 100 may be configured in advance with an operation (such as warning, monitoring, or notification) for an irregular event. The information processing system 100 may have a conversation with a family member regarding the suspicious visitor and store a response policy as a conversation history.

[0119] As another example, suppose the sensor 13 detects that the temperature of the gas stove is rising and that someone is in the house. At this time, the information processing system 100 determines the content of the conversation, such as "It's dangerous if you leave it like this," "Is the water in the pot getting low?", or "It's boiling. Let's turn down the heat." In this way, the information processing system 100 can also predict future dangers by utilizing AI (LLM1, LLM2, etc.). In other words, the information processing system 100 can also realize conversations that link the AI ​​system and sensors. In this way, the information processing system 100 can realize conversations that are close to the user by linking with sensors.

[0120] As yet another example, assume that a person is in a room talking to the information processing system 100, the room temperature rises, and the sensor 13 (temperature sensor) detects 27°C.

[0121] The information conversion unit 23 converts the information output from the sensor 13 (temperature sensor) into text information in natural language, and outputs the text information to the context analysis unit 26 .

[0122] When there is no input from the voice input unit, the context analysis unit 26 starts context analysis using the information "Temperature: 27°C" from the information conversion unit 23. Furthermore, the context analysis unit 26 searches the conversation history management unit 31 for a temperature of 27°C and finds a conversation such as "It's hot today, so please turn on the air conditioner." The context analysis unit 26 also confirms that there is no information that says "hot" for a temperature of 26°C. The context analysis unit 26 generates a conversation response request "Temperature 27°C, tag (air conditioner control), past conversation history (history IDs related to 27°C and 26°C)" and outputs it to the LLM determination unit 27. The subsequent processing is as described above.

[0123] Here, if the current temperature detected by the sensor 13 is 30°C, the information processing system 100 may decide to turn on the air conditioner without asking, "Do you want to turn on the air conditioner?" This is significantly different from a conversation in which the user is simply asked, "Do you want to turn on the air conditioner?" when the temperature reaches 27°C. For example, if the conversation history records that the air conditioner was turned on at 25°C, the information processing system 100 will turn on the air conditioner without asking the user, even if the current temperature is the same 27°C. By accumulating such conversation history, the information processing system 100 can realize conversations and home appliance control that are even more attuned to the user.

[0124] The conversation history may include not only the voice conversation between the user and the information processing system 100, but also the sensor information converted into text. This allows the time of speech, the content of speech, the speaker, and the sensor information at that time to be registered as a series of text and available as part of the conversation or as a conversation history. The sensor information may include information about the location and state when the input from the sensor changed, allowing a history linking the processing performed at each location and state to be recorded. By recording the conversation history and sensor information together in this manner, the information processing system 100 can provide the user with a more personalized conversation based on past responses and / or the context of the current conversation. Using the example above, the information processing system 100 can gradually determine whether the current temperature is hot for the user.

[0125] In this way, the information processing system 100 can realize a conversation that is situation-appropriate and attuned to the user by collecting not only conversation context but also sensor information as appropriate and incorporating the sensor information as part of the conversation.

[0126] [Embodiment 2] Another embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0127] 4 is a block diagram illustrating the configuration of an information processing system 200 according to embodiment 2. As shown in FIG. 4, the information processing system 200 includes an information processing device 210 and a server 220.

[0128] The information processing device 210 is a conversation terminal that includes elements other than the first LLM 51 and the second LLM 52 in the information processing system 100. In particular, the information processing device 210 integrates the microphone 11, camera 12, sensor 13, and fingerprint sensor 14 into a single device. This allows accurate information about the user to be acquired with a high degree of certainty. Furthermore, integrating these input devices reduces the possibility of the user's personal information being leaked.

[0129] The server 220 includes the first LLM 51 and the second LLM 52 in the information processing system 100. A publicly available server or a secure server managed by an individual or organization may be used as the server 220. In FIG. 4 , a single server 220 includes the first LLM 51 and the second LLM 52. However, in the information processing system 200, the server including the first LLM 51 and the server including the second LLM 52 may be separate servers.

[0130] If the server 220 manages multiple LLMs that exist inside or outside the server 220, the server 220 may have a function to call a publicly accessible server from among the LLMs it manages. For example, if the first LLM 51 is located inside the server 220 and the second LLM 52 is located on a different server, the server 220 may use the second LLM 52 via the different server.

[0131] The server 220 may have a function for switching the LLMs it manages. In this case, the server 220 may manage attribute tags of the LLMs it manages and share the attribute information with the LLM determination unit 27. This allows the LLM determination unit 27 to obtain information for selecting an appropriate LLM from currently available LLMs. The server 220 may have information on switchable LLMs and servers containing LLMs, or may be configured to obtain information from a separate database.

[0132] In the example shown in FIG. 4 , the voice input unit 21, the image input unit 22, the information conversion unit 23, the authentication unit 25, the context analysis unit 26, the LLM determination unit 27, the simple response generation unit 28, the response control unit 29, the response output unit 30, the conversation history management unit 31, the experience information generation unit 32, and the home appliance control unit 53 are all present on the same information processing device 210. These units may be integrated on a chip as the information processing device 210. Alternatively, these units may be distributed and located on a secure server or the like. The information processing system 200 may be realized as a program executed by computers provided in the information processing device 210 and the server 220.

[0133] 5 is a perspective view illustrating the terminal device 230. The terminal device 230 is a portable electronic device equipped with the information processing device 210.

[0134] As shown in Fig. 5, the terminal device 230 includes a mounting unit 235. The mounting unit 235 is a member for mounting the terminal device 230 around the user's neck. The mounting unit 235 has an open-ring shape that can be hooked around the user's neck. With this configuration, the user's movements are less likely to be hindered even when the terminal device 230 is mounted.

[0135] The attachment unit 235 has a microphone 11. Specifically, the microphone 11 is disposed at one end of the attachment unit 235, which has an open ring shape. This position is near the user's mouth when the user wears the terminal device 230 around their neck. Therefore, the user can easily input voice via the microphone 11 while wearing the terminal device 230 around their neck.

[0136] The mounting unit 235 has a camera 12. Specifically, the camera 12 is arranged near the end of the open-ring-shaped mounting unit 235 so as to face outward. This allows the orientation of the camera 12 to roughly match the orientation of the user's face when the user is wearing the terminal device 230 around their neck. This makes it possible for the camera 12 to capture what the user sees.

[0137] The attachment part 235 has a fingerprint sensor 14. Specifically, the fingerprint sensor 14 is disposed at the end of the open-ring-shaped attachment part 235 opposite to the end where the microphone 11 is disposed. This position allows the user to easily touch the fingerprint sensor 14 with their finger when the user is wearing the terminal device 230 around their neck. Therefore, the user can easily perform personal authentication.

[0138] The wearing unit 235 has an output device 15. Specifically, speakers serving as the output device 15 are arranged on each of the left and right sides of the wearing unit 235, which has an open ring shape, when the open part is in the front. These positions are near the user's ears when the user wears the terminal device 230 around their neck. The user can easily hear the output from the output device 15 when wearing the terminal device 230 around their neck.

[0139] Alternatively, a user's instruction regarding the operation of the terminal device 230 may be input to the terminal device 230 via another terminal device, such as a smartphone, that transmits the user's instruction regarding the operation of the terminal device 230 to the terminal device 230 .

[0140] The information processing system according to the present disclosure may be portable outside the home, or may be realized in a manner that allows switching between an edge terminal that can be portable outside the home and a fixed terminal that is fixed within the home. The user's conversation history outside the home and the user's conversation history within the home may be shared with each other. This allows the user's conversations or actions outside the home to be utilized in conversations within the home, enabling conversations that are more in line with the user's needs.

[0141] [Example of implementation by software] The functions of the information processing device (hereinafter referred to as the "device") can be realized by an information processing program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (in particular, the voice input unit 21, the image input unit 22, the information conversion unit 23, the authentication unit 25, the context analysis unit 26, the LLM determination unit 27, the simple response generation unit 28, the response control unit 29, the response output unit 30, the conversation history management unit 31, and the experience information generation unit 32).

[0142] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and storage device.

[0143] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0144] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0145] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0146] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

[0147] [Cross-reference to related applications] This application claims the benefit of priority to Japanese Patent Application No. 2024-151623, filed on September 3, 2024, the entire contents of which are incorporated herein by reference.

[0148] DESCRIPTION OF SYMBOLS 11 Microphone 12 Camera 13 Sensor 14 Fingerprint sensor 15 Output device 16 Memory unit 21 Voice input unit 22 Image input unit 23 Information conversion unit 25 Authentication unit 26 Context analysis unit (request generation unit) 27 LLM determination unit 28 Simple response generation unit (response generation unit) 29 Response control unit (output control unit) 30 Response output unit 31 Conversation history management unit 32 Experience information generation unit 51 First LLM (response generation unit) 52 Second LLM (response generation unit) 53 Home appliance control unit (response generation unit) 100, 200 Information processing system 210 Information processing device 220 Server 230 Terminal device 235 Wearing unit

Claims

1. An information processing device that includes an information conversion unit that converts information output from a sensor that acquires information about the user's external environment into text information in natural language, and processes the information output from the sensor using the text information.

2. An information processing device as described in claim 1, comprising: a request generation unit that generates a request for generating a response using the text information; and an output control unit that receives the response generated in response to the request and controls the output of the response.

3. An information processing device according to claim 2, further comprising a conversation history management unit that manages a conversation history with the user, wherein the request generation unit generates the request using the text information and the conversation history.

4. The information processing device according to claim 3, wherein the conversation history management unit manages the conversation history by including the text information in the conversation history.

5. The information processing device according to any one of claims 2 to 4, wherein the output control unit speaks to the user based on the response.

6. The information processing device according to any one of claims 2 to 5, wherein the output control unit controls a home appliance based on the response.

7. The information processing device according to any one of claims 1 to 6, wherein the sensor is installed in the user's living environment.

8. A terminal device comprising the information processing device according to any one of claims 1 to 7.

9. A computer-executable information processing program that causes a computer to execute the following steps: converting information output from a sensor that acquires information about the user's external environment into text information in natural language; and processing the information output from the sensor using the text information.

10. An information processing system comprising an information conversion unit that converts information output from a sensor that acquires information about the user's external environment into text information in natural language, and processes the information output from the sensor using the text information.

11. An information processing method including the steps of: converting information output from a sensor that acquires information about the user's external environment into text information in natural language; and processing the information output from the sensor using the text information.

Images