Information processing device, terminal device, information processing system, and information processing method

Abstract

The present invention improves quality of conversation with a user. An information processing system (100) comprises: a sensor information input unit that acquires sensor information; a language input unit that accepts language input from a user; a request generation unit that generates a conversation response request on the basis of the sensor information and the language input; a response generation unit that generates a response in response to the conversation response request; and a response control unit that receives the response generated in response to the conversation response request and performs output control of the response.

Description

Information processing device, terminal device, information processing system, and information processing method

[0001] The present disclosure relates to an information processing device that responds to a language input from a user, and a terminal device that includes the information processing device.

[0002] Patent Literature 1 discloses a mobile communication terminal including a display and a camera. The mobile communication terminal is configured to execute a health detection function in response to a user's input received at the mobile communication terminal while the display is off. The health detection function detects the user's eyes with the camera and acquires health information from the detected eye characteristics. Furthermore, the mobile communication terminal transmits the acquired health information to a predetermined server.

[0003] International Publication No. WO2013 / 058533

[0004] In a conversation system, it is desirable to improve the quality of conversation with a user.

[0005] An object of one aspect of the present disclosure is to realize an information processing device that can improve the quality of conversation with a user.

[0006] In order to solve the above problem, an information processing device according to one embodiment of the present disclosure includes a sensor information input unit that acquires sensor information, a language input unit that accepts language input from a user, a request generation unit that generates a conversational response request based on the sensor information and the language input, and a response control unit that receives a response generated in response to the conversational response request and controls the output of the response.

[0007] An information processing system according to one aspect of the present disclosure includes a sensor information input unit that acquires sensor information, a language input unit that accepts language input from a user, a request generation unit that generates a conversational response request based on the sensor information and the language input, a response generation unit that generates a response in response to the conversational response request, and a response control unit that receives the response generated in response to the conversational response request and controls the output of the response.

[0008] An information processing method according to one aspect of the present disclosure includes a step of acquiring sensor information, a step of accepting language input from a user, a step of generating a conversational response request based on the sensor information and the language input, a step of receiving a response generated in response to the conversational response request, and a step of controlling the output of the response.

[0009] According to one aspect of the present disclosure, the quality of conversation with a user can be improved.

[0010] FIG. 1 is a block diagram illustrating a configuration of an information processing system according to a first embodiment. FIG. 2 is a flowchart illustrating an example of overall processing in the information processing system according to the first embodiment. FIG. 3 is a flowchart illustrating an example of processing for acquiring vital information in the information processing system. FIG. 4 is a flowchart illustrating another example of overall processing in the information processing system according to the first embodiment. FIG. 5 is a flowchart illustrating an example of processing for acquiring vital information and experience information in the information processing system. FIG. 6 is a block diagram illustrating a configuration of an information processing system according to a second embodiment. FIG. 7 is a perspective view illustrating a terminal device according to the second embodiment.

[0011] First Embodiment Hereinafter, one embodiment of the present disclosure will be described in detail.

[0012] FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system 100 according to a first embodiment. The information processing system 100 is an AI (Artificial Intelligence) conversation system. As illustrated in FIG. 1 , the information processing system 100 includes a microphone 11, a camera 12, a vital sign sensor 13, a fingerprint sensor 14, an output device 15, and a storage unit 16. The information processing system 100 also includes a voice input unit 21 (language input unit), an image input unit 22, a vital sign information input unit 23 (sensor information input unit), a vital sign information analysis unit 24, an authentication unit 25 (sensor information input unit), a context analysis unit 26 (request generation unit), an LLM (Large Language Model) determination unit 27 (destination determination unit), a simple response generation unit 28 (response generation unit), a response control unit 29, a response output unit 30, a conversation history management unit 31 (experience information management unit, determination result correction unit, database correction unit, past information management unit), and an experience information generation unit 32 (experience information management unit, past information management unit). Furthermore, the information processing system 100 includes a first LLM 51 (response generation unit) and a second LLM 52 (response generation unit). The information processing system 100 may further include a position sensor (not shown) to acquire experience information (described later).

[0013] The microphone 11 is a voice input device that accepts voice input from the user to the information processing system 100. The camera 12 is an imaging device that captures images in accordance with user operations or predetermined setting conditions. There are no particular limitations on the type, arrangement, number, imaging range, and imaging conditions of the cameras 12. The fingerprint sensor 14 is a fingerprint detection device that detects the user's fingerprint.

[0014] The vital sensor 13 acquires biological information of the user and outputs a signal corresponding to the biological information. Examples of the biological information acquired by the vital sensor 13 include body temperature, pulse rate, blood pressure, sweating, and respiratory rate.

[0015] The output device 15 is a device that transmits information to the user. Information is transmitted to the user in response to a voice input by the user. The output device 15 is, for example, an image display device and a speaker. The image display device is a device that displays responses to the user's conversation, displays images captured by the camera 12, and notifies the user by display. The speaker is an audio output device that outputs audio responses to the user's conversation and notifies the user by voice. The output device 15 may also include a vibration device for emphasizing notifications to the user, a transmission device for emergency contact, etc.

[0016] The storage unit 16 stores information necessary for controlling the information processing system 100. The information processing system 100 may be communicably connected to an external storage device serving as the storage unit 16. That is, the storage unit 16 may be provided outside the information processing system 100.

[0017] The voice input unit 21 functions as a language input unit that accepts voice input (language input) from the user via the microphone 11. The voice input unit 21 converts the input voice into text data. Alternatively, the user may input language by text input. When the user inputs language by text input, the information processing system 100 may be provided with a text input unit that accepts text input as language input. Examples of the text input unit include a touch panel or a smartphone connected to the context analysis unit 26 so as to be able to communicate with it. In the following description, it is assumed that the user inputs language by voice.

[0018] The image input unit 22 acquires an image from the camera 12 and converts it into embedded data (text data). The conversion is performed to convert the image into a data format that can be understood by the first LLM 51 or the second LLM 52.

[0019] There are several known methods for converting image data into text data. In the field of AI, the following known methods can be appropriately selected or combined for use.

[0020] For example, the conversion method may be Base64 encoding. Base64 encoding is a method for converting binary data into ASCII text and is widely used when handling binary data such as image files in text format. Base64 encoding is often used when embedding images as data URIs in applications, and can also be used in the information processing system 100.

[0021] The conversion method may also be hexadecimal encoding. Hexadecimal encoding is a method of converting binary data into a hexadecimal string. Hexadecimal encoding is generally used more often when visualizing binary data for debugging or data analysis than for images, and is rarely used in the information processing system 100. However, if the image provided is a CG image or, in particular, a mechanical design image, it is easy to capture the characteristics of the image and may be used depending on the application.

[0022] Alternatively, the conversion method may be URL encoding. URL encoding is often used in applications such as directly embedding images in HTML, and is widely used because it is easy to redisplay the converted image. In the information processing system 100, it is preferable that the data format after conversion is one that is easy for LLM to use. For this reason, URL encoding, in which the data format after conversion is an image, is not particularly selected as a conversion method for the information processing system 100 as a chat system. However, URL encoding may be adopted in applications where it is important to redisplay the input image on an image display device.

[0023] Although it is not a direct conversion method, there is also a method called JSON encoding. This method encodes binary data using Base64 and stores the result in JSON format. Currently available open LLMs and closed LLMs that support many image modes support this method, so it can be used effectively in the information processing system 100.

[0024] In any case, the conversion method can be selected based on the learning method of the LLM to be used, as long as it is in a format that can be interpreted by the LLM that ultimately generates the response. Various encoding methods are well known, and conversion can be performed at the input stage of various LLMs. Therefore, the method can be selected taking into account resources, response time, and other qualities. The important thing is that these conversion methods convert image data into text data, thereby providing a method for the context analysis unit 26 and storage unit 16 to handle image input in the same way as normal conversational input.

[0025] The vital information input unit 23 functions as a sensor information input unit that acquires the user's vital information as sensor information. The vital information input unit 23 organizes data such as pulse rate and blood pressure into a form that can be understood as biological data by classifying the output of the vital sensor 13 into specified items and numerical values. The information organized in this manner is referred to as first vital information. The first vital information may be text data.

[0026] For example, converting the output of an infrared sensor into a temperature number that humans can understand is called "calibration" and "data conversion." These processes are procedures that allow the sensor's output to correspond to an accurate physical quantity (in this case, temperature). These processes involve corrections and / or adjustments to convert the sensor's output voltage or digital signal into an actual temperature.

[0027] In the information processing system 100, the data conversion part is included in the processing by the vital information input unit 23. On the other hand, the part corresponding to the calibration is performed at the stage of creating the information processing system 100 or at the stage of first starting up the information processing system 100, and is stored as basic data in the vital information input unit 23. However, it goes without saying that the processing by the vital information input unit 23 of the information processing system 100 may include a calibration function.

[0028] Here, we will provide a basic explanation of the work from calibration to data conversion. Of course, specific operations may differ depending on the type of sensor. However, since the relationship between the sensor and physical quantity is well-known, such differences in specific operations can be addressed using well-known techniques.

[0029] Accurately establishing the relationship between a sensor's output and the actual physical quantity (temperature) essentially involves the following processes: ・Measurement of a reference point: Measure the sensor's output in a known temperature environment to obtain a reference point. ・Determining the relationship: Determine the relationship between the measured output and the actual temperature (e.g., linear or non-linear). ・Adjustment: Adjust the sensor's output to match the actual temperature. This may mean changing the sensor's hardware and / or software settings. ・Verification: Verify that the sensor's output is accurate after calibration. This may involve re-measuring in a different temperature environment.

[0030] Data conversion uses calibration data to convert the sensor's output (e.g., an analog voltage or digital value) into a human-understandable format (e.g., a temperature number). In other words, data conversion is the mapping of the sensor's output to a meaningful physical quantity. This mapping involves calculations and / or corrections based on the sensor's characteristics.

[0031] For example, in temperature conversion using an infrared sensor, the infrared sensor measures the intensity of infrared radiation emitted from an object and outputs that data as a voltage or digital signal (vital sensor 13). Converting this output to temperature requires the following steps: Understand the sensor's characteristics: Check the sensor's datasheet to understand the relationship between output and temperature. For example, check how the output voltage corresponds to temperature over a specific range. Obtain reference data: Measure the sensor's output in a known temperature environment to obtain reference data. For example, record the output at a known temperature point, such as the freezing point or boiling point. Create a conversion formula: Create a conversion formula to calculate temperature from the sensor's output. This can involve linear interpolation, nonlinear interpolation, or fitting a calibration curve. Then, using the sensor's output and the conversion formula, data such as "Temperature: 36.5°C" can be converted.

[0032] The vital information analysis unit 24 analyzes the user's mental state based on the first vital information. Examples of mental states include excitement, anger, sadness, joy, pleasure, and distress. Information indicating the analysis results is referred to as second vital information. The first vital information and the second vital information may also be collectively referred to as vital information. The vital information analysis unit 24 analyzes the first vital information and derives the second vital information using a pre-set database indicating the correspondence between the first vital information and the second vital information. The second vital information may be text data. The vital information analysis unit 24 sends the second vital information to the context analysis unit 26. The vital information analysis unit 24 may also send the first vital information to the context analysis unit 26.

[0033] The vital information analysis unit 24 may be a small-scale AI (CNN, RNN) trained for the purpose of deriving the second vital information, or may be a judgment circuit based on an appropriate database. When the vital information analysis unit 24 is the above-mentioned AI, it is considered that the first vital information (physiological indicators) and the second vital information (mental state) are closely related during learning, and a learning or judgment database can be created based on this information.

[0034] Here, the relationship between the first vital information and the second vital information will be explained using some general principles. However, in the information processing system 100, a learning or judgment database may be created using a unique index.

[0035] For example, the relationship between body temperature and mental state has been shown to increase with increasing stress and / or anxiety. In other cases, it is known that body temperature tends to become unstable in people with depression. Such a relationship between body temperature and mental state has been well-known through numerous studies, and by applying appropriate weighting, it is possible to infer the mental state.

[0036] There is also a great deal of research and clinical data on the relationship between pulse (heart rate) and mental state, and it is known that stress, excitement, or anxiety can cause the pulse to speed up, while relaxation can slow the pulse.

[0037] In addition, it is known that breathing rate and mental state are related such that breathing becomes faster in response to stress, anxiety, or fear, and that breathing becomes deeper and slower in a relaxed state.

[0038] By learning this information in advance and processing it statistically, mental state can be assessed using simple AI or a database.

[0039] Furthermore, the information processing system 100 is a system for conversation. The information processing system 100 may include a feedback system (not shown) that enables the LLM to infer the user's mental state from the content of the conversation. In this case, the feedback system feeds back to the vital information analysis unit 24 a combination of the first and / or second vital information and the mental state inferred by the LLM from the content of the conversation. By including the feedback system, the information processing system 100 can respond to user-specific situations and support the generation of conversations that are more tailored to the user.

[0040] The vital information analysis unit 24 may be capable of detecting vital information that is life-threatening to the user or vital information that has been preset as information that should trigger an alert. When the vital information analysis unit 24 detects the vital information, the vital information analysis unit 24 may issue an emergency call to a medical institution without performing any other processing. Although not shown in the drawings, the emergency call may include contact via a public line, a loud voice and / or an alarm sound notification requesting help from those in the vicinity, a message function to a pre-designated party, etc.

[0041] The authentication unit 25 performs personal authentication of the user based on information from the fingerprint sensor 14. The authentication unit 25 functions as a sensor information input unit that acquires personal authentication information as sensor information from the fingerprint sensor 14. The authentication unit 25 compares the fingerprint acquired from the fingerprint sensor 14 with a pre-registered fingerprint of the user, and determines whether the acquired fingerprint is that of the registered user. If the acquired fingerprint is that of the registered user, the authentication unit 25 verifies the user's personal ID or the organization to which the user belongs, and grants access rights to necessary information.

[0042] The authentication unit 25 outputs the determination result to the context analysis unit 26. The context analysis unit 26 may change the personal information used for context analysis depending on whether the user is a registered user or a guest user.

[0043] 1 illustrates a fingerprint sensor 14 as the personal authentication sensor, but other personal authentication sensors such as an iris authentication sensor, a voiceprint authentication sensor, or a vein sensor may also be used. Personal authentication may also be performed using an image of the user's face or voice. For example, the authentication unit 25 may perform personal authentication of the user through iris authentication or voiceprint authentication. Of course, authentication may also be performed using multiple sensors.

[0044] The context analysis unit 26 performs a context analysis of a general conversation and also performs a context analysis based on the second vital information. At this time, the context analysis unit 26 may perform the context analysis using past conversation history and / or the user's personal information. The context analysis unit 26 outputs a conversation response request reflecting the second vital information to the LLM determination unit 27. Furthermore, when first vital information is sent from the vital information analysis unit 24 to the context analysis unit 26, the context analysis unit 26 may output a conversation response request reflecting the first vital information in addition to the second vital information to the LLM determination unit 27.

[0045] The context analysis by the context analysis unit 26 may be, for example, an analysis using a small-scale language model that extracts keywords from the user's language input based on past history and organizes correlations between the keywords. Alternatively, the context analysis may include a process of determining attributes of the language input by analyzing keywords extracted from the user's language input using information from a database.

[0046] The attributes of the language input are simple tags corresponding to the content of the language input, such as a question, a greeting, an impression, a request for analysis, a request, or a knowledge field. The LLM determination unit 27 can select a response generation unit based on these tags.

[0047] If the user's authentication information (personal information) is present, the context analysis unit 26 may perform context analysis by using the user's own conversation history and, if necessary, the conversation history of a group (such as a family or business group). If the context does not depend on personal information, the previous history can be referenced. The conversation history stores the conversation user ID or conversation group ID and the conversation text itself, and the context analysis unit 26 may perform context analysis by using this information.

[0048] Furthermore, before generating a conversation response request, the context analysis unit 26 may determine whether to generate the request based on the user's authority. For example, if the question input by the user's voice is one for which the user does not have the authority to obtain an answer, the context analysis unit 26 may determine not to generate a conversation response request. Alternatively, the context analysis unit 26 may generate a request for generating a response such as "I cannot answer (with a reason depending on the situation)."

[0049] The context analysis unit 26 functions as a request generation unit that generates a conversation response request based on the sensor information and the user's language input. The conversation response request may include, in addition to the sensor information, the user's language input, and the results of the context analysis, image data converted into embedded data, conversation history or experience information, or a combination of these. The context analysis unit 26 outputs the generated conversation response request to the LLM determination unit 27. That is, the context analysis unit 26 may analyze the user's request based on the conversation history (conversation content and experience information, vital information), current conversation input, current vital information, and input image, and transmit the analysis result to the LLM determination unit 27 as a conversation response request.

[0050] The LLM determination unit 27 determines to which of the plurality of response generation units the conversation response request should be sent based on the content of the conversation response request. The response generation units will be described later. The LLM determination unit 27 sends the conversation response request to at least one of the plurality of response generation units. The LLM determination unit 27 may send the conversation response request to a plurality of response generation units. The context analysis unit 26 and the LLM determination unit 27 may be implemented as a single block.

[0051] The first LLM 51, the second LLM 52, and the simple response generator 28 function as a response generator that generates a response to a user's linguistic input. The LLM determination unit 27 determines to which of the first LLM 51, the second LLM 52, and the simple response generator 28 a conversation response request should be sent based on the content of the conversation response request. In this disclosure, when there is no need to limit the type of response generator, the LLMs and the simple response generator are collectively referred to simply as the response generator. The number of LLMs is not limited to two and may be three or more. Based on the determination of the LLM determination unit 27, multiple LLMs can be switched as needed to select the optimal LLM.

[0052] The simple response generator 28 may be a response generator that generates simple responses and is provided in a terminal that includes the LLM determination unit 27. Meanwhile, the first LLM 51 and the second LLM 52 may be response generators that generate complex responses and are provided in a server or the like external to the terminal that includes the LLM determination unit 27. Furthermore, the first LLM 51 and the second LLM 52 may be LLMs that differ in the generation of responses to conversational response requests. Specifically, the first LLM 51 and the second LLM 52 may be language models that differ in the type and amount of trained data, the maximum input size that can be processed, the response time, and / or the response accuracy.

[0053] The LLM determination unit 27 selects a response generation unit suitable for generating a response to the conversation response request according to the content of the conversation response request, and transmits the conversation response request. Furthermore, the LLM determination unit 27 may transmit a conversation response request including confidential information that should not be sent to an external server to the simple response generation unit 28 without transmitting the request to the external server.

[0054] Furthermore, for example, if the first LLM 51 is located on an internal server, the LLM determination unit 27 may select the first LLM 51 as the destination of a conversation response request containing confidential information that should not be sent to an external server. Each LLM may be tagged with an attribute indicating its location (affiliated organization) and / or its handling authority for confidential information. The LLM determination unit 27 may refer to the attribute tag of each response generator in addition to the content of the response request to select a response generator suitable for generating a response.

[0055] The response control unit 29 functions as a response control unit that receives a response generated in response to a conversation response generation request and controls the output of the response. Specifically, when there are multiple responses, the response control unit 29 may control the output order of the responses. Furthermore, when multiple responses can be integrated or summarized, the response control unit 29 may integrate or summarize the multiple responses. Furthermore, the response control unit 29 may generate a response message by formatting the response generated by the response generation unit so that it is a natural conversation in response to the user's voice input. Furthermore, the response control unit 29 may output the generated response message to the conversation history management unit 31. The response control unit 29 outputs the generated response message to the response output unit 30.

[0056] For example, when there is a response from the simple response generation unit 28 and a response from the LLM, the response control unit 29 may prioritize and output the response from the simple response generation unit 28 first. Also, for example, when the response from the LLM contains a tag that restricts speech, or a tag such as characters that are not suitable for pronunciation, vital information, or attributes, the response control unit 29 may extract and format the speechable portion and output it to the response output unit 30.

[0057] The response output unit 30 outputs the response message generated by the response control unit 29 via the output device 15. Furthermore, if the context analysis unit 26 does not generate a conversation response request, the response output unit 30 may output a message indicating that there will be no response to the user's voice input.

[0058] The conversation history management unit 31 records and manages the authentication information, first vital information, and second vital information as additional information in the conversation history, in addition to the conversation text including the user's linguistic input and the response to the linguistic input. This additional information is managed by appropriate tagging. For example, if any of the additional information has not changed from the previous time or if the context analysis unit 26 does not need any of the additional information, the conversation history management unit 31 does not need to record the additional information in the conversation history.

[0059] For example, for information that anyone can access, the conversation history management unit 31 may not need to record authentication information in the conversation history. For example, when a normal conversation is being held, the conversation history management unit 31 may not need to record first vital information in the conversation history. For example, when a normal conversation is being held but the second vital information changes suddenly, the conversation history management unit 31 may add the first vital information to the conversation history and record it. In this way, the conversation history management unit 31 may select vital information to manage and record based on the results of the context analysis unit 26. By appropriately managing the additional information in this way, the conversation history management unit 31 can appropriately manage the conversation history without wasting memory resources used for the conversation history.

[0060] In particular, the conversation history management unit 31 may function as a determination result correction unit that corrects the second vital information based on the result of the context analysis by the context analysis unit 26. For example, the context analysis unit 26 may analyze the user's mental state from the user's linguistic input in addition to the first vital information. In this case, if the user's mental state analyzed by the context analysis unit 26 does not match the second vital information, the conversation history management unit 31 corrects the second vital information to match the user's mental state analyzed by the context analysis unit 26. The conversation history management unit 31 may associate the corrected second vital information with the linguistic input and its response and manage them as a conversation history.

[0061] The conversation history is recorded in the storage unit 16. The conversation history management unit 31 can access the storage unit 16 as necessary and pass the conversation history to the context analysis unit 26. Furthermore, the conversation history management unit 31 may include a description of the image generated by the response generation unit in the conversation history.

[0062] Furthermore, the conversation history management unit 31 may modify, based on the conversation history, the database used by the vital information analysis unit 24 to analyze the first vital information. That is, the conversation history management unit 31 may function as a database modification unit that modifies the database based on the user's conversation history.

[0063] For example, if second vital information derived by analyzing certain first vital information is corrected based on the results of the above-described context analysis, it is possible that the correspondence between the first vital information and the second vital information in the database is far from the actual correspondence. In this case, the conversation history management unit 31 corrects the second vital information corresponding to the first vital information in the database to the second vital information corrected based on the results of the context analysis. This improves the accuracy of the analysis of the first vital information by the vital information analysis unit 24.

[0064] Note that past conversation history data becomes large as the usage time and frequency of the information processing system 100 increases. Therefore, storing all conversation history data is undesirable from the perspective of increasing memory resources and data processing time. Of course, memory capacity and memory access speeds are still improving year by year, and it is highly likely that in the future it will be possible to record virtually all conversations, if not all. However, at the time a system is implemented, available resources are limited, and it is desirable to be able to efficiently manage conversation history using limited resources. Therefore, the conversation history management unit 31 may have a function to maintain the data stored in the memory unit 16 at an appropriate size. There are several methods for maintaining an appropriate data size for conversation history, and any of these methods can be applied to the information processing system 100.

[0065] The simplest method is to set a limit on the data size of the conversation history in advance, and delete the oldest data when the limit is exceeded. This method is easy to implement and is effective in reducing the size. However, this method has the problem that because old information is automatically deleted, it is difficult to maintain consistency in the conversation, especially with old information. Therefore, this method is suitable for applications where consistency in the conversation is required over a relatively short period of time, such as one day's worth of data.

[0066] Another method is to set an upper limit on the data size of the conversation history and use a separate LLM (which can be the same model) to summarize it and reduce it to a predetermined amount of data. This method removes meaningless conversation history through summarization, increasing the ratio of valid conversations in the record, and therefore makes it possible to keep important conversations, even if they are old, for a relatively long time.

[0067] Furthermore, a suitable method for the information processing system 100 is to use Retrieval-Augmented Generation (RAG). RAG is an approach that combines information retrieval and generative models to generate more accurate answers to user questions. Below, we will briefly explain the basic operation of RAG, from creating embedded data to the search method.

[0068] 1. Creating Embedded Data: First, the conversation history is divided into segments of a predetermined size or a predetermined period, such as one day's worth of data, and each segment is summarized. The summarized data is then converted into vectors (embedded data) using a known embedding model, such as a pre-trained model like BERT, RoBERTa, or Sentence-BERT. An appropriate index may then be constructed. Furthermore, image data can be separated during the summaries and indexes, and stored, for example, on a cloud or on other inexpensive, high-capacity media where the image data can be referenced by the index. This improves memory utilization efficiency and enables longer-term storage. Furthermore, by including a brief description of the separated images in the summaries, older images can be recalled from the conversation as needed.

[0069] 2. Processing of User Questions: A user's question is converted into vector data using the same model as the history data. Using methods such as inner product calculations, the top three most similar history data are extracted and sent to the context analysis unit and answer generation unit, the LLM. Furthermore, if there is no data showing a similarity above a predetermined threshold, the history data can be prevented from being sent. This prevents the generation of an incorrect answer due to being influenced by less relevant information. Therefore, no matter how large the history data, the context analysis and answer generation can utilize history data that is closely related to the question within a practical data size.

[0070] Furthermore, the conversation history management unit may delete the vectorized data itself in order to manage the size of the history data itself. As mentioned above, the deletion method may be to delete the oldest data first, or to appropriately re-summarize the data. Even in this case, important data can be retained for a much longer period of time than if non-vectorized information were deleted.

[0071] This method using RAG allows the contents of conversation history data to be retained relatively accurately and for a long period of time, making it particularly suitable for use in the information processing system 100. However, in order for RAG to function effectively, a certain limit on the number of histories and memory capacity is required. Therefore, a method using RAG or another method can be selected depending on the application and available resources. There are also many well-known methods for appropriately managing memory, and any of these may be used. Furthermore, multiple methods of managing history may be used in combination.

[0072] In either method, the conversation history is organized at an appropriate timing during breaks in the conversation. However, in order to maintain the consistency of the most recent conversation, it is preferable that the most recent conversation history, for example, a conversation history of about 10 turns, is neither compressed by summarization nor deleted.

[0073] The experience information generation unit 32 generates experience information in which vital information of the user at a certain point in time is associated with information indicating the user's situation when the vital information at the certain point in time was acquired. Examples of information indicating the user's situation include location information, images, and time. The conversation history management unit 31 records and manages the experience information in the storage unit 16 by including it in the conversation history. In other words, the conversation history management unit 31 may function as an experience information management unit that manages vital information as a result of determining the user's mental state as experience information together with information indicating the user's situation. The conversation history management unit 31 may manage the experience information as part of the conversation history.

[0074] The experience information generation unit 32 may store the generated experience information directly in the storage unit 16, and the experience information generation unit 32 may manage the experience information. In other words, the experience information generation unit 32 may function as an experience information management unit.

[0075] The experience information is, for example, information indicating when (time), where (location information), what (image) the user saw, and what mental state the user was in (first and second vital information). One piece of experience information may include information identifying a subject in an image or a description of the image.

[0076] As described above, the information based on sensor information, such as the first vital sign information and the second vital sign information, may be text data, which allows the conversation system according to the present embodiment to handle the information based on the sensor information together with linguistic input.

[0077] As described above, the conversation history management unit 31 may manage at least one of the conversation history and the experience information. Both the management of the conversation history and the management of the experience information are considered to be management of past information. Therefore, in the present disclosure, the conversation history management unit 31 is also referred to as a past information management unit. As described above, the experience information generation unit 32 may also function as the experience information management unit, and therefore the experience information generation unit 32 is also referred to as a past information management unit.

[0078] (Example of Processing Flow) Figure 2 is a flowchart showing an example of the overall processing (information processing method) in the information processing system 100. In the information processing system 100, the voice input unit 21 accepts voice input from the user (S11). When the voice input unit 21 accepts the voice input, the context analysis unit 26 acquires the user's vital information and any necessary conversation history (S12), and analyzes the context of the input voice using the vital information and conversation history (S13). The context analysis unit 26 generates a conversation response request based on the analysis result of the context (S14). That is, the context analysis unit 26 generates the conversation response request based on the second vital information as the determination result of the user's mental state and the linguistic input.

[0079] Specifically, when the context analysis unit 26 receives the user input and the second vital information, it infers the user's wishes from past conversation history or general considerations and generates a directional request. The resulting response is generated in a direction that reflects the user's unique wishes.

[0080] For example, consider a case where a voice input of "I wonder if I should do something" is made and the second vital sign is "I feel depressed." In this case, the context analysis unit 26 generates a response request of "I want to do something to cheer myself up" in addition to the above voice input. Examples of responses to the response request are "Maybe it's a good idea to talk to a close friend or family member. Sometimes just talking can make you feel better," or "How about watching a movie or TV show you've been interested in?"

[0081] Also, consider the case where, as in the above case, a voice input of "I think I'll do something" is made and the second vital sign is "I feel excited. I seem happy." In this case, in addition to the above voice input, the context analysis unit 26 generates a response request saying, "I want to share this feeling. I want to have as much fun as I can." Examples of responses to the response request include, "How about going to the gym and working out? Or, how about trying dancing?" and "Why not share that happy feeling with your friends and family? It's even more fun when you can celebrate together."

[0082] Based on the content of the conversation response request, the LLM determination unit 27 selects a response generation unit and transmits the conversation response request (S15). The response generation unit selected by the LLM determination unit 27 generates a response to the conversation response request (S16). Based on the content of the response generated by the response generation unit, the response control unit 29 generates a response message to be output to the user (S17). The conversation history management unit 31 updates the user's conversation history with the response message generated by the response control unit 29 (S18). The response output unit 30 outputs the response message to the user via the output device 15 (S19). Note that the conversation history update (S18) may be performed after the response message output (S19).

[0083] 3 is a flowchart showing an example of the process in which the context analysis unit 26 acquires vital information in step S12 of FIG. 2 . First, the vital information input unit 23 acquires the output signal of the vital sensor 13 and generates first vital information by organizing the signal into a form that can be understood as biometric data (S21). Next, the vital information analysis unit 24 analyzes the user's mental state based on the first vital information and generates second vital information (S22). Furthermore, the vital information analysis unit 24 outputs the second vital information to the context analysis unit 26 (S23). The vital information analysis unit 24 may also output the first vital information to the context analysis unit 26. The context analysis unit 26 acquires the vital information output by the vital information analysis unit 24 in step S23 in the above-mentioned step S12.

[0084] (Variation of Processing Flow) FIG. 4 is a flowchart showing an example of a processing flow (information processing method) when experience information is used in the information processing system 100. In the example shown in FIG. 4, when the voice input unit 21 accepts voice input (S11), the context analysis unit 26 acquires the user's vital information, experience information, and conversation history (S12A). The context analysis unit 26 analyzes the context of the input voice using the acquired vital information, experience information, and conversation history (S13), and then steps S14 to S19 are executed as in FIG. 2. In this case, different responses may be obtained for the same request depending on the experience information. Furthermore, when multiple pieces of experience information exist for the same request, different responses may be obtained, for example, by majority vote on the experience information.

[0085] For example, consider a case where the user inputs a voice message saying, "I think I'll go out somewhere," and the second vital sign indicates, "I'm feeling down." In this case, the context analysis unit 26 generates a response request saying, "I want to change my mood."

[0086] In response to this response request, if there is experience information such as "A nearby park, children playing (image), and a happy mood," an example response to the response request would be "Why not take a walk in the park? You might meet someone interesting." On the other hand, if there is experience information such as "A nearby park, it's kind of noisy and unsettling (conversation), and it feels unpleasant," an example response to the response request would be "On a day like today, why not stay home and relax and listen to your favorite music instead of going out? The weather might be bad (a reasonable excuse)."

[0087] Furthermore, if there are multiple pieces of experience information for this response request, an example response to the response request will be, for example, a suggestion to the user of the destination that made the user feel happy the most times based on a majority vote of the experience information.

[0088] 5 is a flowchart showing an example of the process in which the context analysis unit 26 acquires vital information and experience information in step S12A of FIG. 4. First, the vital information input unit 23 acquires the output signal of the vital sensor 13 and generates first vital information by organizing the signal into a form that can be understood as biometric data (S31). Next, the vital information analysis unit 24 analyzes the user's mental state based on the first vital information and generates second vital information (S32). In parallel with steps S31 and S32, the experience information generation unit 32 acquires images from the camera and location information from the position sensor (S33).

[0089] The experience information generation unit 32 generates experience information based on the vital information, the image, and the position information (S34). The vital information analysis unit 24 outputs the vital information to the context analysis unit 26, and the experience information generation unit 32 outputs the experience information to the context analysis unit 26 (S35). The context analysis unit 26 acquires the vital information output by the vital information analysis unit 24 in step S35 and the experience information generated by the experience information generation unit 32 in step S12A described above.

[0090] [Embodiment 2] Another embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0091] 6 is a block diagram illustrating the configuration of an information processing system 200 according to embodiment 2. As shown in FIG. 6, the information processing system 200 includes an information processing device 210 and a server 220.

[0092] The information processing device 210 is a conversation terminal that includes elements other than the first LLM 51 and the second LLM 52 in the information processing system 100. In particular, the information processing device 210 integrates the microphone 11, camera 12, vital sign sensor 13, and fingerprint sensor 14 into a single device. This allows accurate information about the user to be acquired with a high degree of certainty. Furthermore, integrating these input devices reduces the possibility of the user's personal information being leaked.

[0093] The server 220 includes the first LLM 51 and the second LLM 52 in the information processing system 100. A publicly available server or a secure server managed by an individual or organization may be used as the server 220. In FIG. 6 , a single server 220 includes the first LLM 51 and the second LLM 52. However, in the information processing system 200, the server including the first LLM 51 and the server including the second LLM 52 may be separate servers.

[0094] If the server 220 manages multiple LLMs that exist inside or outside the server 220, the server 220 may have a function to call a publicly accessible server from among the LLMs it manages. For example, if the first LLM 51 is located inside the server 220 and the second LLM 52 is located on a different server, the server 220 may access the second LLM 52 via the different server.

[0095] The server 220 may have a function for switching the LLMs it manages. In this case, the server 220 may manage attribute tags of the LLMs it manages and share the attribute information with the LLM determination unit 27. This allows the LLM determination unit 27 to obtain information for selecting an appropriate LLM from currently available LLMs. The server 220 may have information on switchable LLMs and servers containing LLMs, or may be configured to obtain information from a separate database.

[0096] In the example shown in Figure 6, the voice input unit 21, image input unit 22, vital information input unit 23, vital information analysis unit 24, authentication unit 25, context analysis unit 26, LLM determination unit 27, simple response generation unit 28, response control unit 29, response output unit 30, conversation history management unit 31, and experience information generation unit 32 are all present on the same information processing device 210. These units may be integrated into the information processing device 210 in a state where they are integrated on a chip. Alternatively, these units may be distributed and arranged on a secure server or the like. The information processing system 200 may be realized as a program executed by computers provided in the information processing device 210 and the server 220.

[0097] 7 is a perspective view illustrating the terminal device 230. The terminal device 230 is a portable electronic device equipped with the information processing device 210.

[0098] As shown in Fig. 7, the terminal device 230 includes a mounting unit 235. The mounting unit 235 is a member for mounting the terminal device 230 around the user's neck. The mounting unit 235 has an open-ring shape that can be hooked around the user's neck. With this configuration, the user's movements are less likely to be hindered even when the terminal device 230 is mounted.

[0099] The attachment unit 235 has a microphone 11. Specifically, the microphone 11 is disposed at one end of the attachment unit 235, which has an open ring shape. This position is near the user's mouth when the user wears the terminal device 230 around their neck. Therefore, the user can easily input voice via the microphone 11 while wearing the terminal device 230 around their neck.

[0100] The mounting unit 235 has a camera 12. Specifically, the camera 12 is arranged near the end of the open-ring-shaped mounting unit 235 so as to face outward. This allows the orientation of the camera 12 to roughly match the orientation of the user's face when the user is wearing the terminal device 230 around their neck. This makes it possible for the camera 12 to capture what the user sees.

[0101] The attachment unit 235 has a vital sensor 13. Specifically, the vital sensor 13 is disposed inside a portion of the attachment unit 235 that is open-ring shaped, opposite to the open portion. As a result, when the user wears the terminal device 230 around their neck, the vital sensor 13 comes into contact with the user's neck. Because the vital sensor 13 is integrated with the attachment unit 235, vital information of the user wearing the terminal device 230 can be reliably acquired, and vital information can be acquired whenever necessary.

[0102] The attachment part 235 has a fingerprint sensor 14. Specifically, the fingerprint sensor 14 is disposed at the end of the open-ring-shaped attachment part 235 opposite to the end where the microphone 11 is disposed. This position allows the user to easily touch the fingerprint sensor 14 with their finger when the user is wearing the terminal device 230 around their neck. Therefore, the user can easily perform personal authentication.

[0103] The wearing unit 235 has an output device 15. Specifically, speakers serving as the output device 15 are arranged on each of the left and right sides of the wearing unit 235, which has an open ring shape, when the open part is in the front. These positions are near the user's ears when the user wears the terminal device 230 around their neck. The user can easily hear the output from the output device 15 when wearing the terminal device 230 around their neck.

[0104] The terminal device 230 may allow the user to specify the first vital information that should be acquired or the first vital information that should not be acquired in the vital information input unit 23. An example of a case in which the user specifies the first vital information that should be acquired or the first vital information that should not be acquired is when the user intends to monitor only their own blood pressure. In this case, the user may specify blood pressure as the first vital information that should be acquired and specify first vital information other than blood pressure as the first vital information that should not be acquired. The terminal device 230 may allow the user to specify the first vital information that should be acquired or the first vital information that should not be acquired by voice input. Alternatively, the terminal device 230 may allow the user to specify the first vital information that should be acquired or the first vital information that should not be acquired via another terminal such as a smartphone.

[0105] The terminal device 230 may also be configured to be able to turn the function of the vital sensor 13 on and off. For example, if the vital sensor 13 has functions for measuring body temperature, pulse rate, blood pressure, sweat rate, and respiratory rate, the user may be able to select not to measure one or more of these. In this case, the attachment unit 235 may be provided with an interface, such as a button, for turning the function of the vital sensor 13 on and off. Furthermore, the user's instructions regarding the operation of the terminal device 230 may be input to the terminal device 230 via another terminal device, such as a smartphone, that transmits the user's instructions to the terminal device 230. That is, the terminal device 230 may itself be provided with an interface that allows the user to configure settings related to the acquisition of sensor information, or the interface may be provided by using a device capable of communicating with the terminal device 230.

[0106] [Example of implementation by software] The functions of the information processing device (hereinafter referred to as the "device") can be realized by an information processing program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (in particular, the voice input unit 21, the image input unit 22, the vital information input unit 23, the vital information analysis unit 24, the authentication unit 25, the context analysis unit 26, the LLM determination unit 27, the simple response generation unit 28, the response control unit 29, the response output unit 30, the conversation history management unit 31, and the experience information generation unit 32).

[0107] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and storage device.

[0108] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0109] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0110] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0111] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

[0112] (Cross-reference to related applications) This application claims the benefit of priority to Japanese Patent Application No. 2024-151622, filed on September 3, 2024, the entire contents of which are incorporated herein by reference.

[0113] DESCRIPTION OF SYMBOLS 13 Vital sensor 14 Fingerprint sensor 21 Voice input unit 22 Image input unit 23 Vital information input unit (sensor information input unit) 24 Vital information analysis unit 25 Authentication unit (sensor information input unit) 26 Context analysis unit (request generation unit) 27 LLM determination unit (destination determination unit) 28 Simple response generation unit (response generation unit) 29 Response control unit 30 Response output unit 31 Conversation history management unit (experience information management unit, determination result correction unit, database correction unit, past information management unit) 32 Experience information generation unit (experience information management unit, past information management unit) 51 First LLM (response generation unit) 52 Second LLM (response generation unit) 100, 200 Information processing system 210 Information processing device 230 Terminal device

Claims

1. An information processing device comprising: a sensor information input unit that acquires sensor information; a language input unit that accepts language input from a user; a request generation unit that generates a conversational response request based on the sensor information and the language input; and a response control unit that receives a response generated in response to the conversational response request and controls the output of the response.

2. The information processing device according to claim 1, wherein the sensor information includes vital sign information of the user.

3. The information processing device according to claim 1 or 2, wherein the sensor information includes personal authentication information of the user.

4. The information processing device according to claim 2, further comprising a vital information analysis unit that determines the mental state of the user by analyzing the vital information acquired by the sensor information input unit.

5. The information processing device according to claim 4, wherein the request generation unit generates the conversation response request based on the mental state determination result and the linguistic input.

6. An information processing device according to claim 4 or 5, wherein the vital information analysis unit determines the mental state using a predetermined database, and further comprises a database correction unit that corrects the database based on the user's conversation history.

7. An information processing device according to any one of claims 4 to 6, further comprising a past information management unit that manages the mental state assessment results together with information indicating the user's situation as experience information.

8. The information processing device according to claim 7, wherein the past information management unit manages the experience information as part of a conversation history.

9. The information processing device according to any one of claims 4 to 8, further comprising a determination result correction unit that corrects the mental state determination result based on the conversation history of the user.

10. An information processing device according to any one of claims 1 to 5, further comprising a past information management unit that records a history of the language input, the response, and the sensor information as a conversation history, and the request generation unit generates the conversation response request based on the sensor information, the language input, and the conversation history.

11. A terminal device comprising an information processing device according to any one of claims 1 to 10.

12. The terminal device according to claim 11, further comprising a sensor that outputs the sensor information.

13. The terminal device according to claim 11 or 12, further comprising an interface for the user to configure settings relating to the acquisition of the sensor information.

14. An information processing program for causing a computer to function as the information processing device of claim 1, the information processing program causing a computer to function as the sensor information input unit, the language input unit, the request generation unit, and the response control unit.

15. An information processing system comprising: a sensor information input unit that acquires sensor information; a language input unit that accepts language input from a user; a request generation unit that generates a conversational response request based on the sensor information and the language input; a response generation unit that generates a response in response to the conversational response request; and a response control unit that receives the response generated in response to the conversational response request and controls the output of the response.

16. The information processing system of claim 15, wherein the response generation units are multiple, and further comprises a destination determination unit that determines to which of the multiple response generation units the conversation response request should be sent based on the content of the conversation response request, and the request generation unit analyzes the user's request based on the user's conversation content and experience information, conversation history including vital information, current conversation input, current vital information, and input image, and outputs the analysis result to the destination determination unit as a conversation response request.

17. An information processing method including the steps of: acquiring sensor information; accepting language input from a user; generating a conversational response request based on the sensor information and the language input; receiving a response generated in response to the conversational response request; and controlling the output of the response.

Images