Information processing device, terminal device, information processing system, information processing method, and information processing program

Abstract

The present invention makes it possible to obtain a high-quality response to a user instruction. An information processing system (100) is provided with a language input unit for receiving a language input from a user, a context analysis unit for analyzing a context indicated by the language input, an image input unit for acquiring an image from a camera, a specific instruction management unit for storing an instruction within the language input being related to the image as a specific instruction on the basis of a result of the context analysis, and a trigger generation unit for issuing a trigger signal indicating timing at which the specific instruction is to be executed.

Description

Information processing device, terminal device, information processing system, information processing method, and information processing program

[0001] The present disclosure relates to an information processing device that responds to a user's language input, and a terminal device that includes the information processing device.

[0002] In recent years, chat systems using artificial intelligence, such as ChatGPT, have been developed. Some of these chat systems allow users to input images, and in response to a question about an image such as "What is this image?", the system will respond with "This image is of a dog."

[0003] Retrieved August 20, 2024, Internet (URL: https: / / chatgpt.com / )

[0004] The conversation structure of conventional chat systems is basically one question and one answer, so for example, if a user gives an instruction regarding an image, once a response is received or another conversation takes place before a response is received, the previous instruction regarding the image is canceled.

[0005] Therefore, depending on the content of the user's instruction, it may be necessary to repeat the same instruction each time, which may result in a decrease in convenience.

[0006] An information processing device according to one aspect of the present disclosure includes a language input unit that accepts language input from a user, a context analysis unit that analyzes the context indicated by the language input, an image input unit that acquires an image from a camera, a specific instruction management unit that stores instructions related to the image from the language input as specific instructions based on the results of the context analysis, and a trigger generation unit that emits a trigger signal indicating the timing to execute the specific instruction.

[0007] An information processing system according to one aspect of the present disclosure includes a language input unit that accepts language input from a user, a context analysis unit that analyzes the context indicated by the language input, an image input unit that acquires images from a camera, a specific instruction management unit that manages instructions related to the image among the language input as specific instructions based on the results of the context analysis, a trigger generation unit that emits a trigger signal indicating the timing to execute the specific instruction, and a response generation unit that generates a response related to the specific instruction.

[0008] An information processing method according to one aspect of the present disclosure is executed by a computer and includes the steps of accepting a user's language input, analyzing the context indicated by the language input, acquiring an image from a camera, storing an instruction related to the image from the language input as a specific instruction based on the analysis result of the context, and emitting a trigger signal indicating the timing to execute the specific instruction.

[0009] According to one aspect of the present disclosure, it is possible to improve the convenience of a system that performs processing based on a user's language input.

[0010] Fig. 1 is a block diagram illustrating a configuration of an information processing system according to embodiment 1. Fig. 2 is a flowchart illustrating an example of overall processing in the information processing system according to embodiment 1. Fig. 3 is a block diagram illustrating a configuration of an information processing system according to embodiment 2. Fig. 4 is a perspective view illustrating a terminal device according to embodiment 2.

[0011] [Embodiment 1] An embodiment of the present disclosure will be described in detail below. FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system 100 according to embodiment 1. The information processing system 100 is an AI (Artificial Intelligence) conversation system. As shown in FIG. 1, the information processing system 100 includes a microphone 11, a camera 12, a sensor 17, a fingerprint sensor 14, an output device 15, and a storage unit 16. The information processing system 100 also includes a voice input unit 21 (language input unit), an image input unit 22, an authentication unit 25, a context analysis unit 26, an LLM (Large Language Model) determination unit 27, a simple response generation unit 28, a response control unit 29, a response output unit 30, a conversation history management unit 31, and an experience information generation unit 32. The information processing system 100 also includes a first LLM 51 and a second LLM 52.

[0012] The microphone 11 is a voice input device that accepts voice input from the user to the information processing system 100. The camera 12 is an imaging device that captures images in accordance with user operations or predetermined setting conditions. There are no particular limitations on the type, arrangement, number, imaging range, and imaging conditions of the cameras 12. The fingerprint sensor 14 is a fingerprint detection device that detects the user's fingerprint.

[0013] The output device 15 is a device that transmits information to the user. Information is transmitted to the user in response to a voice input by the user. The output device 15 is, for example, an image display device and a speaker. The image display device is a device that displays responses to the user's conversation, displays images captured by the camera 12, and notifies the user by display. The speaker is an audio output device that outputs audio responses to the user's conversation and notifies the user by voice. The output device may also be equipped with a vibration device for emphasizing notifications to the user, a transmission device for emergency contact, etc.

[0014] The storage unit 16 stores information necessary for controlling the information processing system 100. The information processing system 100 may be communicably connected to an external storage device as the storage unit 16. That is, the storage unit 16 may be provided outside the information processing system 100.

[0015] The voice input unit 21 functions as a language input unit that accepts voice input (language input) from the user via the microphone 11. The voice input unit 21 converts the input voice into text data. Alternatively, the user may input language by text input. When inputting language by text input, a text input unit that accepts text as language input via a text input device such as a touch panel or a smartphone connected for communication may be provided. In the following description, it is assumed that the user inputs language by voice.

[0016] The image input unit 22 acquires images from the camera 12. Unless otherwise specified in this disclosure, the images include still images and moving images. The image input unit 22 may also output the input images from the camera 12 to the context analysis unit 26. The timing of image acquisition and output by the image input unit 22 may be based on a trigger signal transmitted from a trigger generation unit, which will be described later.

[0017] The image input unit 22 acquires an image from the camera 12 and converts it into embedded data (text data). The conversion is performed to convert the image into a data format that can be understood by the first LLM 51 or the second LLM 52.

[0018] There are several known methods for converting image data into text data. In the field of AI, the following known methods can be appropriately selected or combined for use.

[0019] For example, the conversion method may be Base64 encoding. Base64 encoding is a method for converting binary data into ASCII text and is widely used when handling binary data such as image files in text format. Base64 encoding is often used when embedding images as data URIs in applications, and can also be used in the information processing system 100.

[0020] The conversion method may also be hexadecimal encoding. Hexadecimal encoding is a method of converting binary data into a hexadecimal string. Hexadecimal encoding is generally used more often when visualizing binary data for debugging or data analysis than for images, and is rarely used in the information processing system 100. However, if the image provided is a CG image or, in particular, a mechanical design image, it is easy to capture the characteristics of the image and may be used depending on the application.

[0021] Alternatively, the conversion method may be URL encoding. URL encoding is often used in applications such as embedding images directly into HTML, and is widely used because it is easy to redisplay the converted image. In the information processing system 100, it is preferable that the LLM be an easy-to-use format, so it is not particularly selected as a conversion method for the chat system. However, URL encoding may be used in applications where it is important to redisplay the input image on an image display device.

[0022] Although it is not a direct conversion method, there is also a method called JSON encoding. This method encodes binary data using Base64 and stores the result in JSON format. Currently available open LLMs and closed LLMs that support many image modes support this method, so it can be used effectively in the information processing system 100.

[0023] In any case, the conversion method can be selected based on the learning method of the LLM to be used, as long as it is in a format that can be interpreted by the LLM that ultimately generates the response. Various encoding methods are well known, and conversion can be performed at the input stage of various LLMs. Therefore, the method can be selected taking into account resources, response time, and other qualities. The important thing is that these conversion methods convert image data into text data, thereby providing a method for the context analysis unit 26 and storage unit 16 to handle image input in the same way as normal conversational input.

[0024] The authentication unit 25 performs personal authentication of the user based on information from the fingerprint sensor 14. The authentication unit 25 functions as a sensor information input unit that acquires personal authentication information as sensor information from the fingerprint sensor 14. The authentication unit 25 compares the fingerprint acquired from the fingerprint sensor 14 with a pre-registered fingerprint of the user, and determines whether the acquired fingerprint is that of the registered user. If the acquired fingerprint is that of the registered user, the authentication unit 25 verifies the user's personal ID or the organization to which the user belongs, and grants access rights to necessary information.

[0025] The authentication unit 25 outputs the determination result to the context analysis unit 26. The context analysis unit 26 may change the personal information used for context analysis depending on whether the user is a registered user or a guest user.

[0026] 1 illustrates a fingerprint sensor 14 as the personal authentication sensor, but other personal authentication sensors such as an iris authentication sensor, a voiceprint authentication sensor, or a vein sensor may also be used. Personal authentication may also be performed using an image of the user's face or voice. For example, the authentication unit 25 may perform personal authentication of the user through iris authentication or voiceprint authentication. Of course, authentication may also be performed using multiple sensors.

[0027] The conversation history management unit 31 records and manages the conversation text, including the user's linguistic input and the response to the linguistic input, as well as authentication information and image data as additional information in the conversation history. This additional information is managed by appropriate tagging. For example, if any of the additional information has not changed from the previous time or if the context analysis unit 26 does not need any of the additional information, the conversation history management unit 31 does not need to record the additional information in the conversation history.

[0028] For example, for information that anyone can access, the conversation history management unit 31 does not need to record authentication information in the conversation history.

[0029] The conversation history is recorded in the storage unit 16. The conversation history management unit 31 can access the storage unit 16 as necessary and pass the conversation history to the context analysis unit 26. Furthermore, the conversation history management unit 31 may include a description of the image generated by the response generation unit in the conversation history.

[0030] The conversation history is recorded in the storage unit 16. The conversation history management unit 31 can access the storage unit 16 as necessary and pass the conversation history to the context analysis unit 26. Furthermore, the conversation history management unit 31 may include a description of the image generated by the response generation unit in the conversation history.

[0031] Note that past conversation history data becomes large as the usage time and frequency of the information processing system 100 increases. Therefore, storing all conversation history data is undesirable from the perspective of increasing memory resources and data processing time. Of course, memory capacity and memory access speeds are still improving year by year, and it is highly likely that in the future it will be possible to record virtually all conversations, if not all. However, at the time a system is implemented, available resources are limited, and it is desirable to be able to efficiently manage conversation history using limited resources. Therefore, the conversation history management unit 31 may have a function to maintain data stored in the storage unit at an appropriate size. There are several methods for maintaining an appropriate data size for conversation history, and any of these methods can be applied to the information processing system 100.

[0032] The simplest method is to set a limit on the data size of the conversation history in advance, and delete the oldest data when the limit is exceeded. This method is easy to implement and is effective in reducing the size. However, this method has the problem that because old information is automatically deleted, it is difficult to maintain consistency in the conversation, especially with old information. Therefore, this method is suitable for applications where consistency in the conversation is sufficient over a relatively short period of time, such as one day's worth of data.

[0033] Another method is to set an upper limit on the data size of the conversation history and use a separate LLM (which can be the same model) to summarize it and reduce it to a predetermined amount of data. This method removes meaningless conversation history through summarization, increasing the ratio of valid conversations in the record, and therefore makes it possible to keep important conversations, even if they are old, for a relatively long time.

[0034] Furthermore, a suitable method for the information processing system 100 is to use Retrieval-Augmented Generation (RAG). RAG is an approach that combines information retrieval and generative models to generate more accurate answers to user questions. Below, we will briefly explain the basic operation of RAG, from creating embedded data to the search method.

[0035] 1. Creating Embedded Data: First, the conversation history is divided into segments of a predetermined size or a predetermined period, such as one day's worth of data, and each segment is summarized. The summarized data is then converted into vectors (embedded data) using a known embedding model, such as a pre-trained model like BERT, RoBERTa, or Sentence-BERT. An appropriate index may then be constructed. Furthermore, image data can be separated during the summaries and indexes, and stored, for example, on a cloud or on other inexpensive, high-capacity media where the image data can be referenced by the index. This improves memory utilization efficiency and enables longer-term storage. Furthermore, by including a brief description of the separated images in the summaries, older images can be recalled from the conversation as needed.

[0036] 2. Processing of User Questions: A user's question is converted into vector data using the same model as the history data. Using methods such as inner product calculations, the top three most similar historical data are extracted and sent to the context analysis unit and answer generation unit, the LLM. Furthermore, if there is no data showing a similarity level above a predetermined threshold, it can be prevented from being sent. This prevents the generation of an incorrect answer due to being influenced by less relevant information. Therefore, no matter how large the history data, context analysis and answer generation can utilize historical data that is closely related to the question within a practical data size.

[0037] Furthermore, the conversation history management unit may delete the vectorized data itself in order to manage the size of the history data itself. As mentioned above, the deletion method may be to delete the oldest data first, or to appropriately re-summarize the data. Even in this case, important data can be retained for a much longer period of time than if non-vectorized information were deleted.

[0038] This method using RAG allows the contents of conversation history data to be retained relatively accurately and for a long period of time, making it particularly suitable for use in the information processing system 100. However, in order for RAG to function effectively, a certain limit on the number of histories and memory capacity is required. Therefore, a method using RAG or another method may be selected depending on the application and available resources. There are also several well-known methods for appropriately managing memory, and any of these may be used. Furthermore, multiple methods for managing history may be used in combination.

[0039] In either method, the conversation history is organized at an appropriate timing during breaks in the conversation. However, in order to maintain the consistency of the most recent conversation, it is preferable that the most recent conversation history, for example, a conversation history of about 10 turns, is neither compressed by summarization nor deleted.

[0040] The context analysis unit 26 performs a context analysis of a general conversation. At this time, the context analysis unit 26 may perform the context analysis using past conversation history and / or personal information of the user. The context analysis by the context analysis unit 26 may be, for example, an analysis using a small-scale language model that extracts keywords from the user's language input based on the past history and organizes correlations between the keywords. Alternatively, the context analysis may include a process of determining attributes of the language input by analyzing keywords extracted from the user's language input using information in a database.

[0041] The attribute of the language input is a simple tag corresponding to the content of the language input, such as a question, a greeting, an impression, an analysis request, a request, or a knowledge field.

[0042] If the user's authentication information (personal information) is present, the context analysis unit 26 may perform context analysis by using the user's own conversation history and, if necessary, the conversation history of a group (such as a family or business group). If the context does not depend on personal information, the previous history can be referenced. The conversation history stores the conversation user ID or conversation group ID and the conversation text itself, and the context analysis unit 26 may perform context analysis by using this information.

[0043] The context analysis unit 26 functions as a request generation unit that generates a response request based on the user's language input. The response request includes at least one of a conversation response request and an image analysis request. In other words, the response request is a higher-level concept than the conversation response request and the image analysis request. In addition to the user's language input and the results of the context analysis, the response request may also include image data converted into embedded data, conversation history, experience information, or a combination of these. The context analysis unit 26 outputs the generated conversation response request to the LLM determination unit 27. In other words, the context analysis unit 26 may analyze the user's request based on the conversation history (conversation content and experience information), the current conversation input, and the input image, and transmit the analysis result to the LLM determination unit 27 as a response request.

[0044] Furthermore, before generating a response request, the context analysis unit 26 may determine whether to generate the request based on the user's authority. For example, if the question input by the user's voice is one for which the user does not have the authority to obtain an answer, the context analysis unit 26 may determine not to generate a response request. Alternatively, the context analysis unit 26 may generate a request for generating a response such as "I cannot answer (with a reason depending on the situation)."

[0045] The context analysis unit 26 includes a specific instruction management unit 261. Based on the context analysis results, the specific instruction management unit 261 determines whether the instruction indicated by the user's language input is a specific instruction. Furthermore, if the instruction indicated by the user's language input is a specific instruction, the specific instruction management unit 261 stores and manages the instruction as a specific instruction. A specific instruction is an instruction related to an image acquired by the image input unit 22. For example, a specific instruction may be an instruction that the information processing system 100 should execute for a certain period of time or multiple times, and by designating and storing it as a specific instruction, the number of user instructions related to the image can be reduced. An example of a specific instruction is an instruction to check multiple times whether an image of a specific subject is included in an image acquired by the image input unit 22.

[0046] The specific instruction management unit 261 may format the user's linguistic input into a format suitable for the LLM based on the result of the context analysis, and store and manage the specific instruction. The context analysis unit 26 may generate a basic instruction such as "Provide an appropriate response to the user input. User input: {user input}" based on the user's linguistic input. If the context analysis unit 26 includes the specific instruction management unit 261 and the specific instruction management unit 261 determines that the instruction by the linguistic input is an instruction related to an image, the context analysis unit 26 may combine the specific instruction and the basic instruction to generate a response request such as the following: "Provide an appropriate response to the user input. Also, provide a response based on the specific instruction for the provided image. User input: {user input}, specific instruction: {specific instruction}, image: {embedded data of the image}."

[0047] For example, if the language input is "Let me know if you see any signs indicating danger," the specific instruction management unit 261 determines that the instruction is an instruction related to an image. The specific instruction management unit 261 may store and manage the language input as a specific instruction such as "Analyze the provided image, determine whether there is a sign, and if there is a character string or symbol indicating danger on the sign, generate a warning message indicating the nature of the danger." The specific instruction management unit 261 may determine that the instruction is an instruction to be executed multiple times. The context analysis unit 26 generates a response request including the specific instruction. A response request including the specific instruction is referred to as an image analysis request. On the other hand, a response request generated by the context analysis unit 26 that does not include a specific instruction is referred to as a conversation response request. The response request includes at least one of a conversation response request and an image analysis request. The image analysis request may be generated by the specific instruction management unit 261.

[0048] The specific instructions stored in the specific instruction management unit 261 can be changed and / or deleted by user input.

[0049] The specific instruction management unit 261 instructs the language model to analyze an image based on the specific instruction when a trigger signal as an instruction to execute the specific instruction is issued from the trigger generation unit 40, which will be described later. In other words, the specific instruction management unit 261 can function as an image analysis request generation unit that generates an image analysis request for the language model based on the trigger signal.

[0050] The trigger generating unit 40 issues a trigger signal indicating the timing for executing a specific instruction. The trigger generating unit 40 may include an image analyzing unit with an image analysis function. In this case, the image analyzing unit may analyze multiple still images (or videos) continuously (in other words, over time) acquired from a camera and issue a trigger signal when an image (or a frame in the case of a video) that satisfies the conditions indicated by the specific instruction is included. For example, the trigger generating unit 40 may issue a trigger signal to control the timing for outputting images continuously or intermittently acquired by the image input unit 22 to the context analyzing unit 26. This trigger signal may be generated by any method, including known methods.

[0051] The image analysis unit included in the trigger generation unit 40 analyzes the camera image acquired by the image input unit 22. The image analysis unit, in particular, performs object recognition in the image. While a high-performance MM-LLM may be used for this function, considering the cost and determination time, it is preferable to use a lightweight AI such as a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network) specialized for object recognition. Recognizable objects are those that have been trained in advance to suit various use cases, but it is preferable that information on compatible objects is shared with the context analysis unit 26 so that appropriate search targets can be selected.

[0052] Object recognition is a very lightweight AI that can be easily implemented using only a portion of the resources of an edge device. Detailed analysis is not important for the purpose; for example, detecting the presence of a target object (human face, human hand, red sign, dog, cat, car, sign, etc.) By identifying the image or frame at the time of detection, it is possible to determine the image to be analyzed and the timing to issue a trigger signal.

[0053] The object recognition AI used as the image analysis unit may have a mechanism for updating the model to match the objects that can be recognized depending on the application. For example, the object recognition AI used as the image analysis unit may be updated by downloading update programs over a network.

[0054] When the trigger generating unit 40 has an image analysis unit, the trigger signal may be a signal indicating the timing to generate an image analysis request to the specific instruction managing unit 261. The trigger generating unit 40 having an image analysis unit may transmit, together with the trigger signal, image data of an image that satisfies the conditions indicated by the specific instruction to the specific instruction managing unit 261.

[0055] When the specific instruction management unit 261 receives the trigger signal, it generates an image analysis request for the image data transmitted at the same time, and outputs it to the LLM determination unit 27, thereby executing the specific instruction.

[0056] Alternatively, the trigger generating unit 40 may transmit a trigger signal to the image input unit 22. Upon receiving the trigger signal, the image input unit 22 transmits, based on the trigger signal, image data of the image acquired at the time of receiving the signal to the specific instruction managing unit 261. The specific instruction managing unit 261 may generate an image analysis request at the timing of receiving the image data transmitted from the image input unit 22, and output the image analysis request to the LLM determining unit 27.

[0057] When image data is transmitted from the image input unit 22 based on a trigger signal from the trigger generation unit 40, the specific instruction management unit 261 generates an image analysis request even when there is no voice input from the user, and transmits the request to a language model serving as a response generation unit that generates a response to the specific instruction via the LLM determination unit 27. That is, even when an image analysis instruction timing is directly transmitted to the specific instruction management unit 261 due to the occurrence of a trigger, when the input image from the image input unit 22 is updated due to the occurrence of a trigger, or even when both are performed simultaneously, an image analysis request is generated regardless of whether there is voice input from the user, and a response request is transmitted to the response generation unit via the LLM determination unit 27.

[0058] The response generation unit that receives the image analysis request generates a response related to the specific instruction, and the generated response is output from the response output unit 30. That is, in a situation where the specific instruction is generated, the trigger generation unit 40 specifies appropriate timing, whereby processing corresponding to the input image is automatically executed, and a response based on the specific instruction is output spontaneously and actively without any explicit instruction from the user.

[0059] The trigger generating unit 40 may issue a trigger signal based on the output of the sensor 17. The sensor 17 may be a clock (timer) and issue a trigger signal at regular intervals. The sensor 17 may be a step sensor and issue a trigger signal for each predetermined number of steps indicated by the step sensor. The sensor 17 may be an acceleration sensor and issue a trigger signal when the amount of change per unit time in acceleration indicated by the acceleration sensor exceeds a predetermined threshold. The sensor 17 may be a touch sensor and issue a trigger signal each time the user touches a touch sensor provided on the terminal device.

[0060] The sensor 17 may be a myoelectric potential sensor that emits a trigger signal each time it detects that the user has moved a specific muscle, such as shaking their head or winking. The sensor 17 may be a light (brightness) sensor that emits a trigger signal when the amount of light detected by the light sensor changes beyond a predetermined threshold range. The sensor 17 may be a global positioning system (GPS) that uses GPS position information to emit a trigger signal when the user's direction of travel changes or when the user's travel distance exceeds a predetermined distance. Alternatively, the sensor 17 may be a motion sensor or an acoustic sensor that emits a trigger signal each time the amount detected by these sensors changes.

[0061] The trigger generating unit 40 and the sensor 17 may be implemented in separate devices. The trigger generating unit 40 may be equipped with a connector for communicating with the sensor 17. Alternatively, the trigger generating unit 40 may be equipped with a wireless device for communicating with the sensor 17.

[0062] The trigger generating unit 40 can change the sensor control and trigger generating conditions depending on the type of sensor provided in the device. Furthermore, if the sensor 17 is device-dependent, the trigger generating conditions and / or trigger issuing program optimized for the device may be updated by, for example, downloading them from a database on the cloud.

[0063] When the trigger generating unit 40 issues a trigger signal based on the output of the sensor 17, the trigger signal may be a signal indicating to the image input unit 22 the timing to output an image to the context analyzing unit 26. Upon receiving the trigger signal, the image input unit 22 transmits image data of the image acquired at the time of receiving the signal to the specific instruction managing unit 261. The specific instruction managing unit 261 generates an image analysis request at the timing of receiving the image data transmitted from the image input unit 22, and outputs the image analysis request to the LLM determining unit 27, thereby executing the specific instruction.

[0064] The LLM determination unit 27 determines to which of the multiple response generation units the response request should be sent, based on the content of the response request, which includes one or both of a conversation response request and an image analysis request. The response generation units will be described later. The LLM determination unit 27 sends the response request to at least one of the multiple response generation units. The LLM determination unit 27 may send the response request to multiple response generation units. The context analysis unit 26 and the LLM determination unit 27 may be implemented as a single block. Furthermore, the LLM determination unit 27 may, for example, send a response request containing confidential information that should not be sent to an external server to the simple response generation unit 28, rather than to the external server.

[0065] Furthermore, for example, if the first LLM 51 is located on an internal server, the LLM determination unit 27 may select the first LLM 51 as the destination of a reply request containing confidential information that should not be sent to an external server. Each LLM may be tagged with an attribute indicating its location (affiliated organization) and / or its handling authority for the confidential information. The LLM determination unit 27 may refer to the attribute tag of each reply generator in addition to the content of the reply request to select a reply generator suitable for generating a reply.

[0066] The first LLM 51, the second LLM 52, and the simple response generator 28 function as response generators that generate responses to user language input. The LLM determination unit 27 determines to which of the first LLM 51, the second LLM 52, and the simple response generator 28 a response request should be sent based on the content of the response request. In this disclosure, when there is no need to limit the type of response generator, the LLMs and the simple response generators are collectively referred to simply as the response generator. The number of LLMs is not limited to two and may be three or more. Based on the determination of the LLM determination unit 27, multiple LLMs can be switched as needed to select the optimal LLM.

[0067] At least one of the multiple response generation units is an LLM having attributes suitable for the specific instruction. In the present disclosure, the first LLM 51 is a language model suitable for image processing. When the LLM determination unit 27 receives an image analysis request, it transmits the image analysis request to the first LLM 51. The first LLM 51 has an attribute tag indicating that it has attributes suitable for the specific instruction, and the LLM determination unit 27 may select the first LLM as the destination of the image analysis request based on the attribute tag. In the information processing system 100 according to this embodiment, the first LLM 51 may always be selectable by the LLM determination unit 27. In other words, the first LLM 51 may always be present in the information processing system 100.

[0068] For example, when an image analysis request is sent to a closed cloud LLM that realizes very high-performance image analysis, it may take time to generate a response. If the linguistic input is an instruction such as "Tell me if you see a sign indicating danger," the purpose may not be achieved if it takes time to generate a response. Therefore, it is preferable that the first LLM 51 have characteristics suitable for outputting a purpose-specific analysis with a realistic waiting time and as little time as necessary. The first LLM 51 may be an LLM optimized for a particular application and may have, for example, attributes such as those shown in (1) to (5) below.

[0069] (1) It can receive and analyze images as input. (2) It can interpret the character information contained in images as text. (3) It is specialized for image analysis and can operate at high speed. (4) It is not suitable for constructing complex logic or making inferences. (5) It is not suitable for generating very long sentences, but it excels at generating the minimum necessary sentences possible.

[0070] Although attributes (4) and (5) above may at first glance appear to degrade performance, by considering attributes such as (4) and (5) as suitable attributes for the first LLM 51, it is possible to select an appropriate LLM that is in line with the purpose of the conversation.

[0071] The simple response generator 28 may be a response generator provided in a terminal including the LLM determination unit 27 and generating simple responses. Meanwhile, the first LLM 51 and the second LLM 52 may be response generators provided in an external server or the like of the terminal including the LLM determination unit 27 and generating complex responses. The second LLM 52 may have different attributes from the first LLM 51. That is, the second LLM 52 may be a model that generates different responses from the first LLM 51 for general response requests, including image analysis requests. Specifically, the first LLM 51 and the second LLM 52 may be language models that differ in the type and amount of learned data, the maximum input size that can be processed, the response time, and / or the response accuracy. To put it more simply, the second LLM 52 may be a model that, unlike the first LLM 51, is incapable of image analysis. Of course, the first LLM 51 and the second LLM 52 may have similar attributes. For example, the second LLM 52 may be configured to perform image analysis without specific instructions.

[0072] The response control unit 29 receives a response generated in response to a response request and controls the output of the response. Specifically, when there are multiple responses, the response control unit 29 may control the output order of the responses. Furthermore, when multiple responses can be integrated or summarized, the response control unit 29 may integrate or summarize the multiple responses. Furthermore, the response control unit 29 may generate a response message by formatting the response generated by the response generation unit so that it is a natural conversation in response to the user's voice input. Furthermore, the response control unit 29 may output the generated response message to the conversation history management unit 31. The response control unit 29 outputs the generated response message to the response output unit 30.

[0073] For example, when there is a response from the simple response generation unit 28 and a response from an LLM such as the first LLM 51 or the second LLM 52, the response control unit 29 may prioritize the response from the simple response generation unit 28 and output it to the response output unit 30. For example, when the response from an LLM includes a tag that restricts speech or includes tags such as characters that are not suitable for pronunciation, vital information, or attributes, the response control unit 29 may extract and format the speechable portion and output it to the response output unit 30.

[0074] The response output unit 30 outputs the response message generated by the response control unit 29 via the output device 15. Furthermore, if the context analysis unit 26 does not generate an image analysis request, the response output unit 30 may output a message indicating that there will be no response to the user's voice input.

[0075] For example, when analyzing an image according to specific instructions, if the first LLM 51 determines that there is no danger in response to a response request such as "Let me know if there is any danger," it is conceivable that the first LLM 51 may generate a response such as "(no response, null)" or "There is no danger. (No need to speak)." In the former case, the response control unit 29 that received the generated response does not output to the response output unit 30. On the other hand, in the latter case, in response to the response "(No need to speak)," the response control unit 29 may suppress output to the response output unit 30 or explicitly instruct the response output unit 30 not to speak.

[0076] The conversation history is recorded in the storage unit 16. The conversation history management unit 31 can access the storage unit 16 as necessary and pass the conversation history to the context analysis unit 26. The conversation history management unit 31 may manage, as a conversation history, a linguistic input corresponding to a specific instruction, image data of an image analyzed based on the specific instruction, and a response to the specific instruction in association with each other.

[0077] The experience information generation unit 32 generates experience information associated with information indicating the user's past experience. The information indicating the user's past experience may be, for example, location information, images, time, or other information. The experience information generation unit 32 outputs the generated experience information to the conversation history management unit 31. The conversation history management unit 31 records and manages the experience information in the storage unit 16 while including it in the conversation history. Note that the experience information generation unit 32 may directly store the generated experience information in the storage unit 16, and the experience information generation unit 32 may manage the experience information.

[0078] The experience information is, for example, information indicating when (time), where (location information), and what (image) the user saw. One piece of the experience information may include information identifying a subject appearing in the image.

[0079] In the present disclosure, the experience information generation unit is not an essential component.

[0080] (Example of Processing Flow) FIG. 2 is a flowchart showing an example of the overall processing (information processing method) when the information processing system 100 receives a language input, which is speech, from a user.

[0081] First, the speech input unit 21 accepts a language input, which is speech, from the user (S11). When the speech input unit 21 accepts the speech input, the context analysis unit 26 analyzes the context of the input speech using a conversation history (S12). The context analysis unit 26 may perform the process of step S12 without using the conversation history. For example, there may be a case where the information processing system 100 has just been used and there is no conversation history, or there is no conversation history for which access rights exist due to security authentication.

[0082] Based on the analysis result of the context analysis unit 26, the specific instruction management unit 261 determines whether the instruction indicated by the user's linguistic input is an instruction corresponding to a specific instruction, specifically, an instruction regarding an image acquired by the image input unit 22 (S13). For example, if the linguistic input received from the user is "Let me know if you see a sign," the instruction indicated by the linguistic input is an instruction regarding an image that should be executed multiple times, and the specific instruction management unit 261 determines that the instruction is a specific instruction.

[0083] If the instruction corresponds to a specific instruction (S13: YES), the specific instruction management unit 261 stores the instruction as a specific instruction in the storage unit 16 (S14). In step S14, the specific instruction management unit 261 may format the user's linguistic input into a format suitable for LLM and store the formatted linguistic input as a specific instruction. Furthermore, if the trigger generation unit 40 has an image analysis unit, in step S14, the specific instruction management unit 261 instructs the trigger generation unit 40 to start processing based on the specific instruction.

[0084] If the user's language input corresponds to a specific instruction and a specific instruction related to the image has already been stored, the specific instruction management unit 261 may overwrite and update the stored specific instruction with the newly input specific instruction. If the specific instruction management unit 261 determines that the instruction indicated by the new user's language input is the same as the stored specific instruction, the specific instruction may not be updated and the current specific instruction may continue to be stored.

[0085] In the next step S14A, the specific instruction management unit 261 determines whether the specific instruction has been canceled. If the specific instruction has not been canceled (S14A: NO), the process proceeds to step S15.

[0086] The trigger generating unit 40 determines whether the image captured in the image acquired by the image input unit 22 satisfies a predetermined condition indicated by the specific instruction. In the case of an instruction to "let me know if you see a signboard," the predetermined condition is that the image contains an image of a signboard, and the trigger generating unit 40 determines whether the image contains an image of a signboard. If the trigger generating unit 40 determines that the image contains an image of a signboard, it transmits a trigger signal to the specific instruction managing unit 261 along with image data of the image. In this case, the trigger generating unit 40 does not necessarily need to transmit the trigger signal along with the image data to the specific instruction managing unit 261; the transmission of the image data may be sufficient to transmit the trigger signal. In other words, the specific instruction managing unit 261 may use the transmission of image data from the image input unit 22 in response to an instruction from the trigger generating unit 40 as a trigger for executing the specific instruction.

[0087] When the trigger generating unit 40 generates a trigger signal based on the output of the sensor 17 , the trigger generating unit 40 transmits the trigger signal to the image input unit 22 .

[0088] When a trigger signal is issued from the trigger generating unit 40 (S15: YES), the image input unit 22 transmits image data of the image acquired when the trigger signal was received to the specific instruction management unit 261. Note that the system may be designed so that S15 always returns YES when there is voice input from the user. In this case, the voice input from the user can be added as a trigger generation condition.

[0089] When the specific instruction management unit 261 receives the image data, it generates an image analysis request (S16).

[0090] The LLM determination unit 27 selects a response generation unit based on the content of the image analysis request and transmits the image analysis request (S17). The response generation unit selected by the LLM determination unit 27 generates a response to the image analysis request (S18). The response control unit 29 generates a response message to be output to the user based on the content of the response generated by the response generation unit (S19).

[0091] The conversation history management unit 31 updates the user's conversation history (S20). Specifically, the conversation history management unit 31 updates the user's conversation history by adding to the conversation history a series of data that associates the response generated by the response control unit 29, the language input corresponding to the response, and the image data of the analyzed image.

[0092] The response output unit 30 outputs a response message to the user via the output device 15 (S21). Note that the conversation history may be updated (S20) after the response message is output (S21).

[0093] After the process of step S21, the process returns to step S14A, and the processes of S14A to S21 are repeated until a new language input is received. As a result, a response message is output based on the trigger signal even if there is no language input from the user.

[0094] In step S13, if the user's language input does not correspond to a specific instruction related to the image (S13: NO), the specific instruction management unit 261 determines whether a specific instruction related to the image has already been stored (S22). If a specific instruction related to the image has already been stored (S22: YES), the specific instruction management unit 261 determines whether the language input is an instruction to cancel the specific instruction (S23). If the language input is content to cancel the current specific instruction (S23: YES), the specific instruction management unit 261 cancels and deletes the currently stored specific instruction (S15A). For example, if the user's language input includes a context such as "That's enough" or "Image analysis completed," the specific instruction management unit 261 erases the current specific instruction and ends the series of processes.

[0095] In step S23, if the user's language input does not cancel the current specific instruction (S23: NO), the context analysis unit 26 generates a conversational response request corresponding to the instruction and transmits it to the LLM determination unit 27 (S25). The LLM determination unit 27 selects a response generation unit based on the content of the conversational response request and transmits the conversational response request to the selected response generation unit (S26). The response generation unit selected by the LLM determination unit 27 generates a response to the image analysis request (S27). The response control unit 29 generates a response message to be output to the user based on the content of the response generated by the response generation unit (S28).

[0096] The conversation history management unit 31 updates the user's conversation history (S29). Specifically, the conversation history management unit 31 updates the user's conversation history by adding to the conversation history a series of data that associates the response generated by the response control unit 29, the language input corresponding to the response, and the image data of the analyzed image.

[0097] The response output unit 30 outputs a response message to the user via the output device 15, and the series of processes ends (S30). Note that the conversation history may be updated (S29) after the response message is output (S30).

[0098] In step S22, if no specific instruction regarding the image is stored (S22: NO), the process proceeds to step S25. The process from step S25 onwards is as described above.

[0099] If a first language input corresponds to a specific instruction and a loop process from step S14A to step S21 is being performed, and a second language input that does not correspond to a specific instruction is made, the process for the second language input proceeds from step S13 to step S22. Meanwhile, the process for the first language input is also performed in parallel. As a result, the specific instruction is maintained even if another response is made after the specific instruction.

[0100] Furthermore, if an instruction to cancel a specific instruction is given in the second language input, the specific instruction is deleted in step S24, and the loop processing for the first language input ends in step S14A, where it is determined that the specific instruction has been canceled (S14: YES), and the series of processing for the specific instruction is terminated.

[0101] (Specific Examples) Specific examples will be described below in accordance with the processing shown in Fig. 2. In specific examples 1 and 2, it is assumed that the information processing system 100 includes a trigger generating unit 40 having an image analyzing unit.

[0102] Specific Example 1 In step S11, the voice input unit 21 receives a user's linguistic input such as "Please let me know if there are any signs that I should be careful of."

[0103] In step S12, the context analysis unit 26 analyzes the context of the linguistic input.

[0104] In step S13, the specific instruction management unit 261 determines that the language input is a specific instruction regarding the image acquired by the image input unit 22 (S13: YES), stores the instruction as a specific instruction in the memory unit 16, and instructs the trigger generation unit 40 to start processing based on the specific instruction (S14).

[0105] In step S14A, the specific instruction management unit 261 determines whether a specific instruction is stored in the storage unit 16. If a specific instruction is stored in the storage unit 16 (S14A: YES), the specific instruction management unit 261 determines whether a trigger signal has been issued in step S15. On the other hand, if the specific instruction has been deleted or has not yet been created and is not stored in the storage unit 16 (S14A: NO), the trigger signal detection process is not performed and the loop ends.

[0106] Regardless of the progress of the processing flow, the image analysis unit of the trigger generation unit 40 analyzes images continuously acquired from the image input unit 22 by object recognition. If a sign is included in the acquired image, a trigger signal is sent together with the image data of the image to the specific instruction management unit 261. In this specific example 1, it is assumed that an image of a sign indicating a private road is acquired.

[0107] In the first operation of the loop, i.e., the first loop related to "Tell me if you see any signs that require attention," if there is language input, a specific instruction, and no sign in the current image, the specific instruction management unit 261 determines YES in step S15 because there is language input, and executes the subsequent processing. After going through the subsequent steps, in step S21, the response output unit 30 outputs a response message to the user, such as "I understand. You want to be notified if you see a sign that requires attention. There is no such sign in the location you are currently looking at."

[0108] When the specific instruction management unit 261 receives a trigger signal and image data from the trigger generation unit 40 (S15: YES), in step S16, the specific instruction management unit 261 generates an image analysis request asking for information about what is written on the sign image indicated by the image data.

[0109] The context analysis unit 26 generates basic instructions, and the specific instruction management unit 261 generates specific instructions tailored to the LLM. A basic instruction is, for example, "Give an appropriate response to the user input. Also, give a response based on the specific instructions for the provided image. User input: {user input}, specific instruction: {specific instruction}, image: {embedded data of the image}." A specific instruction is, for example, "Analyze the provided image, determine whether there is a signboard, and if there is a character string or symbol in the signboard that draws the observer's attention, generate a warning message indicating the content of the warning."

[0110] By generating such basic instructions by the context analysis unit 26, it is possible to cope with all situations, such as voice input only, image input only, and both voice input and image input.

[0111] In this specific example, the specific instruction management unit 261 stores basic instructions and specific instructions in the storage unit 16, and generates a response request (image analysis request) by combining user input, specific instructions, and image input. However, the context analysis unit 26 may generate a response request by combining user input, specific instructions, and image input outside the specific instruction management unit 261. More specifically, the context analysis unit 26 may call up specific instructions generated and stored by the specific instruction management unit 261 and generate an image analysis request.

[0112] In addition, if the language input is an instruction to cancel or modify an existing specific instruction, the specific instruction management unit 261 erases or modifies the specific instruction stored in the memory unit 16 (S13 → S22 → S23 → S24).

[0113] In this specific example, an image analysis request is generated by combining the user input, specific instructions, and image input with the basic instructions created by the context analysis unit 26. Of course, normal operation is possible even if the user input is not included in the image analysis request.

[0114] In step S17, when the image analysis request is received, the LLM determination unit 27 selects the first LLM 51 suitable for image processing as the destination of the image analysis request.

[0115] In step S18, the first LLM 51 analyzes the image data, interprets the text information contained in the image, and generates a response such as, "It says 'private road'. You may not be able to enter."

[0116] In step S19, the response control unit 29 generates a response message such as "You can't go further because it's a private road. Let's turn back," based on the response of the first LLM 51, so as to create a natural conversation. Of course, if the first LLM 51 generates a response such as "You can't go further because it's a private road. Let's turn back," depending on the capabilities of the LLM, this response may be used as is.

[0117] In step S20, the conversation history management unit 31 adds a series of data that associates the linguistic input "Let me know if there are any signs that I should be careful of," image data of an image including the sign, and a response message "This is a private road, so you cannot proceed. Let's turn back," to the conversation history, and updates the conversation history.

[0118] In step S21, the response output unit 30 outputs a response message to the user via the output device 15 saying, "You cannot proceed because this is a private road. Let's turn back."

[0119] In the above-described specific example 1, the image analysis request is not generated depending on the timing of the user's language input, but can be actively generated based on the image analysis of the trigger generating unit 40, even if there is no voice input from the user. Furthermore, since the specific instruction is stored and managed by the specific instruction managing unit 261, even if another conversation takes place after the above-described language input, the image analysis request is generated spontaneously based on the image analysis of the trigger generating unit 40. This can improve user convenience and obtain a high-quality response to the user's instructions.

[0120] Specific Example 2 In step S11, the speech input unit 21 receives a user's linguistic input of "Please let me know if you see a speed limit sign."

[0121] In step S12, the context analysis unit 26 analyzes the context of the linguistic input.

[0122] In step S13, the specific instruction management unit 261 determines that the language input is an instruction related to an image (S13: YES) and stores the instruction as a specific instruction (S14). Here, the specific instruction management unit 261 converts the user's language input into a format suitable for the LLM and stores the instruction in the storage unit 16 as a specific instruction such as, "If the sign in the input image has information about a speed limit, please output that information. If there is no information about a speed limit in the image, please output null." Other basic instructions related to conversation are the same as those in Specific Example 1. Furthermore, if the language input is to delete a stored specific instruction, the specific instruction management unit 261 deletes the specific instruction (S13 → S22 → S24). If the language input contains a context for modifying the specific instruction, the specific instruction management unit 261 deletes the stored specific instruction or modifies it to a new specific instruction (S14).

[0123] In step S14A, if a specific instruction is stored and has not been erased (S14A: YES), the specific instruction management unit 261 determines in step S15 whether a trigger signal has been issued.

[0124] Regardless of the progress of the processing flow, the image analysis unit of the trigger generation unit 40 analyzes images continuously acquired from the image input unit 22 by object recognition. If a sign is included in the acquired image, a trigger signal is sent together with the image data of the image to the specific instruction management unit 261. In this step S14, it is assumed that an image of a sign indicating a speed limit of 30 km / h has been acquired.

[0125] Here, the analysis content of the image analysis unit of the trigger generation unit 40 will be described. When a small-scale AI such as CNN or RNN is used as the image analysis unit, complex recognition such as "if the sign in the input image has information about the speed limit written on it" is not possible. Therefore, when a small-scale AI is used as the image analysis unit, the specific instruction management unit 261 may issue to the image analysis unit simple instructions that the small-scale AI can understand. For example, in Japan, speed signs with the speed limit written in numbers inside a red circle are often seen. Therefore, in this specific example, "sign," "numbers," "red circle," etc. are set in the instructions to the image analysis unit instead of "information about the speed limit."

[0126] The actual instructions may be given, for example, by creating a database that associates frequently used search words with objects learned by the image analysis unit, and the specific instruction management unit 261 may refer to this database. Alternatively, the specific instruction management unit 261 may request the LLM to identify an object that can be recognized by the image analysis unit and that should be included in the instructions. For example, the specific instruction management unit 261 may cause the LLM to select an object to be recognized by an instruction such as, "From the following list of currently learned objects, please select the xx that best matches 'Speed ​​Limit Sign'. List: xx, xx, xx, xx, xx, xx, ."

[0127] When the specific instruction management unit 261 receives a trigger signal and image data from the trigger generation unit 40 (S15: YES), in step S16, the specific instruction management unit 261 generates an image analysis request stating, "If the image of the sign shown in the image data contains any information regarding a speed limit, please tell us what that information is. If there is no information regarding a speed limit in the image, please output null," and transmits the request to the LLM determination unit 27. If the specific instruction management unit 261 has not received a trigger signal from the trigger generation unit 40 (S15: YES), step S15 is repeated until a trigger signal is issued.

[0128] In step S17, when the image analysis request is received, the LLM determination unit 27 selects the first LLM 51 suitable for image processing as the destination of the image analysis request.

[0129] In step S18, the first LLM 51 analyzes the image data, interprets the text information contained in the image, and generates a response such as "The speed limit is 30 km / h."

[0130] In step S19, the response control unit 29 generates a response message such as "There is a speed limit of 30 km / h ahead" so as to create a natural conversation. Of course, if the first LLM 51 generates a response that can be used as is, such as "There is a speed limit of 30 km / h ahead," it goes without saying that this may be used.

[0131] In step S20, the conversation history management unit 31 adds a series of data that associates the linguistic input "Let me know if you see a speed limit sign," image data of an image including the sign, and a response message "There is a speed limit of 30 km / h up ahead" to the conversation history, and updates the conversation history.

[0132] In step S21, the response output unit 30 outputs a response message to the user via the output device 15 saying, "There is a speed limit of 30 km / h ahead," and the process returns to step S14A.

[0133] If the specific instruction is not canceled in step S14A (S14A: NO), in step S15, the image analysis unit of the trigger generation unit 40 analyzes, by object recognition, images continuously acquired from the image input unit 22. If a sign is included in the acquired image, a trigger signal is sent to the specific instruction management unit 261 along with image data of the image. In this step S15, it is assumed that an image of a sign unrelated to the speed limit has been acquired.

[0134] When the image data is received (S15: YES), in step S16, the specific instruction management unit 261 generates an image analysis request stating, "If the image of the sign shown in the image data contains any information regarding the speed limit, please tell us what it is. If there is no information regarding the speed limit in the image, please output null," and sends it to the LLM determination unit 27.

[0135] In step S17, when the image analysis request is received, the LLM determination unit 27 selects the first LLM 51 suitable for image processing as the destination of the image analysis request.

[0136] In step S18, the first LLM 51 analyzes the image data, interprets the text information contained in the image, and outputs null, in other words, generates a silent response.

[0137] The response generated by the response control unit 29 in step S19 is also silent.

[0138] In step S20, the conversation history management unit 31 adds a series of data that associates the linguistic input "Let me know if you see a speed limit sign," image data of an image including the sign, and the result of the null output to the conversation history, thereby updating the conversation history.

[0139] In step S21, the response output unit 30 does not output a response message.

[0140] In step S14A, the same process is repeated unless the specific instruction is cancelled.

[0141] In the above-described specific example 2, the image analysis request is not generated based on the timing of the user's language input, but is actively generated based on the results of image analysis by the trigger generation unit 40. Furthermore, since the specific instruction management unit 261 stores the user's language input as a specific instruction, responses to multiple signs are generated spontaneously in response to a single instruction. This makes it possible to obtain high-quality responses to the user's instructions.

[0142] Specific Example 3 In step S11, the speech input unit 21 accepts a linguistic input from the user, such as "Please explain what you see here."

[0143] In step S12, the context analysis unit 26 analyzes the context of the linguistic input.

[0144] In step S13, the specific instruction management unit 261 determines that the language input is an instruction related to an image (S13: YES), and stores the instruction as a specific instruction (S14). Here, the specific instruction management unit 261 converts the user's language input into a format suitable for LLM, and stores the instruction as a specific instruction such as "Please briefly explain the objects included in the image. However, there is no need to explain human hands and fingers."

[0145] If the specific instruction management unit 261 determines in step S14 that a specific instruction has been stored (S14A: YES), the specific instruction management unit 261 determines in step S15 whether a trigger signal has been issued.

[0146] The image analysis unit of the trigger generation unit 40 analyzes images continuously acquired from the image input unit 22 by object recognition. If the acquired image includes a fingertip, a trigger signal is sent to the specific instruction management unit 261 along with image data of the image. If a fingertip is detected, the image analysis unit may generate an enlarged image, for example, enlarged twice, with the fingertip at the center, and send a trigger signal to the specific instruction management unit 261 along with image data of the enlarged image. In this step S15, it is assumed that an image of someone pointing at a pencil has been acquired.

[0147] That is, the image analysis unit is instructed to recognize a "human finger." The image analysis unit may be able to acquire the coordinates of the recognized object on the image. The image analysis unit may specify imaging conditions for the image input unit 22, and may zoom the acquired image by two times, centering on the recognized finger. The recognized object may also be processed into a rectangle for processing.

[0148] In step S15, if the specific instruction management unit 261 determines that a trigger signal has been issued by receiving image data (S15: YES), the context analysis unit 26 generates an image analysis request in step S16. The context analysis unit 26 may generate a response request that combines the image analysis request, basic instructions, specific instructions, and the input image.

[0149] In step S17, when the image analysis request is received, the LLM determination unit 27 selects the first LLM 51 suitable for image processing as the destination of the image analysis request.

[0150] In step S18, the first LLM 51 analyzes the image data and generates a response of "It's a pencil."

[0151] In step S19, the response control unit 29 generates a response such as "A pencil has fallen" so as to sound like a natural conversation.

[0152] In step S20, the conversation history management unit 31 adds a series of data that associates the linguistic input "Describe what you see here," image data of an image pointing to a pencil, and a response message "The pencil has fallen" to the conversation history, thereby updating the conversation history.

[0153] In step S21, the response output unit 30 outputs a response message "A pencil has fallen" to the user via the output device 15.

[0154] In step S14A, if it is determined that the specific instruction has been canceled by a user input or the like (S14A: YES), the series of processes ends. If the specific instruction has not been canceled (S14A: NO), when the image analysis unit of the trigger generation unit 40 recognizes the fingertip again, a trigger signal is issued (S15).

[0155] In the above-described specific example 3, the image analysis request is also generated based on the image analysis by the trigger generating unit 40. This makes it possible to obtain a high-quality response to a user's instruction.

[0156] [Embodiment 2] Another embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0157] 3 is a block diagram illustrating an example of the configuration of an information processing system 200 according to embodiment 2. As shown in FIG. 3, the information processing system 200 includes an information processing device 210 and a server 220.

[0158] The information processing device 210 is a conversation terminal that includes elements other than the first LLM 51 and the second LLM 52 in the information processing system 100. In particular, the information processing device 210 integrates the microphone 11, camera 12, sensor 17, and fingerprint sensor 14 into a single device. This allows accurate information about the user to be acquired with a high degree of certainty. Furthermore, integrating these input devices reduces the possibility of the user's personal information being leaked.

[0159] The server 220 includes the first LLM 51 and the second LLM 52 in the information processing system 100. A publicly available server or a secure server managed by an individual or organization may be used as the server 220. In FIG. 3 , a single server 220 includes the first LLM 51 and the second LLM 52. However, in the information processing system 200, the server including the first LLM 51 and the server including the second LLM 52 may be separate servers.

[0160] If the server 220 manages multiple LLMs that exist inside or outside the server 220, the server 220 may have a function to call a publicly accessible server from among the LLMs it manages. For example, if the first LLM 51 is located inside the server 220 and the second LLM 52 is located on a different server, the server 220 may use the second LLM 52 via the different server.

[0161] The server 220 may have a function for switching the LLMs it manages. In this case, the server 220 may manage attribute tags of the LLMs it manages and share the attribute information with the LLM determination unit 27. This allows the LLM determination unit 27 to obtain information for selecting an appropriate LLM from currently available LLMs. The server 220 may have information on switchable LLMs and servers containing LLMs, or may be configured to obtain information from a separate database.

[0162] In the example shown in Figure 3, the voice input unit 21, image input unit 22, trigger generation unit 40, authentication unit 25, context analysis unit 26, LLM determination unit 27, simple response generation unit 28, response control unit 29, response output unit 30, conversation history management unit 31, and experience information generation unit 32 are all present on the same information processing device 210. These units may be integrated on a chip as the information processing device 210. Alternatively, these units may be distributed and located on a secure server or the like. The information processing system 200 may be realized as a program executed by computers provided in the information processing device 210 and the server 220.

[0163] 4 is a perspective view illustrating the terminal device 230. The terminal device 230 is a portable electronic device equipped with the information processing device 210.

[0164] As shown in Fig. 4, the terminal device 230 includes a mounting unit 235. The mounting unit 235 is a member for mounting the terminal device 230 around the user's neck. The mounting unit 235 has an open-ring shape that can be hooked around the user's neck. With this configuration, the user's movements are less likely to be hindered even when the terminal device 230 is mounted.

[0165] The attachment unit 235 has a microphone 11. Specifically, the microphone 11 is disposed at one end of the attachment unit 235, which has an open ring shape. This position is near the user's mouth when the user wears the terminal device 230 around their neck. Therefore, the user can easily input voice via the microphone 11 while wearing the terminal device 230 around their neck.

[0166] The mounting unit 235 has a camera 12. Specifically, the camera 12 is arranged near the end of the open-ring-shaped mounting unit 235 so as to face outward. This allows the orientation of the camera 12 to roughly match the orientation of the user's face when the user is wearing the terminal device 230 around their neck. This makes it possible for the camera 12 to capture what the user sees.

[0167] The attachment part 235 has a fingerprint sensor 14. Specifically, the fingerprint sensor 14 is disposed at the end of the open-ring-shaped attachment part 235 opposite to the end where the microphone 11 is disposed. This position allows the user to easily touch the fingerprint sensor 14 with their finger when the user is wearing the terminal device 230 around their neck. Therefore, the user can easily perform personal authentication.

[0168] The wearing unit 235 has an output device 15. Specifically, speakers serving as the output device 15 are arranged on each of the left and right sides of the wearing unit 235, which has an open ring shape, when the open portion is facing forward. These speakers are located near the user's ears when the user wears the terminal device 230 around their neck. The user can easily hear the output from the output device 15 while wearing the terminal device 230 around their neck. Furthermore, user instructions regarding the operation of the terminal device 230 may be input to the terminal device 230 via another terminal device, such as a smartphone, that transmits the user's instructions to the terminal device 230. That is, the terminal device 230 may itself be provided with an interface that allows the user to make settings related to the acquisition of sensor information, or the interface may be provided by using a device that can communicate with the terminal device 230.

[0169] [Example of implementation by software] The functions of the information processing device (hereinafter referred to as the "device") can be realized by an information processing program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (in particular, the voice input unit 21, the image input unit 22, the trigger generation unit 40, the authentication unit 25, the context analysis unit 26, the LLM determination unit 27, the simple response generation unit 28, the response control unit 29, the response output unit 30, the conversation history management unit 31, and the experience information generation unit 32).

[0170] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and storage device.

[0171] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0172] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0173] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0174] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment.

[0175] [Cross-reference to related applications] This application claims the benefit of priority to Japanese Patent Application No. 2024-151625, filed on September 3, 2024, the entire contents of which are incorporated herein by reference.

[0176] DESCRIPTION OF SYMBOLS 17 Sensor 21 Voice input unit (language input unit) 22 Image input unit 25 Authentication unit (sensor information input unit) 26 Context analysis unit (request generation unit) 261 Specific instruction management unit 27 LLM determination unit 28 Simple response generation unit (response generation unit) 29 Response control unit 30 Response output unit 40 Trigger generation unit 31 Conversation history management unit 51 First LLM (response generation unit) 52 Second LLM (response generation unit) 100, 200 Information processing system 210 Information processing device 230 Terminal device

Claims

1. An information processing device comprising: a language input unit that accepts language input from a user; a context analysis unit that analyzes the context indicated by the language input; an image input unit that acquires images from a camera; a specific instruction management unit that stores instructions related to the image from the language input as specific instructions based on the results of the context analysis; and a trigger generation unit that emits a trigger signal indicating the timing to execute the specific instruction.

2. The information processing device according to claim 1, wherein the specific instruction is an instruction relating to an image to be executed multiple times.

3. The information processing device according to claim 1 or 2, wherein the trigger generating section generates the trigger signal based on the output of a sensor.

4. An information processing device as described in claim 1 or 2, wherein the trigger generating unit has an image analysis unit with an image analysis function, and the image analysis unit inputs the image and emits the trigger signal at the timing when an object that satisfies the conditions indicated by the specific instruction is recognized in the image.

5. The information processing device according to claim 4, wherein the image analysis unit is a pre-trained convolutional neural network (CNN) or a recurrent neural network (RNN).

6. The information processing device according to claim 4 or 5, wherein the image analysis unit is updated by downloading an update program via a network.

7. An information processing device as claimed in any one of claims 1 to 6, wherein the specific instruction management unit formats the language input into a format that can be processed by a language model, stores the formatted instruction as the specific instruction, and instructs the language model to analyze the image based on the specific instruction when the trigger signal is issued.

8. The information processing device according to claim 7, further comprising a response control unit that receives a response to the specific instruction generated based on the result of analyzing the image, and controls output of the response.

9. An information processing device according to claim 8, further comprising a storage unit that stores the language input corresponding to the specific instruction, image data of the image analyzed based on the specific instruction, and a conversation history including the response regarding the specific instruction.

10. A terminal device comprising the information processing device according to claim 1.

11. A terminal device as described in claim 10, comprising: a microphone that captures the user's voice; a camera that captures the image; and an output device that receives a response to the specific instruction and outputs the response generated by a response control unit that controls the output of the response.

12. An information processing program for causing a computer to function as the information processing device described in claim 1, the information processing program causing a computer to function as the language input unit, the context analysis unit, the image input unit, the specific instruction management unit and the trigger generation unit.

13. An information processing system comprising: a language input unit that accepts language input from a user; a context analysis unit that analyzes the context indicated by the language input; an image input unit that acquires images from a camera; a specific instruction management unit that manages instructions related to the image among the language input as specific instructions based on the results of the context analysis; a trigger generation unit that emits a trigger signal indicating the timing to execute the specific instruction; and a response generation unit that generates a response related to the specific instruction.

14. An information processing system according to claim 13, further comprising a plurality of response generation units, at least one of which is a language model having attributes suitable for the specific instruction.

15. An information processing system as described in claim 14, wherein a language model having attributes suitable for the specific instruction has an attribute tag indicating that it has attributes suitable for the specific instruction, and a response regarding the specific instruction is generated by a language model having the attribute tag.

16. An information processing method executed by a computer, comprising: a step of accepting a user's language input; a step of analyzing a context indicated by the language input; a step of acquiring an image from a camera; a step of storing an instruction related to the image from the language input as a specific instruction based on the analysis result of the context; and a step of issuing a trigger signal indicating the timing to execute the specific instruction.

Images