Information processing device, information processing method, and information processing program

Abstract

To automatically generate a prompt from sensor information.SOLUTION: An information processing device includes: a collection unit that collects sensor information from various sensors; an estimation unit that estimates a context such as the state and situation of a user from the sensor information; a prompt generation unit that generates a context prompt according to the context; a transmission unit that transmits and inputs the context prompt to a language model; and a reception unit that receives an answer according to the context prompt from the language model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] There is disclosed a technique for generating a prompt in which an effective sentence for an input question sentence is added as reference information (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is room for improvement in the above prior art. For example, in the above prior art, it is possible to generate an additional sentence related to the question sentence with a total number of characters that does not exceed the character limit that can be input to the answer generation unit of the large language model (LLM) in accordance with the number of characters of the question sentence, but there is room for improvement in generating an additional sentence that reflects the context such as the state or situation of the user who input the question sentence. Therefore, a technique for automatically generating a prompt from sensor information is proposed.

[0005] The present application has been made in view of the above, and an object thereof is to automatically generate a prompt from sensor information.

Means for Solving the Problems

[0006] The information processing apparatus according to the present application includes a collection unit that collects sensor information from various sensors, an estimation unit that estimates a context related to the state or situation of the user from the collected sensor information, A context information generation unit generates and pre-stores context information indicating the context estimated from the sensor information; an input unit receives a question sentence from the user; a calculation unit calculates the correlation between the input question sentence and the stored context information; and an extraction unit extracts context information from the stored context information that has a high correlation with the question sentence. Context prompt from context information The system generates an auxiliary statement and adds it to the question statement to create a prompt.A prompt generation unit that generates a prompt, The prompt including the aforementioned auxiliary statement and the aforementioned question statement A transmitting unit that sends and inputs to the language model, before From the language model Corresponding to the above auxiliary sentence and the above question It is characterized by comprising a receiving unit that receives a response. [Effects of the Invention]

[0007] According to one embodiment, prompts can be automatically generated from sensor information. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is an explanatory diagram showing an overview of the information processing system according to the embodiment. [Figure 2] Figure 2 shows an example of a question, contextual information, and prompt. [Figure 3] Figure 3 shows an example of the configuration of a terminal device according to this embodiment. [Figure 4] Figure 4 shows an example of the configuration of a server device according to this embodiment. [Figure 5] Figure 5 is a flowchart showing the processing procedure according to the embodiment. [Figure 6] Figure 6 shows an example of a hardware configuration. [Modes for carrying out the invention]

[0009] The following describes in detail, with reference to the drawings, embodiments for implementing the information processing device, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing device, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in the following embodiments, and redundant descriptions are omitted.

[0010] [1. Overview of the Information Processing System] First, with reference to Figure 1, an overview of the information processing system according to the embodiment will be described. Figure 1 is an explanatory diagram showing an overview of the information processing system according to the embodiment. As shown in Figure 1, the information processing system 1 according to the embodiment includes a terminal device 10 and a server device 100. These various devices are connected to each other via a network N, either by wire or wireless means, enabling communication. As a result, the terminal device 10 can cooperate with the server device 100. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0011] Terminal device 10 is an information processing device used by user U. For example, terminal device 10 may be a smart device such as a smartphone or tablet, a mobile phone such as a feature phone, a PC (Personal Computer), a PDA (Personal Digital Assistant), a game console or AV equipment with communication functions, an information appliance or digital appliance, a car navigation system, a wearable device such as a smartwatch, head-mounted display, or smart glasses. Alternatively, terminal device 10 may be a house or building compatible with the Internet of Things (IoT), a car, a home appliance, or an electronic device.

[0012] In this embodiment, the terminal device 10 is a smart device such as a smartphone or tablet used by user U, and is a mobile terminal device capable of communicating with any server device via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation). The terminal device 10 also has a screen such as a liquid crystal display with touch panel functionality, and accepts various operations on displayed data such as content from user U using a finger or stylus, such as tapping, sliding, and scrolling. Operations performed on the area of ​​the screen where content is displayed may also be considered as operations on the content. Furthermore, the terminal device 10 may be an information processing device such as a desktop PC (Personal Computer) or a notebook PC, not just a smart device.

[0013] Furthermore, the terminal device 10 can connect to the network N via wireless communication networks such as LTE, 4G, 5G, Bluetooth®, or wireless LAN (Local Area Network), and communicate with the server device 100.

[0014] The server device 100 is, for example, a computer such as a PC or blade server, or a mainframe or workstation. The server device 100 may also be implemented through cloud computing.

[0015] In this embodiment, the server device 100 is an information processing device that works in conjunction with each user U's terminal device 10 and provides each user U's terminal device 10 with API (Application Programming Interface) services for various applications (hereinafter referred to as "apps") and various data, and is implemented by a computer or cloud system.

[0016] Further, the server device 100 may be an information processing device that provides some kind of web service online to each terminal device 10 of each user U. For example, as a web service, the server device 100 may provide services such as Internet connection, search service, SNS (Social Networking Service), e-commerce (EC: Electronic Commerce), electronic payment, online game, online banking, online trading, accommodation and ticket reservation, video and music distribution, news, map, route search, route guidance, route information, operation information, weather forecast, etc. In reality, the server device 100 may cooperate with various servers that provide the above web services and mediate the web services, or be responsible for the processing of the web services.

[0017] In addition, the server device 100 can acquire user information regarding the user U. For example, the server device 100 acquires information regarding the attributes of the user U such as the gender, age, and residential area of the user U. Then, the server device 100 stores and manages the information regarding the attributes of the user U together with the identification information (such as user ID) indicating the user U.

[0018] In addition, the server device 100 acquires various types of history information (log data) indicating the actions of the user U from the terminal device 10 of the user U or from various servers, etc. based on the user ID, etc. For example, the server device 100 acquires a location history, which is a history of the location and time of the user U, from the terminal device 10. In addition, the server device 100 acquires a search history, which is a history of search queries input by the user U, from a search server (search engine). In addition, the server device 100 acquires a browsing history, which is a history of the content viewed by the user U, from a content server. In addition, the server device 100 acquires a purchase history (settlement history), which is a history of the user U's product purchases and settlement processing, from an e-commerce server or a settlement processing server. In addition, the server device 100 may acquire a listing history or a sales history, which is a history of the user U's listings on the marketplace, from an e-commerce server or a settlement processing server. In addition, the server device 100 acquires a posting history, which is a history of the user U's posts, from a posting server or an SNS server that provides a word-of-mouth posting service. Note that each of the above-mentioned various servers, etc. may be the server device 100 itself. That is, the server device 100 may function as each of the above-mentioned various servers, etc.

[0019] In addition, the number of each device included in the information processing system 1 shown in FIG. 1 is not limited to that shown. For example, in FIG. 1, only one terminal device 10 is shown for simplicity of illustration, but this is merely an example and is not limited, and two or more may be used.

[0020] 〔2. Automatic generation of prompts (instruction sentences) from behavior information (sensor information)〕 Conventionally, there have been problems such as the answers and consultation contents of GPT (Generative Pretrained Transformer), which is a text generation AI (Artificial Intelligence) and a language model capable of generating sentences using natural language processing, not being quite suitable for oneself, and not being able to reach the itchy spot one step further.

[0021] Therefore, in this embodiment, context information indicating the state or situation of user U is automatically generated from user U's behavior information (sensor information) and stored in advance in user U's terminal device 10 (within the device). Then, context information that seems to be related to the question or consultation text (question text) entered by user U into the GPT is extracted (selected), auxiliary text (context prompt) is generated from the extracted context information, a prompt (instruction text) is generated by attaching the auxiliary text to the question text, and the generated prompt is entered into the GPT.

[0022] For example, as shown in Figure 1, the user U's terminal device 10 collects sensor information (location information, motion information, etc.) from various sensors mounted or connected to it (step S1). The user U's terminal device 10 may target only pre-configured sensors as the sensors from which to collect sensor information, or it may target all sensors mounted or connected to the terminal device 10. Furthermore, the user U's terminal device 10 may receive sensor information related to the user U from external sensors.

[0023] Next, the user U's terminal device 10 estimates the context of user U, such as their state or situation, from sensor information (location information, motion information, etc.) (step S2). For example, the user U's terminal device 10 estimates the context of user U, such as "this is what user U is usually like," "this is the current situation," "this is the environment," and "lifestyle." Note that the context of user U's state or situation is not limited to user U's own state or situation, but may also include the state or situation of the environment surrounding user U. Furthermore, user U may be indoors or outdoors, or riding in a vehicle (car, train, etc.).

[0024] Next, the user U's terminal device 10 generates context information indicating the user U's state or situation, and stores it in advance within the device (step S3). The context information is information that expresses the context in natural language. At this time, the user U's terminal device 10 may update the context information as needed. Furthermore, the user U's terminal device 10 may generate not just one instance of context information, but multiple instances. This allows for the accumulation of context information based on the user U's usual behavior, daily life, or attributes.

[0025] Next, the terminal device 10 of user U receives a question from user U (step S4). The question from user U may be a question that user U directly entered or selected, or a question that is automatically entered as a result of user U's statements or actions.

[0026] Next, the user U's terminal device 10 calculates the correlation between the input question (input sentence) and the pre-stored context information (step S5). For example, the user U's terminal device 10 vectorizes (quantifies) the question and context information using natural language processing and calculates an index (such as a correlation coefficient) that shows the strength of the relationship (correlation) between the question and context information in the vector space. At this time, the user U's terminal device 10 may also calculate the correlation between the input question and the pre-stored context information by considering real-time sensor information at the time the question was input. That is, the user U's terminal device 10 may calculate the correlation between real-time sensor information and the question and context information.

[0027] Next, user U's terminal device 10 extracts contextual information highly correlated with the question from the pre-stored contextual information (step S6). Here, contextual information highly correlated with the question refers to, for example, contextual information in which an indicator showing the strength of the relationship (correlation) with the question (such as a correlation coefficient) is above a threshold (or within a predetermined rank). Note that user U's terminal device 10 may extract not only one piece of contextual information, but also multiple pieces of contextual information.

[0028] Next, user U's terminal device 10 generates a supplementary statement (context prompt) based on the extracted context information (step S7).

[0029] Next, user U's terminal device 10 adds the generated auxiliary sentences to the question sentence to generate a series of prompts (instruction sentences) (step S8).

[0030] Next, user U's terminal device 10 sends the prompt to the GPT (step S9). In this embodiment, the GPT is assumed to be running on a server (server device 100). That is, user U's terminal device 10 sends a prompt including the auxiliary sentence and the question to the server device 100 via the network N. As a result, user U's terminal device 10 can provide the GPT with information about the context, such as user U's state or situation that underlies the question, as well as the question itself, without requiring user U to make any input other than the question (additional operations). The GPT can then generate and return an answer (a more appropriate answer) that reflects the context, such as user U's state or situation, that underlies the question.

[0031] Next, user U's terminal device 10 receives a response from GPT corresponding to the prompt it entered (step S10). That is, user U's terminal device 10 receives a response corresponding to the auxiliary sentence and the question sentence from the server device 100 via the network N.

[0032] Figure 2 shows an example of a question, contextual information, and prompts. For example, as shown in Figure 2, user U's terminal device 10 pre-generates and stores contextual information that indicates the context estimated from sensor information, such as "I often go to Ikebukuro," "I'm in Ginza now," "I'm outdoors, the temperature is 35 degrees," and "I'm an outdoorsy person," on the device. Then, when user U's terminal device 10 receives a question from the user, such as "What's a good sweets shop?", it extracts contextual information that is highly correlated with the question from the pre-stored contextual information. Then, user U's terminal device 10 generates a supplementary sentence based on the extracted contextual information, such as "I'm walking outside in Ginza now, and the temperature is around 35 degrees. Please give me some recommendations." Then, user U's terminal device 10 adds the generated supplementary sentence to the question to generate a series of prompts, and feeds the generated prompts into GPT.

[0033] In this embodiment, the user U's terminal device 10 stores contextual information about the context estimated by analyzing sensor information, etc., and converts it into text. First, context estimation by analyzing sensor information starts with two perspectives: "normal state" and "abnormal state". In other words, it is necessary to determine whether the current required context state is "normal state" or "abnormal state". The normal state is often determined from statistical information of sensor data or learned pattern information. The abnormal state refers to a state where the difference from the normal state is large, such as an abnormal value / outlier in the sensor data.

[0034] User U's terminal device 10 stores the usual state (=sensor statistics) as contextual information on various axes. For example, it stores it on axes such as "time," "location," and "state of doing XX." Then, when User U's terminal device 10 generates supplementary text to add to the input question (chat information) from this data, it follows the process of 1) selecting appropriate contextual information and 2) writing the selected contextual information into text.

[0035] The selection of appropriate contextual information hinges on the aforementioned "axis." User U's terminal device 10 selects an "axis" that is closely related to the input question (chat information) and first extracts the contextual information organized and stored along that axis. Then, User U's terminal device 10 broadly classifies the extracted contextual information into three categories: environment, User U's attributes (immutable), and User U's state or situation (variable), and then creates sentences for each classified category. Finally, User U's terminal device 10 adds these supplementary sentences, which are sentences based on the contextual information for each category, to the input question to generate a series of prompts, and then inputs the generated prompts into GPT.

[0036] In this way, the user U's terminal device 10 stores context information based on sensor data, generates a prompt based on the context information along with the question text, and outputs it to the server device 100.

[0037] User U's terminal device 10 utilizes contextual information that correlates (is relevant to) the question. Contextual information includes the user's state, situation, attributes, etc. For example, places the user frequently visits, their current situation, preferences (e.g., outdoor activities), etc.

[0038] User U's terminal device 10 converts sensor information into natural language (context information) that indicates the context and stores it. When a question is input from User U to GPT, User U's terminal device 10 generates a supplementary sentence (context prompt) from the stored context information and outputs the supplementary sentence and the question as a series of prompts to the server device 100. The supplementary sentence is a sentence that explains the user's context related to the question.

[0039] Furthermore, user U's terminal device 10 may input context prompts corresponding to each of the stored context information, along with the question, to the GPT to ask which context information has a high correlation (relevance) with the question. In this case, the GPT may vectorize (digitize) the question and context information using natural language processing, select the context information closest to the question in the vector space, and generate and output an answer corresponding to the selected context information and the question.

[0040] [3. Example of terminal device configuration] Next, the configuration of the terminal device 10 will be explained using Figure 3. Figure 3 is a diagram showing an example of the configuration of the terminal device 10. As shown in Figure 3, the terminal device 10 comprises a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

[0041] (Communications Section 11) The communication unit 11 is connected to the network N by wire or wireless connection and transmits and receives information to and from the server device 100 which has GPT via the network N. For example, the communication unit 11 can be implemented by a NIC (Network Interface Card) or an antenna.

[0042] (Display section 12) The display unit 12 is a display device that displays various information such as location information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). The display unit 12 is a touch panel display, but is not limited to this. In this embodiment, the display unit 12 displays the response from GPT.

[0043] (Input section 13) The input unit 13 is an input device that receives various operations from the user U. For example, the input unit 13 has buttons for inputting characters, numbers, etc. The input unit 13 may also be an input / output port (I / O port) or a USB (Universal Serial Bus) port. If the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. The input unit 13 may also be a microphone that receives voice input from the user U. The microphone may be wireless. In this embodiment, the input unit 13 receives input of a question from the user regarding the GPT.

[0044] (Positioning unit 14) The positioning unit 14 receives signals (radio waves) transmitted from GPS (Global Positioning System) satellites and, based on the received signals, acquires position information (e.g., latitude and longitude) indicating the current position of the terminal device 10. In other words, the positioning unit 14 determines the position of the terminal device 10. Note that GPS is just one example of a GNSS (Global Navigation Satellite System).

[0045] Furthermore, the positioning unit 14 can determine its position using various methods other than GPS. For example, the positioning unit 14 may use various communication functions of the terminal device 10 to determine its position as an auxiliary positioning means for position correction, etc., as described below.

[0046] (Wi-Fi positioning) For example, the positioning unit 14 determines the location of the terminal device 10 by utilizing the Wi-Fi® communication function of the terminal device 10 and the communication network provided by each telecommunications company. Specifically, the positioning unit 14 determines the location of the terminal device 10 by performing Wi-Fi communication, etc., and determining the distance to nearby base stations and access points.

[0047] (Beacon positioning) Furthermore, the positioning unit 14 may determine the location using the Bluetooth® function of the terminal device 10. For example, the positioning unit 14 determines the location of the terminal device 10 by connecting to a beacon transmitter connected via the Bluetooth® function.

[0048] (Geomagnetic positioning) Furthermore, the positioning unit 14 determines the position of the terminal device 10 based on the geomagnetic pattern of the structure, which has been measured in advance, and the geomagnetic sensor provided by the terminal device 10.

[0049] (RFID positioning) Furthermore, if, for example, the terminal device 10 is equipped with an RFID (Radio Frequency Identification) tag function equivalent to that of a contactless IC card used at a train station ticket gate or in a store, or if it is equipped with a function to read RFID tags, the location where it was used will be recorded along with the information on the payment or other transactions made by the terminal device 10. The positioning unit 14 may determine the location of the terminal device 10 by acquiring such information. Alternatively, the location may be determined by an optical sensor or infrared sensor equipped in the terminal device 10.

[0050] The positioning unit 14 may, if necessary, determine the position of the terminal device 10 using one or a combination of the positioning means described above.

[0051] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection can be wired or wireless. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices or wireless devices. In the example shown in Figure 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, a barometric pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.

[0052] The sensors 21-28 described above are merely examples and not limiting. In other words, the sensor unit 20 may be configured to include some of the sensors 21-28, or it may include other sensors such as humidity sensors, human presence sensors, motion sensors, millimeter-wave sensors, inertial measurement units (IMUs), etc., in addition to or instead of the sensors 21-28. Furthermore, the communication unit 11, input unit 13, and positioning unit 14 described above may also be included in the sensor unit 20.

[0053] The acceleration sensor 21 is, for example, a 3-axis acceleration sensor and detects the physical movement of the terminal device 10, such as its direction of movement, velocity, and acceleration. The gyro sensor 22 detects the physical movement of the terminal device 10, such as its tilt in the three axes, based on its angular velocity. The barometric pressure sensor 23 detects the atmospheric pressure around the terminal device 10, for example.

[0054] Since the terminal device 10 is equipped with the acceleration sensor 21, gyroscope 22, barometric pressure sensor 23, etc., it becomes possible to determine the position of the terminal device 10 using technologies such as pedestrian dead-reckoning (PDR) that utilize these sensors 21 to 23. This makes it possible to obtain indoor location information that is difficult to obtain with positioning systems such as GPS.

[0055] For example, a pedometer using an accelerometer 21 can calculate the number of steps, walking speed, and distance walked. Additionally, a gyroscope 22 can be used to determine the user U's direction of movement, gaze direction, and body tilt. Furthermore, the barometric pressure detected by the barometric pressure sensor 23 can be used to determine the altitude and floor number of the user U's terminal device 10.

[0056] The temperature sensor 24 detects, for example, the ambient temperature around the terminal device 10. The sound sensor 25 detects, for example, the ambient sound around the terminal device 10. The light sensor 26 detects the ambient illumination around the terminal device 10. The magnetic sensor 27 detects, for example, the Earth's magnetic field around the terminal device 10. The image sensor 28 captures an image of the area around the terminal device 10.

[0057] The aforementioned pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the surrounding environment and conditions of the terminal device 10 by detecting atmospheric pressure, temperature, sound, and illuminance, respectively, and by capturing images of the surroundings. Furthermore, it becomes possible to improve the accuracy of the location information of the terminal device 10 based on the surrounding environment and conditions.

[0058] (Control Unit 30) The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM, input / output ports, and various circuits. Alternatively, the control unit 30 may be composed of hardware such as an integrated circuit (ASIC) or FPGA (Field Programmable Gate Array). The control unit 30 includes a transmission unit 31, a reception unit 32, and a processing unit 33.

[0059] (Transmitter 31) The transmission unit 31 can transmit various information, such as information input by the user U using the input unit 13, various information detected by sensors 21-28 mounted on or connected to the terminal device 10, and location information of the terminal device 10 determined by the positioning unit 14, to the server device 100 via the communication unit 11.

[0060] (Receiver 32) The receiving unit 32 can receive various information provided by the server device 100 and requests for various information from the server device 100 via the communication unit 11.

[0061] (Processing 33) The processing unit 33 controls the entire terminal device 10, including the display unit 12. For example, the processing unit 33 can output and display various information transmitted by the transmission unit 31 and various information received from the server device 100 by the reception unit 32 to the display unit 12.

[0062] Furthermore, the processing unit 33 may function (operate) as a collection unit 33A, estimation unit 33B, context information generation unit 33C, calculation unit 33D, extraction unit 33E, and prompt generation unit 33F, as shown below, by launching an application or the like.

[0063] (Collection Unit 33A) The data collection unit 33A collects sensor information from various sensors. For example, the data collection unit 33A collects sensor information from each of the sensors 21 to 28 of the sensor unit 20. The data collection unit 33A may also collect information obtained from the communication unit 11, the input unit 13, and the positioning unit 14 as a type of sensor information.

[0064] (Estimation part 33B) The estimation unit 33B estimates the context, such as the state and situation of the user U, from the sensor information. In addition, when estimating the context, the estimation unit 33B determines the usual state from the statistical values ​​of the sensor information.

[0065] (Context information generation unit 33C) The context information generation unit 33C generates and stores context information that indicates the context estimated from the sensor information. The context information generation unit 33C also stores context information that indicates the usual state, along axes such as time, place, or during a predetermined action.

[0066] (Calculation unit 33D) The calculation unit 33D calculates the correlation between the input question and the accumulated contextual information. At this time, the input unit 13 receives the input question from the user.

[0067] For example, the calculation unit 33D vectorizes the question text and context information using natural language processing, and calculates an index that shows the strength of the correlation between the question text and context information in the vector space.

[0068] (Extraction part 33E) The extraction unit 33E extracts contextual information that is highly correlated with the question from the accumulated contextual information. The extraction unit 33E also selects axes that are closely related to the question from the contextual information stored on the axes, and extracts the contextual information stored on the selected axes.

[0069] (Prompt generation unit 33F) The prompt generation unit 33F generates a context prompt according to the context. At this time, the transmission unit 31 sends the context prompt to the language model for input. The receiving unit 32 receives a response from the language model according to the context prompt.

[0070] For example, the prompt generation unit 33F generates an auxiliary sentence as a context prompt from the extracted context information, and adds the auxiliary sentence to the question sentence to generate a prompt. At this time, the transmission unit 31 sends the prompt, including the auxiliary sentence and the question sentence, to the language model. The receiving unit 32 receives the answer corresponding to the auxiliary sentence and the question sentence from the language model.

[0071] Furthermore, the prompt generation unit 33F broadly classifies the extracted context information into environment, user attributes, and user state or situation, and then creates sentences for each classified block. It then adds these sentences containing the context information to the input question sentence to generate a series of prompts.

[0072] Alternatively, the transmission unit 31 may input a context prompt corresponding to each of the stored contextual information items, along with the question, into the language model to query which contextual information items have a high correlation with the question.

[0073] (Storage unit 40) The storage unit 40 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as HDD (Hard Disk Drive), SSD (Solid State Drive), and optical discs. Various programs and various data are stored in this storage unit 40.

[0074] [4. Example of Server Device Configuration] Next, the configuration of the server device 100 according to the embodiment will be described using Figure 4. Figure 4 is a diagram showing an example of the configuration of the server device 100 according to the embodiment. As shown in Figure 4, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0075] (Communications Department 110) The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection.

[0076] (Storage unit 120) The storage unit 120 is implemented by, for example, semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as HDDs, SSDs, and optical discs. The storage unit 120 may store identification information (such as a user ID) indicating user U, as well as attribute information and history information (log data) of user U.

[0077] (Control unit 130) The control unit 130 is a controller, and is realized by executing various programs (corresponding to an example of an information processing program) stored in the internal memory of the server device 100 using a memory area such as RAM as a working area, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array). In the example shown in Figure 4, the control unit 130 has an acquisition unit 131, a reception unit 132, a learning unit 133, an inference unit 134, and a provision unit 135.

[0078] (Acquisition part 131) The acquisition unit 131 acquires the search query entered by the user U. For example, when the user U enters a search query into a search engine or the like and performs a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. In other words, the acquisition unit 131 acquires the keyword entered by the user U into the search box of a search engine, website, or application via the communication unit 110.

[0079] Furthermore, the acquisition unit 131 acquires user information about user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information (such as user ID), location information, and attribute information of user U from user U's terminal device 10. The acquisition unit 131 may also acquire identification information and attribute information of user U when user U is registered. The acquisition unit 131 then stores the user information in the storage unit 120.

[0080] Furthermore, the acquisition unit 131 acquires various historical information (log data) indicating the user U's actions via the communication unit 110. For example, the acquisition unit 131 acquires various historical information indicating the user U's actions from the user U's terminal device 10, or from various servers based on the user ID, etc. The acquisition unit 131 then stores the various historical information in the storage unit 120.

[0081] (Reception desk 132) The reception unit 132 receives prompts (instructions) to be sent to the GPT (Generative Pretrained Transformer) from each user U's terminal device 10 via the communication unit 110. Note that the reception unit 132 may be part of the acquisition unit 131 described above.

[0082] (Learning Section 133) The learning unit 133 generates and constructs the GPT used by the server device 100 using machine learning. In practice, it is not limited to GPT; large language models (LLMs) with similar functionality may also be used.

[0083] (Inference part 134) The inference unit 134 uses GPT to infer an appropriate response to a prompt input to the large-scale language model. For example, the inference unit 134 inputs a prompt to GPT and outputs an appropriate response corresponding to that prompt.

[0084] (Provider 135) The provisioning unit 135 provides each user U's terminal device 10 with an appropriate response (inferred response) corresponding to the input prompt via the communication unit 110. The provisioning unit 135 may also provide each user U's terminal device 10 with a free version, paid version, trial version of GPT, or a plugin usable with GPT, etc., via the communication unit 110.

[0085] [5. Processing Procedure] Next, the processing procedure by the user U's terminal device 10 according to this embodiment will be explained using Figure 5. Figure 5 is a flowchart of the processing procedure according to this embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 30 of the user U's terminal device 10.

[0086] For example, as shown in Figure 5, the collection unit 33A of the user U's terminal device 10 collects sensor information output from various sensors mounted on or connected to the terminal device 10 (step S101).

[0087] Next, the estimation unit 33B of the user U's terminal device 10 estimates the context, such as the state and situation of the user U, from the collected sensor information (step S102).

[0088] Next, the context information generation unit 33C of the user U's terminal device 10 generates and stores context information indicating the context estimated from the sensor information (step S103). The context information is information that expresses the context in natural language.

[0089] Next, the input unit 13 of user U's terminal device 10 receives a question from the user (step S104).

[0090] Next, the calculation unit 33D of user U's terminal device 10 calculates the correlation between the input question and the accumulated contextual information (step S105).

[0091] Next, the extraction unit 33E of user U's terminal device 10 extracts contextual information from the accumulated contextual information that is highly correlated with the question (step S106).

[0092] Next, the prompt generation unit 33F of user U's terminal device 10 generates an auxiliary sentence as a context prompt from the extracted context information, and adds the auxiliary sentence to the question sentence to generate a prompt (step S107).

[0093] Next, the transmission unit 31 of user U's terminal device 10 sends a prompt including the auxiliary text and the question text to the GPT on the server device 100 via the communication unit 11 (step S108).

[0094] Next, the receiving unit 32 of user U's terminal device 10 receives the answers corresponding to the auxiliary text and question text from the GPT on the server device 100 via the communication unit 11 and displays them on the display unit 12 of the terminal device 10 (step S109).

[0095] [6. Variant Example] The terminal device 10 and server device 100 described above may be implemented in various other forms besides those of the embodiment described above. Therefore, the following describes modifications of the embodiment.

[0096] In the above embodiment, some or all of the processing performed by the server device 100 may actually be performed by the terminal device 10 (or an application running on the terminal). For example, the processing may be completed in a standalone manner (by the terminal device 10 alone). In this case, the terminal device 10 is assumed to have the functions of the server device 100 in the above embodiment. Furthermore, in the above embodiment, since the terminal device 10 is in cooperation with the server device 100, from the perspective of the user U, it appears as if the processing of the server device 100 is also being performed by the terminal device 10. In other words, from another perspective, it can be said that the terminal device 10 is equipped with the server device 100.

[0097] Furthermore, in the above embodiment, the user U's terminal device 10 generates an auxiliary sentence (context prompt) from context information indicating the context of user U's state or situation, generates a prompt (instruction sentence) by attaching the auxiliary sentence to the question sentence, and inputs the generated prompt into the GPT. However, in practice, only the auxiliary sentence (context prompt) may be input into the GPT. For example, the user U's terminal device 10 may automatically extract context information indicating the context of user U's state or situation from sensor information automatically collected in real time, automatically generate an auxiliary sentence (context prompt) from the automatically extracted context information, and automatically input the automatically generated auxiliary sentence (context prompt) into the GPT.

[0098] Furthermore, in the above embodiment, the user U's terminal device 10 stores the usual state (=sensor statistics) on various axes as contextual information, but in reality, it may also store abnormal states (=sensor outliers) on various axes as contextual information. For example, the user U's terminal device 10 may store contextual information indicating an abnormal state on axes such as time, place, or during a predetermined action, as contextual information indicating a unique state or situation that is different from the usual. Then, when contextual information indicating an abnormal state is required from the content of the question, the user U's terminal device 10 selects an axis that is closely related to the question from among the axes related to abnormal states and extracts the contextual information stored on the selected axis.

[0099] [7. Effects] As described above, the information processing device (terminal device 10 and server device 100) according to the present invention is characterized by comprising: an information collection unit 33A that collects sensor information from various sensors; an estimation unit 33B that estimates context such as the user's state and situation from the sensor information; a prompt generation unit 33F that generates a context prompt according to the context; a transmission unit 31 that transmits and inputs the context prompt to a language model; and a receiving unit 32 that receives a response from the language model according to the context prompt.

[0100] Furthermore, the information processing device according to the present invention further comprises: a context information generation unit 33C that generates and pre-stores context information indicating the context estimated from sensor information; an input unit 13 that receives input of a question sentence from a user; a calculation unit 33D that calculates the correlation between the input question sentence and the stored context information; and an extraction unit 33E that extracts context information from the stored context information that has a high correlation with the question sentence. The prompt generation unit 33F generates an auxiliary sentence as a context prompt from the extracted context information and adds the auxiliary sentence to the question sentence to generate a prompt. The transmission unit 31 transmits the prompt, including the auxiliary sentence and the question sentence, to the language model. The reception unit 32 receives the answer corresponding to the auxiliary sentence and the question sentence from the language model.

[0101] The calculation unit 33D vectorizes the question text and context information using natural language processing, and calculates an index that indicates the strength of the correlation between the question text and context information in the vector space.

[0102] When estimating context, the estimation unit 33B determines the usual state from the statistical values ​​of the sensor information. The context information generation unit 33C stores context information indicating the usual state on axes such as time, place, or during a predetermined action. The extraction unit 33E selects axes that are closely related to the question and extracts the context information stored on the selected axes.

[0103] The prompt generation unit 33F broadly classifies the extracted context information into environment, user attributes, and user state or situation, and then creates sentences for each classified block. It then adds these sentences containing the context information to the input question sentence to generate a series of prompts.

[0104] The transmission unit 31 sends the question along with a context prompt corresponding to each of the stored contextual information to the language model, and queries the language model to determine which contextual information is highly correlated with the question.

[0105] By any or a combination of the above-described processes, the information processing device according to the present invention can automatically generate prompts from sensor information.

[0106] [8. Hardware Configuration] Furthermore, the terminal device 10 and server device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 6. The following explanation will use the terminal device 10 as an example. Figure 6 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0107] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0108] The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as USB (Universal Serial Bus) memory or SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), file server, etc.

[0109] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010, such as displays, projectors, and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners, and is implemented using, for example, USB.

[0110] Furthermore, the output interface 1060 and input interface 1070 may be wirelessly connected to the output device 1010 and input device 1020, respectively. In other words, the output device 1010 and input device 1020 may be wireless devices.

[0111] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0112] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0113] The network interface 1080 receives data from other devices via network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via network N.

[0114] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0115] For example, when computer 1000 functions as terminal device 10, the arithmetic unit 1030 of computer 1000 realizes the functions of control unit 30 by executing a program loaded onto primary storage device 1040. Alternatively, computer 1000's arithmetic unit 1030 may load a program obtained from another device via network interface 1080 onto primary storage device 1040 and execute the loaded program. Furthermore, computer 1000's arithmetic unit 1030 may cooperate with other devices via network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0116] [9. Other] Although embodiments of the present invention have been described above, the present invention is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above.

[0117] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0118] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0119] For example, the server device 100 described above may be implemented using multiple server computers, and the configuration can be flexibly changed, such as by calling external platforms via APIs (Application Programming Interfaces) or network computing depending on the function.

[0120] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent.

[0121] Furthermore, the terms "section, module, unit" mentioned above can be replaced with "means" or "circuit," etc. For example, the acquisition unit can be replaced with acquisition means or acquisition circuit. [Explanation of symbols]

[0122] 1. Information Processing System 10 Terminal devices 11 Communications Department 12 Display section 13 Input section 14 Positioning Unit 20 Sensor section 31 Transmitter 32 Receiver 33 Processing Unit 33A Collection Department 33B Estimation part 33C Context Information Generation Unit 33D Calculation Unit 33E Extraction part 33F Prompt generation unit 100 Server Devices 110 Communications Department 120 Storage section 130 Control Unit 131 Acquisition Department 132 Reception Department 133 Learning Department 134 Reasoning part 135 Provision Department

Claims

1. A collection unit that collects sensor information from various sensors, An estimation unit that estimates the context of the user's state or situation from the collected sensor information, A context information generation unit generates and pre-stores context information indicating the context estimated from the sensor information, An input section that accepts question text from the user, A calculation unit that calculates the correlation between the input question text and the accumulated contextual information, An extraction unit extracts contextual information from the accumulated contextual information that is highly correlated with the question text, A prompt generation unit generates auxiliary sentences as context prompts from extracted context information, and adds the auxiliary sentences to the question sentence to generate a prompt; A transmission unit that sends and inputs the prompt, including the auxiliary sentence and the question sentence, to a language model, A receiving unit that receives answers corresponding to the auxiliary sentence and the question sentence from the language model, An information processing device characterized by comprising:

2. The calculation unit vectorizes the question text and the context information using natural language processing, and calculates an index that indicates the strength of the correlation between the question text and the context information in the vector space. The information processing apparatus according to feature 1.

3. When estimating the context, the estimation unit determines the usual state from the statistical values ​​of the sensor information. The context information generation unit stores context information that represents the usual state, along axes such as time, place, or during a predetermined action, as the context information. The extraction unit selects axes that are closely related to the question and extracts contextual information stored on the selected axes. The information processing apparatus according to feature 1.

4. The prompt generation unit broadly classifies the extracted contextual information into environment, user attributes, and user state or situation, creates sentences for each classified block, and adds these sentences containing the contextual information to the input question sentence to generate a series of prompts. The information processing apparatus according to claim 3.

5. The transmission unit, along with the question, inputs a contextual prompt corresponding to each of the stored contextual pieces of information to the language model, and queries the language model to determine which contextual pieces of information have a high correlation with the question. The information processing apparatus according to feature 1.

6. An information processing method performed by an information processing device, A data collection process that collects sensor information from various sensors, An estimation step of estimating the context of the user's state or situation from the collected sensor information, A context information generation step involves generating and pre-storing context information that indicates the context estimated from the sensor information, An input process that receives the question text from the user, A calculation process to calculate the correlation between the input question text and the accumulated contextual information, An extraction process for extracting contextual information that is highly correlated with the question from the accumulated contextual information, A prompt generation step involves generating an auxiliary sentence as a context prompt from the extracted context information, and adding the auxiliary sentence to the question sentence to generate a prompt. A transmission step of sending and inputting the prompt, which includes the auxiliary sentence and the question sentence, to a language model; A receiving step of receiving answers corresponding to the auxiliary sentence and the question sentence from the language model, An information processing method characterized by including

7. A collection procedure for collecting sensor information from various sensors, An estimation procedure for estimating the context of the user's state or situation from the collected sensor information, A context information generation procedure for generating and pre-storing context information indicating the context estimated from the sensor information, An input procedure for receiving user questions, A calculation procedure for determining the correlation between the input question text and the accumulated contextual information, An extraction procedure for extracting contextual information that is highly correlated with the aforementioned question from the accumulated contextual information, A prompt generation procedure that generates a supplementary sentence as a context prompt from extracted context information, and adds the supplementary sentence to the question sentence to generate a prompt, A transmission procedure for sending and inputting the prompt, which includes the auxiliary sentence and the question sentence, to a language model; A receiving procedure for receiving answers corresponding to the auxiliary sentence and the question sentence from the language model, An information processing program characterized by causing a computer to execute it.

Images