Information processing device, information processing method, and information processing program

Abstract

To automatically generate a prompt from sensor information.SOLUTION: An information processing device includes: a collection unit that collects sensor information from various sensors; an estimation unit that estimates a context such as the state and situation of a user from the sensor information; a prompt generation unit that generates a context prompt according to the context; a transmission unit that transmits and inputs the context prompt to a language model; and a reception unit that receives an answer according to the context prompt from the language model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] A technique has been disclosed for generating a prompt that includes a valid sentence as reference information in response to an input question sentence (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7325152 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is room for improvement in the above conventional technology. For example, the above conventional technology can generate additional text related to a question text with a total number of characters, including the number of characters in the question text, that does not exceed the limit on the number of characters that can be input to the answer generation section of a large-scale language model (LLM), but there is room for improvement in generating additional text that reflects the context, such as the state or situation, of the user who input the question text. Therefore, we propose a technology to automatically generate prompts from sensor information.

[0005] The present application has been made in consideration of the above, and has an object to automatically generate prompts from sensor information. [Means for solving the problem]

[0006] The information processing device of the present application is characterized by comprising a collection unit that collects sensor information from various sensors, an estimation unit that estimates a context such as a user's state or situation from the sensor information, a prompt generation unit that generates a context prompt according to the context, a transmission unit that transmits the context prompt to a language model and inputs it into the language model, and a receiving unit that receives an answer according to the context prompt from the language model. Effect of the Invention

[0007] According to one aspect of the embodiment, prompts may be automatically generated from the sensor information. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of an information processing system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a question, context information, and a prompt. [Diagram 3] FIG. 3 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a server device according to the embodiment. [Diagram 5] FIG. 5 is a flowchart showing a processing procedure according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to the embodiments. In addition, the same components in the following embodiments are denoted by the same reference numerals, and duplicated descriptions are omitted.

[0010] [1. Overview of the information processing system] First, an overview of an information processing system according to an embodiment will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram showing an overview of an information processing system according to an embodiment. As shown in Fig. 1, an information processing system 1 according to an embodiment includes a terminal device 10 and a server device 100. These various devices are connected to each other via a network N in a wired or wireless manner so as to be able to communicate with each other. This allows the terminal device 10 to cooperate with the server device 100. The network N is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network) such as the Internet.

[0011] The terminal device 10 is an information processing device used by a user U. For example, the terminal device 10 is a smart device such as a smartphone or a tablet terminal, a mobile phone such as a feature phone, a personal computer (PC), a personal digital assistant (PDA), a game machine or AV device equipped with a communication function, an information appliance or digital appliance, a car navigation system, a smart watch or a head-mounted display, a wearable device such as smart glasses, etc. The terminal device 10 may also be a house or building, a car, a home appliance, an electronic device, etc. that supports the Internet of Things (IOT).

[0012] In this embodiment, the terminal device 10 is a smart device such as a smartphone or tablet terminal used by a user U, and is a mobile terminal device that can communicate with any server device via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5th generation mobile communication system). The terminal device 10 has a screen such as a liquid crystal display having a touch panel function, and accepts various operations on display data such as content, such as tapping, sliding, and scrolling, performed by a finger or a stylus from the user U. An operation performed on an area of ​​the screen where the content is displayed may be regarded as an operation on the content. The terminal device 10 may be not only a smart device, but also an information processing device such as a desktop PC (Personal Computer) or a notebook PC.

[0013] In addition, the terminal device 10 can connect to the network N via a wireless communication network such as LTE, 4G, or 5G, or a short-range wireless communication such as Bluetooth (registered trademark) or a wireless LAN (Local Area Network), and communicate with the server device 100.

[0014] The server device 100 is, for example, a computer such as a PC or a blade server, or a mainframe or a workstation, etc. The server device 100 may be realized by cloud computing.

[0015] In this embodiment, the server device 100 is an information processing device that cooperates with the terminal device 10 of each user U and provides API (Application Programming Interface) services for various applications (hereinafter, apps), etc., as well as various data, to the terminal device 10 of each user U, and is realized by a computer, a cloud system, etc.

[0016] The server device 100 may also be an information processing device that provides some kind of online Web service to the terminal device 10 of each user U. For example, the server device 100 may provide services such as Internet connection, search service, SNS (Social Networking Service), electronic commerce (EC), electronic payment, online games, online banking, online trading, hotel and ticket reservations, video and music distribution, news, maps, route search, route guidance, line information, operation information, and weather forecasts as Web services. In practice, the server device 100 may cooperate with various servers that provide the above-mentioned Web services, and may mediate the Web services or be responsible for processing the Web services.

[0017] The server device 100 can acquire user information about the user U. For example, the server device 100 acquires information about the attributes of the user U, such as the gender, age, and residential area of ​​the user U. The server device 100 then stores and manages the information about the attributes of the user U together with identification information (such as a user ID) indicating the user U.

[0018] The server device 100 also acquires various types of history information (log data) indicating the actions of the user U from the terminal device 10 of the user U or from various servers based on the user ID or the like. For example, the server device 100 acquires a location history, which is a history of the location and date and time of the user U, from the terminal device 10. The server device 100 also acquires a search history, which is a history of search queries input by the user U, from a search server (search engine). The server device 100 also acquires a browsing history, which is a history of content browsed by the user U, from a content server. The server device 100 also acquires a purchase history (payment history), which is a history of product purchases and payment processing by the user U, from an electronic commerce server or a payment processing server. The server device 100 may also acquire a listing history, which is a history of listings on the marketplace by the user U, and a sales history, from the electronic commerce server or the payment processing server. The server device 100 also acquires a posting history, which is a history of posts by the user U, from a posting server that provides a word-of-mouth posting service or an SNS server. The above-mentioned various servers and the like may be the server device 100 itself. That is, the server device 100 may function as the above-mentioned various servers and the like.

[0019] Furthermore, the number of devices included in the information processing system 1 shown in Fig. 1 is not limited to that shown in the figure. For example, in Fig. 1, for the sake of simplicity, only one terminal device 10 is shown, but this is merely an example and is not limiting, and two or more devices may be included.

[0020] [2. Automatic generation of prompts (instructions) from behavioral information (sensor information)] In the past, there were problems with the answers and questions provided by GPT (Generative Pretrained Transformer), a text generation AI (Artificial Intelligence) and language model capable of generating sentences using natural language processing, not being quite suited to the user or not addressing the user's specific needs.

[0021] Therefore, in this embodiment, context information indicating a context such as the state or situation of user U is automatically generated from the behavioral information (sensor information) of user U, and is stored in advance in the terminal device 10 (within the device) of user U. Then, context information likely to be related to the question or consultation text (question text) entered by user U into the GPT is extracted (selected), an auxiliary text (context prompt) is generated from the extracted context information, a prompt (instruction text) is generated by adding the auxiliary text to the question text, and the generated prompt is entered into the GPT.

[0022] 1, for example, the terminal device 10 of the user U collects sensor information (position information, movement information, etc.) from various sensors mounted or connected thereto (step S1). Note that the terminal device 10 of the user U may collect sensor information from only preset sensors, or from all sensors mounted or connected to the terminal device 10. The terminal device 10 of the user U may also receive sensor information related to the user U from an external sensor.

[0023] Next, the terminal device 10 of the user U estimates a context, such as the state or situation of the user U, from the sensor information (location information, movement information, etc.) (step S2). For example, the terminal device 10 of the user U estimates context, such as "what kind of person the user U is usually," "what kind of situation he is in now," "what kind of environment he is in," and "lifestyle," for the user U. Note that the context, such as the state or situation of the user U, is not limited to the state or situation of the user U himself, but may also be a context, such as the state or situation of the environment surrounding the user U. Furthermore, the user U may be indoors or outdoors, or may be on a vehicle (car, train, etc.).

[0024] Next, the terminal device 10 of the user U generates context information indicating a context such as the state or situation of the user U, and stores the generated context information in advance in the device (step S3). The context information is information that expresses the context in natural language. At this time, the terminal device 10 of the user U may update the context information as needed. Furthermore, the terminal device 10 of the user U may generate multiple pieces of context information, not limited to one piece of context information. This makes it possible to accumulate context information based on the usual behavior, daily life, attributes, etc. of the user U.

[0025] Next, the terminal device 10 of the user U accepts input of a question from the user U (step S4). The question from the user U may be a question directly input or selected by the user U himself / herself, or may be a question automatically input due to a statement or action of the user U, etc.

[0026] Next, the terminal device 10 of the user U calculates the correlation between the input question sentence (input sentence) and the pre-stored context information (step S5). For example, the terminal device 10 of the user U vectorizes (quantifies) the question sentence and the context information by natural language processing, and calculates an index (such as a correlation coefficient) indicating the strength of the association (correlation) between the question sentence and the context information in the vector space. At this time, the terminal device 10 of the user U may calculate the correlation between the input question sentence and the pre-stored context information, taking into account real-time sensor information at the time the question sentence is input. In other words, the terminal device 10 of the user U may calculate the correlation between the real-time sensor information and the question sentence, and the context information.

[0027] Next, the terminal device 10 of the user U extracts context information that is highly correlated with the question sentence from the pre-stored context information (step S6). Here, the context information that is highly correlated with the question sentence refers to, for example, context information whose index (such as a correlation coefficient) indicating the strength of the relevance (correlation) with the question sentence is equal to or greater than a threshold (or within a predetermined rank). Note that the terminal device 10 of the user U may extract multiple pieces of context information, not limited to one piece of context information.

[0028] Next, the terminal device 10 of the user U generates an auxiliary sentence (context prompt) based on the extracted context information (step S7).

[0029] Next, the terminal device 10 of the user U generates a series of prompts (instructions) by adding the generated auxiliary sentences to the question sentence (step S8).

[0030] Next, the terminal device 10 of the user U inputs the prompt into the GPT (step S9). In this embodiment, it is assumed that the GPT runs on a server (server device 100). That is, the terminal device 10 of the user U transmits a prompt including an auxiliary sentence and a question sentence to the server device 100 via the network N. This allows the terminal device 10 of the user U to provide the GPT with information on not only the question entered by the user U, but also the context, such as the state or situation of the user U, that is the background of the question, without requiring the user U to enter anything other than the question (additional operation). The GPT can then generate and return an answer (more appropriate answer) to the question entered by the user U that reflects the context, such as the state or situation of the user U, that is the background of the question.

[0031] Next, the terminal device 10 of the user U receives an answer corresponding to the input prompt from the GPT (step S10). That is, the terminal device 10 of the user U receives an answer corresponding to the auxiliary sentence and the question sentence from the server device 100 via the network N.

[0032] FIG. 2 is a diagram showing an example of a question sentence, context information, and a prompt. For example, as shown in FIG. 2, the terminal device 10 of the user U generates and stores information such as "I often go to Ikebukuro," "I'm currently in Ginza," "I'm outdoors, the temperature is 35 degrees," and "I'm an outdoorsy person" on the device in advance as context information indicating a context estimated from sensor information. Then, when the terminal device 10 of the user U inputs a question sentence of a user such as "What sweets shop do you recommend?", the terminal device 10 of the user U extracts context information that is highly correlated with the question sentence from the context information stored in advance. Then, the terminal device 10 of the user U generates a sentence such as "Please give me some recommended information for someone who is currently walking outside the town of Ginza, where the temperature is about 35 degrees," as an auxiliary sentence based on the extracted context information. Then, the terminal device 10 of the user U adds the generated auxiliary sentence to the question sentence to generate a series of prompts, and inputs the generated prompts into the GPT.

[0033] In this embodiment, the terminal device 10 of the user U saves context information on the context estimated by analyzing sensor information, etc., and documents it. First, in estimating the context by analyzing the sensor information, the first two viewpoints are "usual state" and "abnormal state." In other words, it is necessary to determine whether the state of the context currently required is "usual state" or "abnormal state." The usual state is often obtained from statistical information of the sensor data or learned pattern information. An abnormal state refers to a state that is significantly different from the usual state, such as an abnormal value / outlier of the sensor data.

[0034] The terminal device 10 of the user U stores the usual state (=sensor statistics) on various axes as context information. For example, it stores the states on axes such as "time," "place," and "state of doing XX." Then, when the terminal device 10 of the user U generates an auxiliary text to be added to the input question text (chat information) from among these, the steps are 1) selecting appropriate context information, and 2) converting the selected context information into text.

[0035] The above-mentioned "axis" is the key to selecting appropriate context information. User U's terminal device 10 selects an "axis" that is closely related to the input question (chat information), and first extracts the context information organized and saved on that axis. User U's terminal device 10 then broadly classifies the extracted context information into environment, attributes of user U (what is unchanging), and state or situation of user U (what is variable), and creates sentences for each classified chunk. User U's terminal device 10 then generates a series of prompts by attaching auxiliary sentences, which are sentences made by dividing the context information into chunks (classifications), to the input question, and inputs the generated prompts into GPT.

[0036] In this way, the terminal device 10 of the user U accumulates context information based on sensor information, and generates a prompt based on the context information together with a question text, and outputs the prompt to the server device 100.

[0037] The terminal device 10 of the user U uses context information that is correlated (relevant) to the question sentence. The context information is the user's state, situation, attributes, etc. For example, the user's favorite places, current situation, preferences (outdoors, etc.), etc.

[0038] The terminal device 10 of the user U converts the sensor information into natural language (context information) indicating the context and stores it. When the user U inputs a question to the GPT, the terminal device 10 of the user U generates an auxiliary sentence (context prompt) from the stored context information and outputs the auxiliary sentence and the question sentence as a series of prompts to the server device 100. The auxiliary sentence is a sentence that explains the user's context related to the question.

[0039] The terminal device 10 of the user U may input a context prompt corresponding to each of all the accumulated context information together with the question text into the GPT to ask the GPT which context information has a high correlation (relevance) with the question text. At this time, the GPT may vectorize (quantify) the question text and the context information by natural language processing, adopt the context information close to the question text in the vector space, and generate and output an answer corresponding to the adopted context information and the question text.

[0040] [3. Example of terminal device configuration] Next, the configuration of the terminal device 10 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the terminal device 10. As shown in Fig. 3, the terminal device 10 includes a communication unit 11, a display unit 12, an input unit 13, a positioning unit 14, a sensor unit 20, a control unit 30 (controller), and a storage unit 40.

[0041] (Communications Department 11) The communication unit 11 is connected to a network N by wire or wirelessly, and transmits and receives information to and from a server device 100 having a GPT via the network N. For example, the communication unit 11 is realized by a NIC (Network Interface Card), an antenna, or the like.

[0042] (Display section 12) The display unit 12 is a display device that displays various information such as position information. For example, the display unit 12 is a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). The display unit 12 is also a touch panel display, but is not limited to this. In this embodiment, the display unit 12 displays a response from the GPT.

[0043] (Input section 13) The input unit 13 is an input device that accepts various operations from the user U. For example, the input unit 13 has buttons for inputting characters, numbers, and the like. The input unit 13 may be an input / output port (I / O port), a USB (Universal Serial Bus) port, or the like. In addition, when the display unit 12 is a touch panel display, a part of the display unit 12 functions as the input unit 13. In addition, the input unit 13 may be a microphone that accepts voice input from the user U. The microphone may be wireless. In this embodiment, the input unit 13 accepts input of a question sentence from the user to the GPT.

[0044] (Positioning unit 14) The positioning unit 14 receives a signal (radio wave) transmitted from a satellite of a GPS (Global Positioning System), and acquires position information (e.g., latitude and longitude) indicating the current position of the terminal device 10, which is the device itself, based on the received signal. That is, the positioning unit 14 measures the position of the terminal device 10. Note that the GPS is merely an example of a GNSS (Global Navigation Satellite System).

[0045] The positioning unit 14 can measure the position by various methods other than GPS. For example, the positioning unit 14 may measure the position by using various communication functions of the terminal device 10 as described below as an auxiliary positioning means for position correction or the like.

[0046] (Wi-Fi positioning) For example, the positioning unit 14 uses a Wi-Fi (registered trademark) communication function of the terminal device 10 or a communication network provided by each communication company to measure the position of the terminal device 10. Specifically, the positioning unit 14 performs Wi-Fi communication or the like and measures the distance to a nearby base station or access point to measure the position of the terminal device 10.

[0047] (Beacon positioning) The positioning unit 14 may measure the position by using a Bluetooth (registered trademark) function of the terminal device 10. For example, the positioning unit 14 measures the position of the terminal device 10 by connecting to a beacon transmitter connected by the Bluetooth (registered trademark) function.

[0048] (geomagnetic positioning) The positioning unit 14 also measures the position of the terminal device 10 based on a geomagnetic pattern of a structure measured in advance and a geomagnetic sensor included in the terminal device 10.

[0049] (RFID positioning) Furthermore, for example, if the terminal device 10 has a function of an RFID (Radio Frequency Identification) tag equivalent to a contactless IC card used at station ticket gates, stores, etc., or has a function of reading an RFID tag, the location of use is recorded together with information on a payment or the like made by the terminal device 10. The positioning unit 14 may obtain such information to measure the location of the terminal device 10. Furthermore, the location may be measured by an optical sensor, an infrared sensor, or the like provided in the terminal device 10.

[0050] The positioning unit 14 may, if necessary, measure the position of the terminal device 10 using one or a combination of the above-mentioned positioning means.

[0051] (Sensor unit 20) The sensor unit 20 includes various sensors mounted on or connected to the terminal device 10. The connection may be wired or wireless. For example, the sensors may be detection devices other than the terminal device 10, such as wearable devices and wireless devices. In the example shown in FIG. 3, the sensor unit 20 includes an acceleration sensor 21, a gyro sensor 22, an air pressure sensor 23, a temperature sensor 24, a sound sensor 25, a light sensor 26, a magnetic sensor 27, and an image sensor (camera) 28.

[0052] The above-mentioned sensors 21 to 28 are merely examples and are not limited to the above. That is, the sensor unit 20 may be configured to include some of the sensors 21 to 28, or may include other sensors such as a humidity sensor, a human sensor, a motion sensor, a millimeter wave sensor, an inertial measurement unit (IMU), etc. in addition to or instead of the sensors 21 to 28. The communication unit 11, the input unit 13, and the positioning unit 14 may also be included in the sensor unit 20.

[0053] The acceleration sensor 21 is, for example, a three-axis acceleration sensor, and detects physical movements of the terminal device 10, such as the moving direction, speed, and acceleration of the terminal device 10. The gyro sensor 22 detects physical movements of the terminal device 10, such as tilts in three axial directions, based on the angular velocity, etc. of the terminal device 10. The air pressure sensor 23 detects, for example, the air pressure around the terminal device 10.

[0054] Since the terminal device 10 includes the above-mentioned acceleration sensor 21, gyro sensor 22, and air pressure sensor 23, it is possible to measure the position of the terminal device 10 using a technique such as Pedestrian Dead-Reckoning (PDR) that uses these sensors 21 to 23. This makes it possible to obtain indoor position information that is difficult to obtain using a positioning system such as GPS.

[0055] For example, the number of steps, walking speed, and distance walked can be calculated by a pedometer using the acceleration sensor 21. In addition, the moving direction, line of sight direction, and body inclination of the user U can be known by using the gyro sensor 22. In addition, the altitude and floor number on which the terminal device 10 of the user U is located can be known from the air pressure detected by the air pressure sensor 23.

[0056] The air temperature sensor 24 detects, for example, the air temperature around the terminal device 10. The sound sensor 25 detects, for example, the sound around the terminal device 10. The light sensor 26 detects the illuminance around the terminal device 10. The magnetic sensor 27 detects, for example, the geomagnetism around the terminal device 10. The image sensor 28 captures an image around the terminal device 10.

[0057] The above-mentioned air pressure sensor 23, temperature sensor 24, sound sensor 25, light sensor 26, and image sensor 28 can detect the air pressure, temperature, sound, and illuminance, and capture images of the surroundings, thereby detecting the environment and situation around the terminal device 10. In addition, the accuracy of the location information of the terminal device 10 can be improved based on the environment and situation around the terminal device 10.

[0058] (Control unit 30) The control unit 30 includes, for example, a microcomputer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM, an input / output port, and various other circuits. The control unit 30 may also be configured with hardware such as an integrated circuit, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 includes a transmitting unit 31, a receiving unit 32, and a processing unit 33.

[0059] (Transmitter 31) The transmission unit 31 can transmit, for example, various information input by the user U using the input unit 13, various information detected by each sensor 21-28 mounted on or connected to the terminal device 10, and location information of the terminal device 10 measured by the positioning unit 14 to the server device 100 via the communication unit 11.

[0060] (Receiving unit 32) The receiving unit 32 can receive various types of information provided by the server device 100 and requests for various types of information from the server device 100 via the communication unit 11.

[0061] (Processing section 33) The processing unit 33 controls the entire terminal device 10, including the display unit 12, etc. For example, the processing unit 33 can output various information transmitted by the transmission unit 31 and various information received from the server device 100 by the reception unit 32 to the display unit 12 for display.

[0062] In addition, the processing unit 33 may function (operate) as a collection unit 33A, an estimation unit 33B, a context information generation unit 33C, a calculation unit 33D, an extraction unit 33E, and a prompt generation unit 33F, as described below, by launching an app or the like.

[0063] (Collection section 33A) The collection unit 33A collects sensor information from various sensors. For example, the collection unit 33A collects sensor information from the sensors 21 to 28 of the sensor unit 20. The collection unit 33A may also collect information obtained from the communication unit 11, the input unit 13, and the positioning unit 14 as a type of sensor information.

[0064] (Estimation part 33B) The estimation unit 33B estimates a context, such as a state or situation of the user U (user), from the sensor information. When estimating the context, the estimation unit 33B determines a usual state from a statistical value of the sensor information.

[0065] (Context information generating unit 33C) The context information generating unit 33C generates context information indicating a context estimated from the sensor information and stores the generated context information in advance. In addition, the context information generating unit 33C stores, as the context information, context information indicating a usual state along an axis such as a time, a place, or during a predetermined action.

[0066] (Calculation unit 33D) The calculation unit 33D calculates the correlation between the input question sentence and the accumulated context information. At this time, the input unit 13 accepts the input of the question sentence from the user.

[0067] For example, the calculation unit 33D vectorizes each of the question sentence and the context information by natural language processing, and calculates an index indicating the strength of the correlation between the question sentence and the context information in the vector space.

[0068] (Extraction part 33E) The extraction unit 33E extracts context information that is highly correlated with the question sentence from the accumulated context information. In addition, the extraction unit 33E selects an axis that is closely related to the question sentence from the context information stored in the axes, and extracts the context information stored in the selected axis.

[0069] (Prompt generation unit 33F) The prompt generating unit 33F generates a context prompt according to the context. At this time, the sending unit 31 sends the context prompt to the language model and inputs it. Also, the receiving unit 32 receives an answer according to the context prompt from the language model.

[0070] For example, the prompt generation unit 33F generates an auxiliary sentence as a context prompt from the extracted context information, and generates a prompt by adding the auxiliary sentence to the question sentence. At this time, the sending unit 31 sends the prompt including the auxiliary sentence and the question sentence to the language model and inputs it. The receiving unit 32 receives an answer corresponding to the auxiliary sentence and the question sentence from the language model.

[0071] In addition, the prompt generation unit 33F broadly classifies the extracted context information into environment, user attributes, and user state or situation, turns each classified block into sentences, and attaches the auxiliary sentences that represent the context information to the input question sentence to generate a series of prompts.

[0072] In addition, the transmission unit 31 may input, together with the question sentence, context prompts corresponding to each of all the accumulated context information into the language model, and inquire of the language model as to which context information has a high correlation with the question sentence.

[0073] (Storage unit 40) The storage unit 40 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. Various programs and various data are stored in the storage unit 40.

[0074] [4. Example of Server Device Configuration] Next, the configuration of the server device 100 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the server device 100 according to the embodiment. As shown in Fig. 4, the server device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0075] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is also connected to a network N in a wired or wireless manner.

[0076] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a HDD, an SSD, an optical disk, etc. The storage unit 120 may store attribute information and history information (log data) of the user U together with identification information (such as a user ID) indicating the user U.

[0077] (Control unit 130) The control unit 130 is a controller, and is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the server device 100 using a storage area such as a RAM as a working area. In the example shown in FIG. 4, the control unit 130 has an acquisition unit 131, a reception unit 132, a learning unit 133, an inference unit 134, and a provision unit 135.

[0078] (Acquisition part 131) The acquisition unit 131 acquires a search query input by a user U (user). For example, when the user U inputs a search query into a search engine or the like to perform a keyword search, the acquisition unit 131 acquires the search query via the communication unit 110. That is, the acquisition unit 131 acquires, via the communication unit 110, a keyword input by the user U into a search box of a search engine, a site, or an app.

[0079] Furthermore, the acquisition unit 131 acquires user information regarding the user U via the communication unit 110. For example, the acquisition unit 131 acquires identification information (such as a user ID) indicating the user U, location information of the user U, attribute information of the user U, and the like from the terminal device 10 of the user U. Furthermore, the acquisition unit 131 may acquire the identification information indicating the user U, attribute information of the user U, and the like when the user U is registered. Then, the acquisition unit 131 stores the user information in the storage unit 120.

[0080] Furthermore, the acquisition unit 131 acquires various types of history information (log data) indicating the behavior of the user U via the communication unit 110. For example, the acquisition unit 131 acquires various types of history information indicating the behavior of the user U from the terminal device 10 of the user U, or from various servers based on a user ID or the like. Then, the acquisition unit 131 stores the various types of history information in the storage unit 120.

[0081] (Reception department 132) The reception unit 132 receives a prompt (instruction sentence) to be input to a GPT (Generative Pretrained Transformer) from the terminal device 10 of each user U via the communication unit 110. Note that the reception unit 132 may be a part of the acquisition unit 131 described above.

[0082] (Learning Section 133) The learning unit 133 generates and constructs, by machine learning, a GPT to be used in the server device 100. Note that in practice, the GPT is not limited to a GPT, and may be a large-scale language model (LLM) having a similar function.

[0083] (Inference part 134) The inference unit 134 uses the GPT to infer an appropriate answer corresponding to the prompt input to the large-scale language model. For example, the inference unit 134 inputs a prompt to the GPT and outputs an appropriate answer corresponding to the prompt.

[0084] (Provider 135) The providing unit 135 provides an appropriate answer (inferred answer) corresponding to the input prompt to the terminal device 10 of each user U via the communication unit 110. Note that the providing unit 135 may provide a free version, a paid version, a trial version of GPT, or a plug-in available for GPT to the terminal device 10 of each user U via the communication unit 110.

[0085] 5. Processing Procedure Next, a processing procedure by the terminal device 10 of the user U according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the processing procedure according to the embodiment. Note that the processing procedure shown below is repeatedly executed by the control unit 30 of the terminal device 10 of the user U.

[0086] For example, as shown in FIG. 5, the collection unit 33A of the terminal device 10 of the user U collects sensor information output from various sensors mounted on or connected to the terminal device 10 (step S101).

[0087] Next, the estimation unit 33B of the terminal device 10 of the user U estimates a context such as a state or situation of the user U (user) from the collected sensor information (step S102).

[0088] Next, the context information generating unit 33C of the terminal device 10 of the user U generates context information indicating the context estimated from the sensor information and stores it in advance (step S103). Note that the context information is information that expresses the context in a natural language.

[0089] Next, the input unit 13 of the terminal device 10 of the user U accepts input of a question from the user (step S104).

[0090] Next, the calculation unit 33D of the terminal device 10 of the user U calculates the correlation between the input question sentence and the accumulated context information (step S105).

[0091] Next, the extraction unit 33E of the terminal device 10 of the user U extracts, from the accumulated context information, context information that is highly correlated with the question sentence (step S106).

[0092] Next, the prompt generating unit 33F of the terminal device 10 of the user U generates an auxiliary sentence as a context prompt from the extracted context information, and generates a prompt by adding the auxiliary sentence to the question sentence (step S107).

[0093] Next, the transmission unit 31 of the terminal device 10 of the user U transmits the prompt including the auxiliary text and the question text to the GPT on the server device 100 via the communication unit 11 and inputs it (step S108).

[0094] Next, the receiving unit 32 of the terminal device 10 of the user U receives the answer corresponding to the auxiliary text and the question text from the GPT on the server device 100 via the communication unit 11, and displays it on the display unit 12 of the terminal device 10 (step S109).

[0095] 6. Modifications The above-described terminal device 10 and server device 100 may be implemented in various different forms other than the above-described embodiment. Therefore, the following describes modifications of the embodiment.

[0096] In the above embodiment, some or all of the processing executed by the server device 100 may actually be executed by the terminal device 10 (or an application running on the terminal). For example, the processing may be completed in a stand-alone manner (by the terminal device 10 alone). In this case, the terminal device 10 is assumed to have the functions of the server device 100 in the above embodiment. Also, in the above embodiment, since the terminal device 10 works in conjunction with the server device 100, from the perspective of the user U, it appears that the processing of the server device 100 is also executed by the terminal device 10. In other words, from another perspective, it can be said that the terminal device 10 is equipped with the server device 100.

[0097] In the above embodiment, the terminal device 10 of the user U generates an auxiliary sentence (context prompt) from context information indicating a context such as the state or situation of the user U, generates a prompt (instruction sentence) by attaching the auxiliary sentence to the question sentence, and inputs the generated prompt into the GPT, but in reality, only the auxiliary sentence (context prompt) may be input into the GPT. For example, the terminal device 10 of the user U may automatically extract context information indicating a context such as the state or situation of the user U from sensor information automatically collected in real time, automatically generate an auxiliary sentence (context prompt) from the automatically extracted context information, and automatically input the automatically generated auxiliary sentence (context prompt) into the GPT.

[0098] In the above embodiment, the terminal device 10 of the user U stores the usual state (=sensor statistics) on various axes as the context information, but in reality, the terminal device 10 of the user U may store the abnormal state (=sensor outlier) on various axes as the context information. For example, the terminal device 10 of the user U may store the context information indicating the abnormal state on an axis such as time, place, or during a specific action as the context information indicating the context such as a unique state or situation different from the usual. Then, when the context information indicating the abnormal state is required from the content of the question, the terminal device 10 of the user U selects an axis closely related to the question from the axes related to the abnormal state, and extracts the context information stored on the selected axis.

[0099] 7. Effects As described above, the information processing device (terminal device 10 and server device 100) of the present application is characterized by comprising a collection unit 33A that collects sensor information from various sensors, an estimation unit 33B that estimates a context such as a user's state or situation from the sensor information, a prompt generation unit 33F that generates a context prompt according to the context, a transmission unit 31 that transmits the context prompt to a language model and inputs it therein, and a receiving unit 32 that receives an answer according to the context prompt from the language model.

[0100] The information processing device according to the present application further includes a context information generating unit 33C that generates context information indicating a context estimated from sensor information and stores the context information in advance, an input unit 13 that accepts an input of a question sentence from a user, a calculating unit 33D that calculates a correlation between the input question sentence and the stored context information, and an extracting unit 33E that extracts context information that is highly correlated with the question sentence from the stored context information. The prompt generating unit 33F generates an auxiliary sentence as a context prompt from the extracted context information, and generates a prompt by adding the auxiliary sentence to the question sentence. The transmitting unit 31 transmits a prompt including the auxiliary sentence and the question sentence to the language model and inputs it. The receiving unit 32 receives an answer corresponding to the auxiliary sentence and the question sentence from the language model.

[0101] The calculation unit 33D vectorizes each of the question sentence and the context information by natural language processing, and calculates an index indicating the strength of the correlation between the question sentence and the context information in the vector space.

[0102] When estimating a context, the estimation unit 33B determines a usual state from the statistical values ​​of the sensor information. The context information generation unit 33C stores, as context information, context information indicating a usual state on an axis such as time, place, or during a specific action. The extraction unit 33E selects an axis closely related to the question sentence, and extracts the context information stored on the selected axis.

[0103] The prompt generation unit 33F broadly classifies the extracted context information into environment, user attributes, and user state or situation, turns each classified block into sentences, and attaches the auxiliary sentences that represent the context information to the input question sentence to generate a series of prompts.

[0104] The transmission unit 31 inputs the question sentence and context prompts corresponding to each of all the accumulated context information into the language model, and queries the language model as to which context information has a high correlation with the question sentence.

[0105] By performing any one or a combination of the above-described processes, the information processing device according to the present application can automatically generate a prompt from sensor information.

[0106] [8. Hardware Configuration] Moreover, the terminal device 10 and the server device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration as shown in Fig. 6, for example. The terminal device 10 will be described below as an example. Fig. 6 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected by a bus 1090.

[0107] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0108] The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that primarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may be a cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

[0109] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, a printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (registered trademark) (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, a button, a scanner, etc., and is realized by a USB, etc.

[0110] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0111] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device, such as an input / output I / F.

[0112] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0113] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0114] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0115] For example, when the computer 1000 functions as the terminal device 10, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 30. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I / F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with the other device via the network I / F 1080 and use functions and data of a program by calling them from another program of the other device.

[0116] [9. Others] Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0117] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0118] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0119] For example, the above-mentioned server device 100 may be realized by multiple server computers, and depending on the functions, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

[0120] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0121] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, an acquisition section can be read as an acquisition means or an acquisition circuit. [Explanation of symbols]

[0122] 1. Information Processing Systems 10 Terminal Equipment 11 Communications Department 12 Display section 13 Input section 14 Positioning unit 20 Sensor section 31 Transmitter 32 Receiving section 33 Processing section 33A Collection Department 33B Estimation part 33C Context information generation unit 33D Calculation Section 33E Extraction part 33F Prompt Generation Unit 100 Server device 110 Communications Department 120 Storage section 130 Control section 131 Acquisition Department 132 Reception 133 Learning Department 134 Reasoning part 135 Provision Department

Claims

1. a collection unit that collects sensor information from various sensors; An estimation unit that estimates a context such as a state or situation of a user from the sensor information; a prompt generator for generating a contextual prompt according to the context; a sending unit for sending the context prompts to populate a language model; a receiver for receiving an answer responsive to the contextual prompt from the language model; An information processing device comprising:

2. a context information generation unit that generates context information indicating the context estimated from the sensor information and stores the context information in advance; an input unit for accepting input of a question sentence from a user; A calculation unit that calculates a correlation between an input question sentence and accumulated context information; an extraction unit that extracts context information that is highly correlated with the question sentence from the accumulated context information; Further equipped with the prompt generation unit generates an auxiliary sentence as the context prompt from the extracted context information, and generates a prompt by adding the auxiliary sentence to the question sentence; the sending unit sends the prompt including the auxiliary sentence and the question sentence to the language model and inputs the prompt into the language model; The receiving unit receives an answer corresponding to the auxiliary sentence and the question sentence from the language model.

2. The information processing apparatus according to claim 1,

3. The calculation unit vectorizes the question sentence and the context information by natural language processing, and calculates an index indicating a strength of a correlation between the question sentence and the context information in a vector space.

3. The information processing apparatus according to claim 2.

4. When estimating the context, the estimation unit determines a usual state from a statistical value of the sensor information; The context information generation unit stores, as the context information, context information indicating a usual state along an axis such as a time, a place, or a predetermined action, The extraction unit selects an axis that is closely related to the question sentence, and extracts context information stored in the selected axis.

3. The information processing apparatus according to claim 2.

5. The prompt generation unit roughly classifies the extracted context information into an environment, a user attribute, and a user state or situation, converts each classified chunk into a sentence, and generates a series of prompts by adding auxiliary sentences obtained by converting the context information into a sentence to the input question sentence.

5. The information processing apparatus according to claim 4.

6. The transmission unit inputs a context prompt corresponding to each of all the stored context information into the language model together with the question sentence, and queries the language model as to which context information is highly correlated with the question sentence.

3. The information processing apparatus according to claim 2.

7. An information processing method executed by an information processing device, A collection step of collecting sensor information from various sensors; an estimation step of estimating a context such as a state or situation of a user from the sensor information; a prompt generating step of generating a contextual prompt according to the context; a sending step of sending the context prompts to populate a language model; receiving a response from the language model in response to the contextual prompt; 13. An information processing method comprising:

8. a collection procedure for collecting sensor information from various sensors; an estimation step of estimating a context such as a state or situation of a user from the sensor information; a prompt generation step of generating a contextual prompt according to the context; a sending step of sending the context prompts to populate a language model; a receiving step of receiving an answer from the language model in response to the contextual prompt; An information processing program characterized by causing a computer to execute the above.

Images