system

Abstract

Provide a system. A voice input means; means for converting voice data into text data; A means for analyzing text data to recognize the intent of a question; A means for acquiring current location information; A means for searching for information on an object based on current location information; a means for analyzing the search results to generate an answer; means for converting the generated answers into audio data; and means for playing the audio data.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Young children are very interested in what they see from inside the car and often ask their parents questions. However, parents have to ensure safety while driving, making it difficult for them to answer these questions immediately. As a result, their children's intellectual curiosity is not fully satisfied. Furthermore, parents themselves are not always able to fully answer their children's questions. Therefore, there is a need for a system that can safely respond to children's questions while driving and also help parents learn. [Means for solving the problem]

[0005] The present invention provides a system including a voice input means, a means for converting voice data into text data, a means for analyzing the text data to recognize the intent of the question, a means for acquiring current location information, a means for searching for information on an object based on the current location information, a means for analyzing the search results and generating an answer, a means for converting the generated answer into voice data, and a means for playing back the voice data, which allows a user to safely respond to a child's question while driving. Furthermore, by further including a means for generating additional information related to the generated answer, both parents and children can acquire new knowledge, enhancing the educational effect. In this way, the system satisfies a child's intellectual curiosity and contributes to parental learning.

[0006] "Audio input means" is a device that captures speech uttered by a user and provides it as input to the system.

[0007] "Means for converting voice data into text data" refers to technology or devices for converting input voice data into text format data.

[0008] "Means for analyzing text data and recognizing the intent of the question" refers to natural language processing technology for understanding the converted text data and understanding what the user is asking.

[0009] "Means for obtaining current location information" refers to technologies or devices that allow the system to determine its current geographical location using location information services such as GPS.

[0010] "Means for searching for information on an object based on current location information" refers to a technology for searching a database for detailed information on an object (e.g., an airplane, a building, etc.) based on acquired location information.

[0011] "Means for analyzing search results and generating answers" refers to technology for automatically generating appropriate answers to users' questions based on the searched information.

[0012] The "means for converting the generated answer into voice data" is a voice synthesis technology for converting the text-format answer into voice format and conveying it to the user.

[0013] The "means for reproducing audio data" refers to a device such as a speaker or a headphone that reproduces the generated audio data and allows the user to hear it.

[0014] The "means for generating additional information related to the generated answer" refers to a technique for generating additional supplementary information or knowledge related to the answer provided to the user, thereby increasing the educational value. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This system responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system not only satisfies children's intellectual curiosity, but also creates an environment where parents can learn together by providing useful information to parents.

[0037] System Configuration

[0038] This system consists of the following main components:

[0039] 1. Terminal

[0040] Voice input means (microphone)

[0041] A means of converting voice data into text data (voice recognition API)

[0042] A means of obtaining current location information (GPS module)

[0043] A means of playing audio data (speaker)

[0044] 2. Server

[0045] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0046] A method for searching for information about an object based on its current location (database query)

[0047] A means of analyzing search results and generating answers (generative AI)

[0048] A means of generating additional information related to the generated answer (knowledge database)

[0049] Program processing overview

[0050] 1. Enter the user's question

[0051] User: A child in the car asks, "Where is this plane going?"

[0052] Device: Built-in microphone captures audio.

[0053] 2. Speech-to-text

[0054] On the device: Use a speech recognition API to convert the voice data into text. For example, use the Google® Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[0055] 3. Question Analysis

[0056] Terminal: Sends text data to the server.

[0057] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[0058] 4. Obtaining location information

[0059] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[0060] 5. Executing a Database Query

[0061] Device: Sends the acquired location information to the server.

[0062] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[0063] 6. Answer Generation

[0064] Server: Analyzes the received flight information and generates an appropriate response, such as "This flight is heading to Los Angeles International Airport in the United States."

[0065] Server: In addition, it retrieves additional information about Los Angeles from the knowledge database and adds it, for example, "Los Angeles is a large city famous for its film industry."

[0066] 7. Voice output of answers

[0067] Server: Sends the generated text data to the device.

[0068] Terminal: Convert the received text data into voice data using a speech synthesis API, such as the Google Text-to-Speech API.

[0069] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0070] Specific examples

[0071] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[0072] In this way, the system satisfies children's intellectual curiosity and provides useful information to parents, enhancing the educational benefits of in-car experiences.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] User: A child in the car asks, "Where is this plane going?"

[0076] Device: A built-in microphone captures your child's voice.

[0077] Step 2:

[0078] On the device: Uses a speech recognition API to convert captured voice data into text data.

[0079] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[0080] Step 3:

[0081] Terminal: Sends text data to the server.

[0082] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[0083] Example: Use the BERT model to determine "The person is asking about the flight destination."

[0084] Step 4:

[0085] Device: Uses the GPS module to obtain current location information.

[0086] Example: Get current location as "Tokyo".

[0087] Step 5:

[0088] Device: Sends the acquired current location information to the server.

[0089] Step 6:

[0090] Server: Queries the aircraft database based on the received current location information.

[0091] Example: Running an SQL query to get information about planes flying near your current location.

[0092] Step 7:

[0093] Server: Parses the query results and generates an answer to the user's question.

[0094] Example: Generate the answer "This plane is heading to Los Angeles International Airport in the United States."

[0095] Step 8:

[0096] Server: Generates additional information related to the answer.

[0097] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[0098] Step 9:

[0099] Server: Sends the generated answer and additional information to the device.

[0100] Step 10:

[0101] On the device: Use a speech synthesis API to convert text data into audio data.

[0102] Example: Call the Google Text-to-Speech API to convert text to audio data.

[0103] Step 11:

[0104] Terminal: Plays back the generated audio data using a speaker.

[0105] Example: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0106] This series of processes provides appropriate answers to the child's questions, allowing both parent and child to learn.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] There is a demand for a system that can quickly and accurately answer questions that children in the car ask about the scenery outside, instead of the parent who is driving, thereby satisfying the child's intellectual curiosity, reducing the burden on the parent, and providing an educational environment inside the car.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for analyzing text data to recognize the intent of the question, means for searching for information on the object based on the acquired location information, and means for analyzing the search results to generate an answer, thereby enabling the server to quickly and accurately provide information on places and objects based on the child's question.

[0112] The "voice input means" is an input device for capturing voices uttered by the user.

[0113] The "means for converting voice data into text data" is a technology for analyzing captured voice data and converting it into corresponding text data.

[0114] "Means for analyzing text data and recognizing the intent of a question" refers to natural language processing technology for analyzing text data, understanding its content, and determining the intent of a question.

[0115] "Means for acquiring location information" refers to technology that acquires information on the latitude and longitude of the current location using a GPS module or the like.

[0116] The "means for searching for information on an object based on acquired location information" is a technology for searching a database for information on a specific object based on acquired location information.

[0117] "Means for analyzing search results and generating answers" refers to technology for analyzing information obtained through a search and generating appropriate answers to the user's questions.

[0118] The "means for generating additional information related to the generated answer" is a technique for adding further related knowledge or information to the generated answer to provide a more detailed answer.

[0119] The "means for converting the generated answer into voice data" is a technology for converting the answer in text format into voice data, and making it into a format that can be output as voice.

[0120] "Means for reproducing audio data" refers to a technique for reproducing audio data using an output device such as a speaker, thereby providing information to the user auditorily.

[0121] This invention is a system that responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system satisfies children's intellectual curiosity and creates an environment where both parents can learn by providing useful information.

[0122] System Configuration

[0123] This system consists of the following main components:

[0124] 1. Terminal

[0125] Voice input means (microphone)

[0126] A means of converting voice data into text data (voice recognition API)

[0127] Means of obtaining location information (GPS module)

[0128] A means of playing audio data (speaker)

[0129] 2. Server

[0130] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0131] A method for searching for information about an object based on the acquired location information (database query)

[0132] A means of analyzing search results and generating answers (generative AI model)

[0133] A means of generating additional information related to the generated answer (knowledge database)

[0134] A means to convert the generated answers into voice data (voice synthesis API)

[0135] System Operation

[0136] 1. Enter the user's question

[0137] User: A child asks, "Where is this plane going?"

[0138] Device: A sensitive microphone inside the car captures the child's voice.

[0139] 2. Speech-to-text

[0140] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[0141] 3. Question Analysis

[0142] Terminal: The converted text data is sent to the server using the HTTP POST method.

[0143] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[0144] 4. Obtaining location information

[0145] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[0146] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[0147] 5. Executing a Database Query

[0148] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[0149] Server: Query the database to get data on planes flying around the current location.

[0150] 6. Answer Generation

[0151] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[0152] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[0153] 7. Voice output of answers

[0154] Server: Sends the generated text data to the device.

[0155] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[0156] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0157] Specific examples

[0158] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[0159] Example prompt sentence:

[0160] When asked "Where is this plane going?", provide an appropriate answer by determining the plane's destination based on its current location, and provide any additional information about the destination.

[0161] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0162] Step 1:

[0163] Entering user questions

[0164] User: A child asks, "Where is this plane going?"

[0165] Device: A sensitive microphone installed inside the car captures the child's voice.

[0166] Input: Child's voice.

[0167] Processing: Captured as audio data by microphone.

[0168] Output: The captured audio data.

[0169] Step 2:

[0170] Speech-to-text

[0171] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[0172] Input: The captured audio data.

[0173] Processing: Send the audio data to the speech recognition API and convert it into corresponding text data.

[0174] Output: The converted text data (e.g., "Where is this plane going?").

[0175] Step 3:

[0176] Question Analysis

[0177] Terminal: The converted text data is sent to the server using the HTTP POST method.

[0178] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[0179] Input: Text data (e.g., "Where is this plane going?").

[0180] Processing: Text data is sent to the server via an HTTP POST request, and the server analyzes it using a natural language processing engine.

[0181] Output: The intent of the question (e.g., "Question about flight destinations").

[0182] Step 4:

[0183] Obtaining location information

[0184] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[0185] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[0186] Input: Current location (latitude and longitude).

[0187] Processing: Obtain location information using the GPS module and send it to the server via an HTTP POST request.

[0188] Output: The location information sent.

[0189] Step 5:

[0190] Executing a Database Query

[0191] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[0192] Server: Query the database to get data on planes flying around the current location.

[0193] Input: Location and question intent.

[0194] Processing: Query the database to get the relevant plane information.

[0195] Output: Plane information data.

[0196] Step 6:

[0197] Generate answers

[0198] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[0199] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[0200] Input: Airplane information data.

[0201] Processing: A generative AI model generates an answer and adds additional information from a knowledge database.

[0202] Output: The generated answer with additional information (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city known for its film industry.").

[0203] Step 7:

[0204] Voice output of answers

[0205] Server: Sends the generated text data to the device.

[0206] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[0207] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0208] Input: Generated text data.

[0209] Processing: Convert into audio data using speech synthesis API and play on speaker.

[0210] Output: An audio response that the user can hear.

[0211] (Application example 1)

[0212] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0213] In recent years, with the increasing popularity of self-driving vehicles, people are spending more time in their cars. Children, in particular, are increasingly curious about the scenery outside and asking questions. However, it is difficult, inconvenient, and potentially dangerous for parents to respond to such questions while driving. Furthermore, there are limited means to satisfy children's intellectual curiosity and provide an environment where parents and children can learn together. This results in lost opportunities for educational interaction between families in the car. The present invention aims to solve these problems by providing a system that responds appropriately and quickly to children's questions.

[0214] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0215] In this invention, the server includes means for searching for information on objects based on current location information, means for analyzing the search results and generating an answer, and means for acquiring additional information related to the generated answer from a knowledge base. As a result, when a child asks a question about the scenery outside while in the car, an appropriate answer can be automatically and quickly generated, allowing the whole family to share new knowledge.

[0216] "Voice input means" is a general term for devices that capture voices uttered by a user.

[0217] The "means for converting voice data into text data" is a technology that uses voice recognition technology to convert input voice into corresponding text data.

[0218] "Means for analyzing text data and recognizing the intent of the question" refers to a technology that uses natural language processing technology to understand the user's intent and the content of the question from text data.

[0219] "Means for obtaining current location information" refers to a technology that allows the system to obtain current geographical location information using a GPS module or the like.

[0220] The "means for searching for information on an object based on current location information" is a technology for searching for information on a specific object from a database or the like based on the acquired location information.

[0221] "Means for analyzing search results and generating answers" refers to technology that processes the data obtained through a search and constructs an appropriate answer to the user's question.

[0222] The "means for obtaining additional information related to the generated answer from a knowledge base" is a technique for obtaining supplemental information from a database or the like in order to add the supplemental information to the generated answer.

[0223] A "generative AI model" is an artificial intelligence technology that can learn from large amounts of data and generate natural, human-like answers.

[0224] "Means for reproducing audio data" refers to technology that converts text data into audio using speech synthesis technology and outputs it from a speaker or the like.

[0225] A "prompt sentence" is an input sentence that causes a generative AI model to generate an appropriate answer.

[0226] The present invention is an educational system in which a system installed in an autonomous vehicle provides appropriate answers to questions posed by a child inside the vehicle about the scenery outside. This system includes a voice input means, a means for converting voice data into text data, a means for analyzing the text data and recognizing the intent of the question, a means for acquiring current location information, a means for searching for information on an object based on the current location information, a means for analyzing the search results and generating an answer, a means for acquiring additional information related to the generated answer from a knowledge base, a means for converting the generated answer into voice data, and a means for playing back the voice data.

[0227] The server first captures the child's voice using a voice input method (e.g., a microphone).The voice data is then converted into text data using a tool such as the Google Speech-to-Text API.The text data is then analyzed using natural language processing technology (e.g., the Google Cloud Natural Language API) to recognize the intent of the question.

[0228] Next, the current location information is acquired using a GPS module and sent to the server along with the text data. The server then searches for information about the object from sources such as a database based on the current location information. For example, if information about a specific mountain is to be acquired, the server queries the database for data related to that mountain.

[0229] The search results are analyzed using a generative AI model (e.g., GPT-4 (registered trademark)) to generate an appropriate answer to the user's question. For example, an answer such as "This mountain is Mount Fuji. It is the highest mountain in Japan" can be obtained. Furthermore, additional relevant information can be obtained from a knowledge base and added to the answer, thereby providing a more detailed answer.

[0230] Finally, the generated answers are converted into audio data using the Google Text-to-Speech API and played through the vehicle's speakers, allowing the child to get a spoken answer to their question.

[0231] As a concrete example, if a child asks, "What mountain is this?", the microphone captures the voice and speech recognition technology generates the text data "What mountain is this?". Natural language processing technology analyzes the intent of the question, and if it determines that the current location is, for example, Nagano Prefecture, the server searches a database for information on nearby mountains. The answer generated might be, "That mountain is the Hakuba Mountains," with additional information added, such as, "The Hakuba Mountains are famous for their beautiful hiking trails." This answer is then played back as audio from the microphone.

[0232] An example prompt here is, "Please describe the mountain the child saw based on the following data: Hakuba Mountains."

[0233] In this way, the present invention not only satisfies a child's intellectual curiosity and enhances the educational effect in the car, but also provides an opportunity for parents and children to learn together.

[0234] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0235] Step 1:

[0236] The device uses the microphone in the car to capture the voice of the user, a child, asking, "What mountain is this?" The input is the user's voice, and the output is the voice data.

[0237] Step 2:

[0238] The device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the captured voice data into text data. The input is the voice data, and the output is the text data "What mountain is this?"

[0239] Step 3:

[0240] The terminal sends the converted text data to the server. The input here is the text data "What is this mountain?" and the output is the transmission of the text data to the server.

[0241] Step 4:

[0242] The server receives the text data and analyzes it using a natural language processing engine (e.g., Google Cloud Natural Language API) to recognize the intent of the question. The input is the text data "What is this mountain?", and the output is the analyzed intent of the question (e.g., "I want to know the name of the mountain").

[0243] Step 5:

[0244] The terminal obtains the current location information using the GPS module. The input is the current geographical location, and the output is the current location information (e.g., "Nagano Prefecture").

[0245] Step 6:

[0246] The terminal sends the acquired current location information to the server. The input is the current location information "Nagano Prefecture," and the output is the transmission of the location information to the server.

[0247] Step 7:

[0248] The server executes a database query based on the received text data and location information to search for the target mountain. The input is the text data "What mountain is this?" and the current location information "Nagano Prefecture," and the output is the corresponding mountain information (e.g., "Hakuba Mountains").

[0249] Step 8:

[0250] The server uses a generative AI model (for example, GPT-4) based on the search results to generate an appropriate answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails." The input is the search result "Hakuba Mountain," and the output is the generated answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails."

[0251] Step 9:

[0252] The server retrieves additional information related to the generated answer from the knowledge base and adds it to the answer. The input is the generated answer "This mountain is Hakuba Mountains." The output is the answer with the additional information added: "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails."

[0253] Step 10:

[0254] The server sends the final answer to the terminal. The input is the answer "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails." with additional information added, and the output is the sending of the answer to the terminal.

[0255] Step 11:

[0256] The device converts the received text data into voice data using a speech synthesis API (for example, Google Text-to-Speech API). The input is the text data "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails.", and the output is voice data.

[0257] Step 12:

[0258] The terminal uses a speaker to play back the generated voice data and provide an answer to the user (child). The input is the voice data, and the output is voice output from the speaker.

[0259] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0260] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[0261] System Configuration

[0262] This system consists of the following main components:

[0263] 1. Terminal

[0264] Voice input means (microphone)

[0265] A means of converting voice data into text data (voice recognition API)

[0266] Emotion engine (means of analyzing user emotions)

[0267] A means of obtaining current location information (GPS module)

[0268] A means of playing audio data (speaker)

[0269] 2. Server

[0270] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0271] A method for searching for information about an object based on its current location (database query)

[0272] A means of analyzing search results and generating answers (generative AI)

[0273] A means of generating additional information related to the generated answer (knowledge database)

[0274] Program processing overview

[0275] 1. Enter the user's question

[0276] User: A child in the car asks, "Where is this plane going?"

[0277] Device: Built-in microphone captures audio.

[0278] 2. Speech-to-text

[0279] On the device: Use a speech recognition API to convert the voice data into text data. For example, use the Google Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[0280] 3. Question Analysis

[0281] Terminal: Sends text data to the server.

[0282] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[0283] 4. Emotion Analysis

[0284] On the device: The emotion engine analyzes the voice data and recognizes the user's emotions, for example, whether a child is excited or curious.

[0285] 5. Obtaining location information

[0286] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[0287] 6. Executing a Database Query

[0288] Device: Sends the acquired location and emotion information to the server.

[0289] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[0290] 7. Answer Generation

[0291] Server: Analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[0292] 8. Generating Additional Information

[0293] Server: Additionally, generate additional information related to the answer, for example, "Los Angeles is a large city famous for its film industry."

[0294] 9. Voice output of answers

[0295] Server: Sends the generated answer and additional information to the device.

[0296] On the device: Convert text data into audio data using a speech synthesis API, for example, the Google Text-to-Speech API.

[0297] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0298] Specific examples

[0299] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[0300] This process provides accurate and personalized answers to a child's questions, helping both parents and children learn.

[0301] The processing flow will be explained below.

[0302] Step 1:

[0303] User: A child in the car asks, "Where is this plane going?"

[0304] Device: A built-in microphone captures your child's voice.

[0305] Step 2:

[0306] On the device: Uses a speech recognition API to convert captured voice data into text data.

[0307] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[0308] Step 3:

[0309] Terminal: Sends text data to the server.

[0310] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[0311] Example: Use the BERT model to determine "The person is asking about the flight destination."

[0312] Step 4:

[0313] Device: The emotion engine analyzes the voice data and recognizes the user's emotions.

[0314] For example: Analyzing speech tone, volume, rhythm, etc. to determine whether a child is excited or curious.

[0315] Step 5:

[0316] Device: Uses the GPS module to obtain current location information.

[0317] Example: Get current location as "Tokyo".

[0318] Step 6:

[0319] Device: Sends the acquired current location information and recognized emotion information to the server.

[0320] Step 7:

[0321] Server: Queries the aircraft database based on the received location and emotion information.

[0322] Example: Running an SQL query to get information about planes flying near your current location.

[0323] Step 8:

[0324] Server: Analyzes the query results and generates an answer to the user's question, taking into account information from the emotion engine and adjusting the tone and content appropriately.

[0325] For example: If a child is excited, generate the answer "Wow! That plane is heading to Los Angeles International Airport in the US! How exciting!"

[0326] Step 9:

[0327] Server: Also generates additional information related to the answer.

[0328] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[0329] Step 10:

[0330] Server: Sends the generated answer and additional information to the device.

[0331] Step 11:

[0332] On the device: Use a speech synthesis API to convert text data into audio data.

[0333] Example: Call the Google Text-to-Speech API to convert text to audio data.

[0334] Step 12:

[0335] Terminal: Plays back the generated audio data using a speaker.

[0336] Example: Play audio that says, "Wow! The plane is heading to Los Angeles International Airport in the United States! Los Angeles is a big city famous for its film industry. How exciting!"

[0337] If the user is excited, the tone and content of the response can be adjusted to provide a more personalized experience. This process ensures that the child's questions are answered appropriately and emotionally, creating a rich learning experience for both parent and child.

[0338] Example 2

[0339] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0340] Conventional in-car voice response systems only provide simple answers to user questions and are unable to provide personalized responses that take into account the user's emotional state or current location. This has been a challenge, particularly in satisfying children's intellectual curiosity and providing an environment where parents and children can learn together. Furthermore, there has been a lack of systems that provide appropriate additional information based on the user's emotions and current location.

[0341] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0342] In this invention, the server includes a means for converting voice data into text data, a means for analyzing the user's emotions, and a means for acquiring current location information, thereby enabling the provision of personalized answers and additional information that take into account the user's emotional state and current location.

[0343] "Audio input means" is a device for capturing audio uttered by a user.

[0344] A "means for converting audio data into text data" is a process or technique that analyzes captured audio data and converts it into corresponding text data.

[0345] The "means for analyzing user emotions" is a function or engine that recognizes and analyzes the user's emotional state (e.g., joy, excitement, interest, etc.) from voice data.

[0346] The "means for analyzing text data and recognizing the intent of the question" is a technology for analyzing the converted text data and understanding the user's question and intent.

[0347] "Means for obtaining current location information" refers to a device or function for obtaining latitude and longitude information of the user's current location using location measurement technology such as GPS.

[0348] The "means for searching for information about an object based on current location information and emotion data" is a process for searching for appropriate information about an object from related databases or information sources using the acquired location information and emotion information.

[0349] "Means for analyzing search results and generating answers" refers to technology that generates appropriate answers to user questions based on information acquired through searches.

[0350] The "means for generating additional information related to the generated answer" is a technique for adding supplementary information or interesting information further related to the generated answer.

[0351] The "means for converting the generated answers into voice data" is a voice synthesis technology that converts answers in text format into voice format.

[0352] The "means for reproducing audio data" is a technique for providing the converted audio data to the user through an output device such as a speaker.

[0353] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[0354] System Configuration

[0355] This system consists of the following main components:

[0356] 1. Terminal

[0357] Voice input means (microphone)

[0358] A means of converting voice data into text data (voice recognition API)

[0359] Emotion engine (means of analyzing user emotions)

[0360] A means of obtaining current location information (GPS module)

[0361] A means of playing audio data (speaker)

[0362] 2. Server

[0363] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0364] A method for searching for information about an object based on its current location (database query)

[0365] A means of analyzing search results and generating answers (generative AI)

[0366] A means of generating additional information related to the generated answer (knowledge database)

[0367] System Operation

[0368] The operation of this system is carried out in the following procedure.

[0369] 1. Enter the user's question

[0370] A child user in a car asks, "Where is this plane going?" The device's built-in microphone captures the voice.

[0371] 2. Speech-to-text

[0372] The device uses a speech recognition API (for example, Google Speech-to-Text API) to convert the voice data into text data, which becomes "Where is this plane going?"

[0373] 3. Question Analysis

[0374] The device sends the text data to a server, where a natural language processing engine analyzes the text and recognizes the intent of the child's question, such as asking about the flight's destination.

[0375] 4. Emotion Analysis

[0376] The device's emotion engine analyzes the voice data and recognizes the user's emotions, for example, determining whether a child is excited or curious.

[0377] 5. Obtaining location information

[0378] The device uses the GPS module to obtain the current location information. For example, it verifies that the current location is "Tokyo."

[0379] 6. Executing a Database Query

[0380] The device sends the acquired location and emotion information to a server, which queries a database containing information on airplanes to obtain data on airplanes flying near the current location.

[0381] 7. Answer Generation

[0382] The server analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[0383] 8. Generating Additional Information

[0384] The server also generates additional information related to the answer, for example, "Los Angeles is a large city known for its film industry."

[0385] 9. Voice output of answers

[0386] The server sends the generated answer and additional information to the device. The device converts the text data into audio data using a speech synthesis API (e.g., Google Text-to-Speech API). The audio data is played back using the device's speaker, providing the answer to the user. For example, it could say, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a major city famous for its film industry."

[0387] Specific examples

[0388] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[0389] Prompt Sentence Examples

[0390] "When a child asks, 'Where is this plane going?', provide an appropriate voice response based on real-time aircraft data and current location. Choose an appropriate tone for the response and include any additional relevant information."

[0391] This system satisfies children's intellectual curiosity and provides an environment where parents and children can learn together.

[0392] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0393] Step 1:

[0394] A user asks a question in the car. A child says, "Where is this plane going?" This is the input voice data for the system. The device's microphone receives this voice and captures it as voice data.

[0395] Step 2:

[0396] The device converts the voice data into text data. The device uses a voice recognition API (for example, a voice recognition cloud service) to convert the captured voice data into text data such as "Where is this plane going?" This is a process that performs phonemic analysis on the voice data and converts it into the corresponding text. The output is the converted text data.

[0397] Step 3:

[0398] The terminal sends text data to the server. Based on the text data input, the terminal securely sends the data to the server using the HTTPS protocol. The output is the text data sent to the server.

[0399] Step 4:

[0400] The server analyzes the text data and recognizes the intent of the question. The server uses a natural language processing engine to analyze the text "Where is this plane going?", thereby recognizing the meaning "asking about the plane's destination." The output is the intent information of the question.

[0401] Step 5:

[0402] The device analyzes emotions. The device's emotion engine analyzes the voice data and identifies the user's (child's) emotional state. For example, it determines whether the child is excited or curious. The input is the captured voice data, and the output is the analyzed emotion data.

[0403] Step 6:

[0404] The device obtains location information. The device's GPS module is used to obtain the user's current location. For example, confirm that the current location is within Tokyo's 23 wards. The input is a signal from a GPS satellite, and the output is latitude and longitude information.

[0405] Step 7:

[0406] The device sends emotion data and location information to the server. The device sends the existing emotion data and location information to the server using the HTTPS protocol. The output is the emotion data and location information sent to the server.

[0407] Step 8:

[0408] The server executes a database query. Based on the location information it receives, it retrieves the relevant data from a database containing information about airplanes. The server issues a query such as "Search for information about airplanes in flight" and retrieves data from the database such as "Japan Airlines Flight 123 is heading to Los Angeles." The output is the query result.

[0409] Step 9:

[0410] The server analyzes the query results and generates an answer. The server uses an automatic generation engine to prepare an answer to the user's question. At this time, it also takes into account emotional data and generates an answer such as "This plane is heading to Los Angeles, USA." The output is a specific answer sentence.

[0411] Step 10:

[0412] The server generates additional information. The server searches for relevant information from a knowledge database and adds additional information related to the answer. For example, it adds information such as "Los Angeles is a city famous for its film industry." The output is a set of the answer and the additional information.

[0413] Step 11:

[0414] The server sends the generated answer and additional information to the terminal. The server sends the answer and additional information as text data to the terminal. The output is the text data sent to the terminal.

[0415] Step 12:

[0416] The device converts the text data into voice data. The device uses a voice synthesis API (e.g., a voice synthesis cloud service) to convert the text data into voice data. The output is the converted voice data.

[0417] Step 13:

[0418] The terminal plays the audio data. The converted audio data is played using the terminal's speaker, and the content is provided to the user as "This plane is heading to Los Angeles, USA. Los Angeles is a city famous for its film industry." The output is a voice response.

[0419] (Application example 2)

[0420] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0421] Conventional in-car knowledge provision systems make it difficult for parents to provide immediate and appropriate answers when their children ask about the scenery. Furthermore, rather than simply providing answers, personalized responses based on the child's emotions and interests are necessary. Furthermore, there is a need for real-time information provision based on the car's location and the situation at hand while the car is moving. It is necessary to solve these problems and provide a fun learning environment for both parents and children.

[0422] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing voice data to recognize the intent of the question, means for analyzing the user's emotions, and means for searching for information on the object based on current location information and emotion information. This makes it possible to provide appropriate and personalized answers to children's questions that take into account emotions and current location.

[0423] The "voice input means" is a means for acquiring voice and processing the acquired voice.

[0424] The "means for converting voice data into text data" is a means for converting acquired voice data into character string data.

[0425] The "means for analyzing text data and recognizing the intent of the question" is a means for analyzing the converted text data and understanding the purpose and intent of the user's question from its contents.

[0426] The "means for analyzing the user's emotions" is a means for analyzing the emotions of the user from voice data and other input data, and grasping the emotional state.

[0427] "Means for obtaining current location information" refers to means for identifying the current geographic location using GPS or other location information technology.

[0428] The "means for searching for information on an object based on current location information and emotion information" is a means for combining the acquired current location information and emotion information to search for information on related objects from a database or other information sources.

[0429] The "means for analyzing search results and generating an answer" refers to a means for analyzing information obtained through a search and generating an appropriate answer to a user's question based on that information.

[0430] The "means for converting the generated answer into voice data" is a means for converting the generated answer in text format into voice format.

[0431] The "means for reproducing audio data" refers to a means for transmitting the generated audio data to a user through a speaker or other reproducing device.

[0432] System Configuration

[0433] The present invention is a system that provides knowledge on behalf of parents in response to questions posed by children in the car, and further provides more personalized responses by recognizing the user's emotions and adjusting the content of the answers. The system consists of the following main components:

[0434] 1. Terminal

[0435] Voice input means (microphone)

[0436] A means of converting voice data into text data (Google Speech-to-Text API)

[0437] A means of analyzing user emotions (IBM Watson (registered trademark) Tone Analyzer)

[0438] A means of obtaining current location information (GPS module)

[0439] A means of playing audio data (speaker)

[0440] 2. Server

[0441] A means of analyzing text data and recognizing the intent of the question (Google Natural Language API)

[0442] A method to search for information about an object based on its current location and emotion (Google Cloud Firestore)

[0443] A means of analyzing search results and generating answers (generative AI)

[0444] A way to convert the generated answers into audio data (Google Text-to-Speech API)

[0445] System operation explanation

[0446] Voice to text conversion

[0447] Any questions asked by the user in the car are captured by a microphone on the device, and the voice data is converted to text data using the Google Speech-to-Text API, which is then sent to a server to understand what the user is asking.

[0448] Question analysis and sentiment analysis

[0449] The server receives the text data and analyzes the intent of the question using the Google Natural Language API, while simultaneously analyzing the user's sentiment using IBM Watson Tone Analyzer. This sentiment data is used to adjust the tone and content of the response.

[0450] Obtaining current location information

[0451] The device's GPS module obtains the current location information, which is also sent to the server, and is used to search for objects related to the query.

[0452] Database Querying and Answer Generation

[0453] The server searches for relevant information using Google Cloud Firestore based on the acquired current location and emotion information. The search results are analyzed and a generative AI generates an answer to the question. The generated answer is converted into audio data using the Google Text-to-Speech API and sent to the device.

[0454] Audio playback

[0455] Finally, the generated audio data is played back through the device's speaker to provide the answer to the user, allowing for an appropriate and personalized response to the child's question.

[0456] Specific examples

[0457] As an example of actual operation, consider a child in a car asking, "Where is this plane going?" The device captures the audio and converts it to text using the Google Speech-to-Text API. The text data is then sent to the server, where the Google Natural Language API analyzes the intent of the question. At the same time, IBM Watson Tone Analyzer analyzes the audio data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server queries Google Cloud Firestore to obtain information about the plane, adds emotional information, and generates an appropriate answer. This answer is converted into audio data using the Google Text-to-Speech API and played through the device's speaker.

[0458] Prompt Sentence Examples

[0459] You are an assistant that generates personalized answers to questions your child asks in the car.

[0460] Here are some questions: "Where is this plane going?"

[0461] Your current location is Tokyo. Your flight destination is Los Angeles International Airport.

[0462] Answer: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!"

[0463] This process allows parents to concentrate on driving with peace of mind, while creating an environment where children can have fun and satisfy their curiosity.

[0464] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0465] Step 1:

[0466] The user speaks a question while in the car, and the device's microphone captures the voice and saves it as audio data.

[0467] Input: User's voice question

[0468] Output: Captured audio data

[0469] Step 2:

[0470] The device uses a speech recognition API (Google Speech-to-Text API) to convert the voice data into text data.

[0471] Input: Audio data

[0472] Output: Text data (e.g., "Where is this plane going?")

[0473] Step 3:

[0474] The device sends text data to the server, which then uses a natural language processing engine (Google Natural Language API) to analyze the intent of the question.

[0475] Input: Text data

[0476] Output: The intent of the question (e.g., asking for flight destination)

[0477] Step 4:

[0478] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) to analyze the voice data and recognize the user's emotions.

[0479] Input: Audio data

[0480] Output: User emotion (e.g., excitement, curiosity)

[0481] Step 5:

[0482] The device's GPS module acquires the current location information and sends it to the server.

[0483] Input: Location measurement

[0484] Output: Current location (e.g., Tokyo)

[0485] Step 6:

[0486] The server uses Google Cloud Firestore to search for information about related objects (e.g., airplanes in flight) based on the current location and emotion information.

[0487] Input: Current location, emotion

[0488] Output: Object information (e.g., flight destination)

[0489] Step 7:

[0490] The server analyzes the search results and generates an answer to the user's question, adjusting the tone and content of the answer based on emotional information.

[0491] Input: Object information, emotion information, question intent

[0492] Output: Answer (e.g., "This plane is heading to Los Angeles International Airport in the United States.")

[0493] Step 8:

[0494] The server converts the generated answer into audio data using the Google Text-to-Speech API and sends it to the device.

[0495] Input: Answer

[0496] Output: Audio data

[0497] Step 9:

[0498] The terminal uses a speaker to play back the audio data and provide the answer to the user.

[0499] Input: Audio data

[0500] Output: Speech to be played to the user (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!")

[0501] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0502] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0503] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0504] [Second embodiment]

[0505] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0506] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0507] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0508] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0509] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0510] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0511] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0512] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0513] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0514] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0515] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0516] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0517] This system responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system not only satisfies children's intellectual curiosity, but also creates an environment where parents can learn together by providing useful information to parents.

[0518] System Configuration

[0519] This system consists of the following main components:

[0520] 1. Terminal

[0521] Voice input means (microphone)

[0522] A means of converting voice data into text data (voice recognition API)

[0523] A means of obtaining current location information (GPS module)

[0524] A means of playing audio data (speaker)

[0525] 2. Server

[0526] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0527] A method for searching for information about an object based on its current location (database query)

[0528] A means of analyzing search results and generating answers (generative AI)

[0529] A means of generating additional information related to the generated answer (knowledge database)

[0530] Program processing overview

[0531] 1. Enter the user's question

[0532] User: A child in the car asks, "Where is this plane going?"

[0533] Device: Built-in microphone captures audio.

[0534] 2. Speech-to-text

[0535] On the device: Use a speech recognition API to convert the voice data into text. For example, use the Google Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[0536] 3. Question Analysis

[0537] Terminal: Sends text data to the server.

[0538] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[0539] 4. Obtaining location information

[0540] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[0541] 5. Executing a Database Query

[0542] Device: Sends the acquired location information to the server.

[0543] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[0544] 6. Answer Generation

[0545] Server: Analyzes the received flight information and generates an appropriate response, such as "This flight is heading to Los Angeles International Airport in the United States."

[0546] Server: In addition, it retrieves additional information about Los Angeles from the knowledge database and adds it, for example, "Los Angeles is a large city famous for its film industry."

[0547] 7. Voice output of answers

[0548] Server: Sends the generated text data to the device.

[0549] Terminal: Convert the received text data into voice data using a speech synthesis API, such as the Google Text-to-Speech API.

[0550] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0551] Specific examples

[0552] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[0553] In this way, the system satisfies children's intellectual curiosity and provides useful information to parents, enhancing the educational benefits of in-car experiences.

[0554] The processing flow will be explained below.

[0555] Step 1:

[0556] User: A child in the car asks, "Where is this plane going?"

[0557] Device: A built-in microphone captures your child's voice.

[0558] Step 2:

[0559] On the device: Uses a speech recognition API to convert captured voice data into text data.

[0560] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[0561] Step 3:

[0562] Terminal: Sends text data to the server.

[0563] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[0564] Example: Use the BERT model to determine "The person is asking about the flight destination."

[0565] Step 4:

[0566] Device: Uses the GPS module to obtain current location information.

[0567] Example: Get current location as "Tokyo".

[0568] Step 5:

[0569] Device: Sends the acquired current location information to the server.

[0570] Step 6:

[0571] Server: Queries the aircraft database based on the received current location information.

[0572] Example: Running an SQL query to get information about planes flying near your current location.

[0573] Step 7:

[0574] Server: Parses the query results and generates an answer to the user's question.

[0575] Example: Generate the answer "This plane is heading to Los Angeles International Airport in the United States."

[0576] Step 8:

[0577] Server: Generates additional information related to the answer.

[0578] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[0579] Step 9:

[0580] Server: Sends the generated answer and additional information to the device.

[0581] Step 10:

[0582] On the device: Use a speech synthesis API to convert text data into audio data.

[0583] Example: Call the Google Text-to-Speech API to convert text to audio data.

[0584] Step 11:

[0585] Terminal: Plays back the generated audio data using a speaker.

[0586] Example: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0587] This series of processes provides appropriate answers to the child's questions, allowing both parent and child to learn.

[0588] Example 1

[0589] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0590] There is a demand for a system that can quickly and accurately answer questions that children in the car ask about the scenery outside, instead of the parent who is driving, thereby satisfying the child's intellectual curiosity, reducing the burden on the parent, and providing an educational environment inside the car.

[0591] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0592] In this invention, the server includes means for analyzing text data to recognize the intent of the question, means for searching for information on the object based on the acquired location information, and means for analyzing the search results to generate an answer, thereby enabling the server to quickly and accurately provide information on places and objects based on the child's question.

[0593] The "voice input means" is an input device for capturing voices uttered by the user.

[0594] The "means for converting voice data into text data" is a technology for analyzing captured voice data and converting it into corresponding text data.

[0595] "Means for analyzing text data and recognizing the intent of a question" refers to natural language processing technology for analyzing text data, understanding its content, and determining the intent of a question.

[0596] "Means for acquiring location information" refers to technology that acquires information on the latitude and longitude of the current location using a GPS module or the like.

[0597] The "means for searching for information on an object based on acquired location information" is a technology for searching a database for information on a specific object based on acquired location information.

[0598] "Means for analyzing search results and generating answers" refers to technology for analyzing information obtained through a search and generating appropriate answers to the user's questions.

[0599] The "means for generating additional information related to the generated answer" is a technique for adding further related knowledge or information to the generated answer to provide a more detailed answer.

[0600] The "means for converting the generated answer into voice data" is a technology for converting the answer in text format into voice data, and making it into a format that can be output as voice.

[0601] "Means for reproducing audio data" refers to a technique for reproducing audio data using an output device such as a speaker, thereby providing information to the user auditorily.

[0602] This invention is a system that responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system satisfies children's intellectual curiosity and creates an environment where both parents can learn by providing useful information.

[0603] System Configuration

[0604] This system consists of the following main components:

[0605] 1. Terminal

[0606] Voice input means (microphone)

[0607] A means of converting voice data into text data (voice recognition API)

[0608] Means of obtaining location information (GPS module)

[0609] A means of playing audio data (speaker)

[0610] 2. Server

[0611] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0612] A method for searching for information about an object based on the acquired location information (database query)

[0613] A means of analyzing search results and generating answers (generative AI model)

[0614] A means of generating additional information related to the generated answer (knowledge database)

[0615] A means to convert the generated answers into voice data (voice synthesis API)

[0616] System Operation

[0617] 1. Enter the user's question

[0618] User: A child asks, "Where is this plane going?"

[0619] Device: A sensitive microphone inside the car captures the child's voice.

[0620] 2. Speech-to-text

[0621] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[0622] 3. Question Analysis

[0623] Terminal: The converted text data is sent to the server using the HTTP POST method.

[0624] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[0625] 4. Obtaining location information

[0626] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[0627] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[0628] 5. Executing a Database Query

[0629] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[0630] Server: Query the database to get data on planes flying around the current location.

[0631] 6. Answer Generation

[0632] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[0633] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[0634] 7. Voice output of answers

[0635] Server: Sends the generated text data to the device.

[0636] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[0637] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0638] Specific examples

[0639] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[0640] Example prompt sentence:

[0641] When asked "Where is this plane going?", provide an appropriate answer by determining the plane's destination based on its current location, and provide any additional information about the destination.

[0642] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0643] Step 1:

[0644] Entering user questions

[0645] User: A child asks, "Where is this plane going?"

[0646] Device: A sensitive microphone installed inside the car captures the child's voice.

[0647] Input: Child's voice.

[0648] Processing: Captured as audio data by microphone.

[0649] Output: The captured audio data.

[0650] Step 2:

[0651] Speech-to-text

[0652] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[0653] Input: The captured audio data.

[0654] Processing: Send the audio data to the speech recognition API and convert it into corresponding text data.

[0655] Output: The converted text data (e.g., "Where is this plane going?").

[0656] Step 3:

[0657] Question Analysis

[0658] Terminal: The converted text data is sent to the server using the HTTP POST method.

[0659] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[0660] Input: Text data (e.g., "Where is this plane going?").

[0661] Processing: Text data is sent to the server via an HTTP POST request, and the server analyzes it using a natural language processing engine.

[0662] Output: The intent of the question (e.g., "Question about flight destinations").

[0663] Step 4:

[0664] Obtaining location information

[0665] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[0666] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[0667] Input: Current location (latitude and longitude).

[0668] Processing: Obtain location information using the GPS module and send it to the server via an HTTP POST request.

[0669] Output: The location information sent.

[0670] Step 5:

[0671] Executing a Database Query

[0672] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[0673] Server: Query the database to get data on planes flying around the current location.

[0674] Input: Location and question intent.

[0675] Processing: Query the database to get the relevant plane information.

[0676] Output: Plane information data.

[0677] Step 6:

[0678] Generate answers

[0679] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[0680] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[0681] Input: Airplane information data.

[0682] Processing: A generative AI model generates an answer and adds additional information from a knowledge database.

[0683] Output: The generated answer with additional information (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city known for its film industry.").

[0684] Step 7:

[0685] Voice output of answers

[0686] Server: Sends the generated text data to the device.

[0687] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[0688] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0689] Input: Generated text data.

[0690] Processing: Convert into audio data using speech synthesis API and play on speaker.

[0691] Output: An audio response that the user can hear.

[0692] (Application example 1)

[0693] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0694] In recent years, with the increasing popularity of self-driving vehicles, people are spending more time in their cars. Children, in particular, are increasingly curious about the scenery outside and asking questions. However, it is difficult, inconvenient, and potentially dangerous for parents to respond to such questions while driving. Furthermore, there are limited means to satisfy children's intellectual curiosity and provide an environment where parents and children can learn together. This results in lost opportunities for educational interaction between families in the car. The present invention aims to solve these problems by providing a system that responds appropriately and quickly to children's questions.

[0695] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0696] In this invention, the server includes means for searching for information on objects based on current location information, means for analyzing the search results and generating an answer, and means for acquiring additional information related to the generated answer from a knowledge base. As a result, when a child asks a question about the scenery outside while in the car, an appropriate answer can be automatically and quickly generated, allowing the whole family to share new knowledge.

[0697] "Voice input means" is a general term for devices that capture voices uttered by a user.

[0698] The "means for converting voice data into text data" is a technology that uses voice recognition technology to convert input voice into corresponding text data.

[0699] "Means for analyzing text data and recognizing the intent of the question" refers to a technology that uses natural language processing technology to understand the user's intent and the content of the question from text data.

[0700] "Means for obtaining current location information" refers to a technology that allows the system to obtain current geographical location information using a GPS module or the like.

[0701] The "means for searching for information on an object based on current location information" is a technology for searching for information on a specific object from a database or the like based on the acquired location information.

[0702] "Means for analyzing search results and generating answers" refers to technology that processes the data obtained through a search and constructs an appropriate answer to the user's question.

[0703] The "means for obtaining additional information related to the generated answer from a knowledge base" is a technique for obtaining supplemental information from a database or the like in order to add the supplemental information to the generated answer.

[0704] A "generative AI model" is an artificial intelligence technology that can learn from large amounts of data and generate natural, human-like answers.

[0705] "Means for reproducing audio data" refers to technology that converts text data into audio using speech synthesis technology and outputs it from a speaker or the like.

[0706] A "prompt sentence" is an input sentence that causes a generative AI model to generate an appropriate answer.

[0707] The present invention is an educational system in which a system installed in an autonomous vehicle provides appropriate answers to questions posed by a child inside the vehicle about the scenery outside. This system includes a voice input means, a means for converting voice data into text data, a means for analyzing the text data and recognizing the intent of the question, a means for acquiring current location information, a means for searching for information on an object based on the current location information, a means for analyzing the search results and generating an answer, a means for acquiring additional information related to the generated answer from a knowledge base, a means for converting the generated answer into voice data, and a means for playing back the voice data.

[0708] The server first captures the child's voice using a voice input method (e.g., a microphone).The voice data is then converted into text data using a tool such as the Google Speech-to-Text API.The text data is then analyzed using natural language processing technology (e.g., the Google Cloud Natural Language API) to recognize the intent of the question.

[0709] Next, the current location information is acquired using a GPS module and sent to the server along with the text data. The server then searches for information about the object from sources such as a database based on the current location information. For example, if information about a specific mountain is to be acquired, the server queries the database for data related to that mountain.

[0710] The search results are analyzed using a generative AI model (e.g., GPT-4) to generate an appropriate answer to the user's question. For example, the answer may be, "This mountain is Mount Fuji. It is the highest mountain in Japan." Furthermore, additional relevant information can be retrieved from a knowledge base and added to the answer, making the provided answer more detailed.

[0711] Finally, the generated answers are converted into audio data using the Google Text-to-Speech API and played through the vehicle's speakers, allowing the child to get a spoken answer to their question.

[0712] As a concrete example, if a child asks, "What mountain is this?", the microphone captures the voice and speech recognition technology generates the text data "What mountain is this?". Natural language processing technology analyzes the intent of the question, and if it determines that the current location is, for example, Nagano Prefecture, the server searches a database for information on nearby mountains. The answer generated might be, "That mountain is the Hakuba Mountains," with additional information added, such as, "The Hakuba Mountains are famous for their beautiful hiking trails." This answer is then played back as audio from the microphone.

[0713] An example prompt here is, "Please describe the mountain the child saw based on the following data: Hakuba Mountains."

[0714] In this way, the present invention not only satisfies a child's intellectual curiosity and enhances the educational effect in the car, but also provides an opportunity for parents and children to learn together.

[0715] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0716] Step 1:

[0717] The device uses the microphone in the car to capture the voice of the user, a child, asking, "What mountain is this?" The input is the user's voice, and the output is the voice data.

[0718] Step 2:

[0719] The device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the captured voice data into text data. The input is the voice data, and the output is the text data "What mountain is this?"

[0720] Step 3:

[0721] The terminal sends the converted text data to the server. The input here is the text data "What is this mountain?" and the output is the transmission of the text data to the server.

[0722] Step 4:

[0723] The server receives the text data and analyzes it using a natural language processing engine (e.g., Google Cloud Natural Language API) to recognize the intent of the question. The input is the text data "What is this mountain?", and the output is the analyzed intent of the question (e.g., "I want to know the name of the mountain").

[0724] Step 5:

[0725] The terminal obtains the current location information using the GPS module. The input is the current geographical location, and the output is the current location information (e.g., "Nagano Prefecture").

[0726] Step 6:

[0727] The terminal sends the acquired current location information to the server. The input is the current location information "Nagano Prefecture," and the output is the transmission of the location information to the server.

[0728] Step 7:

[0729] The server executes a database query based on the received text data and location information to search for the target mountain. The input is the text data "What mountain is this?" and the current location information "Nagano Prefecture," and the output is the corresponding mountain information (e.g., "Hakuba Mountains").

[0730] Step 8:

[0731] The server uses a generative AI model (for example, GPT-4) based on the search results to generate an appropriate answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails." The input is the search result "Hakuba Mountain," and the output is the generated answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails."

[0732] Step 9:

[0733] The server retrieves additional information related to the generated answer from the knowledge base and adds it to the answer. The input is the generated answer "This mountain is Hakuba Mountains." The output is the answer with the additional information added: "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails."

[0734] Step 10:

[0735] The server sends the final answer to the terminal. The input is the answer "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails." with additional information added, and the output is the sending of the answer to the terminal.

[0736] Step 11:

[0737] The device converts the received text data into voice data using a speech synthesis API (for example, Google Text-to-Speech API). The input is the text data "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails.", and the output is voice data.

[0738] Step 12:

[0739] The terminal uses a speaker to play back the generated voice data and provide an answer to the user (child). The input is the voice data, and the output is voice output from the speaker.

[0740] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0741] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[0742] System Configuration

[0743] This system consists of the following main components:

[0744] 1. Terminal

[0745] Voice input means (microphone)

[0746] A means of converting voice data into text data (voice recognition API)

[0747] Emotion engine (means of analyzing user emotions)

[0748] A means of obtaining current location information (GPS module)

[0749] A means of playing audio data (speaker)

[0750] 2. Server

[0751] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0752] A method for searching for information about an object based on its current location (database query)

[0753] A means of analyzing search results and generating answers (generative AI)

[0754] A means of generating additional information related to the generated answer (knowledge database)

[0755] Program processing overview

[0756] 1. Enter the user's question

[0757] User: A child in the car asks, "Where is this plane going?"

[0758] Device: Built-in microphone captures audio.

[0759] 2. Speech-to-text

[0760] On the device: Use a speech recognition API to convert the voice data into text data. For example, use the Google Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[0761] 3. Question Analysis

[0762] Terminal: Sends text data to the server.

[0763] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[0764] 4. Emotion Analysis

[0765] On the device: The emotion engine analyzes the voice data and recognizes the user's emotions, for example, whether a child is excited or curious.

[0766] 5. Obtaining location information

[0767] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[0768] 6. Executing a Database Query

[0769] Device: Sends the acquired location and emotion information to the server.

[0770] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[0771] 7. Answer Generation

[0772] Server: Analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[0773] 8. Generating Additional Information

[0774] Server: Additionally, generate additional information related to the answer, for example, "Los Angeles is a large city famous for its film industry."

[0775] 9. Voice output of answers

[0776] Server: Sends the generated answer and additional information to the device.

[0777] On the device: Convert text data into audio data using a speech synthesis API, for example, the Google Text-to-Speech API.

[0778] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[0779] Specific examples

[0780] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[0781] This process provides accurate and personalized answers to a child's questions, helping both parents and children learn.

[0782] The processing flow will be explained below.

[0783] Step 1:

[0784] User: A child in the car asks, "Where is this plane going?"

[0785] Device: A built-in microphone captures your child's voice.

[0786] Step 2:

[0787] On the device: Uses a speech recognition API to convert captured voice data into text data.

[0788] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[0789] Step 3:

[0790] Terminal: Sends text data to the server.

[0791] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[0792] Example: Use the BERT model to determine "The person is asking about the flight destination."

[0793] Step 4:

[0794] Device: The emotion engine analyzes the voice data and recognizes the user's emotions.

[0795] For example: Analyzing speech tone, volume, rhythm, etc. to determine whether a child is excited or curious.

[0796] Step 5:

[0797] Device: Uses the GPS module to obtain current location information.

[0798] Example: Get current location as "Tokyo".

[0799] Step 6:

[0800] Device: Sends the acquired current location information and recognized emotion information to the server.

[0801] Step 7:

[0802] Server: Queries the aircraft database based on the received location and emotion information.

[0803] Example: Running an SQL query to get information about planes flying near your current location.

[0804] Step 8:

[0805] Server: Analyzes the query results and generates an answer to the user's question, taking into account information from the emotion engine and adjusting the tone and content appropriately.

[0806] For example: If a child is excited, generate the answer "Wow! That plane is heading to Los Angeles International Airport in the US! How exciting!"

[0807] Step 9:

[0808] Server: Also generates additional information related to the answer.

[0809] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[0810] Step 10:

[0811] Server: Sends the generated answer and additional information to the device.

[0812] Step 11:

[0813] On the device: Use a speech synthesis API to convert text data into audio data.

[0814] Example: Call the Google Text-to-Speech API to convert text to audio data.

[0815] Step 12:

[0816] Terminal: Plays back the generated audio data using a speaker.

[0817] Example: Play audio that says, "Wow! The plane is heading to Los Angeles International Airport in the United States! Los Angeles is a big city famous for its film industry. How exciting!"

[0818] If the user is excited, the tone and content of the response can be adjusted to provide a more personalized experience. This process ensures that the child's questions are answered appropriately and emotionally, creating a rich learning experience for both parent and child.

[0819] Example 2

[0820] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0821] Conventional in-car voice response systems only provide simple answers to user questions and are unable to provide personalized responses that take into account the user's emotional state or current location. This has been a challenge, particularly in satisfying children's intellectual curiosity and providing an environment where parents and children can learn together. Furthermore, there has been a lack of systems that provide appropriate additional information based on the user's emotions and current location.

[0822] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0823] In this invention, the server includes a means for converting voice data into text data, a means for analyzing the user's emotions, and a means for acquiring current location information, thereby enabling the provision of personalized answers and additional information that take into account the user's emotional state and current location.

[0824] "Audio input means" is a device for capturing audio uttered by a user.

[0825] A "means for converting audio data into text data" is a process or technique that analyzes captured audio data and converts it into corresponding text data.

[0826] The "means for analyzing user emotions" is a function or engine that recognizes and analyzes the user's emotional state (e.g., joy, excitement, interest, etc.) from voice data.

[0827] The "means for analyzing text data and recognizing the intent of the question" is a technology for analyzing the converted text data and understanding the user's question and intent.

[0828] "Means for obtaining current location information" refers to a device or function for obtaining latitude and longitude information of the user's current location using location measurement technology such as GPS.

[0829] The "means for searching for information about an object based on current location information and emotion data" is a process for searching for appropriate information about an object from related databases or information sources using the acquired location information and emotion information.

[0830] "Means for analyzing search results and generating answers" refers to technology that generates appropriate answers to user questions based on information acquired through searches.

[0831] The "means for generating additional information related to the generated answer" is a technique for adding supplementary information or interesting information further related to the generated answer.

[0832] The "means for converting the generated answers into voice data" is a voice synthesis technology that converts answers in text format into voice format.

[0833] The "means for reproducing audio data" is a technique for providing the converted audio data to the user through an output device such as a speaker.

[0834] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[0835] System Configuration

[0836] This system consists of the following main components:

[0837] 1. Terminal

[0838] Voice input means (microphone)

[0839] A means of converting voice data into text data (voice recognition API)

[0840] Emotion engine (means of analyzing user emotions)

[0841] A means of obtaining current location information (GPS module)

[0842] A means of playing audio data (speaker)

[0843] 2. Server

[0844] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[0845] A method for searching for information about an object based on its current location (database query)

[0846] A means of analyzing search results and generating answers (generative AI)

[0847] A means of generating additional information related to the generated answer (knowledge database)

[0848] System Operation

[0849] The operation of this system is carried out in the following procedure.

[0850] 1. Enter the user's question

[0851] A child user in a car asks, "Where is this plane going?" The device's built-in microphone captures the voice.

[0852] 2. Speech-to-text

[0853] The device uses a speech recognition API (for example, Google Speech-to-Text API) to convert the voice data into text data, which becomes "Where is this plane going?"

[0854] 3. Question Analysis

[0855] The device sends the text data to a server, where a natural language processing engine analyzes the text and recognizes the intent of the child's question, such as asking about the flight's destination.

[0856] 4. Emotion Analysis

[0857] The device's emotion engine analyzes the voice data and recognizes the user's emotions, for example, determining whether a child is excited or curious.

[0858] 5. Obtaining location information

[0859] The device uses the GPS module to obtain the current location information. For example, it verifies that the current location is "Tokyo."

[0860] 6. Executing a Database Query

[0861] The device sends the acquired location and emotion information to a server, which queries a database containing information on airplanes to obtain data on airplanes flying near the current location.

[0862] 7. Answer Generation

[0863] The server analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[0864] 8. Generating Additional Information

[0865] The server also generates additional information related to the answer, for example, "Los Angeles is a large city known for its film industry."

[0866] 9. Voice output of answers

[0867] The server sends the generated answer and additional information to the device. The device converts the text data into audio data using a speech synthesis API (e.g., Google Text-to-Speech API). The audio data is played back using the device's speaker, providing the answer to the user. For example, it could say, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a major city famous for its film industry."

[0868] Specific examples

[0869] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[0870] Prompt Sentence Examples

[0871] "When a child asks, 'Where is this plane going?', provide an appropriate voice response based on real-time aircraft data and current location. Choose an appropriate tone for the response and include any additional relevant information."

[0872] This system satisfies children's intellectual curiosity and provides an environment where parents and children can learn together.

[0873] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0874] Step 1:

[0875] A user asks a question in the car. A child says, "Where is this plane going?" This is the input voice data for the system. The device's microphone receives this voice and captures it as voice data.

[0876] Step 2:

[0877] The device converts the voice data into text data. The device uses a voice recognition API (for example, a voice recognition cloud service) to convert the captured voice data into text data such as "Where is this plane going?" This is a process that performs phonemic analysis on the voice data and converts it into the corresponding text. The output is the converted text data.

[0878] Step 3:

[0879] The terminal sends text data to the server. Based on the text data input, the terminal securely sends the data to the server using the HTTPS protocol. The output is the text data sent to the server.

[0880] Step 4:

[0881] The server analyzes the text data and recognizes the intent of the question. The server uses a natural language processing engine to analyze the text "Where is this plane going?", thereby recognizing the meaning "asking about the plane's destination." The output is the intent information of the question.

[0882] Step 5:

[0883] The device analyzes emotions. The device's emotion engine analyzes the voice data and identifies the user's (child's) emotional state. For example, it determines whether the child is excited or curious. The input is the captured voice data, and the output is the analyzed emotion data.

[0884] Step 6:

[0885] The device obtains location information. The device's GPS module is used to obtain the user's current location. For example, confirm that the current location is within Tokyo's 23 wards. The input is a signal from a GPS satellite, and the output is latitude and longitude information.

[0886] Step 7:

[0887] The device sends emotion data and location information to the server. The device sends the existing emotion data and location information to the server using the HTTPS protocol. The output is the emotion data and location information sent to the server.

[0888] Step 8:

[0889] The server executes a database query. Based on the location information it receives, it retrieves the relevant data from a database containing information about airplanes. The server issues a query such as "Search for information about airplanes in flight" and retrieves data from the database such as "Japan Airlines Flight 123 is heading to Los Angeles." The output is the query result.

[0890] Step 9:

[0891] The server analyzes the query results and generates an answer. The server uses an automatic generation engine to prepare an answer to the user's question. At this time, it also takes into account emotional data and generates an answer such as "This plane is heading to Los Angeles, USA." The output is a specific answer sentence.

[0892] Step 10:

[0893] The server generates additional information. The server searches for relevant information from a knowledge database and adds additional information related to the answer. For example, it adds information such as "Los Angeles is a city famous for its film industry." The output is a set of the answer and the additional information.

[0894] Step 11:

[0895] The server sends the generated answer and additional information to the terminal. The server sends the answer and additional information as text data to the terminal. The output is the text data sent to the terminal.

[0896] Step 12:

[0897] The device converts the text data into voice data. The device uses a voice synthesis API (e.g., a voice synthesis cloud service) to convert the text data into voice data. The output is the converted voice data.

[0898] Step 13:

[0899] The terminal plays the audio data. The converted audio data is played using the terminal's speaker, and the content is provided to the user as "This plane is heading to Los Angeles, USA. Los Angeles is a city famous for its film industry." The output is a voice response.

[0900] (Application example 2)

[0901] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0902] Conventional in-car knowledge provision systems make it difficult for parents to provide immediate and appropriate answers when their children ask about the scenery. Furthermore, rather than simply providing answers, personalized responses based on the child's emotions and interests are necessary. Furthermore, there is a need for real-time information provision based on the car's location and the situation at hand while the car is moving. It is necessary to solve these problems and provide a fun learning environment for both parents and children.

[0903] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing voice data to recognize the intent of the question, means for analyzing the user's emotions, and means for searching for information on the object based on current location information and emotion information. This makes it possible to provide appropriate and personalized answers to children's questions that take into account emotions and current location.

[0904] The "voice input means" is a means for acquiring voice and processing the acquired voice.

[0905] The "means for converting voice data into text data" is a means for converting acquired voice data into character string data.

[0906] The "means for analyzing text data and recognizing the intent of the question" is a means for analyzing the converted text data and understanding the purpose and intent of the user's question from its contents.

[0907] The "means for analyzing the user's emotions" is a means for analyzing the emotions of the user from voice data and other input data, and grasping the emotional state.

[0908] "Means for obtaining current location information" refers to means for identifying the current geographic location using GPS or other location information technology.

[0909] The "means for searching for information on an object based on current location information and emotion information" is a means for combining the acquired current location information and emotion information to search for information on related objects from a database or other information sources.

[0910] The "means for analyzing search results and generating an answer" refers to a means for analyzing information obtained through a search and generating an appropriate answer to a user's question based on that information.

[0911] The "means for converting the generated answer into voice data" is a means for converting the generated answer in text format into voice format.

[0912] The "means for reproducing audio data" refers to a means for transmitting the generated audio data to a user through a speaker or other reproducing device.

[0913] System Configuration

[0914] The present invention is a system that provides knowledge on behalf of parents in response to questions posed by children in the car, and further provides more personalized responses by recognizing the user's emotions and adjusting the content of the answers. The system consists of the following main components:

[0915] 1. Terminal

[0916] Voice input means (microphone)

[0917] A means of converting voice data into text data (Google Speech-to-Text API)

[0918] A means of analyzing user emotions (IBM Watson Tone Analyzer)

[0919] A means of obtaining current location information (GPS module)

[0920] A means of playing audio data (speaker)

[0921] 2. Server

[0922] A means of analyzing text data and recognizing the intent of the question (Google Natural Language API)

[0923] A method to search for information about an object based on its current location and emotion (Google Cloud Firestore)

[0924] A means of analyzing search results and generating answers (generative AI)

[0925] A way to convert the generated answers into audio data (Google Text-to-Speech API)

[0926] System operation explanation

[0927] Voice to text conversion

[0928] Any questions asked by the user in the car are captured by a microphone on the device, and the voice data is converted to text data using the Google Speech-to-Text API, which is then sent to a server to understand what the user is asking.

[0929] Question analysis and sentiment analysis

[0930] The server receives the text data and analyzes the intent of the question using the Google Natural Language API, while simultaneously analyzing the user's sentiment using IBM Watson Tone Analyzer. This sentiment data is used to adjust the tone and content of the response.

[0931] Obtaining current location information

[0932] The device's GPS module obtains the current location information, which is also sent to the server, and is used to search for objects related to the query.

[0933] Database Querying and Answer Generation

[0934] The server searches for relevant information using Google Cloud Firestore based on the acquired current location and emotion information. The search results are analyzed and a generative AI generates an answer to the question. The generated answer is converted into audio data using the Google Text-to-Speech API and sent to the device.

[0935] Audio playback

[0936] Finally, the generated audio data is played back through the device's speaker to provide the answer to the user, allowing for an appropriate and personalized response to the child's question.

[0937] Specific examples

[0938] As an example of actual operation, consider a child in a car asking, "Where is this plane going?" The device captures the audio and converts it to text using the Google Speech-to-Text API. The text data is then sent to the server, where the Google Natural Language API analyzes the intent of the question. At the same time, IBM Watson Tone Analyzer analyzes the audio data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server queries Google Cloud Firestore to obtain information about the plane, adds emotional information, and generates an appropriate answer. This answer is converted into audio data using the Google Text-to-Speech API and played through the device's speaker.

[0939] Prompt Sentence Examples

[0940] You are an assistant that generates personalized answers to questions your child asks in the car.

[0941] Here are some questions: "Where is this plane going?"

[0942] Your current location is Tokyo. Your flight destination is Los Angeles International Airport.

[0943] Answer: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!"

[0944] This process allows parents to concentrate on driving with peace of mind, while creating an environment where children can have fun and satisfy their curiosity.

[0945] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0946] Step 1:

[0947] The user speaks a question while in the car, and the device's microphone captures the voice and saves it as audio data.

[0948] Input: User's voice question

[0949] Output: Captured audio data

[0950] Step 2:

[0951] The device uses a speech recognition API (Google Speech-to-Text API) to convert the voice data into text data.

[0952] Input: Audio data

[0953] Output: Text data (e.g., "Where is this plane going?")

[0954] Step 3:

[0955] The device sends text data to the server, which then uses a natural language processing engine (Google Natural Language API) to analyze the intent of the question.

[0956] Input: Text data

[0957] Output: The intent of the question (e.g., asking for flight destination)

[0958] Step 4:

[0959] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) to analyze the voice data and recognize the user's emotions.

[0960] Input: Audio data

[0961] Output: User emotion (e.g., excitement, curiosity)

[0962] Step 5:

[0963] The device's GPS module acquires the current location information and sends it to the server.

[0964] Input: Location measurement

[0965] Output: Current location (e.g., Tokyo)

[0966] Step 6:

[0967] The server uses Google Cloud Firestore to search for information about related objects (e.g., airplanes in flight) based on the current location and emotion information.

[0968] Input: Current location, emotion

[0969] Output: Object information (e.g., flight destination)

[0970] Step 7:

[0971] The server analyzes the search results and generates an answer to the user's question, adjusting the tone and content of the answer based on emotional information.

[0972] Input: Object information, emotion information, question intent

[0973] Output: Answer (e.g., "This plane is heading to Los Angeles International Airport in the United States.")

[0974] Step 8:

[0975] The server converts the generated answer into audio data using the Google Text-to-Speech API and sends it to the device.

[0976] Input: Answer

[0977] Output: Audio data

[0978] Step 9:

[0979] The terminal uses a speaker to play back the audio data and provide the answer to the user.

[0980] Input: Audio data

[0981] Output: Speech to be played to the user (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!")

[0982] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0983] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0984] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0985] [Third embodiment]

[0986] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0987] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0988] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0989] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0990] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0991] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0992] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0993] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0994] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0995] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0996] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0997] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0998] This system responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system not only satisfies children's intellectual curiosity, but also creates an environment where parents can learn together by providing useful information to parents.

[0999] System Configuration

[1000] This system consists of the following main components:

[1001] 1. Terminal

[1002] Voice input means (microphone)

[1003] A means of converting voice data into text data (voice recognition API)

[1004] A means of obtaining current location information (GPS module)

[1005] A means of playing audio data (speaker)

[1006] 2. Server

[1007] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1008] A method for searching for information about an object based on its current location (database query)

[1009] A means of analyzing search results and generating answers (generative AI)

[1010] A means of generating additional information related to the generated answer (knowledge database)

[1011] Program processing overview

[1012] 1. Enter the user's question

[1013] User: A child in the car asks, "Where is this plane going?"

[1014] Device: Built-in microphone captures audio.

[1015] 2. Speech-to-text

[1016] On the device: Use a speech recognition API to convert the voice data into text. For example, use the Google Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[1017] 3. Question Analysis

[1018] Terminal: Sends text data to the server.

[1019] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[1020] 4. Obtaining location information

[1021] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[1022] 5. Executing a Database Query

[1023] Device: Sends the acquired location information to the server.

[1024] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[1025] 6. Answer Generation

[1026] Server: Analyzes the received flight information and generates an appropriate response, such as "This flight is heading to Los Angeles International Airport in the United States."

[1027] Server: In addition, it retrieves additional information about Los Angeles from the knowledge database and adds it, for example, "Los Angeles is a large city famous for its film industry."

[1028] 7. Voice output of answers

[1029] Server: Sends the generated text data to the device.

[1030] Terminal: Convert the received text data into voice data using a speech synthesis API, such as the Google Text-to-Speech API.

[1031] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1032] Specific examples

[1033] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[1034] In this way, the system satisfies children's intellectual curiosity and provides useful information to parents, enhancing the educational benefits of in-car experiences.

[1035] The processing flow will be explained below.

[1036] Step 1:

[1037] User: A child in the car asks, "Where is this plane going?"

[1038] Device: A built-in microphone captures your child's voice.

[1039] Step 2:

[1040] On the device: Uses a speech recognition API to convert captured voice data into text data.

[1041] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[1042] Step 3:

[1043] Terminal: Sends text data to the server.

[1044] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[1045] Example: Use the BERT model to determine "The person is asking about the flight destination."

[1046] Step 4:

[1047] Device: Uses the GPS module to obtain current location information.

[1048] Example: Get current location as "Tokyo".

[1049] Step 5:

[1050] Device: Sends the acquired current location information to the server.

[1051] Step 6:

[1052] Server: Queries the aircraft database based on the received current location information.

[1053] Example: Running an SQL query to get information about planes flying near your current location.

[1054] Step 7:

[1055] Server: Parses the query results and generates an answer to the user's question.

[1056] Example: Generate the answer "This plane is heading to Los Angeles International Airport in the United States."

[1057] Step 8:

[1058] Server: Generates additional information related to the answer.

[1059] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[1060] Step 9:

[1061] Server: Sends the generated answer and additional information to the device.

[1062] Step 10:

[1063] On the device: Use a speech synthesis API to convert text data into audio data.

[1064] Example: Call the Google Text-to-Speech API to convert text to audio data.

[1065] Step 11:

[1066] Terminal: Plays back the generated audio data using a speaker.

[1067] Example: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1068] This series of processes provides appropriate answers to the child's questions, allowing both parent and child to learn.

[1069] Example 1

[1070] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1071] There is a demand for a system that can quickly and accurately answer questions that children in the car ask about the scenery outside, instead of the parent who is driving, thereby satisfying the child's intellectual curiosity, reducing the burden on the parent, and providing an educational environment inside the car.

[1072] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1073] In this invention, the server includes means for analyzing text data to recognize the intent of the question, means for searching for information on the object based on the acquired location information, and means for analyzing the search results to generate an answer, thereby enabling the server to quickly and accurately provide information on places and objects based on the child's question.

[1074] The "voice input means" is an input device for capturing voices uttered by the user.

[1075] The "means for converting voice data into text data" is a technology for analyzing captured voice data and converting it into corresponding text data.

[1076] "Means for analyzing text data and recognizing the intent of a question" refers to natural language processing technology for analyzing text data, understanding its content, and determining the intent of a question.

[1077] "Means for acquiring location information" refers to technology that acquires information on the latitude and longitude of the current location using a GPS module or the like.

[1078] The "means for searching for information on an object based on acquired location information" is a technology for searching a database for information on a specific object based on acquired location information.

[1079] "Means for analyzing search results and generating answers" refers to technology for analyzing information obtained through a search and generating appropriate answers to the user's questions.

[1080] The "means for generating additional information related to the generated answer" is a technique for adding further related knowledge or information to the generated answer to provide a more detailed answer.

[1081] The "means for converting the generated answer into voice data" is a technology for converting the answer in text format into voice data, and making it into a format that can be output as voice.

[1082] "Means for reproducing audio data" refers to a technique for reproducing audio data using an output device such as a speaker, thereby providing information to the user auditorily.

[1083] This invention is a system that responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system satisfies children's intellectual curiosity and creates an environment where both parents can learn by providing useful information.

[1084] System Configuration

[1085] This system consists of the following main components:

[1086] 1. Terminal

[1087] Voice input means (microphone)

[1088] A means of converting voice data into text data (voice recognition API)

[1089] Means of obtaining location information (GPS module)

[1090] A means of playing audio data (speaker)

[1091] 2. Server

[1092] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1093] A method for searching for information about an object based on the acquired location information (database query)

[1094] A means of analyzing search results and generating answers (generative AI model)

[1095] A means of generating additional information related to the generated answer (knowledge database)

[1096] A means to convert the generated answers into voice data (voice synthesis API)

[1097] System Operation

[1098] 1. Enter the user's question

[1099] User: A child asks, "Where is this plane going?"

[1100] Device: A sensitive microphone inside the car captures the child's voice.

[1101] 2. Speech-to-text

[1102] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[1103] 3. Question Analysis

[1104] Terminal: The converted text data is sent to the server using the HTTP POST method.

[1105] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[1106] 4. Obtaining location information

[1107] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[1108] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[1109] 5. Executing a Database Query

[1110] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[1111] Server: Query the database to get data on planes flying around the current location.

[1112] 6. Answer Generation

[1113] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[1114] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[1115] 7. Voice output of answers

[1116] Server: Sends the generated text data to the device.

[1117] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[1118] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1119] Specific examples

[1120] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[1121] Example prompt sentence:

[1122] When asked "Where is this plane going?", provide an appropriate answer by determining the plane's destination based on its current location, and provide any additional information about the destination.

[1123] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1124] Step 1:

[1125] Entering user questions

[1126] User: A child asks, "Where is this plane going?"

[1127] Device: A sensitive microphone installed inside the car captures the child's voice.

[1128] Input: Child's voice.

[1129] Processing: Captured as audio data by microphone.

[1130] Output: The captured audio data.

[1131] Step 2:

[1132] Speech-to-text

[1133] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[1134] Input: The captured audio data.

[1135] Processing: Send the audio data to the speech recognition API and convert it into corresponding text data.

[1136] Output: The converted text data (e.g., "Where is this plane going?").

[1137] Step 3:

[1138] Question Analysis

[1139] Terminal: The converted text data is sent to the server using the HTTP POST method.

[1140] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[1141] Input: Text data (e.g., "Where is this plane going?").

[1142] Processing: Text data is sent to the server via an HTTP POST request, and the server analyzes it using a natural language processing engine.

[1143] Output: The intent of the question (e.g., "Question about flight destinations").

[1144] Step 4:

[1145] Obtaining location information

[1146] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[1147] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[1148] Input: Current location (latitude and longitude).

[1149] Processing: Obtain location information using the GPS module and send it to the server via an HTTP POST request.

[1150] Output: The location information sent.

[1151] Step 5:

[1152] Executing a Database Query

[1153] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[1154] Server: Query the database to get data on planes flying around the current location.

[1155] Input: Location and question intent.

[1156] Processing: Query the database to get the relevant plane information.

[1157] Output: Plane information data.

[1158] Step 6:

[1159] Generate answers

[1160] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[1161] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[1162] Input: Airplane information data.

[1163] Processing: A generative AI model generates an answer and adds additional information from a knowledge database.

[1164] Output: The generated answer with additional information (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city known for its film industry.").

[1165] Step 7:

[1166] Voice output of answers

[1167] Server: Sends the generated text data to the device.

[1168] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[1169] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1170] Input: Generated text data.

[1171] Processing: Convert into audio data using speech synthesis API and play on speaker.

[1172] Output: An audio response that the user can hear.

[1173] (Application example 1)

[1174] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1175] In recent years, with the increasing popularity of self-driving vehicles, people are spending more time in their cars. Children, in particular, are increasingly curious about the scenery outside and asking questions. However, it is difficult, inconvenient, and potentially dangerous for parents to respond to such questions while driving. Furthermore, there are limited means to satisfy children's intellectual curiosity and provide an environment where parents and children can learn together. This results in lost opportunities for educational interaction between families in the car. The present invention aims to solve these problems by providing a system that responds appropriately and quickly to children's questions.

[1176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1177] In this invention, the server includes means for searching for information on objects based on current location information, means for analyzing the search results and generating an answer, and means for acquiring additional information related to the generated answer from a knowledge base. As a result, when a child asks a question about the scenery outside while in the car, an appropriate answer can be automatically and quickly generated, allowing the whole family to share new knowledge.

[1178] "Voice input means" is a general term for devices that capture voices uttered by a user.

[1179] The "means for converting voice data into text data" is a technology that uses voice recognition technology to convert input voice into corresponding text data.

[1180] "Means for analyzing text data and recognizing the intent of the question" refers to a technology that uses natural language processing technology to understand the user's intent and the content of the question from text data.

[1181] "Means for obtaining current location information" refers to a technology that allows the system to obtain current geographical location information using a GPS module or the like.

[1182] The "means for searching for information on an object based on current location information" is a technology for searching for information on a specific object from a database or the like based on the acquired location information.

[1183] "Means for analyzing search results and generating answers" refers to technology that processes the data obtained through a search and constructs an appropriate answer to the user's question.

[1184] The "means for obtaining additional information related to the generated answer from a knowledge base" is a technique for obtaining supplemental information from a database or the like in order to add the supplemental information to the generated answer.

[1185] A "generative AI model" is an artificial intelligence technology that can learn from large amounts of data and generate natural, human-like answers.

[1186] "Means for reproducing audio data" refers to technology that converts text data into audio using speech synthesis technology and outputs it from a speaker or the like.

[1187] A "prompt sentence" is an input sentence that causes a generative AI model to generate an appropriate answer.

[1188] The present invention is an educational system in which a system installed in an autonomous vehicle provides appropriate answers to questions posed by a child inside the vehicle about the scenery outside. This system includes a voice input means, a means for converting voice data into text data, a means for analyzing the text data and recognizing the intent of the question, a means for acquiring current location information, a means for searching for information on an object based on the current location information, a means for analyzing the search results and generating an answer, a means for acquiring additional information related to the generated answer from a knowledge base, a means for converting the generated answer into voice data, and a means for playing back the voice data.

[1189] The server first captures the child's voice using a voice input method (e.g., a microphone).The voice data is then converted into text data using a tool such as the Google Speech-to-Text API.The text data is then analyzed using natural language processing technology (e.g., the Google Cloud Natural Language API) to recognize the intent of the question.

[1190] Next, the current location information is acquired using a GPS module and sent to the server along with the text data. The server then searches for information about the object from sources such as a database based on the current location information. For example, if information about a specific mountain is to be acquired, the server queries the database for data related to that mountain.

[1191] The search results are analyzed using a generative AI model (e.g., GPT-4) to generate an appropriate answer to the user's question. For example, the answer may be, "This mountain is Mount Fuji. It is the highest mountain in Japan." Furthermore, additional relevant information can be retrieved from a knowledge base and added to the answer, making the provided answer more detailed.

[1192] Finally, the generated answers are converted into audio data using the Google Text-to-Speech API and played through the vehicle's speakers, allowing the child to get a spoken answer to their question.

[1193] As a concrete example, if a child asks, "What mountain is this?", the microphone captures the voice and speech recognition technology generates the text data "What mountain is this?". Natural language processing technology analyzes the intent of the question, and if it determines that the current location is, for example, Nagano Prefecture, the server searches a database for information on nearby mountains. The answer generated might be, "That mountain is the Hakuba Mountains," with additional information added, such as, "The Hakuba Mountains are famous for their beautiful hiking trails." This answer is then played back as audio from the microphone.

[1194] An example prompt here is, "Please describe the mountain the child saw based on the following data: Hakuba Mountains."

[1195] In this way, the present invention not only satisfies a child's intellectual curiosity and enhances the educational effect in the car, but also provides an opportunity for parents and children to learn together.

[1196] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1197] Step 1:

[1198] The device uses the microphone in the car to capture the voice of the user, a child, asking, "What mountain is this?" The input is the user's voice, and the output is the voice data.

[1199] Step 2:

[1200] The device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the captured voice data into text data. The input is the voice data, and the output is the text data "What mountain is this?"

[1201] Step 3:

[1202] The terminal sends the converted text data to the server. The input here is the text data "What is this mountain?" and the output is the transmission of the text data to the server.

[1203] Step 4:

[1204] The server receives the text data and analyzes it using a natural language processing engine (e.g., Google Cloud Natural Language API) to recognize the intent of the question. The input is the text data "What is this mountain?", and the output is the analyzed intent of the question (e.g., "I want to know the name of the mountain").

[1205] Step 5:

[1206] The terminal obtains the current location information using the GPS module. The input is the current geographical location, and the output is the current location information (e.g., "Nagano Prefecture").

[1207] Step 6:

[1208] The terminal sends the acquired current location information to the server. The input is the current location information "Nagano Prefecture," and the output is the transmission of the location information to the server.

[1209] Step 7:

[1210] The server executes a database query based on the received text data and location information to search for the target mountain. The input is the text data "What mountain is this?" and the current location information "Nagano Prefecture," and the output is the corresponding mountain information (e.g., "Hakuba Mountains").

[1211] Step 8:

[1212] The server uses a generative AI model (for example, GPT-4) based on the search results to generate an appropriate answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails." The input is the search result "Hakuba Mountain," and the output is the generated answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails."

[1213] Step 9:

[1214] The server retrieves additional information related to the generated answer from the knowledge base and adds it to the answer. The input is the generated answer "This mountain is Hakuba Mountains." The output is the answer with the additional information added: "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails."

[1215] Step 10:

[1216] The server sends the final answer to the terminal. The input is the answer "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails." with additional information added, and the output is the sending of the answer to the terminal.

[1217] Step 11:

[1218] The device converts the received text data into voice data using a speech synthesis API (for example, Google Text-to-Speech API). The input is the text data "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails.", and the output is voice data.

[1219] Step 12:

[1220] The terminal uses a speaker to play back the generated voice data and provide an answer to the user (child). The input is the voice data, and the output is voice output from the speaker.

[1221] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1222] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[1223] System Configuration

[1224] This system consists of the following main components:

[1225] 1. Terminal

[1226] Voice input means (microphone)

[1227] A means of converting voice data into text data (voice recognition API)

[1228] Emotion engine (means of analyzing user emotions)

[1229] A means of obtaining current location information (GPS module)

[1230] A means of playing audio data (speaker)

[1231] 2. Server

[1232] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1233] A method for searching for information about an object based on its current location (database query)

[1234] A means of analyzing search results and generating answers (generative AI)

[1235] A means of generating additional information related to the generated answer (knowledge database)

[1236] Program processing overview

[1237] 1. Enter the user's question

[1238] User: A child in the car asks, "Where is this plane going?"

[1239] Device: Built-in microphone captures audio.

[1240] 2. Speech-to-text

[1241] On the device: Use a speech recognition API to convert the voice data into text data. For example, use the Google Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[1242] 3. Question Analysis

[1243] Terminal: Sends text data to the server.

[1244] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[1245] 4. Emotion Analysis

[1246] On the device: The emotion engine analyzes the voice data and recognizes the user's emotions, for example, whether a child is excited or curious.

[1247] 5. Obtaining location information

[1248] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[1249] 6. Executing a Database Query

[1250] Device: Sends the acquired location and emotion information to the server.

[1251] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[1252] 7. Answer Generation

[1253] Server: Analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[1254] 8. Generating Additional Information

[1255] Server: Additionally, generate additional information related to the answer, for example, "Los Angeles is a large city famous for its film industry."

[1256] 9. Voice output of answers

[1257] Server: Sends the generated answer and additional information to the device.

[1258] On the device: Convert text data into audio data using a speech synthesis API, for example, the Google Text-to-Speech API.

[1259] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1260] Specific examples

[1261] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[1262] This process provides accurate and personalized answers to a child's questions, helping both parents and children learn.

[1263] The processing flow will be explained below.

[1264] Step 1:

[1265] User: A child in the car asks, "Where is this plane going?"

[1266] Device: A built-in microphone captures your child's voice.

[1267] Step 2:

[1268] On the device: Uses a speech recognition API to convert captured voice data into text data.

[1269] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[1270] Step 3:

[1271] Terminal: Sends text data to the server.

[1272] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[1273] Example: Use the BERT model to determine "The person is asking about the flight destination."

[1274] Step 4:

[1275] Device: The emotion engine analyzes the voice data and recognizes the user's emotions.

[1276] For example: Analyzing speech tone, volume, rhythm, etc. to determine whether a child is excited or curious.

[1277] Step 5:

[1278] Device: Uses the GPS module to obtain current location information.

[1279] Example: Get current location as "Tokyo".

[1280] Step 6:

[1281] Device: Sends the acquired current location information and recognized emotion information to the server.

[1282] Step 7:

[1283] Server: Queries the aircraft database based on the received location and emotion information.

[1284] Example: Running an SQL query to get information about planes flying near your current location.

[1285] Step 8:

[1286] Server: Analyzes the query results and generates an answer to the user's question, taking into account information from the emotion engine and adjusting the tone and content appropriately.

[1287] For example: If a child is excited, generate the answer "Wow! That plane is heading to Los Angeles International Airport in the US! How exciting!"

[1288] Step 9:

[1289] Server: Also generates additional information related to the answer.

[1290] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[1291] Step 10:

[1292] Server: Sends the generated answer and additional information to the device.

[1293] Step 11:

[1294] On the device: Use a speech synthesis API to convert text data into audio data.

[1295] Example: Call the Google Text-to-Speech API to convert text to audio data.

[1296] Step 12:

[1297] Terminal: Plays back the generated audio data using a speaker.

[1298] Example: Play audio that says, "Wow! The plane is heading to Los Angeles International Airport in the United States! Los Angeles is a big city famous for its film industry. How exciting!"

[1299] If the user is excited, the tone and content of the response can be adjusted to provide a more personalized experience. This process ensures that the child's questions are answered appropriately and emotionally, creating a rich learning experience for both parent and child.

[1300] Example 2

[1301] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1302] Conventional in-car voice response systems only provide simple answers to user questions and are unable to provide personalized responses that take into account the user's emotional state or current location. This has been a challenge, particularly in satisfying children's intellectual curiosity and providing an environment where parents and children can learn together. Furthermore, there has been a lack of systems that provide appropriate additional information based on the user's emotions and current location.

[1303] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1304] In this invention, the server includes a means for converting voice data into text data, a means for analyzing the user's emotions, and a means for acquiring current location information, thereby enabling the provision of personalized answers and additional information that take into account the user's emotional state and current location.

[1305] "Audio input means" is a device for capturing audio uttered by a user.

[1306] A "means for converting audio data into text data" is a process or technique that analyzes captured audio data and converts it into corresponding text data.

[1307] The "means for analyzing user emotions" is a function or engine that recognizes and analyzes the user's emotional state (e.g., joy, excitement, interest, etc.) from voice data.

[1308] The "means for analyzing text data and recognizing the intent of the question" is a technology for analyzing the converted text data and understanding the user's question and intent.

[1309] "Means for obtaining current location information" refers to a device or function for obtaining latitude and longitude information of the user's current location using location measurement technology such as GPS.

[1310] The "means for searching for information about an object based on current location information and emotion data" is a process for searching for appropriate information about an object from related databases or information sources using the acquired location information and emotion information.

[1311] "Means for analyzing search results and generating answers" refers to technology that generates appropriate answers to user questions based on information acquired through searches.

[1312] The "means for generating additional information related to the generated answer" is a technique for adding supplementary information or interesting information further related to the generated answer.

[1313] The "means for converting the generated answers into voice data" is a voice synthesis technology that converts answers in text format into voice format.

[1314] The "means for reproducing audio data" is a technique for providing the converted audio data to the user through an output device such as a speaker.

[1315] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[1316] System Configuration

[1317] This system consists of the following main components:

[1318] 1. Terminal

[1319] Voice input means (microphone)

[1320] A means of converting voice data into text data (voice recognition API)

[1321] Emotion engine (means of analyzing user emotions)

[1322] A means of obtaining current location information (GPS module)

[1323] A means of playing audio data (speaker)

[1324] 2. Server

[1325] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1326] A method for searching for information about an object based on its current location (database query)

[1327] A means of analyzing search results and generating answers (generative AI)

[1328] A means of generating additional information related to the generated answer (knowledge database)

[1329] System Operation

[1330] The operation of this system is carried out in the following procedure.

[1331] 1. Enter the user's question

[1332] A child user in a car asks, "Where is this plane going?" The device's built-in microphone captures the voice.

[1333] 2. Speech-to-text

[1334] The device uses a speech recognition API (for example, Google Speech-to-Text API) to convert the voice data into text data, which becomes "Where is this plane going?"

[1335] 3. Question Analysis

[1336] The device sends the text data to a server, where a natural language processing engine analyzes the text and recognizes the intent of the child's question, such as asking about the flight's destination.

[1337] 4. Emotion Analysis

[1338] The device's emotion engine analyzes the voice data and recognizes the user's emotions, for example, determining whether a child is excited or curious.

[1339] 5. Obtaining location information

[1340] The device uses the GPS module to obtain the current location information. For example, it verifies that the current location is "Tokyo."

[1341] 6. Executing a Database Query

[1342] The device sends the acquired location and emotion information to a server, which queries a database containing information on airplanes to obtain data on airplanes flying near the current location.

[1343] 7. Answer Generation

[1344] The server analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[1345] 8. Generating Additional Information

[1346] The server also generates additional information related to the answer, for example, "Los Angeles is a large city known for its film industry."

[1347] 9. Voice output of answers

[1348] The server sends the generated answer and additional information to the device. The device converts the text data into audio data using a speech synthesis API (e.g., Google Text-to-Speech API). The audio data is played back using the device's speaker, providing the answer to the user. For example, it could say, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a major city famous for its film industry."

[1349] Specific examples

[1350] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[1351] Prompt Sentence Examples

[1352] "When a child asks, 'Where is this plane going?', provide an appropriate voice response based on real-time aircraft data and current location. Choose an appropriate tone for the response and include any additional relevant information."

[1353] This system satisfies children's intellectual curiosity and provides an environment where parents and children can learn together.

[1354] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1355] Step 1:

[1356] A user asks a question in the car. A child says, "Where is this plane going?" This is the input voice data for the system. The device's microphone receives this voice and captures it as voice data.

[1357] Step 2:

[1358] The device converts the voice data into text data. The device uses a voice recognition API (for example, a voice recognition cloud service) to convert the captured voice data into text data such as "Where is this plane going?" This is a process that performs phonemic analysis on the voice data and converts it into the corresponding text. The output is the converted text data.

[1359] Step 3:

[1360] The terminal sends text data to the server. Based on the text data input, the terminal securely sends the data to the server using the HTTPS protocol. The output is the text data sent to the server.

[1361] Step 4:

[1362] The server analyzes the text data and recognizes the intent of the question. The server uses a natural language processing engine to analyze the text "Where is this plane going?", thereby recognizing the meaning "asking about the plane's destination." The output is the intent information of the question.

[1363] Step 5:

[1364] The device analyzes emotions. The device's emotion engine analyzes the voice data and identifies the user's (child's) emotional state. For example, it determines whether the child is excited or curious. The input is the captured voice data, and the output is the analyzed emotion data.

[1365] Step 6:

[1366] The device obtains location information. The device's GPS module is used to obtain the user's current location. For example, confirm that the current location is within Tokyo's 23 wards. The input is a signal from a GPS satellite, and the output is latitude and longitude information.

[1367] Step 7:

[1368] The device sends emotion data and location information to the server. The device sends the existing emotion data and location information to the server using the HTTPS protocol. The output is the emotion data and location information sent to the server.

[1369] Step 8:

[1370] The server executes a database query. Based on the location information it receives, it retrieves the relevant data from a database containing information about airplanes. The server issues a query such as "Search for information about airplanes in flight" and retrieves data from the database such as "Japan Airlines Flight 123 is heading to Los Angeles." The output is the query result.

[1371] Step 9:

[1372] The server analyzes the query results and generates an answer. The server uses an automatic generation engine to prepare an answer to the user's question. At this time, it also takes into account emotional data and generates an answer such as "This plane is heading to Los Angeles, USA." The output is a specific answer sentence.

[1373] Step 10:

[1374] The server generates additional information. The server searches for relevant information from a knowledge database and adds additional information related to the answer. For example, it adds information such as "Los Angeles is a city famous for its film industry." The output is a set of the answer and the additional information.

[1375] Step 11:

[1376] The server sends the generated answer and additional information to the terminal. The server sends the answer and additional information as text data to the terminal. The output is the text data sent to the terminal.

[1377] Step 12:

[1378] The device converts the text data into voice data. The device uses a voice synthesis API (e.g., a voice synthesis cloud service) to convert the text data into voice data. The output is the converted voice data.

[1379] Step 13:

[1380] The terminal plays the audio data. The converted audio data is played using the terminal's speaker, and the content is provided to the user as "This plane is heading to Los Angeles, USA. Los Angeles is a city famous for its film industry." The output is a voice response.

[1381] (Application example 2)

[1382] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1383] Conventional in-car knowledge provision systems make it difficult for parents to provide immediate and appropriate answers when their children ask about the scenery. Furthermore, rather than simply providing answers, personalized responses based on the child's emotions and interests are necessary. Furthermore, there is a need for real-time information provision based on the car's location and the situation at hand while the car is moving. It is necessary to solve these problems and provide a fun learning environment for both parents and children.

[1384] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing voice data to recognize the intent of the question, means for analyzing the user's emotions, and means for searching for information on the object based on current location information and emotion information. This makes it possible to provide appropriate and personalized answers to children's questions that take into account emotions and current location.

[1385] The "voice input means" is a means for acquiring voice and processing the acquired voice.

[1386] The "means for converting voice data into text data" is a means for converting acquired voice data into character string data.

[1387] The "means for analyzing text data and recognizing the intent of the question" is a means for analyzing the converted text data and understanding the purpose and intent of the user's question from its contents.

[1388] The "means for analyzing the user's emotions" is a means for analyzing the emotions of the user from voice data and other input data, and grasping the emotional state.

[1389] "Means for obtaining current location information" refers to means for identifying the current geographic location using GPS or other location information technology.

[1390] The "means for searching for information on an object based on current location information and emotion information" is a means for combining the acquired current location information and emotion information to search for information on related objects from a database or other information sources.

[1391] The "means for analyzing search results and generating an answer" refers to a means for analyzing information obtained through a search and generating an appropriate answer to a user's question based on that information.

[1392] The "means for converting the generated answer into voice data" is a means for converting the generated answer in text format into voice format.

[1393] The "means for reproducing audio data" refers to a means for transmitting the generated audio data to a user through a speaker or other reproducing device.

[1394] System Configuration

[1395] The present invention is a system that provides knowledge on behalf of parents in response to questions posed by children in the car, and further provides more personalized responses by recognizing the user's emotions and adjusting the content of the answers. The system consists of the following main components:

[1396] 1. Terminal

[1397] Voice input means (microphone)

[1398] A means of converting voice data into text data (Google Speech-to-Text API)

[1399] A means of analyzing user emotions (IBM Watson Tone Analyzer)

[1400] A means of obtaining current location information (GPS module)

[1401] A means of playing audio data (speaker)

[1402] 2. Server

[1403] A means of analyzing text data and recognizing the intent of the question (Google Natural Language API)

[1404] A method to search for information about an object based on its current location and emotion (Google Cloud Firestore)

[1405] A means of analyzing search results and generating answers (generative AI)

[1406] A way to convert the generated answers into audio data (Google Text-to-Speech API)

[1407] System operation explanation

[1408] Voice to text conversion

[1409] Any questions asked by the user in the car are captured by a microphone on the device, and the voice data is converted to text data using the Google Speech-to-Text API, which is then sent to a server to understand what the user is asking.

[1410] Question analysis and sentiment analysis

[1411] The server receives the text data and analyzes the intent of the question using the Google Natural Language API, while simultaneously analyzing the user's sentiment using IBM Watson Tone Analyzer. This sentiment data is used to adjust the tone and content of the response.

[1412] Obtaining current location information

[1413] The device's GPS module obtains the current location information, which is also sent to the server, and is used to search for objects related to the query.

[1414] Database Querying and Answer Generation

[1415] The server searches for relevant information using Google Cloud Firestore based on the acquired current location and emotion information. The search results are analyzed and a generative AI generates an answer to the question. The generated answer is converted into audio data using the Google Text-to-Speech API and sent to the device.

[1416] Audio playback

[1417] Finally, the generated audio data is played back through the device's speaker to provide the answer to the user, allowing for an appropriate and personalized response to the child's question.

[1418] Specific examples

[1419] As an example of actual operation, consider a child in a car asking, "Where is this plane going?" The device captures the audio and converts it to text using the Google Speech-to-Text API. The text data is then sent to the server, where the Google Natural Language API analyzes the intent of the question. At the same time, IBM Watson Tone Analyzer analyzes the audio data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server queries Google Cloud Firestore to obtain information about the plane, adds emotional information, and generates an appropriate answer. This answer is converted into audio data using the Google Text-to-Speech API and played through the device's speaker.

[1420] Prompt Sentence Examples

[1421] You are an assistant that generates personalized answers to questions your child asks in the car.

[1422] Here are some questions: "Where is this plane going?"

[1423] Your current location is Tokyo. Your flight destination is Los Angeles International Airport.

[1424] Answer: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!"

[1425] This process allows parents to concentrate on driving with peace of mind, while creating an environment where children can have fun and satisfy their curiosity.

[1426] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1427] Step 1:

[1428] The user speaks a question while in the car, and the device's microphone captures the voice and saves it as audio data.

[1429] Input: User's voice question

[1430] Output: Captured audio data

[1431] Step 2:

[1432] The device uses a speech recognition API (Google Speech-to-Text API) to convert the voice data into text data.

[1433] Input: Audio data

[1434] Output: Text data (e.g., "Where is this plane going?")

[1435] Step 3:

[1436] The device sends text data to the server, which then uses a natural language processing engine (Google Natural Language API) to analyze the intent of the question.

[1437] Input: Text data

[1438] Output: The intent of the question (e.g., asking for flight destination)

[1439] Step 4:

[1440] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) to analyze the voice data and recognize the user's emotions.

[1441] Input: Audio data

[1442] Output: User emotion (e.g., excitement, curiosity)

[1443] Step 5:

[1444] The device's GPS module acquires the current location information and sends it to the server.

[1445] Input: Location measurement

[1446] Output: Current location (e.g., Tokyo)

[1447] Step 6:

[1448] The server uses Google Cloud Firestore to search for information about related objects (e.g., airplanes in flight) based on the current location and emotion information.

[1449] Input: Current location, emotion

[1450] Output: Object information (e.g., flight destination)

[1451] Step 7:

[1452] The server analyzes the search results and generates an answer to the user's question, adjusting the tone and content of the answer based on emotional information.

[1453] Input: Object information, emotion information, question intent

[1454] Output: Answer (e.g., "This plane is heading to Los Angeles International Airport in the United States.")

[1455] Step 8:

[1456] The server converts the generated answer into audio data using the Google Text-to-Speech API and sends it to the device.

[1457] Input: Answer

[1458] Output: Audio data

[1459] Step 9:

[1460] The terminal uses a speaker to play back the audio data and provide the answer to the user.

[1461] Input: Audio data

[1462] Output: Speech to be played to the user (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!")

[1463] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1464] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1465] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1466] [Fourth embodiment]

[1467] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1468] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1469] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1470] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1471] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1472] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1473] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1474] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1475] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1476] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1477] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1478] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1479] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1480] This system responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system not only satisfies children's intellectual curiosity, but also creates an environment where parents can learn together by providing useful information to parents.

[1481] System Configuration

[1482] This system consists of the following main components:

[1483] 1. Terminal

[1484] Voice input means (microphone)

[1485] A means of converting voice data into text data (voice recognition API)

[1486] A means of obtaining current location information (GPS module)

[1487] A means of playing audio data (speaker)

[1488] 2. Server

[1489] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1490] A method for searching for information about an object based on its current location (database query)

[1491] A means of analyzing search results and generating answers (generative AI)

[1492] A means of generating additional information related to the generated answer (knowledge database)

[1493] Program processing overview

[1494] 1. Enter the user's question

[1495] User: A child in the car asks, "Where is this plane going?"

[1496] Device: Built-in microphone captures audio.

[1497] 2. Speech-to-text

[1498] On the device: Use a speech recognition API to convert the voice data into text. For example, use the Google Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[1499] 3. Question Analysis

[1500] Terminal: Sends text data to the server.

[1501] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[1502] 4. Obtaining location information

[1503] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[1504] 5. Executing a Database Query

[1505] Device: Sends the acquired location information to the server.

[1506] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[1507] 6. Answer Generation

[1508] Server: Analyzes the received flight information and generates an appropriate response, such as "This flight is heading to Los Angeles International Airport in the United States."

[1509] Server: In addition, it retrieves additional information about Los Angeles from the knowledge database and adds it, for example, "Los Angeles is a large city famous for its film industry."

[1510] 7. Voice output of answers

[1511] Server: Sends the generated text data to the device.

[1512] Terminal: Convert the received text data into voice data using a speech synthesis API, such as the Google Text-to-Speech API.

[1513] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1514] Specific examples

[1515] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[1516] In this way, the system satisfies children's intellectual curiosity and provides useful information to parents, enhancing the educational benefits of in-car experiences.

[1517] The processing flow will be explained below.

[1518] Step 1:

[1519] User: A child in the car asks, "Where is this plane going?"

[1520] Device: A built-in microphone captures your child's voice.

[1521] Step 2:

[1522] On the device: Uses a speech recognition API to convert captured voice data into text data.

[1523] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[1524] Step 3:

[1525] Terminal: Sends text data to the server.

[1526] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[1527] Example: Use the BERT model to determine "The person is asking about the flight destination."

[1528] Step 4:

[1529] Device: Uses the GPS module to obtain current location information.

[1530] Example: Get current location as "Tokyo".

[1531] Step 5:

[1532] Device: Sends the acquired current location information to the server.

[1533] Step 6:

[1534] Server: Queries the aircraft database based on the received current location information.

[1535] Example: Running an SQL query to get information about planes flying near your current location.

[1536] Step 7:

[1537] Server: Parses the query results and generates an answer to the user's question.

[1538] Example: Generate the answer "This plane is heading to Los Angeles International Airport in the United States."

[1539] Step 8:

[1540] Server: Generates additional information related to the answer.

[1541] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[1542] Step 9:

[1543] Server: Sends the generated answer and additional information to the device.

[1544] Step 10:

[1545] On the device: Use a speech synthesis API to convert text data into audio data.

[1546] Example: Call the Google Text-to-Speech API to convert text to audio data.

[1547] Step 11:

[1548] Terminal: Plays back the generated audio data using a speaker.

[1549] Example: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1550] This series of processes provides appropriate answers to the child's questions, allowing both parent and child to learn.

[1551] Example 1

[1552] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1553] There is a demand for a system that can quickly and accurately answer questions that children in the car ask about the scenery outside, instead of the parent who is driving, thereby satisfying the child's intellectual curiosity, reducing the burden on the parent, and providing an educational environment inside the car.

[1554] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1555] In this invention, the server includes means for analyzing text data to recognize the intent of the question, means for searching for information on the object based on the acquired location information, and means for analyzing the search results to generate an answer, thereby enabling the server to quickly and accurately provide information on places and objects based on the child's question.

[1556] The "voice input means" is an input device for capturing voices uttered by the user.

[1557] The "means for converting voice data into text data" is a technology for analyzing captured voice data and converting it into corresponding text data.

[1558] "Means for analyzing text data and recognizing the intent of a question" refers to natural language processing technology for analyzing text data, understanding its content, and determining the intent of a question.

[1559] "Means for acquiring location information" refers to technology that acquires information on the latitude and longitude of the current location using a GPS module or the like.

[1560] The "means for searching for information on an object based on acquired location information" is a technology for searching a database for information on a specific object based on acquired location information.

[1561] "Means for analyzing search results and generating answers" refers to technology for analyzing information obtained through a search and generating appropriate answers to the user's questions.

[1562] The "means for generating additional information related to the generated answer" is a technique for adding further related knowledge or information to the generated answer to provide a more detailed answer.

[1563] The "means for converting the generated answer into voice data" is a technology for converting the answer in text format into voice data, and making it into a format that can be output as voice.

[1564] "Means for reproducing audio data" refers to a technique for reproducing audio data using an output device such as a speaker, thereby providing information to the user auditorily.

[1565] This invention is a system that responds to questions posed by children in a car who are interested in the scenery outside by providing appropriate answers on behalf of their parents who are driving. This system satisfies children's intellectual curiosity and creates an environment where both parents can learn by providing useful information.

[1566] System Configuration

[1567] This system consists of the following main components:

[1568] 1. Terminal

[1569] Voice input means (microphone)

[1570] A means of converting voice data into text data (voice recognition API)

[1571] Means of obtaining location information (GPS module)

[1572] A means of playing audio data (speaker)

[1573] 2. Server

[1574] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1575] A method for searching for information about an object based on the acquired location information (database query)

[1576] A means of analyzing search results and generating answers (generative AI model)

[1577] A means of generating additional information related to the generated answer (knowledge database)

[1578] A means to convert the generated answers into voice data (voice synthesis API)

[1579] System Operation

[1580] 1. Enter the user's question

[1581] User: A child asks, "Where is this plane going?"

[1582] Device: A sensitive microphone inside the car captures the child's voice.

[1583] 2. Speech-to-text

[1584] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[1585] 3. Question Analysis

[1586] Terminal: The converted text data is sent to the server using the HTTP POST method.

[1587] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[1588] 4. Obtaining location information

[1589] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[1590] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[1591] 5. Executing a Database Query

[1592] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[1593] Server: Query the database to get data on planes flying around the current location.

[1594] 6. Answer Generation

[1595] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[1596] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[1597] 7. Voice output of answers

[1598] Server: Sends the generated text data to the device.

[1599] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[1600] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1601] Specific examples

[1602] As an example of actual operation, consider a case where a child user asks "Where is this plane going?" while in a car. The device captures the voice and converts it into text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, the GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane, generates an appropriate answer, and sends it to the device. Finally, the device uses a speech synthesis API to convert the answer into audio data, which is played through the speaker.

[1603] Example prompt sentence:

[1604] When asked "Where is this plane going?", provide an appropriate answer by determining the plane's destination based on its current location, and provide any additional information about the destination.

[1605] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1606] Step 1:

[1607] Entering user questions

[1608] User: A child asks, "Where is this plane going?"

[1609] Device: A sensitive microphone installed inside the car captures the child's voice.

[1610] Input: Child's voice.

[1611] Processing: Captured as audio data by microphone.

[1612] Output: The captured audio data.

[1613] Step 2:

[1614] Speech-to-text

[1615] On the device: Using a speech recognition API (for example, a general speech recognition service), the captured voice data is converted into text data, for example, "Where is this plane going?"

[1616] Input: The captured audio data.

[1617] Processing: Send the audio data to the speech recognition API and convert it into corresponding text data.

[1618] Output: The converted text data (e.g., "Where is this plane going?").

[1619] Step 3:

[1620] Question Analysis

[1621] Terminal: The converted text data is sent to the server using the HTTP POST method.

[1622] Server: Analyzes the text data using a natural language processing engine (e.g., a general natural language processing service) and recognizes the intent of the question. It recognizes that the question is about flight destinations.

[1623] Input: Text data (e.g., "Where is this plane going?").

[1624] Processing: Text data is sent to the server via an HTTP POST request, and the server analyzes it using a natural language processing engine.

[1625] Output: The intent of the question (e.g., "Question about flight destinations").

[1626] Step 4:

[1627] Obtaining location information

[1628] Terminal: Uses the GPS module to obtain current location information (e.g., latitude and longitude).

[1629] On the device: Send the acquired location information to the server by including the location information in the body of the HTTP POST request.

[1630] Input: Current location (latitude and longitude).

[1631] Processing: Obtain location information using the GPS module and send it to the server via an HTTP POST request.

[1632] Output: The location information sent.

[1633] Step 5:

[1634] Executing a Database Query

[1635] Server: Based on the obtained location information, it queries a database containing information about the aircraft (e.g., a general flight information database).

[1636] Server: Query the database to get data on planes flying around the current location.

[1637] Input: Location and question intent.

[1638] Processing: Query the database to get the relevant plane information.

[1639] Output: Plane information data.

[1640] Step 6:

[1641] Generate answers

[1642] Server: Analyzes the received flight information and generates an appropriate answer using a generative AI model (e.g., a general generative AI model). For example, it generates an answer such as, "This flight is heading to Los Angeles International Airport in the United States."

[1643] Server: Retrieves relevant information from a knowledge database and adds additional information to the answer, such as "Los Angeles is a large city famous for its film industry."

[1644] Input: Airplane information data.

[1645] Processing: A generative AI model generates an answer and adds additional information from a knowledge database.

[1646] Output: The generated answer with additional information (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city known for its film industry.").

[1647] Step 7:

[1648] Voice output of answers

[1649] Server: Sends the generated text data to the device.

[1650] On the device: Use a speech synthesis API (for example, a general speech synthesis service) to convert text data into speech data.

[1651] Terminal: Play the generated audio data using a speaker, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1652] Input: Generated text data.

[1653] Processing: Convert into audio data using speech synthesis API and play on speaker.

[1654] Output: An audio response that the user can hear.

[1655] (Application example 1)

[1656] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1657] In recent years, with the increasing popularity of self-driving vehicles, people are spending more time in their cars. Children, in particular, are increasingly curious about the scenery outside and asking questions. However, it is difficult, inconvenient, and potentially dangerous for parents to respond to such questions while driving. Furthermore, there are limited means to satisfy children's intellectual curiosity and provide an environment where parents and children can learn together. This results in lost opportunities for educational interaction between families in the car. The present invention aims to solve these problems by providing a system that responds appropriately and quickly to children's questions.

[1658] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1659] In this invention, the server includes means for searching for information on objects based on current location information, means for analyzing the search results and generating an answer, and means for acquiring additional information related to the generated answer from a knowledge base. As a result, when a child asks a question about the scenery outside while in the car, an appropriate answer can be automatically and quickly generated, allowing the whole family to share new knowledge.

[1660] "Voice input means" is a general term for devices that capture voices uttered by a user.

[1661] The "means for converting voice data into text data" is a technology that uses voice recognition technology to convert input voice into corresponding text data.

[1662] "Means for analyzing text data and recognizing the intent of the question" refers to a technology that uses natural language processing technology to understand the user's intent and the content of the question from text data.

[1663] "Means for obtaining current location information" refers to a technology that allows the system to obtain current geographical location information using a GPS module or the like.

[1664] The "means for searching for information on an object based on current location information" is a technology for searching for information on a specific object from a database or the like based on the acquired location information.

[1665] "Means for analyzing search results and generating answers" refers to technology that processes the data obtained through a search and constructs an appropriate answer to the user's question.

[1666] The "means for obtaining additional information related to the generated answer from a knowledge base" is a technique for obtaining supplemental information from a database or the like in order to add the supplemental information to the generated answer.

[1667] A "generative AI model" is an artificial intelligence technology that can learn from large amounts of data and generate natural, human-like answers.

[1668] "Means for reproducing audio data" refers to technology that converts text data into audio using speech synthesis technology and outputs it from a speaker or the like.

[1669] A "prompt sentence" is an input sentence that causes a generative AI model to generate an appropriate answer.

[1670] The present invention is an educational system in which a system installed in an autonomous vehicle provides appropriate answers to questions posed by a child inside the vehicle about the scenery outside. This system includes a voice input means, a means for converting voice data into text data, a means for analyzing the text data and recognizing the intent of the question, a means for acquiring current location information, a means for searching for information on an object based on the current location information, a means for analyzing the search results and generating an answer, a means for acquiring additional information related to the generated answer from a knowledge base, a means for converting the generated answer into voice data, and a means for playing back the voice data.

[1671] The server first captures the child's voice using a voice input method (e.g., a microphone).The voice data is then converted into text data using a tool such as the Google Speech-to-Text API.The text data is then analyzed using natural language processing technology (e.g., the Google Cloud Natural Language API) to recognize the intent of the question.

[1672] Next, the current location information is acquired using a GPS module and sent to the server along with the text data. The server then searches for information about the object from sources such as a database based on the current location information. For example, if information about a specific mountain is to be acquired, the server queries the database for data related to that mountain.

[1673] The search results are analyzed using a generative AI model (e.g., GPT-4) to generate an appropriate answer to the user's question. For example, the answer may be, "This mountain is Mount Fuji. It is the highest mountain in Japan." Furthermore, additional relevant information can be retrieved from a knowledge base and added to the answer, making the provided answer more detailed.

[1674] Finally, the generated answers are converted into audio data using the Google Text-to-Speech API and played through the vehicle's speakers, allowing the child to get a spoken answer to their question.

[1675] As a concrete example, if a child asks, "What mountain is this?", the microphone captures the voice and speech recognition technology generates the text data "What mountain is this?". Natural language processing technology analyzes the intent of the question, and if it determines that the current location is, for example, Nagano Prefecture, the server searches a database for information on nearby mountains. The answer generated might be, "That mountain is the Hakuba Mountains," with additional information added, such as, "The Hakuba Mountains are famous for their beautiful hiking trails." This answer is then played back as audio from the microphone.

[1676] An example prompt here is, "Please describe the mountain the child saw based on the following data: Hakuba Mountains."

[1677] In this way, the present invention not only satisfies a child's intellectual curiosity and enhances the educational effect in the car, but also provides an opportunity for parents and children to learn together.

[1678] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1679] Step 1:

[1680] The device uses the microphone in the car to capture the voice of the user, a child, asking, "What mountain is this?" The input is the user's voice, and the output is the voice data.

[1681] Step 2:

[1682] The device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the captured voice data into text data. The input is the voice data, and the output is the text data "What mountain is this?"

[1683] Step 3:

[1684] The terminal sends the converted text data to the server. The input here is the text data "What is this mountain?" and the output is the transmission of the text data to the server.

[1685] Step 4:

[1686] The server receives the text data and analyzes it using a natural language processing engine (e.g., Google Cloud Natural Language API) to recognize the intent of the question. The input is the text data "What is this mountain?", and the output is the analyzed intent of the question (e.g., "I want to know the name of the mountain").

[1687] Step 5:

[1688] The terminal obtains the current location information using the GPS module. The input is the current geographical location, and the output is the current location information (e.g., "Nagano Prefecture").

[1689] Step 6:

[1690] The terminal sends the acquired current location information to the server. The input is the current location information "Nagano Prefecture," and the output is the transmission of the location information to the server.

[1691] Step 7:

[1692] The server executes a database query based on the received text data and location information to search for the target mountain. The input is the text data "What mountain is this?" and the current location information "Nagano Prefecture," and the output is the corresponding mountain information (e.g., "Hakuba Mountains").

[1693] Step 8:

[1694] The server uses a generative AI model (for example, GPT-4) based on the search results to generate an appropriate answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails." The input is the search result "Hakuba Mountain," and the output is the generated answer: "This mountain is Hakuba Mountain. It is famous for its beautiful hiking trails."

[1695] Step 9:

[1696] The server retrieves additional information related to the generated answer from the knowledge base and adds it to the answer. The input is the generated answer "This mountain is Hakuba Mountains." The output is the answer with the additional information added: "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails."

[1697] Step 10:

[1698] The server sends the final answer to the terminal. The input is the answer "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails." with additional information added, and the output is the sending of the answer to the terminal.

[1699] Step 11:

[1700] The device converts the received text data into voice data using a speech synthesis API (for example, Google Text-to-Speech API). The input is the text data "This mountain is Hakuba Mountains. It is famous for its beautiful hiking trails.", and the output is voice data.

[1701] Step 12:

[1702] The terminal uses a speaker to play back the generated voice data and provide an answer to the user (child). The input is the voice data, and the output is voice output from the speaker.

[1703] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1704] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[1705] System Configuration

[1706] This system consists of the following main components:

[1707] 1. Terminal

[1708] Voice input means (microphone)

[1709] A means of converting voice data into text data (voice recognition API)

[1710] Emotion engine (means of analyzing user emotions)

[1711] A means of obtaining current location information (GPS module)

[1712] A means of playing audio data (speaker)

[1713] 2. Server

[1714] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1715] A method for searching for information about an object based on its current location (database query)

[1716] A means of analyzing search results and generating answers (generative AI)

[1717] A means of generating additional information related to the generated answer (knowledge database)

[1718] Program processing overview

[1719] 1. Enter the user's question

[1720] User: A child in the car asks, "Where is this plane going?"

[1721] Device: Built-in microphone captures audio.

[1722] 2. Speech-to-text

[1723] On the device: Use a speech recognition API to convert the voice data into text data. For example, use the Google Speech-to-Text API to send the voice data and get the text "Where is this plane going?"

[1724] 3. Question Analysis

[1725] Terminal: Sends text data to the server.

[1726] Server: Uses a natural language processing engine to analyze the text and recognize the intent of the question, for example, recognizing that the user is asking about flight destinations.

[1727] 4. Emotion Analysis

[1728] On the device: The emotion engine analyzes the voice data and recognizes the user's emotions, for example, whether a child is excited or curious.

[1729] 5. Obtaining location information

[1730] Device: Use the GPS module to obtain the current location. Confirm that the current location is "Tokyo."

[1731] 6. Executing a Database Query

[1732] Device: Sends the acquired location and emotion information to the server.

[1733] Server: Queries a database containing information about planes to retrieve data about planes flying near your current location.

[1734] 7. Answer Generation

[1735] Server: Analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[1736] 8. Generating Additional Information

[1737] Server: Additionally, generate additional information related to the answer, for example, "Los Angeles is a large city famous for its film industry."

[1738] 9. Voice output of answers

[1739] Server: Sends the generated answer and additional information to the device.

[1740] On the device: Convert text data into audio data using a speech synthesis API, for example, the Google Text-to-Speech API.

[1741] Terminal: Plays audio data using a speaker and provides answers to the user, for example, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a large city famous for its film industry."

[1742] Specific examples

[1743] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[1744] This process provides accurate and personalized answers to a child's questions, helping both parents and children learn.

[1745] The processing flow will be explained below.

[1746] Step 1:

[1747] User: A child in the car asks, "Where is this plane going?"

[1748] Device: A built-in microphone captures your child's voice.

[1749] Step 2:

[1750] On the device: Uses a speech recognition API to convert captured voice data into text data.

[1751] Example: Call the Google Speech-to-Text API, send audio data, and get the text "Where is this plane going?"

[1752] Step 3:

[1753] Terminal: Sends text data to the server.

[1754] Server: Analyzes the received text data using a natural language processing engine and recognizes the intent of the question.

[1755] Example: Use the BERT model to determine "The person is asking about the flight destination."

[1756] Step 4:

[1757] Device: The emotion engine analyzes the voice data and recognizes the user's emotions.

[1758] For example: Analyzing speech tone, volume, rhythm, etc. to determine whether a child is excited or curious.

[1759] Step 5:

[1760] Device: Uses the GPS module to obtain current location information.

[1761] Example: Get current location as "Tokyo".

[1762] Step 6:

[1763] Device: Sends the acquired current location information and recognized emotion information to the server.

[1764] Step 7:

[1765] Server: Queries the aircraft database based on the received location and emotion information.

[1766] Example: Running an SQL query to get information about planes flying near your current location.

[1767] Step 8:

[1768] Server: Analyzes the query results and generates an answer to the user's question, taking into account information from the emotion engine and adjusting the tone and content appropriately.

[1769] For example: If a child is excited, generate the answer "Wow! That plane is heading to Los Angeles International Airport in the US! How exciting!"

[1770] Step 9:

[1771] Server: Also generates additional information related to the answer.

[1772] Example: Generate the additional information "Los Angeles is a large city famous for its film industry."

[1773] Step 10:

[1774] Server: Sends the generated answer and additional information to the device.

[1775] Step 11:

[1776] On the device: Use a speech synthesis API to convert text data into audio data.

[1777] Example: Call the Google Text-to-Speech API to convert text to audio data.

[1778] Step 12:

[1779] Terminal: Plays back the generated audio data using a speaker.

[1780] Example: Play audio that says, "Wow! The plane is heading to Los Angeles International Airport in the United States! Los Angeles is a big city famous for its film industry. How exciting!"

[1781] If the user is excited, the tone and content of the response can be adjusted to provide a more personalized experience. This process ensures that the child's questions are answered appropriately and emotionally, creating a rich learning experience for both parent and child.

[1782] Example 2

[1783] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1784] Conventional in-car voice response systems only provide simple answers to user questions and are unable to provide personalized responses that take into account the user's emotional state or current location. This has been a challenge, particularly in satisfying children's intellectual curiosity and providing an environment where parents and children can learn together. Furthermore, there has been a lack of systems that provide appropriate additional information based on the user's emotions and current location.

[1785] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1786] In this invention, the server includes a means for converting voice data into text data, a means for analyzing the user's emotions, and a means for acquiring current location information, thereby enabling the provision of personalized answers and additional information that take into account the user's emotional state and current location.

[1787] "Audio input means" is a device for capturing audio uttered by a user.

[1788] A "means for converting audio data into text data" is a process or technique that analyzes captured audio data and converts it into corresponding text data.

[1789] The "means for analyzing user emotions" is a function or engine that recognizes and analyzes the user's emotional state (e.g., joy, excitement, interest, etc.) from voice data.

[1790] The "means for analyzing text data and recognizing the intent of the question" is a technology for analyzing the converted text data and understanding the user's question and intent.

[1791] "Means for obtaining current location information" refers to a device or function for obtaining latitude and longitude information of the user's current location using location measurement technology such as GPS.

[1792] The "means for searching for information about an object based on current location information and emotion data" is a process for searching for appropriate information about an object from related databases or information sources using the acquired location information and emotion information.

[1793] "Means for analyzing search results and generating answers" refers to technology that generates appropriate answers to user questions based on information acquired through searches.

[1794] The "means for generating additional information related to the generated answer" is a technique for adding supplementary information or interesting information further related to the generated answer.

[1795] The "means for converting the generated answers into voice data" is a voice synthesis technology that converts answers in text format into voice format.

[1796] The "means for reproducing audio data" is a technique for providing the converted audio data to the user through an output device such as a speaker.

[1797] This system responds to questions posed by children about the scenery in the car and safely provides knowledge on behalf of parents. It also recognizes the user's emotions and adjusts the content of the answers to provide more personalized responses. This system satisfies children's intellectual curiosity and creates an environment where parents can learn together.

[1798] System Configuration

[1799] This system consists of the following main components:

[1800] 1. Terminal

[1801] Voice input means (microphone)

[1802] A means of converting voice data into text data (voice recognition API)

[1803] Emotion engine (means of analyzing user emotions)

[1804] A means of obtaining current location information (GPS module)

[1805] A means of playing audio data (speaker)

[1806] 2. Server

[1807] A means of analyzing text data and recognizing the intent of the question (natural language processing engine)

[1808] A method for searching for information about an object based on its current location (database query)

[1809] A means of analyzing search results and generating answers (generative AI)

[1810] A means of generating additional information related to the generated answer (knowledge database)

[1811] System Operation

[1812] The operation of this system is carried out in the following procedure.

[1813] 1. Enter the user's question

[1814] A child user in a car asks, "Where is this plane going?" The device's built-in microphone captures the voice.

[1815] 2. Speech-to-text

[1816] The device uses a speech recognition API (for example, Google Speech-to-Text API) to convert the voice data into text data, which becomes "Where is this plane going?"

[1817] 3. Question Analysis

[1818] The device sends the text data to a server, where a natural language processing engine analyzes the text and recognizes the intent of the child's question, such as asking about the flight's destination.

[1819] 4. Emotion Analysis

[1820] The device's emotion engine analyzes the voice data and recognizes the user's emotions, for example, determining whether a child is excited or curious.

[1821] 5. Obtaining location information

[1822] The device uses the GPS module to obtain the current location information. For example, it verifies that the current location is "Tokyo."

[1823] 6. Executing a Database Query

[1824] The device sends the acquired location and emotion information to a server, which queries a database containing information on airplanes to obtain data on airplanes flying near the current location.

[1825] 7. Answer Generation

[1826] The server analyzes the query results and generates an answer to the user's question, taking into account the information from the emotion engine and adjusting the tone and content accordingly. For example, it generates an answer such as "This plane is heading to Los Angeles International Airport in the United States."

[1827] 8. Generating Additional Information

[1828] The server also generates additional information related to the answer, for example, "Los Angeles is a large city known for its film industry."

[1829] 9. Voice output of answers

[1830] The server sends the generated answer and additional information to the device. The device converts the text data into audio data using a speech synthesis API (e.g., Google Text-to-Speech API). The audio data is played back using the device's speaker, providing the answer to the user. For example, it could say, "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a major city famous for its film industry."

[1831] Specific examples

[1832] As a practical example, consider a child in a car asking, "Where is this plane going?" The device captures the voice and converts it to text using a speech recognition API. The text data is then sent to the server, where a natural language processing engine analyzes the intent of the question. In parallel, an emotion engine analyzes the voice data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server performs a database query to obtain information about the plane and adds emotional information to generate an appropriate answer. This answer and additional information are sent to the device, where it is converted into audio data using a speech synthesis API. Finally, an audio answer is played through the device's speaker, providing appropriate knowledge for the child's question.

[1833] Prompt Sentence Examples

[1834] "When a child asks, 'Where is this plane going?', provide an appropriate voice response based on real-time aircraft data and current location. Choose an appropriate tone for the response and include any additional relevant information."

[1835] This system satisfies children's intellectual curiosity and provides an environment where parents and children can learn together.

[1836] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1837] Step 1:

[1838] A user asks a question in the car. A child says, "Where is this plane going?" This is the input voice data for the system. The device's microphone receives this voice and captures it as voice data.

[1839] Step 2:

[1840] The device converts the voice data into text data. The device uses a voice recognition API (for example, a voice recognition cloud service) to convert the captured voice data into text data such as "Where is this plane going?" This is a process that performs phonemic analysis on the voice data and converts it into the corresponding text. The output is the converted text data.

[1841] Step 3:

[1842] The terminal sends text data to the server. Based on the text data input, the terminal securely sends the data to the server using the HTTPS protocol. The output is the text data sent to the server.

[1843] Step 4:

[1844] The server analyzes the text data and recognizes the intent of the question. The server uses a natural language processing engine to analyze the text "Where is this plane going?", thereby recognizing the meaning "asking about the plane's destination." The output is the intent information of the question.

[1845] Step 5:

[1846] The device analyzes emotions. The device's emotion engine analyzes the voice data and identifies the user's (child's) emotional state. For example, it determines whether the child is excited or curious. The input is the captured voice data, and the output is the analyzed emotion data.

[1847] Step 6:

[1848] The device obtains location information. The device's GPS module is used to obtain the user's current location. For example, confirm that the current location is within Tokyo's 23 wards. The input is a signal from a GPS satellite, and the output is latitude and longitude information.

[1849] Step 7:

[1850] The device sends emotion data and location information to the server. The device sends the existing emotion data and location information to the server using the HTTPS protocol. The output is the emotion data and location information sent to the server.

[1851] Step 8:

[1852] The server executes a database query. Based on the location information it receives, it retrieves the relevant data from a database containing information about airplanes. The server issues a query such as "Search for information about airplanes in flight" and retrieves data from the database such as "Japan Airlines Flight 123 is heading to Los Angeles." The output is the query result.

[1853] Step 9:

[1854] The server analyzes the query results and generates an answer. The server uses an automatic generation engine to prepare an answer to the user's question. At this time, it also takes into account emotional data and generates an answer such as "This plane is heading to Los Angeles, USA." The output is a specific answer sentence.

[1855] Step 10:

[1856] The server generates additional information. The server searches for relevant information from a knowledge database and adds additional information related to the answer. For example, it adds information such as "Los Angeles is a city famous for its film industry." The output is a set of the answer and the additional information.

[1857] Step 11:

[1858] The server sends the generated answer and additional information to the terminal. The server sends the answer and additional information as text data to the terminal. The output is the text data sent to the terminal.

[1859] Step 12:

[1860] The device converts the text data into voice data. The device uses a voice synthesis API (e.g., a voice synthesis cloud service) to convert the text data into voice data. The output is the converted voice data.

[1861] Step 13:

[1862] The terminal plays the audio data. The converted audio data is played using the terminal's speaker, and the content is provided to the user as "This plane is heading to Los Angeles, USA. Los Angeles is a city famous for its film industry." The output is a voice response.

[1863] (Application example 2)

[1864] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1865] Conventional in-car knowledge provision systems make it difficult for parents to provide immediate and appropriate answers when their children ask about the scenery. Furthermore, rather than simply providing answers, personalized responses based on the child's emotions and interests are necessary. Furthermore, there is a need for real-time information provision based on the car's location and the situation at hand while the car is moving. It is necessary to solve these problems and provide a fun learning environment for both parents and children.

[1866] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for analyzing voice data to recognize the intent of the question, means for analyzing the user's emotions, and means for searching for information on the object based on current location information and emotion information. This makes it possible to provide appropriate and personalized answers to children's questions that take into account emotions and current location.

[1867] The "voice input means" is a means for acquiring voice and processing the acquired voice.

[1868] The "means for converting voice data into text data" is a means for converting acquired voice data into character string data.

[1869] The "means for analyzing text data and recognizing the intent of the question" is a means for analyzing the converted text data and understanding the purpose and intent of the user's question from its contents.

[1870] The "means for analyzing the user's emotions" is a means for analyzing the emotions of the user from voice data and other input data, and grasping the emotional state.

[1871] "Means for obtaining current location information" refers to means for identifying the current geographic location using GPS or other location information technology.

[1872] The "means for searching for information on an object based on current location information and emotion information" is a means for combining the acquired current location information and emotion information to search for information on related objects from a database or other information sources.

[1873] The "means for analyzing search results and generating an answer" refers to a means for analyzing information obtained through a search and generating an appropriate answer to a user's question based on that information.

[1874] The "means for converting the generated answer into voice data" is a means for converting the generated answer in text format into voice format.

[1875] The "means for reproducing audio data" refers to a means for transmitting the generated audio data to a user through a speaker or other reproducing device.

[1876] System Configuration

[1877] The present invention is a system that provides knowledge on behalf of parents in response to questions posed by children in the car, and further provides more personalized responses by recognizing the user's emotions and adjusting the content of the answers. The system consists of the following main components:

[1878] 1. Terminal

[1879] Voice input means (microphone)

[1880] A means of converting voice data into text data (Google Speech-to-Text API)

[1881] A means of analyzing user emotions (IBM Watson Tone Analyzer)

[1882] A means of obtaining current location information (GPS module)

[1883] A means of playing audio data (speaker)

[1884] 2. Server

[1885] A means of analyzing text data and recognizing the intent of the question (Google Natural Language API)

[1886] A method to search for information about an object based on its current location and emotion (Google Cloud Firestore)

[1887] A means of analyzing search results and generating answers (generative AI)

[1888] A way to convert the generated answers into audio data (Google Text-to-Speech API)

[1889] System operation explanation

[1890] Voice to text conversion

[1891] Any questions asked by the user in the car are captured by a microphone on the device, and the voice data is converted to text data using the Google Speech-to-Text API, which is then sent to a server to understand what the user is asking.

[1892] Question analysis and sentiment analysis

[1893] The server receives the text data and analyzes the intent of the question using the Google Natural Language API, while simultaneously analyzing the user's sentiment using IBM Watson Tone Analyzer. This sentiment data is used to adjust the tone and content of the response.

[1894] Obtaining current location information

[1895] The device's GPS module obtains the current location information, which is also sent to the server, and is used to search for objects related to the query.

[1896] Database Querying and Answer Generation

[1897] The server searches for relevant information using Google Cloud Firestore based on the acquired current location and emotion information. The search results are analyzed and a generative AI generates an answer to the question. The generated answer is converted into audio data using the Google Text-to-Speech API and sent to the device.

[1898] Audio playback

[1899] Finally, the generated audio data is played back through the device's speaker to provide the answer to the user, allowing for an appropriate and personalized response to the child's question.

[1900] Specific examples

[1901] As an example of actual operation, consider a child in a car asking, "Where is this plane going?" The device captures the audio and converts it to text using the Google Speech-to-Text API. The text data is then sent to the server, where the Google Natural Language API analyzes the intent of the question. At the same time, IBM Watson Tone Analyzer analyzes the audio data and recognizes the user's emotions. The device's GPS module obtains the current location information and sends it to the server. The server queries Google Cloud Firestore to obtain information about the plane, adds emotional information, and generates an appropriate answer. This answer is converted into audio data using the Google Text-to-Speech API and played through the device's speaker.

[1902] Prompt Sentence Examples

[1903] You are an assistant that generates personalized answers to questions your child asks in the car.

[1904] Here are some questions: "Where is this plane going?"

[1905] Your current location is Tokyo. Your flight destination is Los Angeles International Airport.

[1906] Answer: "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!"

[1907] This process allows parents to concentrate on driving with peace of mind, while creating an environment where children can have fun and satisfy their curiosity.

[1908] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1909] Step 1:

[1910] The user speaks a question while in the car, and the device's microphone captures the voice and saves it as audio data.

[1911] Input: User's voice question

[1912] Output: Captured audio data

[1913] Step 2:

[1914] The device uses a speech recognition API (Google Speech-to-Text API) to convert the voice data into text data.

[1915] Input: Audio data

[1916] Output: Text data (e.g., "Where is this plane going?")

[1917] Step 3:

[1918] The device sends text data to the server, which then uses a natural language processing engine (Google Natural Language API) to analyze the intent of the question.

[1919] Input: Text data

[1920] Output: The intent of the question (e.g., asking for flight destination)

[1921] Step 4:

[1922] The server uses an emotion analysis engine (IBM Watson Tone Analyzer) to analyze the voice data and recognize the user's emotions.

[1923] Input: Audio data

[1924] Output: User emotion (e.g., excitement, curiosity)

[1925] Step 5:

[1926] The device's GPS module acquires the current location information and sends it to the server.

[1927] Input: Location measurement

[1928] Output: Current location (e.g., Tokyo)

[1929] Step 6:

[1930] The server uses Google Cloud Firestore to search for information about related objects (e.g., airplanes in flight) based on the current location and emotion information.

[1931] Input: Current location, emotion

[1932] Output: Object information (e.g., flight destination)

[1933] Step 7:

[1934] The server analyzes the search results and generates an answer to the user's question, adjusting the tone and content of the answer based on emotional information.

[1935] Input: Object information, emotion information, question intent

[1936] Output: Answer (e.g., "This plane is heading to Los Angeles International Airport in the United States.")

[1937] Step 8:

[1938] The server converts the generated answer into audio data using the Google Text-to-Speech API and sends it to the device.

[1939] Input: Answer

[1940] Output: Audio data

[1941] Step 9:

[1942] The terminal uses a speaker to play back the audio data and provide the answer to the user.

[1943] Input: Audio data

[1944] Output: Speech to be played to the user (e.g., "This plane is heading to Los Angeles International Airport in the United States. Los Angeles is a big city famous for its film industry!")

[1945] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1946] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1947] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1948] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1949] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1950] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1951] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1952] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1953] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1954] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1955] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1956] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1957] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1958] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1959] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1960] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1961] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1962] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1963] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1964] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1965] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1966] The following is further disclosed regarding the above embodiment.

[1967] (Claim 1)

[1968] A voice input means;

[1969] means for converting voice data into text data;

[1970] A means for analyzing text data to recognize the intent of a question;

[1971] A means for acquiring current location information;

[1972] A means for searching for information on an object based on current location information;

[1973] a means for analyzing the search results to generate an answer;

[1974] means for converting the generated answers into audio data;

[1975] A system including means for playing audio data.

[1976] (Claim 2)

[1977] 10. The system of claim 1, further comprising means for generating additional information related to the generated answer.

[1978] (Claim 3)

[1979] 10. The system of claim 1, wherein the voice input means includes a microphone for capturing voices emitted by the user.

[1980] "Example 1"

[1981] (Claim 1)

[1982] A voice input means;

[1983] means for converting voice data into text data;

[1984] A means for analyzing text data to recognize the intent of a question;

[1985] A means for acquiring location information;

[1986] A means for searching for information on the object based on the acquired location information;

[1987] a means for analyzing the search results to generate an answer;

[1988] means for generating additional information related to the generated answer;

[1989] means for converting the generated answers into audio data;

[1990] A system including means for playing audio data.

[1991] (Claim 2)

[1992] 10. The system of claim 1, further comprising: means for searching a database for information including the object based on the acquired location information.

[1993] (Claim 3)

[1994] 10. The system of claim 1, wherein the voice input means includes an input device that captures voices uttered by the user.

[1995] "Application Example 1"

[1996] (Claim 1)

[1997] A voice input means;

[1998] means for converting voice data into text data;

[1999] A means for analyzing text data to recognize the intent of a question;

[2000] A means for acquiring current location information;

[2001] A means for searching for information on an object based on current location information;

[2002] a means for analyzing the search results to generate an answer;

[2003] means for obtaining additional information from the knowledge base related to the generated answer;

[2004] means for converting the generated answers into audio data;

[2005] A system including means for playing audio data.

[2006] (Claim 2)

[2007] 10. The system of claim 1, further comprising means for generating an answer using a generative AI model.

[2008] (Claim 3)

[2009] 2. The system of claim 1, wherein the voice input means includes a voice pickup device for capturing voices emitted by the user.

[2010] "Example 2: Combining Emotion Engines"

[2011] (Claim 1)

[2012] A voice input means;

[2013] means for converting voice data into text data;

[2014] means for analyzing user emotions;

[2015] A means for analyzing text data to recognize the intent of a question;

[2016] A means for acquiring current location information;

[2017] A means for searching for information about an object based on current location information and emotion data;

[2018] a means for analyzing the search results to generate an answer;

[2019] means for generating additional information related to the generated answer;

[2020] means for converting the generated answers into audio data;

[2021] A system including means for playing audio data.

[2022] (Claim 2)

[2023] 10. The system of claim 1, further comprising means for generating additional information related to the generated answer.

[2024] (Claim 3)

[2025] 10. The system of claim 1, wherein the voice input means includes a microphone for capturing voices emitted by the user.

[2026] "Application example 2 when combining emotion engines"

[2027] (Claim 1)

[2028] A voice input means;

[2029] means for converting voice data into text data;

[2030] A means for analyzing text data to recognize the intent of a question;

[2031] means for analyzing user emotions;

[2032] A means for acquiring current location information;

[2033] A means for searching for information on an object based on current location information and emotion information;

[2034] a means for analyzing the search results to generate an answer;

[2035] means for converting the generated answers into audio data;

[2036] A system including means for playing audio data.

[2037] (Claim 2)

[2038] 10. The system of claim 1, further comprising means for generating additional information related to the generated answer.

[2039] (Claim 3)

[2040] 10. The system of claim 1, wherein the voice input means includes a microphone for capturing voices emitted by the user. [Explanation of symbols]

[2041] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A voice input means; means for converting voice data into text data; A means for analyzing text data to recognize the intent of a question; A means for acquiring current location information; A means for searching for information on an object based on current location information; a means for analyzing the search results to generate an answer; means for converting the generated answers into audio data; and means for playing the audio data.

2. 10. The system of claim 1, further comprising means for generating additional information related to the generated answer.

3. 2. The system of claim 1, wherein the voice input means includes a microphone for capturing voices emitted by the user.

Images